On June 30th, 2020 from 0:00 UTC to 3:56 UTC, customers using Side Conversations on Pod 25 experienced delays with the delivery of messages. A separate and smaller subset of messages failed with an error message during the incident. Push notifications for customers using the Support mobile application were delayed and/or failed to deliver during the incident.
04:36 UTC | 21:36 PT
The issues impacting the performance of side conversations on Pod 25 have been resolved. We apologise for any inconvenience this may have caused you and your team.
02:19 UTC | 19:19 PT
We have identified the cause of the performance issues with side conversations on Pod 25 and are currently working on a solution. We will send through updates as we have additional information to share.
01:50 UTC | 18:50 PT
We’ve received reports of performance issues with side conversations on Pod 25. Our team is currently investigating.
Root Cause Analysis
This incident resulted from an implementation of logic for checking the push notification settings for an agent when sending notifications, which when paired with a data set of users with a high ticket creation rate, generated an unsustainable load and caused load balancer issues.
To fix this issue, the logic was amended for checking for these push notification settings. Once this was completed, our load balancers were able to resume normal functionality.
- Analyze patterns of large data using agent groups and their impact on our load balancers and improve behavior
- Created an automated process to hinder these types of requests to prevent future overload to the load balancers
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.