SUMMARY
On May 4th, 2022, a third-party vendor experienced an outage in a service that enables customers to use the Message channel in Sunshine Conversations (Sales & Support.) As a result, customers were unable to connect to any live agents.
Timeline
15:55 UTC | 08:55 PT
We are happy to report that we have resolved the issues affecting access to our Zendesk ZBot Widget for support (support.zendesk.com). Thank you for your patience during our investigation.
14:33 UTC | 07:33 PT
We are investigating reports of customers being unable to reach our Zendesk ZBot Widget for support (support.zendesk.com). We will provide an update as soon as we have more information. If you need urgent assistance, please reach out to serviceincident@zendesk.com.
POST-MORTEM
Root Cause Analysis
Our third party vendor’s primary database became unresponsive to all of their services, effectively causing a full-service shutdown. Their team investigated the root cause and found that the combination of an inefficient query and a bug in MongoDB Atlas’s caching caused the crash of the database and made it unreachable, initially preventing the team from restoring normal operations.
Resolution
With the support from their database provider’s team, they were able to scale the database manually and restore normal operations.
Remediation Items
- Eliminate the problematic queries.
- Improve overall database efficiency.
- Work with their database provider team to get the bug resolved.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via the email listed above.