SUMMARY
On November 27, 2023 from 10:13 UTC to 12:07 UTC, Zendesk customers across all Pods could have experienced a delay in reaching out to, or failure in certain functionalities of our ZBot, such as a lack of bot loading the welcome message or instances of handoff to agents, during which customers would have received a message stating that ZBot is currently not able to transfer to an agent. This could have led to unsuccessful attempts to create a ticket and reach Zendesk Customer Support when using our ZBot Widget.
Timeline
12:20 UTC | 04:20 PT
Our ZBot is back and operational as normal and our customers can reach us using the normal widget again. Thank you for your patience. We will now consider this incident resolved.
11:41 UTC | 03:41 PT
We are aware of issues with our ZBot, resulting in delayed or no responses from the bot and customers not reaching our support team. We are working with our vendor to resolve the issue as soon as possible and will provide another update in 30 mins.
POST-MORTEM
Root Cause Analysis
This incident was caused as our vendor experienced an overload in one of their database clusters. The database, managed by an external provider to our vendor, had been scaled down automatically based on previous usage.
Resolution
Our vendor attempted to manually scale the database back up, but a malfunction on the external provider’s side prevented this process from completing. The external provider then force-applied this change, restoring our vendor’s system to normal functionality.
Remediation Items
- Enforcing a higher minimum cluster size to avoid excessive automatic downscaling. [Vendor]
- Discovery with external partner about manual changes not applying immediately [Vendor]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.