Summary
On September 29, 2022 from 02:08 UTC to 03:34, Zendesk Chat customers on multiple pods may have experienced an issue where chats were not automatically creating tickets in Zendesk Support.
Timeline
02:47 UTC | 18:47 PT
We are aware of an issue where chats are not automatically creating tickets in Zendesk Support. We are investigating and will provide more information in 30 mins.
03:16 UTC | 19:16 PT
We are glad to inform you that the issue affecting ticket creation from chats has been resolved. Tickets from chats should have all been created. Many thanks for your patience and thank you for your partnership.
Root Cause Analysis
This incident was caused by an escaped defect in a recent deployment of the Chat software.
Resolution
To fix this issue, we rolled back to a previous version of the Chat software. From that point, we monitored until we had confirmation of full recovery.
Remediation Items
- Investigate options to implement preflight checks at deployment phase [Scheduled]
- Additional monitoring to detect this issue before it impacts customers [Scheduled]
- Review smoke tests for the impacted service [Scheduled]
- Review and amend process for deterministic deployments [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out oursystem status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident,please log a ticket with us via ZBot Messaging within the Widget.