On November 8, 2021 from 03:25 UTC to 03:57 UTC, Zendesk Support customers using Agent Workspace on Pod 25 experienced delayed chats and messages.
04:00 UTC | 20:00 PT
We've received reports of delayed chats/messaging through Agent Workspace for customers on Pod 25. Our team has resolved the issue. We are monitoring for full recovery and will send a final update soon. Thanks for your patience.
04:15 UTC | 20:15 PT
The issue with chats and messaging in Agent Workspace on Pod 25 is fully resolved. We apologize for the inconvenience today and greatly appreciate your patience while we worked to fix it.
Root Cause Analysis
This incident was caused by a code deploy process failure. An automated process to rollback deploys on failure was incorrectly initiated in a code deploy. This rollback interrupted an infrastructure rebalance that was in progress and stopped the consumption of chat messages.
To fix this issue, we rolled the deploy forward as normal and the message backlog was cleared.
- Update deploy procedure documentation [Completed]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.