Service Incident - November 8, 2021 - Pod 25 - Chat/Messaging delays in Agent Workspace

Summary

On November 8, 2021 from 03:25 UTC to 03:57 UTC, Zendesk Support customers using Agent Workspace on Pod 25 experienced delayed chats and messages.

Timeline

04:00 UTC | 20:00 PT

We've received reports of delayed chats/messaging through Agent Workspace for customers on Pod 25. Our team has resolved the issue. We are monitoring for full recovery and will send a final update soon. Thanks for your patience.

04:15 UTC | 20:15 PT

The issue with chats and messaging in Agent Workspace on Pod 25 is fully resolved. We apologize for the inconvenience today and greatly appreciate your patience while we worked to fix it.

Root Cause Analysis

This incident was caused by a code deploy process failure. An automated process to rollback deploys on failure was incorrectly initiated in a code deploy. This rollback interrupted an infrastructure rebalance that was in progress and stopped the consumption of chat messages.

Resolution

To fix this issue, we rolled the deploy forward as normal and the message backlog was cleared.

Remediation Items

1. Update deploy procedure documentation [Completed]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.