Service Incident - November 22, 2022 - Support | Multiple Pods - Messaging delays in Agent Workspace

Summary

On November 22, 2022 from 15:00 UTC to 16:31 UTC, Zendesk Support customers on multiple pods experienced delays in sending and receiving messages in Agent Workspace. 

Timeline

16:33 UTC | 08:33 PT

We are investigating reports of delays in sending and receiving messages in Agent Workspaces. More information to follow shortly.

16:50 UTC | 08:50 PT

We have confirmed an issue causing delays sending and receiving messages in Agent Workspace, and our team is investigating. We will post further updates soon.

16:56 UTC | 08:56 PT

We are seeing improvement in the issue causing delays in sending and receiving messages in Agent Workspaces. Please let us know if you continue to experience any problems.

17:11 UTC | 09:11 PT

We are happy to report that the issue causing messaging delays in Agent Workspaces has been resolved. Thank you for your patience while we investigated.

 

Root Cause Analysis

This incident was caused by a network issue that resulted in a disconnection of a backend event streaming service preventing message consumption and transfer to our live chat servers. 

Resolution

Our event streaming service recovered automatically from this network issue.

Remediation Items

1. Additional alerts to catch these errors sooner.
2. Investigate new timeout values to initiate earlier rebalancing.
3. Create a new runbook to allow a quicker response and fix for future issues.
4. Increase tracing and visibility into backend messaging services.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.