Service Incident - March 10th, 2022 - Chat & Messaging | Multiple Pods - Issues affecting ticket creation in Agent Workspace

SUMMARY

On March 10, 2022 from 06:09 UTC to 06:44 UTC, Agent workspace customers using Chat and Messaging experienced ticket creation issues, and ticket updates were delayed during the incident.

Timeline

06:54 UTC | 22:54 PT

We have reports of issues with ticket creation and routing in messaging/chat on all Pods. More information in 15 minutes.

07:04 UTC | 23:04 PT

We are glad to inform you that we have resolved the issues impacting ticket creation and routing in messaging/chat for AW customers on all pods. Thank you for being so patient.

07:14 UTC | 23:14 PT

We have confirmed reports of full recovery for ticket creation and routing in messaging/chat for our AW customers

POST-MORTEM

Root Cause Analysis

This incident was caused by a failure in one of the systems that routes Chat and Messaging to our production Pods as a result of recently deployed code.

Resolution

To fix this issue, we rolled back the code in question, and started seeing recovery.  Upon resolution of the incident, all tickets were successfully created, and agents could respond.

Remediation Items

1. Reproduce the issue in our staging environment and fix the code [In progress]
2. Update and improve alerting mechanisms for quicker response [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.