SUMMARY
On January 18, 2022 from 19:01 UTC to 22:39 UTC, Zendesk Chat customers using Agent Workspace on Pod 13 were unable to receive and accept incoming chats while agents were online and available.
Timeline
21:46 UTC | 13:46 PT
We are investigating reports of Agent Workspace customers on Pod 13 being unable to serve or self-assign incoming Chats. We will provide another update as soon as we can.
22:16 UTC | 14:16 PT
We have confirmed an issue impacting Chat creation and assignment for Agent Workspace customers on Pod 13 and are working on finding a root cause. We will provide another update within an hour.
22:51 UTC | 14:51 PT
We are seeing improvement in Chat creation and assignment in for Agent Workspace customers on Pod 13. Please let us know if you continue to experience any issues.
23:17 UTC | 15:17 PT
We are happy to report that the issue impacting Chat creation and assignment for Agent Workspace customers on Pod 13 has been resolved. Thank you for your patience during our investigation.
Root Cause Analysis
This incident was caused by Zendesk Chat middleware unexpectedly setting a session cookie on an error response. The client-side stored this cookie and applied it to subsequent requests, leading to the behavior customers observed during the incident.
Resolution
To fix this issue, we restarted the affected service to clear the cookie store and recovery was observed shortly after.
Remediation Items
- Temporarily disabled use of client-side cookie store in HTTP client library [Completed]
- Fix middleware authentication behaviour [In Progress]
- Retune the existing monitors and alerting for the affected service [In Progress]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.