SUMMARY
On March 24, 2021 from 16:31 UTC to 18:37 UTC, Zendesk Chat customers utilizing visitor authentication features may have experienced issues with chat visitors being unable to start a chat session.
Timeline
18:36 UTC | 11:36 PT
We are experiencing degradation of the Chat Authentication Server. This will result in Authenticated Chat users not being able to access Chat. Users who have not set up Authenticated Chat are not affected. We are currently investigating the issue.
19:41 UTC | 12:41 PT
We have identified the issue causing a degradation of the Chat Authentication Server and have deployed a fix to resolve it. Please let us know if you're still having issues accessing Chat.
POST-MORTEM
Root Cause Analysis
This incident was caused by insufficient capacity due to an infrastructure configuration error that resulted in traffic to the Zendesk Chat authentication server to be throttled by our infrastructure provider.
Resolution
To fix this issue, our team upgraded and relaunched the affected authentication service infrastructure. Service recovered shortly after.
Remediation Items
- Upgrade authentication server instances in the impacted region [Completed].
- Upgrade remaining authentication server instances [Scheduled].
- Block infrastructure with similar configuration from being provisioned in the production environment [Scheduled].
- Monitor on hosting provider throttling of this infrastructure service [Scheduled].
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.