On August 13, 2020 from 9:39 UTC to 11:07 UTC, some customers experienced errors preventing them from logging in to Zendesk Chat.
11:33 UTC | 04:33 PT
We have received reports from customer facing issues logging into Zendesk Chat. We are currently investigating the root cause and will provide updates soon.
11:54 UTC | 04:54 PT
We continue to monitor issues affecting some customer using Zendesk Chat. Updates to follow shortly.
12:07 UTC | 05:07 PT
We have identified an issue with Chat that caused instability for customers using Zendesk Chat. These problems have now been resolved. If you experience any further issues, please reach out to us at firstname.lastname@example.org
14:35 UTC | 07:35 PT
Regarding the the issue with Zendesk Chat today, please see below for more information: https://support.zendesk.com/hc/en-us/articles/360052210734-Service-Incident-August-13th-2020 This issue is now resolved
Root Cause Analysis
This incident was caused by an issue with a recent Chat code deploy. The code introduced a configuration resulting in elevated connections to the central database user used to connect agents to Chat. The connection limit to this user was exhausted, resulting in login attempts failing with an error message.
To fix this issue, our team increased the connection limit to allow agents to resume Chat logins. We then identified the culprit code and rolled back the deploy.
- Implement additional monitors for central database user connection limits [Completed].
- Increased connection limit for the central database user [Completed].
- Review Chat database connection management [Scheduled].
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.