15:33 UTC | 08:33 PT
Thank you for your patience while we worked to resolve this Chat authentication issue. We are happy to report that the fix our engineers have deployed is working as expected. Please let us know if you experience login issues again.
15:08 UTC | 08:08 PT
Our engineers have deployed a fix for the chat authentication issue. We ask that you attempt logging in to Zendesk Chat again and report back if you find any further issues. Thank you!
13:55 UTC | 06:55 PT
Our teams continue to work towards the resolution of the authentication/server error issues that some Zendesk Chat customers may be experiencing. Clearing your cache and cookies may allow a successful login at this time.
13:54 UTC | 06:54 PT
Our teams continue to work towards the resolution of the authentication/server error issues that some Zendesk Chat customers may be experiencing. Clearing your cache and cookies may allow a successful login at this time
13:11 UTC | 06:11 PT
We are still working towards the resolution of the authentication issues impacting some Zendesk Chat customers. We will provide an update in 60 minutes.
12:09 UTC | 05:09 PT
Our teams are still hard at work investigating the authentication issues impacting some Zendesk chat customers. We will provide another update in 60 minutes. Thank you for your patience and understanding.
11:02 UTC | 04:02 PT
Our teams continue to investigate the authentication issues impacting some Zendesk Chat customers. We will provide an update in 60 minutes.
10:02 UTC | 03:02 PT
We are still working to resolve the logging issues with Zendesk Chat. Next update in 60 minutes or earlier as we get new information.
09:19 UTC | 02:19 PT
We are continuing to investigate logging issues with Zendesk Chat. We have seen improvement by using a different browser or clearing cache and cookies, please try this while we’re working on fixing this issue. Thanks for bearing with us.
08:52 UTC | 01:52 PT
Some customers may be experiencing issues logging into Zendesk Chat. Our engineering teams are currently investigating. We will provide an update shortly.
On October 4, 2019 from 04:06 UTC to 14:56 UTC customers using Zendesk Chat experienced CSRF verification and server errors when logging in to the standalone Chat Dashboard and when attempting password resets.
Root Cause Analysis
This incident was caused by two specific issues impacting two sets of customers. These issues were revealed due to the volume of password reset requests Zendesk triggered following the remediation items for the 2016 Security Incident announced recently.
- CSRF verification errors: A bug in Zendesk Chat caused two CSRF token cookies to be present in the browser containing mismatched values. These mismatched values resulted in a 403 Forbidden error being triggered, resulting in CSRF verification errors when attempting to reset Zendesk Chat passwords.
- Server errors: Zendesk Chat proxy rate limits for various endpoints were triggered due to the volume of password reset requests. These resulted in server errors in the browser preventing agents from logging in to the Chat Dashboard and causing password reset errors.
In our first attempt to fix this issue, we temporarily turned off our proxy rate limits which reduced one source of errors customers were seeing. Since the first fix didn’t completely stop the flow of customer reports, our team continued to investigate other CSRF verification issues eventually identifying the CSRF token issue and exempting the Chat login flow from CSRF validation. This was deemed a low risk mitigation step as the login page requires a password field and that reduces the chances of the situation being exploited. Shortly after the incident ended, our team proceeded to eliminate the double cookie issue (see remediation items 1 & 2 below) before re-enabling CSRF validation.
- [Completed] Set CSRF cookie domain setting to prevent cookie value mismatch.
- [Completed] Implement CSRF token cleanup mechanism to prevent multiple tokens existing in the browser.
- [In progress] Additional monitoring and alerts to bring visibility to CSRF errors.
- [In progress] Additional monitoring and alerts to bring visibility to proxy rate limit errors.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.