Service Incident - March 15, 2023 - Multiple Products | Pod 23 - Login issues

SUMMARY

On March 15, 2023 from 23:13 UTC to 23:50 UTC, customers using Zendesk Support, Chat, Talk or Guide on Pod 23 were either unable to login to the products, or were logged out and were unable to log back in. 

Timeline

23:28 UTC | 16:28 PT
We are aware of an issue affecting logins for customers on Pod 23. We are still assessing further impact. Next update in 30 minutes.

00:00 UTC | 17:00 PT
We are continuing investigations with login issues impacting customers on Pod 23. We appreciate your patience. We will provide another update in the next 30 minutes.

00:07 UTC | 17:07 PT
We have rolled back a change and customers on Pod 23 should now be able to login. We are continuing to monitor in the meantime.

00:33 UTC | 17:33 PT
We have confirmation that customers on Pod 23 are now able to login after a change was reverted. We continue to monitor, and will provide another update in 60 minutes or if we have further information to share.

01:24 UTC | 18:24 PT
After rolling back a change earlier, customers on Pod 23 were able to login once again. This is now in a monitoring phase, and we will update once more when this incident is fully resolved.

04:29 UTC | 21:29 PT

We are glad to inform you that the issue impacting logins on Pod 23 has been resolved.  Many thanks for your patience as we got to this point.

 

POST-MORTEM

Root Cause Analysis

This incident was caused by a code deployment failure as a result of a configuration error.

Resolution

To resolve this issue, a manual fix was performed for Pod 23, and a temporary deployment freeze was put in place for the software deployment platform to prevent this same problem from propagating to other applications.

Remediation Items

1. Identify all applications with similar configurations, and develop documentation to mitigate such configuration errors. [In progress]
2. Evaluate automation and additional tooling for improved paging of teams [Scheduled]

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.