Service Incident - July 28, 2022 - Support | Multiple Pods - Intermittent login issues

SUMMARY

On July 28, 2022 from 14:06 UTC to 14:33 UTC, customers on multiple pods using Zendesk Support may have experienced slowness, errors, and intermittent issues with logins. 

Timeline

14:30 UTC | 07:30 PT

We’re currently investigating issues for customers in Pods 19 and 27. More information soon.

14:52 UTC | 07:52 PT

We confirm issues for customers in Pods 13, 17, 19, 20 and 27 where they were unable to login and access their Zendesk account, but are seeing improvements already. More information in 30 min or if we have further information.

15:20 UTC | 08:20 PT

We are seeing some improvement in the access issues across Pods 13, 17, 19, 20, and 27, but our team continues to monitor the situation. We will provide additional information within the next 30 minutes.

15:48 UTC | 08:48 PT

Access issues across 13, 17, 19, 20, and 27 have improved but multiple products are experiencing some latency. Our team is monitoring and additional updates will be provided when we have new information to share.

16:14 UTC | 09:14 PT

Access issues across Pods 13, 17, 19, 20, and 27 have been resolved and latency is returning to normal. Please let us know if you continue to experience any issues.

POST-MORTEM

Root Cause Analysis

This incident was caused by a configuration change as part of a network resilience program being implemented, resulting in intermittent delays to customers accessing Zendesk Support services.

Resolution

To fix this issue, the change was rolled back, and recovery was observed thereafter.

Remediation Items

1. Identify and resolve the underlying network layer configuration that was included in the change control before attempting to apply it again.
2. Ensure that a test case exists to validate that the specific network layer configuration has been corrected.
3. Improve our network layer deployment pipeline's rollback capabilities to reduce the scope and duration of any future network layer configuration issues.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.