Service Incident - August 23rd, 2021 - Pod 25 Support & Guide Errors

SUMMARY

On August 23, 2021 from 5:19 UTC to 10:57 UTC, Support and Guide customers on Pod 25 may have seen “Oops” or cannot be accessed (500 errors) errors in the requests page where end-users can view tickets.

Timeline

11:37 UTC | 04:37 PT
We’re happy to report that the errors affecting My Requests Guide page and classic ticketing functionality for some customers on Pod 25 has been fully restored. Apologies for any inconvenience caused.

10:48 UTC | 03:48 PT
Our teams continue to work on the fix for the error messages some Pod 25 customers are experiencing. We appreciate your ongoing patience.

10:17 UTC | 03:17 PT
We have identified the root cause for the error messages impacting some Pod 25 customers. Our teams are working towards a resolution.

09:56 UTC | 02:56 PT
We are investigating some customers receiving “Oops” errors on Pod 25. More information to follow.

POST-MORTEM

Root Cause Analysis

This incident was caused by a misconfigured database shard during a scheduled database migration.

Resolution

To fix this issue, the proper configuration was applied and the migration was redeployed. 

Remediation Items

1. Additional validation added to database migration processes.
2. Schedule database configuration check to run more frequently.
3. Reduce error noise to catch similar issues more quickly.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.