Service Incident - July 24, 2023 - Sunshine Custom Objects | Multiple Pods - 503 errors when attempting to fetch Sunshine Custom Objects

SUMMARY

On July 24, 2023 from 19:32 UTC to 20:31 UTC, customers across multiple pods may have experienced 503 errors when they were attempting to fetch legacy Sunshine Custom Objects. 

Timeline

20:29 UTC | 13:29 PT
We are investigating increased error rates for legacy Sunshine Custom Objects across US and APAC based pods. We will provide another update in 15 minutes.

20:40 UTC | 13:40 PT
We have confirmed the issue causing increased error rates for legacy Sunshine Custom Objects across US and APAC based pods and our engineers are investigating. Next update in 30 minutes.

20:59 UTC | 13:59 PT
The issue causing increased error rates for legacy Sunshine Custom Objects across US and APAC based pods is now fully resolved. Please let us know if you have any questions or continue to run into any issues.

POST-MORTEM

Root Cause Analysis

This incident was caused by a database instance running out of local storage space, leaving no available storage for temporary capacity needed to service new queries.

Resolution

To fix this issue, the affected database instance was rebooted and the local storage space was restored.  Recovery was observed from that point on.

Remediation Items

1. Adding an alert when the local storage goes below a certain threshold on our database clusters [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.