On July 24, 2023 from 19:32 UTC to 20:31 UTC, customers across multiple pods may have experienced 503 errors when they were attempting to fetch legacy Sunshine Custom Objects.
20:29 UTC | 13:29 PT
We are investigating increased error rates for legacy Sunshine Custom Objects across US and APAC based pods. We will provide another update in 15 minutes.
20:40 UTC | 13:40 PT
We have confirmed the issue causing increased error rates for legacy Sunshine Custom Objects across US and APAC based pods and our engineers are investigating. Next update in 30 minutes.
20:59 UTC | 13:59 PT
The issue causing increased error rates for legacy Sunshine Custom Objects across US and APAC based pods is now fully resolved. Please let us know if you have any questions or continue to run into any issues.
Root Cause Analysis
This incident was caused by a database instance running out of local storage space, leaving no available storage for temporary capacity needed to service new queries.
To fix this issue, the affected database instance was rebooted and the local storage space was restored. Recovery was observed from that point on.
- Adding an alert when the local storage goes below a certain threshold on our database clusters [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.