SUMMARY
On August 29, 2022 from 14:02 UTC to 15:18 UTC, some Zendesk Explore customers on Pods 20 and 27 may have experienced data refresh delay issues.
Timeline
14:31 UTC | 07:31 PT
We are aware of an issue that is causing some data refresh delay for Explore for some customers in the AMER and APAC regions. We are working to understand the cause of this and will keep you updated as we find out more. Thank you for your patience.
14:47 UTC | 07:47 PT
We are continuing to investigate the cause of the issue that is causing data refresh delays for some customers in the AMER and APAC regions and have engaged additional engineering resources. We will update you in 30 mins or if we have further information.
16:19 UTC | 09:19 PT
The issue causing Explore data refresh delays is now fully resolved. Thank you for your patience and please let us know if you continue to experience issues.
POST-MORTEM
Root Cause Analysis
An error in an automated process caused a database connection issue with one of our internal services on pod 20 and 27.
Resolution
The issue was resolved after an engineer restarted the failing process.
Remediation Items
- Additional monitoring and alerts to assist in responding quicker to similar events in future. [Scheduled]
- Improve runbook to enable quicker response times. [In Progress]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.