SUMMARY
On April 12, 2022 from 9:15 UTC to 12:50 UTC, a small subset of Explore customers in the EU region may have experienced dataset synchronization delays.
Timeline
11:13 UTC | 04:13 PT
We are currently monitoring recovery for a dataset sync issue impacting a subset of Explore customers in the EU region. From 09:15 UTC today, the sync started delaying and it might take another couple of hours to fully refresh. We appreciate your patience.
14:50 UTC | 07:50 PT
We’re happy to report the recovery for the dataset sync issue impacting a subset of Explore customers in the EU region. Thanks for your patience throughout this incident.
POST-MORTEM
Root Cause Analysis
This incident was caused by a change made to the system where a new cluster replaced the existing cluster handling pipelines and this wasn't expected. This led to certain resources not being removed successfully from the infrastructure state file as previously thought, leading to delays in data processing.
Resolution
There was no manual fix for this issue, the system just needed extra time to load the pipeline from the new cluster that was created unexpectedly. After a couple of hours, datasets were correctly synced.
Remediation Items
- Add a monitoring alert for this specific scenario. [To Do]
- Improve Database performance in the user simulator. [Done]
- Update documentation to reflect specific use cases. [Done]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.