SUMMARY
On July 25, 2021 from 13:56 UTC to 14:08 UTC, customers on Zendesk Explore experienced issues with data not being returned on their Explore dashboards and queries.
Timeline
14:22 UTC | 07:22 PT
Thank you for your patience while we resolved the issue in Explore causing data not to be loaded in dashboards. This has been fixed now. We apologise for the inconvenience!
14:09 UTC | 07:09 PT
We are investigating reports of data not loading from Explore dashboards for some of our customers.
POST-MORTEM
Root Cause Analysis
This incident was caused by a misconfigured process claiming exclusive rights to a part of our Explore database, this meant no other queries could be resolved while this process was active for our customers connected to this database.
Resolution
To fix this issue, our engineering team restarted the database. Once the restart finished, all connections could be reestablished and functionality resumed as normal.
Remediation Items
- Review the configuration of the process causing the incident.
- Add additional monitoring for the number of connections to each database.
- Add additional alarms for connection issues in the database.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.