On July 17 2019, from 01:20 UTC to 13:29 UTC, Zendesk Chat customers on all Pods were unable to retrieve results using the Chat incremental API.
14:15 UTC | 07:15 PT
We experienced some data delays with our Chat Incremental API today between the hours of 01:20 UTC & 13:29 UTC. If you use the Chat Incremental API, please refresh your data during that time frame. We apologise for the inconvenience.
Root cause Analysis
A database slave used for Chat reporting stopped getting updates from the master database.
Investigation showed that the MySQL instance on this host was initially killed by a safety process and then automatically restarted.
As an immediate action, we restarted the replication on the database slave and switched to the other spare slaves while waiting for the replication to catch up.
- Adding Database Monitoring and Alerting improvements.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.