On May 20, 2019 from 19:05 UTC to May 21 00:18 UTC customers on Zendesk Support, Guide & Talk on Pods 15 and 23 experienced issues with Support & Guide loading slowly, intermittent connectivity, not loading, and dropped Talk calls. Support.zendesk.com also experienced issues during this time.
00:32 UTC | 17:32 PT
We've successfully rolled back a change on pods 15 and 23 and we're happy to report that all issues have been resolved.
Root cause Analysis
This incident was caused by a database index change which degraded performance for some customers. This sub-optimal index caused a spike in database IO usage and saturated the DB clusters.
To fix this issue, we rolled back the data migration which resolved issues for customers. The rollback was slow to complete due to the sub-optimal index causing significant DB performance issues.
- Updated our rollout process for database index changes to ensure a smooth performance transition
- Improved index hash metrics
- Improved rate limiting
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.