Service Incident - October 6th, 2021 - Twitter Integration Issues

Summary

Between 18:55 on October 5, 2021 and 16:24 on October 6, 2021, customers on Pods 17 and 23 using integrations relying on Twitter webhooks experienced authorization issues. 

Timeline

15:01 UTC | 08:01 PT

We are investigating issues for some of our customers using our Twitter integrations. More updates to follow.

15:32 UTC | 08:32 PT

We have identified the cause of the issues affecting integrations relying on Twitter webhooks and are working towards a fix. We will provide more information as soon as it's available.

16:38 UTC | 09:38 PT

We are happy to report that the issues affecting integrations relying on Twitter webhooks have been resolved. Please let us know if you continue to experience any issues.

POST-MORTEM

Root Cause Analysis

This incident was caused by a bug introduced with an update that was not noticed in testing due to a difference in the testing environment.

Resolution

To fix this issue, the update was rolled back.

Remediation Items

1. Include additional layers of testing for specific channels. 
2. Improve recording of any possible differences between testing and production environments.
3. Include additional testing guidelines and account configurations to ensure coverage for a range of workflows.
4. Investigate possibility of replaying webhooks.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.