SUMMARY
On December 8th, 2020 from 11:39 UTC to December 9th 14:25 UTC, a small subset of our customers using our Social Messaging add-on product in conjunction with having previously used, but no longer on Agent Workspaces, experienced a loss of ticket creation in Support from the integrated social channels.
TIMELINE
15:16 UTC | 07:16 PT
We have deployed a fix to resolve the issue that caused tickets not being created from the Zendesk Web Messenger integration and this should be resolved by now. Please contact us if you are still having any problems.
14:33 UTC | 06:33 PT
We have received reports that the Social Messaging integration is not creating tickets in Support. We have found the issue and we are working on the remediation.
POST-MORTEM
Root Cause Analysis
This incident was caused by a migration in the backend that was meant to upgrade webhooks used by our product. While this ran successfully, accounts reverting from the use of Agent Workspaces created an edge case configuration that was not accounted for with this change, removing their webhooks in the process.
Resolution
To fix this issue, our developers re-added the previously removed webhooks for all impacted accounts. This restored functionality, but they still needed to make sure all messages received by these accounts during the incident were backfilled and successfully delivered to these accounts without generating errors or duplicate updates.
All restoration work has been successfully completed on the 16th of December.
Remediation Items
- Our developers migrated these specific accounts webhooks to be on par with the rest of our customers to prevent this issue for reoccurring.
- A new implementation process for these changes is being created to have better oversight.
- Our team is working on creating improved monitoring and alerting for message ticket generation, to spot any form of issue like this earlier on both platform and account level.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.