SUMMARY
Between February 07, 2024 from 01:30 UTC to February 09, 2024 10:50 UTC, a subset of customers on multiple pods may have experienced their Social Messaging channels being disabled.
Timeline
15:55 UTC | 07:55 PT
Between February 7, 2024 around 01.30 UTC and February 9, 2024 around 10.50 UTC some customers had social messaging disabled automatically. Zendesk are reaching out to any customers impacted by this, If you have not already been in contact with the Zendesk team or do not receive an email informing you, you were not impacted.
Unfortunately it is not possible to restore any messages that were potentially sent by end users during the period in which messaging was disabled.
POST-MORTEM
Root Cause Analysis
This incident was caused by a bug with a backend service that provisions/deprovisions Messaging for an account. This service disabled the Messaging feature for a subset of customers who had a legacy Social Messaging app installed on their Zendesk instance.
Resolution
To fix this issue, a rollback to a previous version of the software was performed, and the Messaging channel was then turned back on for the affected accounts.
Remediation Items
- Remove all references to the legacy Social Messaging app in the code [Scheduled]
- Fix the bug related to the backend provisioning service [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.