SUMMARY
On July 07, 2021 from 02:48 UTC to 14:44 UTC, customers on our Social Messaging platform were unable to send outbound replies to Facebook direct messages.
Timeline
04:33 UTC | 21:33 PT
We're investigating errors encountered when replying to end users via the Facebook Direct Message in Support on all Pods. Meantime as a workaround, you can directly go on your FB page. More to come.
05:36 UTC | 22:36 PT
Issue impacting FB Direct messages has been identified. Workaround for Agent workspace is now in place. Outbound DM is restricted to 24hr since the end-users last FB message. Currently working solutions for non-agent workspace. Next update in 1 hour.
06:44 UTC | 23:44 PT
A temporary fix deployment for non-Agent Workspace accounts to allow messages to be sent is ongoing and will be completed in approximately 4 hours. Thanks for your patience while we work through this issue. We will provide an update when we complete our deployment.
10:46 UTC | 03:46 PT
The temporary fix deployment for non-Agent Workspace accounts regarding the Facebook DM deliverability issue is still in progress. We expect it to be completed shortly. We appreciate for your patience while we work through this issue.
13:43 UTC | 06:43 PT
The temporary fix deployed for non-Agent Workspace accounts for the Facebook DM deliverability issue is undergoing progress. PODs 15, 17, 18, 19, 25, 26 and 27 are stable and should be seeing improvement in the platform integration for the past hour. Updates when we know more.
14:43 UTC | 07:43 PT
We are happy to report that the temporary fix for the Facebook DM deliverability issue has now been applied to all Pods. We will continue to work on a permanent solution. Thank you for your ongoing patience.
06:05 UTC | 23:05 PT
We are happy to report that the permanent fix for the Facebook Direct Message deliverability issue has been rolled out. Thank you so much for being so patient while we worked through this issue.
POST-MORTEM
Root Cause Analysis
This incident was caused by the revoking of the Zendesk access to our partner’s HUMAN_AGENT tag feature, resulting in authentication failures to their Messenger platform.
Resolution
To fix this issue, we bypassed the HUMAN_AGENT tag feature in the interim, whilst waiting for re-approval of access to it. Recovery was observed after that.
Remediation Items
- Re-enable access via HUMAN_AGENT to the Messenger platform [Done]
- Review and improve internal processes to mitigate future recurrences [In Progress]
- Work with our Messenger platform partner on better communication channels for upcoming changes. [In Progress]
- Centralise the monitoring of the Messenger dashboard and associated logs [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.