Summary
On November 16, 2023 from 20:10 UTC to UTC, Zendesk Support customers using Facebook messaging channels experienced intermittent outbound message delivery failures to their end users.
Timeline
01:20 UTC | 17:20 PT
We have received reports of some customers experiencing delays when attempting to send outbound messages to their end users through the Facebook API or instances where these messages are not delivered. We are aware of an outage faced by our service partner and are actively collaborating with them to obtain timely updates. We anticipate providing the next update within 30 minutes to let you know where things are. In the meantime, we are exploring if there are workarounds that can be employed. Your patience during this time is greatly appreciated.
01:54 UTC | 17:54 PT
We have confirmed that a some customers may experience outbound Facebook messaging delays. Our service partner is actively working on resolving this issue. Currently, no immediate workaround has been found and we are continuing to investigate. We anticipate providing our next update within the next 60 minutes.
02:56 UTC | 18:56 PT
While our service partner is currently working on a resolution, we have conducted testing and have successfully sent Facebook messages through the Zendesk interface. We would appreciate your help to review if you are now able to send outbound Facebook messages at your end. In urgent situations, it might be possible to send messages directly from Facebook Messenger as a temporary solution. We aim to provide the next update in 60 minutes.
03:43 UTC | 19:43 PT
Our service partner has put in a fix and has resolved the issue at their end. We also have positive confirmation that outbound Facebook messages can be successfully sent. Thank you for being so patient as we worked through this.
POST-MORTEM
Root Cause Analysis
This incident was caused by a service interruption in the Meta Send API. In addition, our response and issue identification was delayed due to a monitoring issue on the Zendesk side.
Resolution
Meta resolved the API issues within their infrastructure.
Remediation Items
- Review existing monitors to capture all failures and alert appropriately.
- Review forthcoming Meta root cause analysis and explore further reactive improvements.
- Review error handling logic for Facebook API client.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 comment
Dan Beirouty
Post-mortem published November 29, 2023.
0