SUMMARY
On July 12, 2024, Sunshine Conversations and Messaging experienced an issue where Instagram messages were not being received. This was due to a disruption impacting Webhooks on Messenger API for Instagram, which affected all Sunshine Conversations and Messaging accounts integrated with Instagram. We detected the issue promptly and worked with Meta to resolve it. Unfortunately, messages sent during the incident were lost and could not be recovered.
Timeline
July 16, 2024 04:44 PM UTC | July 11, 2024 09:44 AM PT
We’ve been informed by our partner that messages that were sent during the incident between 2024-07-11 15:56 UTC to 2024-07-12 00:00 UTC that were lost due to the configuration change cannot be recovered. Unfortunately, it is not possible to backfill messages that were sent during the incident.
July 12, 2024 01:23 AM UTC | July 11, 2024 06:23 PM PT
We have confirmation that the issue previously preventing Instagram messages from being received has been resolved. Thank you so much for your patience as we got to this point.
July 12, 2024 12:56 AM UTC | July 11, 2024 05:56 PM PT
Our partner has confirmed that the service disruption at their end has been resolved. Messages to the Instagram channel should now be able to be received. Please let us know if you continue to face issues.
July 11, 2024 10:17 PM UTC | July 11, 2024 03:17 PM PT
We have confirmed that messages from the Instagram channel are not being received due to a partner service disruption impacting webhooks on the Messenger API for Instagram. We will provide another update when we have new information to share.
July 11, 2024 08:06 PM UTC | July 11, 2024 01:06 PM PT
Our engineers continue to investigate the issue causing messages from the Instagram channel to not be received. We will provide another update when we have more information to share.
July 11, 2024 05:59 PM UTC | July 11, 2024 10:59 AM PT
We are investigating an issue impacting customers using the Instagram integration. Messages from the Instagram channel are not being received. We will provide additional information as it becomes available.
POST-MORTEM
The incident was caused by disruption impacting Webhooks on Messenger API for Instagram. This resulted in Meta temporarily stopping the delivery of webhooks, which are essential for processing Instagram messages. This affected all users with Instagram integrations. We escalated the issue to Meta and worked closely with them to restore service.
Root Cause Analysis
Sunshine Conversations relies on Webhooks from Instagram to process messages. When Meta stopped sending these Webhooks, we were unable to process any Instagram messages. The root cause was a configuration change on Meta's side, which they later confirmed.
Resolution
We escalated the issue with Meta until the issue was acknowledged and resolved. Meta resumed sending webhooks, and normal service was restored.
Remediation Items
- Improve Escalation Processes: We will review and update our escalation policies to ensure faster resolution in the future.
- Monitor Enhancements: We will enhance our monitoring systems to better detect and respond to similar issues.
We apologize for any inconvenience this incident may have caused and appreciate your understanding as we work to improve our processes.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.