SUMMARY
On February 5, 2022 from 01:40 UTC to February 7, 2022 16:40 UTC, customers were unable to add a new or remove an existing WhatsApp integrations.
17:09 UTC | 09:09 PT
The issue impacting the ability to add and remove existing WhatsApp integrations is now fully resolved.
15:56 UTC | 07:56 PT
We continue to investigate the cause of customers being unable to add new or remove existing WhatsApp integrations. Further updates to follow.
15:23 UTC | 07:23 PT
Our team is investigating an issue with customers being unable to add a new or remove an existing WhatsApp integration. Messaging is currently not impacted. We will provide further updates as available.
POST-MORTEM
Root Cause Analysis
Our whatsapp-prov-service was not able to reach RabbitMQ cluster to send heartbeats. Missing heartbeats will automatically disconnect Celery from RabbitMQ server after a few attempts.
Resolution
A new cluster was deployed and connected between ourwhatsapp-prov-service and RabbitMQ.
Remediation Items
- Add additional monitoring and alerting
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.