On June 6, 2019 from 18:56 UTC to June 7th, 20:59 UTC customers were unable to send broadcast campaigns and some trigger campaigns through Zendesk Connect across all pods.
22:15 UTC | 15:15 PT
The issue with delayed broadcast campaigns with Connect has now been resolved. During the investigation, it was discovered that trigger campaigns with a delay may be delayed more than expected. These campaigns will send and no further action is required.
20:55 UTC | 13:55 PT
We are continuing to investigate the root cause of the delays in sending broadcast campaigns with Connect. These delays do not affect trigger campaigns.
19:42 UTC | 12:42 PT
We are currently investigating delays in sending broadcast campaigns with Connect.
Root cause Analysis
This incident was caused by a conflicting configuration on the servers responsible for queueing and sending Connect broadcast and trigger campaigns.
To fix this issue, we manually configured the servers to allow campaigns to send out immediately, and then fully resolved the issue by updating the conflicting code.
- Add monitoring tools for visibility into delivery services.
- Increase communication between internal networking and product teams.
- Increase the process rate for trigger campaigns with delivery windows.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.