SUMMARY
14:13 UTC | 06:13 PT
We can confirm that the issue with Side Conversations is now resolved. If you notice any issue going forward, please reach out to us at support@zendesk.com
13:27 UTC | 05:27 PT
We have received reports of issues with Side Conversations on Pod 17. We will keep you updated as we investigate further
POST-MORTEM
During this incident, customers experienced side conversations delays on Pod 17 affecting inbound email, outbound email, Slack messages, and ticket channel messages. This incident was caused by a configuration change that was made on November 2, but not all affected services were redeployed to pick up the new change. On November 3, containers that crashed were not able to restart, and around 03:00 UTC on November 4, this started affecting email delivery and side-conversation service in the Support UI. Redeploys were started around 12:09 UTC, and after a few hiccups service was returned to normal around 13:30 UTC. No data was lost, and all pending messages were delivered. In order to prevent this from happening again in the future, we will create a monitor to notify when there are no available kpods for side-conversations services.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.