Service Incident - November 10, 2022 - Sell | Multiple Pods - Disruptions with sending and receiving emails

SUMMARY

On November 10, 2022 from 13:09 UTC to 15:05 UTC, some Zendesk Sell customers in multiple locations experienced a disruption in receiving and sending emails.

Timeline

13:40 UTC | 05:40 PT

Our team is investigating errors when sending emails and delays in receiving emails in Zendesk Sell in multiple locations. We will follow up with another update once we learn more.

14:31 UTC | 06:31 PT

We continue investigating issues and delays with sending and receiving email in Sell across multiple Pods. We are rolling back a recent change that was made to see if it fixes the issue. We appreciate your patience.

15:05 UTC | 07:05 PT

We are happy to confirm the rollback change made to help improve issues with sending and receiving emails in Sell is finished across multiple Pods, and with that, we see backend improvements in email processing. Please bear with us while the system stabilises.

16:35 UTC | 08:35 PT

We are happy to report that the issue disrupting sending and receiving emails in Sell across multiple pods has been resolved. Thank you for your patience during our investigation.

POST-MORTEM

Root Cause Analysis

This incident was caused by a scheduled production upgrade to our database software version, which had the side effect of degradation in the email functionality in our Sell product.

Resolution

To fix this issue, we performed a rollback and reverted to the previous version.  We continued to monitor until the mail functionality recovered.

Remediation Items

1. Review database for read and write improvements [Scheduled]
2. Archive unused data in critical tables to improve performance [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.