SUMMARY
Between November 29, 2023, 16:06 UTC to December 20, 2023 11.20 UTC some Sell customers were experiencing delays for incoming and outgoing emails.
Timeline
12:03 UTC | 04:03 PT (Dec 20)
We have successfully stabilised email deliverability for Sell. A permanent solution has been put in place to prevent any recurrence of similar issues. We consider this incident fully resolved. We thank you for your patience during the investigation and extend our apologies for any disruption it may have caused.
15:31 UTC | 07:31 PT (Dec 19)
Since our previous update, we’ve been observing consistent email deliverability with Sell. We’re continuing our monitoring over this issue until it’s fully resolved. We will provide more updates when we have more information available. Thank you for your patience.
21:52 UTC | 13:52 PT (Dec 18)
Sell email processing and deliverability delays have stabilized and our engineers will continue to monitor performance over the next 24 hours. We will provide another update when we have new information to share.
17:47 UTC | 09:47 PT (Dec 18)
After resuming our investigation earlier today, we are seeing growing latency and delays in incoming and outgoing emails from Sell once again. Our team is working diligently to address the immediate impact and we will provide additional updates as soon as we have new information to share.
17:19 UTC | 09:19 PT (Dec 15)
Our team has been able to reduce the spike of latency and delays in incoming and outbound emails in Sell seen again today. Emails with Sell are processing normally at this time. We will continue our monitoring through the weekend and our investigation into root cause at the beginning of next week.
16:07 UTC | 08:07 PT (Dec 15)
We are currently experiencing another spike in latency and delays which are affecting incoming and outbound emails with Sell. We continue to investigate and will continue to post updates as we learn more.
20:01 UTC | 12:01 PT (Dec 14)
After restarting affected services, latency has dropped and incoming and outbound email in Sell is processing as expected. We will extend our monitoring through to tomorrow to allow the team to investigate further and ensure complete recovery.
19:36 UTC | 11:36 PT (Dec 14)
Our teams have noticed another spike in latency and delays affecting incoming and outbound email with Sell, and have restarted the affected systems once again. Latency has started dropping and we will monitor further to ensure full recovery.
18:43 UTC | 10:43 PT (Dec 14)
Our team has restarted a backend system responsible for processing mail in Sell and latency and delays have returned to normal levels. Incoming and outbound emails in Sell are now processing as expected. Thank you for your patience during our investigation.
18:18 UTC | 10:18 PT (Dec 14)
We have confirmed an issue with sending and receiving emails in Sell and our team is investigating. We will continue to post updates as we learn more.
18:01 UTC | 10:01 PT (Dec 14)
We are investigating reports of delays and errors sending and receiving emails in Sell. Further updates will be posted shortly.
POST-MORTEM
Root Cause Analysis
The root cause for the Sell mail delays and errors was not predicted dependencies introduced during database clusters migration that in effect of subsystems interactions caused database performance degradation to a point where some Mail related features were unavailable to customers.
Resolution
Issues were resolved by changing the pattern of network communication between said subsystems.
Remediation Items
- Document and schedule work to address uncovered issues [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.