Service Incident - May 18th, 2020 - Sell Email Issues

SUMMARY

On May 18, 2020 between 20:35 UTC and 23:35 UTC a subset of Zendesk Sell customers were unable to send or receive emails, attributes visible on Smartlist were outdated or unavailable, and emails visible on object feed were missing or incomplete.

01:14 UTC | 18:14 PT
We're happy to report that the issue impacting Outbound Emails in Zendesk Sell has been resolved.

00:26 UTC | 17:26 PT
We are seeing improvements when trying to send outbound emails in Zendesk Sell. We will provide another update once we've confirmed the issue is resolved.

23:25 UTC | 16:25 PT
We are continuing to investigate the issue impacting Zendesk Sell outbound email delivery. We'll provide updates as we learn more.

22:21 UTC | 15:21 PT
We are working to resolve the issue impacting outbound email delivery for Zendesk Sell. We apologize for the inconvenience and will update you with more news soon.

21:19 UTC | 14:19 PT
We are currently investigating an issue with outbound email delivery for Zendesk Sell. Errors include "There has been a problem sending out your email." We will provide an update in 1 hour if not sooner.

Root Cause Analysis

This incident was caused by low performance in the email processing database and issues with dependent services, which ultimately affected mail delivery and caused a backlog of delayed jobs.

Resolution

A problematic transaction was identified and killed, allowing new transactions and delayed jobs to process and restore functionality.

Remediation Items

Our engineering team is working on implementing an automated and more robust monitoring system to alert on such behavior in the future, as well as migrating to a background processing tool with higher capacity.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.