Service Incident - September 16th, 2020 - Explore Degradation

SUMMARY

From September 15, 2020 at 22:45 UTC to September 16, 2020 at 07:45 UTC, Explore customers did not receive any emails, including scheduled reports and dashboards from the system. 

Timeline

08:37 UTC | 01:37 PT
We have implemented a fix and Explore dashboard email delivery is now back to normal. Emails will be sent as per next scheduled time. Apologies for any inconvenience caused

07:48 UTC | 00:48 PT
We are currently experiencing scheduled email issues impacting Explore. Our engineers are rolling out a fix.

Root Cause Analysis

This incident was caused by the third party service used to send emails from Explore revoking our API keys without prior notice.

Resolution

To fix this issue, once identified, our team updated the respective API keys on both ends and the emailing service returned to normal.

Remediation Items

1. Our team started discussions with our partner to avoid situations like this in the future.
2. Add automated monitoring for any errors of all parts this partner service.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.