Service Incident - December 6, 2022 - Sell | All Pods - Sell not loading

SUMMARY

On December 06, 2022 from 12:33 UTC to 12:57 UTC, some customers experienced issues loading or accessing Zendesk Sell.

Timeline

12:57 UTC | 04:57 PT

We are actively investigating reports of issues accessing or loading Zendesk Sell. We will provide more information as it becomes available.

13:12 UTC | 05:12 PT

We’re happy to report that access to Zendesk Sell has been fully restored. Apologies for any inconvenience caused and we thank you for your patience

14:06 UTC | 06:06 PT

All known issues related to this incident have been completely resolved. Thank you for your cooperation.

POST-MORTEM

Root Cause Analysis

This incident was caused by a minor upgrade that did not operate as expected. This resulted in the Sell web application not serving customer requests by its HTTP workers, causing traffic to stop being served between the times stated above. 

Resolution

To fix the issue our team initiated a failover which allowed the new workers to reconnect to the database and serve traffic again. 

Remediation Items

1. Create a proactive monitoring alert to prevent similar issues. [Scheduled]
2. Change configuration so that no single data source connectivity issue can cause services to stop serving traffic. [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.