SUMMARY
On April 25, 2024 between 07:30 UTC and 08:15 UTC, a subset of customers using the Sell service across all Pods encountered difficulties accessing key features such as Leads, Deals, and Contacts. Instead of the usual functionality, users were met with an exclamation mark, indicating an error.
Timeline
April 25, 2024 08:32 AM UTC | April 25, 2024 01:32 AM PT
We are investigating reports of deals, leads and contacts not loading in Sell for customers across multiple Pods. We will provide more information as it becomes available.
April 25, 2024 08:49 AM UTC | April 25, 2024 01:49 AM PT
We are seeing recovery and customers are able to access leads, deals and contacts as normal again. Thank you for your patience during the investigation. We will now consider this incident as resolved.
POST-MORTEM
Root Cause Analysis
This incident was caused by an unintended consequence following a system upgrade designed to enhance our platform's capabilities. A critical detail in the configuration of web address interpretation was omitted, leading to access issues within key functionalities such as Leads, Deals, and Contacts. This error was not detected by our standard verification procedures prior to release.
Resolution
To fix this issue, upon recognizing the source of the issue, we promptly reverted to the prior stable configuration, restoring full service functionality.
Remediation Items
- Enhancement of automated tests to cover potential errors caused by trailing slashes in API requests.
- Implementation of improved smoke tests, alerts and monitoring.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.