Service Incident - September 22, 2022 - Sell | Multiple Pods - Performance Dashboard widgets not updating

SUMMARY

Between September 22, 2022 at 18:19 UTC and September 23 at 15:03 UTC, Zendesk Sell customers may have experienced multiple Sell Performance Dashboard widgets displaying stale data. 

Timeline

September 22 19:00 UTC | 12:00 PT

Our engineering team is investigating an issue that is preventing multiple Sell widgets from updating. We will provide further updates when we have additional information to share.

September 22 22:25 UTC | 15:25 PT

Our engineers will continue investigation tomorrow during EMEA business hours.

September 22 22:25 UTC | 15:25 PT

In the interim, you can access Revenue and Forecasted Sales through the Reports sidebar with Revenue as “total sales” and Forecasted Sales as the report with the same name. New deals can be viewed by creating a deal smart list in the Deals tab.

September 23 13:23 UTC | 06:23 PT

We appreciate your patience while we work through this issue with multiple Sell Performance Dashboard widgets not updating correctly. We continue investigating and have made some progress with some accounts. [1/2]

September 23 13:24 UTC | 06:24 PT

However, engineering teams are continuing to triage this issue and a fix is expected Monday. We shall share more information as the fix progresses. [2/2]

September 23 15:56 UTC | 08:56 PT

We are happy to report that we have released a fix for the issues with Sell Performance Dashboard widgets not updating. Thank you for your patience during our investigation.

 

POST-MORTEM

Root Cause Analysis

This incident was caused by a bug in the Sell data pipeline.

Resolution

This issue was fixed by redeploying the latest stable version of the data pipeline configuration.

Remediation Items

1. Refactor logging to remove error noise and reduce time to diagnose the issue.
2. Improve monitoring and alerting to catch similar issues faster.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.