SUMMARY
From March 20, 19:00 UTC, to March 21, 01:07 UTC, a number of our customers across multiple Pods encountered an issue with in-product messaging where Admin users were shown banners that incorrectly indicated a payment failure, despite their accounts being current and not in Dunning (overdue payment) status. This led to unnecessary confusion as the message wrongly advised them to contact their account owner to address a payment issue that did not exist for some of the accounts.
Timeline
20:59 UTC | 13:59 PT
We are investigating reports of customers receiving an incorrect "failed payment" error despite their accounts being in good standing. We will provide another update when we have more information to share.
21:28 UTC | 14:28 PT
Our engineers continue to investigate the issue causing an incorrect "failed payment" error to be displayed despite accounts being in good standing. We will provide another update in an hour or when we have more information to share.
22:26 UTC | 15:26 PT
The issue causing incorrect "failed payment" errors to be displayed despite accounts being in good standing remains a top priority for our engineers. We will provide another update when we have more information to share.
01:59 UTC | 18:59 PT
We have rolled out a configuration change to remove the erroneous "failed payment" message appearing on some customers' Zendesk accounts. We have seen positive signs of recovery and will continue to monitor. A final update will be posted as soon as possible.
March 21 - 15:35 UTC | 08:35 PT
We are happy to report that the "failed payment" error message issue is now resolved. This message was incorrectly displayed to agents despite accounts being in good standing. Our teams will be investigating the root cause to ensure it does not happen again. Thanks for your patience and understanding while we worked through this issue.
POST-MORTEM
Root Cause Analysis
The incident was caused by an oversight during an update to our internal system that decides which customers receive certain in-product messages. A critical check related to customers' payment status was omitted by mistake, causing the message to be broadcast to a larger audience than was appropriate.
Resolution
To fix this issue, the incorrect in-product banner was disabled at approximately 00:45 UTC, preventing the display of any further misleading messages to our users.
Remediation Items
- Immediate Actions taken: Reassess and refine our customer grouping process to ensure it accurately includes payment status & Confirm that the information we use to identify customers' payment methods is correct and up-to-date.
- Strengthen our update procedures with better checks and reviews before and after changes go live.
- Introduce alerts to quickly identify and rectify any future issues with customer messaging.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 comment
Jessica G.
Post-Mortem published April 5, 2024.
0