SUMMARY
On September 4, 2024 from 18:21 UTC to 23:22 UTC, customers experienced difficulties in purchasing, previewing items, and performing other operations in the shopping cart due to an upstream issue with our billing partner.
Timeline
September 04, 2024 07:17 PM UTC | September 04, 2024 12:17 PM PT
We are investigating access issues with the Billing admin page. We will provide an update in 30 minutes or when we have more information to share.
September 04, 2024 07:23 PM UTC | September 04, 2024 12:23 PM PT
Our billing partner is currently experiencing a service degradation and our engineers have placed Billing in maintenance mode until the degradation is resolved. Subscription changes will be disabled until the service incident is resolved. We will provide another update in one hour or when we have new information to share.
September 04, 2024 09:33 PM UTC | September 04, 2024 02:33 PM PT
Our billing partner continues to experience a service degradation and Admin Center Billing remains in maintenance mode until the degradation is resolved. Subscription changes will remain disabled until the service incident is resolved. We will provide another when our partner's service incident has been resolved.
September 05, 2024 12:11 AM UTC | September 04, 2024 05:11 PM PT
Our engineers have taken Billing out of maintenance mode and restored Billing access now that our partner's service incident has been resolved. You can now update your account subscriptions again. Please clear your browser cache and cookies for the changes to take effect.
POST-MORTEM
Root Cause Analysis
This incident was caused by an outage in Zuora's production environment, linked to an issue with their Redis caching layer. Zuora's outage prevented our billing system from functioning correctly, which in turn impacted various customer-facing functionalities dependent on Zuora.
Resolution
To address the issue, the billing team placed the billing app into maintenance mode and reached out to Zuora for a resolution. Zuora identified and fixed their issue, allowing the billing app to resume normal operations.
Remediation Items
- Improve robustness of the billing breakers page to handle performance issues triggered by Zuora outages.
- Ensure essential functions can still load and function partially even if Zuora is down.
- Implement proactive alerts and circuit breakers to automatically enter maintenance mode if a Zuora outage is detected.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.
1 comment
Jessica G.
Post-mortem published September 11, 2024.
0