SUMMARY

On July 31, 2024 from 18:30 UTC to 21:05 UTC, we faced an unforeseen issue that impacted our billing admin page. This problem stemmed from an outage experienced by one of our third-party service providers, Zuora. As a result, customers were unable to preview pricing information, purchase items, or update their subscriptions.


Timeline

July 31, 2024 08:57 PM UTC | July 31, 2024 01:57 PM PT
Our engineers have taken Billing out of maintenance mode and restored Billing access now that our partner's service incident has been resolved. You can now update your account subscriptions again. Please clear your browser cache and cookies for the changes to take effect.

July 31, 2024 08:05 PM UTC | July 31, 2024 01:05 PM PT
Our billing partner is currently experiencing a service degradation and our engineers have placed Billing in maintenance mode until the degradation is resolved. Subscription changes will be disabled until the service incident is resolved. We will provide another update in one hour or when we have new information to share.

July 31, 2024 07:58 PM UTC | July 31, 2024 12:58 PM PT
We are investigating access issues with the Billing admin page. We will provide an update in 30 minutes or when we have more information to share.

 

Post-Mortem

During the incident, once the issue was identified, our engineers placed the Billing admin in maintenance mode then restored access once the Zuora service incident resolved.

Root Cause Analysis

The root cause was a caching issue in Zuora's system, leading to their service outage. As our billing system depends on Zuora for pricing and subscription data, this caused disruptions for our customers when trying to make purchases or view pricing information.

Resolution

Our team liaised with Zuora to ensure a quick resolution. The billing system was temporarily put into maintenance mode to prevent further issues and was closely monitored until Zuora resolved their outage.

Remediation Items

To prevent similar issues in the future, we have identified several actions:

  1. Improvements to Maintenance Mode: Ensure that recovery from maintenance mode is seamless for customers, possibly by implementing automatic cache clearing.
  2. Endpoint Review: Assess which endpoints should remain unaffected by billing maintenance mode and document them accordingly.
  3. Automatic Maintenance Mode: Consider implementing additional circuit breakers or automatic notifications within the billing system to enter maintenance mode if Zuora's service degrades or goes down.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.

Powered by Zendesk