SUMMARY
Between 23:00 UTC on September 3, 2021 and 18:32 UTC on September 4, 2021, customers attempting to make payments or update payment methods received an error “Transaction declined.9002 - MISSING_OR_INVALID_AUTHORIZATION”.
19:09 UTC | 12:09 PT
We are happy to report that the issue causing transaction declined errors has been resolved. We apologize for the inconvenience!
18:10 UTC | 11:10 PT
Our team has identified the root cause of the transaction declined errors impacting some of our customers when paying for their subscriptions or updating their payment details. We will continue to provide further updates as they become available.
16:17 UTC | 09:17 PT
Our team continues to investigate, and we will provide further updates about the declined transaction errors reported by our customers once they become available.
15:20 UTC | 08:20 PT
We continue to investigate the errors encountered by some of our customers when trying to pay for their subscriptions. We will provide another update in an hour.
14:50 UTC | 07:50 PT
Our Team is investigating reports of some of our customers receiving payment declined errors when trying to pay for their subscriptions today. More information to follow.
POST-MORTEM
Root Cause Analysis
A key used by our billing service to authenticate credit card transactions with one of our payment gateways, had expired, and as a result, no payment requests to the payment gateway were authorized.
Resolution
Once the issue was identified, renewing and updating the key immediately rectified the problem.
Remediation Items
- Rotate billing key
- Create alerting and monitoring for billing transaction failures
- Improve billing escalation process
- Improve monitoring of key expirations
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.