On September 30, 2019, at 18:00 UTC to October 1st at 20:40 UTC, Zendesk customers across all Pods and Products may have experienced delays in processing subscription updates.
Subscription updates affected were limited to adding agents and add-ons or upgrading existing plans, while new customers making initial purchases, and existing customers downgrading or cancelling accounts were not affected.
04:59 UTC | 21:59 PT
We have received confirmation that delays in subscription changes have stabilized and we continue to monitor the situation. If you experience further subscription change delays, please contact us at firstname.lastname@example.org.
23:59 UTC | 16:59 PT
We are continuing to work with our billing vendor. They have reported improvements in the service, and we will update once new information is available.
17:43 UTC | 10:43 PT
We are still investigating the delay in account subscription changes and are working with our billing vendor to resolve this issue. We will provide updates as new information is available.
16:46 UTC | 09:46 PT
We have identified a delay in subscription changes affecting all accounts. We are working with our billing vendor to resolve this as soon as possible.
16:20 UTC | 09:20 PT
We are aware of and are currently investigating an issue affecting account subscription changes. We will provide more information as soon as it is available.
Root cause Analysis
Our billing vendor experienced an issue causing automatic Callouts to our billing service to be delayed.
While our team was able to manually sync delayed subscription changes to mitigate customer impact, our billing vendor ultimately resolved the issue on their side to ensure Callout notifications were firing as expected, which allowed subscription changes to process as usual.
- Backlog of delayed subscription changes were processed manually to sync all accounts affected by the service interruption
- Investigate additional monitoring for increased visibility into billing vendor delays
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.