22:11 UTC | 14:11 PT
We're happy to report that the billing application issues some customers experienced are fully resolved. Please reach out to us if you're still experiencing issues.
21:06 UTC | 13:06 PT
Our teams have isolated the issue impacting billing services across all products and are actively working on a resolution. We will update in an hour or earlier if we have new information.
20:13 UTC | 12:13 PT
We are continuing to investigate the billing access issues some customers are experiencing. This is not affecting product functionality. We will be updating in hour increments going forward.
19:22 UTC | 11:22 PT
Our team is continuing to work towards a resolution for the billing access issues some customers are experiencing. All other aspects of our products are functioning properly, including ticketing. The next update will be in one hour.
18:27 UTC | 10:27 PT
We are continuing to investigate issues with our billing services. The next update will be in 1 hour or earlier if we have new information.
18:03 UTC | 10:03 PT
We are currently investigating issues with our billing services for all products. We will provide an update as we learn more.
During this incident on December 12, all customers were unable to access the Subscription and Shopping Cart pages within their accounts. Any syncs occurring during this incident that affected billing were also affected. This incident was caused by an issue with an SSL certificate update from our upstream billing software service. Specifically, a function of the client adapter that this service uses had contained a Certificate Authority file that was out of date, which meant that the adapter was unable to validate the most recent Certificate Authority from the SSL change from the upstream service. Once our teams identified that cause, we deployed promptly deployed a patch, and the issue was resolved.
To prevent future occurrences of this particular incident, we are updating all functions of similar client adapters to ensure that they are up to date and that the proper Certificate Authorities are validated in the event of SSL certificate updates from any providers. We are also continually updating our monitoring tools to better detect symptomatic issues relating to the kinds of errors occurring during the incident.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.