SUMMARY
01:05 UTC | 18:05 PT
We have many confirmed reports that this is now resolved. Thank you once again for your partnership and understanding.
00:54 UTC | 17:54 PT
We have resolved the issues causing difficulties with access to login and admin pages. Apologies for the inconvenience this may have caused you, and thank you for your patience as we worked through this.
00:40 UTC | 17:40 PT
We have reports around issues with access to admin and login pages and are currently investigating further. More information in 15 minutes.
POST-MORTEM
During this incident, Support and Guide customers experienced issues logging into their accounts and accessing admin pages.
As part of an ongoing project to migrate and improve our deploy infrastructure, a new asset pipeline was being tested and during this testing a bug and configuration error led to bad assets being uploading to the assets server, overwriting the old ones. The assets that were uploaded did not contain the expected content, causing errors on the pages they were used for, including login and admin page.
Once the issue was identified, the assets were re-uploaded and CDN cache for static assets flushed, which resolved the issue. In order to prevent this from happening again in the future, we will be ensuring asset uploads use sync to avoid overwriting, improving our asset upload deploy process and improving our login error monitoring and testing.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.