Service Incident - April 5th, 2022 - Apps Marketplace Unavailable

SUMMARY

Between 7:43 and 10:50 UTC on April 5, 2022, customers were unable to access the apps marketplace.

Timeline

14:05 UTC | 7:05 PT

This incident has been resolved! We apologize for the inconvenience and thank you for your patience.

11:04 UTC | 04:04 PT

We are happy to report improvement to the Zendesk Marketplace accessibility where the URLs can now be reached. We are monitoring this until full resolution. Thanks for your patience.

10:30 UTC | 03:30 PT

We appreciate your patience while we work through this issue with the Zendesk Marketplace URLs being inaccessible. We have confirmed that this is not affecting its Apps, and all Apps should be loading correctly in full functionality at the moment.

10:09 UTC | 03:09 PT

We have received some reports of Zendesk Marketplace being inaccessible at the moment. Zendesk Apps are loading normally and correctly. Please bear with us while we investigate this further.

POST-MORTEM

Root Cause Analysis

When making an API call to our marketplace data source, we received an "Unreachable hosts" error. The page that is attempting to render does not get the required data, and throws a TypeError: Cannot read property 'props' of undefined, crashing the server.

Resolution

Reverted to backup marketplace instance.

Remediation Items

1. Create redundancy policy for critical application data
2. Update data source packages to latest versions
3. Upgrade ec2 instance sizes
4. Add rate limiting for marketplace

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.