SUMMARY
On November 13, 2025, between 01:40 UTC (November 12, 2025 05:40 PM PT) and 13:35 UTC (November 13, 2025 05:35 AM PT), customers across all pods experienced intermittent 500 errors that impacted their ability to access the dashboard in the Zendesk QA environment.
Timeline
November 13, 2025 08:37 AM UTC | November 13, 2025 12:37 AM PT
Our engineering team is aware of the intermittent 500 errors affecting dashboard access in Zendesk QA and is actively collaborating with our external service provider to thoroughly investigate and resolve the issue. We will provide updates as soon as new information becomes available. Thank you for your patience and understanding.
November 13, 2025 10:46 UTC | November 13, 2025 02:46 AM PT
Our engineers received confirmation from our vendor partner that a mitigation was deployed to address intermittent 500 errors in Zendesk QA. Despite this, our monitoring tools continue to detect ongoing issues. We remain engaged with our vendor partner to investigate and resolve the problem. We apologize for the disruption and will keep you informed as updates become available.
November 13, 2025 12:50 UTC | November 13, 2025 04:50 AM PT
Our Engineering team has identified the cause of the recent intermittent errors affecting Zendesk QA. Our team is actively collaborating with our vendor to resolve the issue and prevent future occurrences. We apologize for the inconvenience and appreciate your patience. Updates will be shared as soon as they become available.
November 13, 2025 13:51 UTC | November 13, 2025 05:51 AM PT
Our engineers have observed that dashboards are now loading on their end following the implementation of a remediation by our vendor partner to address the intermittent 500 errors in Zendesk QA. We kindly request your feedback to confirm whether this fix is effective for you. Please do not hesitate to contact us if the issue continues.
November 13, 2025 17:04 UTC | November 13, 2025 09:04 AM PT
We are pleased to inform you that the intermittent 500 errors affecting Zendesk QA have been successfully resolved. We sincerely appreciate your patience as our team worked diligently to address this matter.
Root Cause Analysis
The incident was likely caused by a bug in a third-party service responsible for embedding the dashboard within the application. During the event, both the vendor service and dashboards remained accessible; the issue was isolated to the embedding function.
Resolution
To resolve the issue, our engineers restarted the vendor service and collaborated with vendor support to investigate and address deadlocks. The vendor has implemented a remediation, and dashboards have been confirmed loading successfully for internal users.
Remediation Items
- Review and update the entire ZQA team’s escalation process, including the Embedded Reporting team and related documentation.
- Verify and adjust access permissions for QA GCP dashboard instances.
- Update the run-book to include involving the Mario team for troubleshooting ZRA dashboards.
- Ensure the designated on-call ZQA engineer has full admin access to Looker logs and the ability to escalate issues as needed.
- Explore integrating Looker logs with Datadog for improved monitoring.
- Consider implementing a smoke test to verify access to the ZQA Dashboard.
- Identify and monitor additional metrics for the Looker dashboard.
- Track and confirm the permanent vendor fix for the Looker deadlock issue causing dashboard errors.
- Establish proper advance notification procedures for Looker instance upgrades and consider scheduling upgrades during off-peak hours.
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.