Service Incident - February 5, 2025 - Guide - Login issues on mobile browsers

SUMMARY

On February 5, 2025 from 10:50 UTC to 12:03 UTC, some Guide customers may have experienced degraded login functionality when accessing Guide from mobile browsers.

TIMELINE

February 05, 2025 11:48 AM UTC | February 05, 2025 03:48 AM PT
We are pleased to inform you that the login issues with our guide service have been identified and resolved by our engineering team. You should now be able to access the service without any problems. Thank you for your patience and understanding during this incident. If you continue to experience any issues, please reach out to our support team.

February 05, 2025 11:29 AM UTC | February 05, 2025 03:29 AM PT
We are experiencing service degradation with our guide service login from mobile browsers. If you encounter errors while accessing your account, please try logging in from a desktop. Our team is actively working to resolve the issue.

POST-MORTEM

Root Cause Analysis

This incident was caused by a bug that re-enabled an old rollout flag instead of removing it. The deployment of that code led to the mobile login feature being disrupted. The bug was able to reach production undetected due to insufficient testing and monitoring.

Resolution

To resolve the issue, the team executed a rollback of the deployment that had introduced the defect. This action restored normal functionality for the mobile authentication process, allowing users to log in successfully again.

Remediation Items

1. Monitor for Guide Login Failures: Implement a monitoring system to detect and alert on Guide login failures promptly.
2. Update Login Overview Dashboard: Enhance the visibility of a login overview dashboard by adding graphs for login methods to better track performance.
3. Canary Style Reduction in rollout flags: Implement a canary deployment strategy for rollout flags to minimize risk during future changes.
4. E2E/Regression Tests for Mobile Authentication: Develop end-to-end and regression tests specifically for Guide mobile authentication flows to catch issues before deployment.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.