Summary
On November 14, 2021 from 03:00 AM UTC to 05:38 AM UTC, some of our Chat customers using our mobile app were not automatically reconnected to our service after our scheduled maintenance for the day had finished.
Timeline
05:21 UTC | 21:21 PT
Our internal monitoring has revealed that a process to reconnect agents using the Zendesk Chat mobile app didn't go according to plan during today's maintenance window. If your agents are using the mobile app, please ask them to log out of the app then log back in again to reconnect. We apologize for any inconvenience caused.
Root Cause Analysis
This incident was caused by an interference of the post maintenance recovery steps. While all of these have been tested prior to the maintenance individually, unfortunately our connection servers went into maintenance mode before a list of all of our at the time active mobile apps using customers could be exported.
This meant that when the servers came back online, the process could not re-add all agents in the correct status.
Resolution
To fix this issue, our impacted customers needed to manually log back into the Chat mobile app.
Remediation Items
- We have updated our maintenance procedures for Chat to avoid a recurrence of this behaviour.