Article updated on March 17, 2023 with post-mortem of unexpected/extended downtime
Summary
On February 27, 2023 from 02:00 UTC to 04:24 UTC, Zendesk Jira integration customers experienced unexpected downtime following a scheduled maintenance window that was intended to finish at 02:00 UTC. Due to unforeseen circumstances, issues with the maintenance resulted in extended downtime for those customers using the integration.
Root Cause Analysis
This incident was caused by a process breakdown during a scheduled database migration. A critical step was missed towards the end of the first run of the maintenance work resulting in the migration needing to be restarted. There was insufficient time to complete the tasks in the advertised window and as such a decision was made to re-run the work and communicate the extension of the window to our customers. Rescheduling the maintenance window was discussed, however on balance the best option was to complete the migration on the same day during the low traffic period to limit inconvenience for our customers.
Resolution
To fix this issue, our team identified the remaining database migration steps, completed those tasks and then restored service to the Jira integration in our new database cluster.
Remediation Items
- Update process for self-service database migrations to ensure no steps are missed in future.
__________________________________________
Advertised maintenance window details
Zendesk will perform critical maintenance that will impact performance for Jira integration customers on February 26-27, 2023 UTC during the time listed below. This is a rescheduled maintenance window that was postponed from the original planned time on February 5, 2023.
Affected products:
- Jira integration
Integration |
Date |
Start Time |
End Time |
Jira |
Sunday, February 26, 2023 |
15:00 PST / |
18:00 PST / |
Expected behaviour:
- During the outage all functionality of the Jira integration will stop working. The sidebar app in both Zendesk and Jira won’t work, and field syncing will also cease to function.
Why we're doing this: The Zendesk Integrations team is upgrading AWS instances for better performance.