SUMMARY
On March 28, 2024 from 15:57 UTC to 17:50 UTC, a subset of customers were unable to update or publish new schedules in Tymeshift.
Timeline
17:53 UTC | 10:53 PT
We are investigating reports that a subset of customers are unable to update or publish new schedules using Tymeshift. We will provide new information shortly.
18:00 UTC | 11:00 PT
We have recovered from the issue causing a subset of customers to be unable to update or publish new schedules using Tymeshift; as a result this incident is now resolved. Thank you for your patience during our investigation.
POST-MORTEM
Root Cause Analysis
This incident was caused by an incorrect application configuration where the service account did not have the correct permissions to edit or create the schedule service, resulting in issues when attempting to update Tymeshift schedules.
Resolution
To fix this issue, the production service account was granted the necessary permissions. Once this was done, recovery was observed.
Remediation Items
- Review procedures to include use of feature flags for future changes[Scheduled]
- Review application configuration across environments to ensure consistency [Scheduled]
- Review smoke tests to cover similar scenarios [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.