20:15 UTC | 12:15 PT
We're happy to report the issue involving changes to Chat settings that some customers experienced is resolved. If you're still experiencing issues, please reach out to us.
20:00 UTC | 12:00 PT
We've identified the Chat accounts affected by this incident and are working directly with those customers. All other Chat accounts are functioning properly.
19:01 UTC | 11:01 PT
We are continuing to investigate an issue causing changes in some Chat accounts. Please accept our apologies for any inconvenience this has caused.
18:24 UTC | 10:24 PT
We are continuing to investigate and isolate an issue that was causing changes in some Chat accounts.
17:49 UTC | 09:49 PT
We are continuing to investigate reports of Chat settings changing unexpectedly for some customers.
17:27 UTC | 09:27 PT
We are currently investigating reports of some settings being unexpectedly changed in Chat.
On February 19, 2019 at 05:00 UTC (February 18, 21:00 PT), we started receiving reports of stale or missing agent information (including name, roles and department information) as well as issues with chat routing and other downstream functionality.
This incident was caused by a background job with heavier-than-normal changes to agent data, which triggered a performance degradation in a related database query. The slow database queries resulted in timeouts on an internal API preventing Chat from fetching updated agent information.
During the investigation, our engineers deployed a change to the database query associated with this degradation to resolve the issue. To prevent this issue from occurring in the future, we are building further robustness around our database queries by implementing additional alerting and refining our database indexes.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.