SUMMARY
On September 19, 2022 from 16:32 UTC to September 20, 2022 03:28 UTC, Zendesk Support customers on Pods 15 and 25, experienced inbound tickets coming from Outlook or Hotmail with East Asian characters resulting in ticket comments appearing distorted.
Timeline
03:27 UTC | 20:27 PT
We are investigating reports of issues in Support where inbound tickets from Outlook or Hotmail results in East Asian characters on ticket comments appearing distorted. We will provide another update when we have more to share. Thank you for your patience in the meantime.
04:02 UTC | 21:02 PT
We have turned off a feature and have confirmed in a test account that East Asian characters are now appearing normally for newly created comments. Kindly confirm at your end, and let us know.
04:24 UTC | 21:24 PT
We are glad to inform you that the issue is resolved with East Asian characters appearing distorted in ticket comments. Previous comments need to be viewed via the original email source as a workaround. New comments will appear normally. Thank you for your understanding.
POST-MORTEM
Root Cause Analysis
This incident was caused by an unexpected error introduced in an update to backend email processing.
Resolution
The issue was fixed by turning off the feature introduced by the email processing update.
Remediation Items
- Improve testing scenarios to catch similar issues before they are introduced to production.
- Revisit email processing update to ensure compatibility with East Asian characters.
- Upgrade internal libraries to reduce complexity of releasing similar updates.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.