SUMMARY
On June 1, 2025 from 22:00 UTC to June 5, 2025 at 12:00 UTC, customers experienced slow response times for conversations and latency issues in AI Agents.
Timeline
June 04, 2025 12:44 PM UTC | June 04, 2025 05:44 AM PT
We have received reports of customers experiencing slow response times in AI Agents. Our engineers are actively investigating this issue and working on a mitigation. We apologize for this inconvenience and will provide updates as it becomes available.
June 04, 2025 02:22 PM UTC | June 04, 2025 07:22 AM PT
Our engineers have identified that the latency issue with uGPT is only affecting 15% of conversations. Nevertheless, they are working proactively to mitigate this at the soonest time. We appreciate your patience on this issue and we will provide updates as they become available.
June 05, 2025 09:41 AM UTC | June 05, 2025 02:41 AM PT
We are pleased to report that the response levels of AI agents have returned to normal, and our team is closely monitoring the situation. We appreciate your patience while we address this issue, and we will provide a final update upon its resolution.
June 06, 2025 12:15 PM UTC | June 06, 2025 05:15 AM PT
We are pleased to inform you that the latency response issue with uGPT has been resolved. We appreciate your patience as our engineers worked to fix this issue.
Root Cause Analysis
This incident was caused by a group of servers that ran out of available network IP addresses to assign to new tasks. This prevented new tasks from starting and caused network delays and connection problems. Moving the work to another group with enough available addresses fixed the problem.
Resolution
To fix this issue, our engineers moved the backed up work to a different group of servers that had enough available IP addresses.
Remediation Items
- Set up notification alerts when the available network addresses reach 80% utilization to prevent the issue occurring and so engineers immediately take necessary actions.
- Reduce alert noise by organizing uGPT notifications into clearer, more meaningful groups.
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.