Change the behaviour for AI responses when an agent is needed

We see customer confusion in scenarios where:

1.  The client's issue requires the agent's involvement
2. the bot generates an AI reply with "To resolve this issue, contact support" based on our Help center articles.
3. The bot goes on to ask “Was that helpful?”.
4. The customer clicks "yes" and the dialogue ends. However, the right thing to do would be to redirect the customer to the agent afterward.

We are thinking about how to reword the knowledge base articles to solve this problem, but it seems to me that we are not the only ones facing this.