Does selecting Only tickets in agent's groups prevent agents from searching other tickets?
Yes, when agents can only access tickets in their groups, search only returns tickets they have access to.
All other tickets are hidden. If agents search for a specific ticket number they do not have access to, Support doesn't search the ticket number. For example, if agents go to the URL (https://subdomain.zendesk.com/agent/tickets/###) of a ticket they do not have access to, Support shows access denied.
Although restricting an agent's access to see only tickets in their group restricts what tickets they can see in the search results, this does not affect the visibility of other search results, such as users or organizations. To learn more about further restricting an agent's permission see the article: Creating custom roles and assigning agents (Enterprise).