Does selecting Only tickets in agent's groups prevent agents from searching other tickets?
Yes, when agents can only access tickets in their group(s), search only returns tickets they have access to.
All other tickets will be hidden. If they search a specific ticket number they do not have access to, it will not search the ticket number. For example, if they go to the url (https://subdomain.zendesk.com/agent/tickets/###) of a ticket they do not have access to, it will display access denied.
Although restricting an agent's access to see only tickets in their group will restrict what tickets they can see in Search results, this does not affect the visibility of other search results, such as users or organizations. To learn more about further restricting an agent's permission see our article on creating custom roles (only available on the Enterprise plan).