Does selecting Only tickets in agent's groups prevent agents from searching other tickets?
Yes, when agents can only access tickets in their group(s), search only returns tickets they have access to.
All other tickets will be hidden. If they search a specific ticket number they do not have access to, it will treat the search as the number itself not a ticket number. If they go to the url (https://subdomain.zendesk.com/agent/tickets/###) of a ticket they do not have access to, it will display access denied.