Does selecting Only tickets in agent's groups prevent agents from searching other tickets?
Yes, when agents can only access tickets in their group(s), search only returns tickets they have access to.
All other tickets will be hidden. If they search a specific ticket number they do not have access to, it will not treat the search ticket number. For example, if they go to the url (https://subdomain.zendesk.com/agent/tickets/###) of a ticket they do not have access to, it will display access denied.