The current groups administration leaves a lot to be desired.

Even with as little as 50 people, the limitations of the chosen "display all agents with a checkbox for group inclusion" approach is lacking.

I'd hope for something along these lines:

• filter the current set of agents & admins by (partial) name, department, ... i.e. include a search box that allows to enter the filter criteria much like in the search box of the ticket view.
• Why not be a bit innovative:
  Create a group pool box into which an admin can drag and drop or select/filter groups
  Create an agent pool box into which the admin can select/filter agents
  A button "Add selected agents to groups" (and a cancel, sure), to allow a single-operation multi-group inclusion of multiple agents.