Default group change is not effective

When changing the "Default group" assignment from one group to another, it shows up in the admin center as expected. But even after making the change it does not allow me for removing members from the group "that once has been the Default group". Surely this can not by design like this ?