If you go to your Groups section in the management console, you can see the rights you grant to each group. There is the option to "Publish group information to all extensions ".
If none of your groups have this, they will only be able to see their own group members.