Yes, it does seem that you can only set it to one group. The only reason is like you are saying, having permissions for one set of user to read/write and one to read, can just be done with 'rw' for the group, and 'r' for the world.
It is a bum deal with the single user thing though. I guess one thing you could do is have a group set to a dir like /home/group with a umask of 032 for the read function. Then you could make another dir for a different group with a umask of 012 for your 'rw' files. At least you can still assign users multiple groups.