Fun with SSH ControlMaster

So, there was this user, wondering why a different group membership is displayed depending on the host name used in the SSH login process:

$ ssh mallory id; ssh 10.0.0.2 id                 # Both point to the same machine!
uid=1000(dummy) gid=100(users) groups=100(users),16(dialout),33(video)
uid=1000(dummy) gid=100(users) groups=100(users),7(lp),16(dialout),33(video)
Huh? What happened here? After quite some digging, I found the following in the user's ~/.ssh/config:
Host *
        ControlMaster   no
        ControlPath     /home/dummy/tmp/ssh-%r@%h
And sure enough there was an active SSH connection and an active socket file in the ControlPath directory with a timestamp from a few weeks ago:
$ netstat -nl | grep tmp
unix  2      [ ACC ]     STREAM     LISTENING     16314    /home/dummy/tmp/ssh-dummy@mallory.Xnmcb2CghSke46qz
$ ls -l /home/dummy/tmp
srw-------. 1 dummy dummy 0 Jan 02 14:22 ssh-dummy@mallory
The use case for ControlMaster is, in short: establish an SSH connection once and then establish later SSH connections to the same host (and as the same user) over the already established socket.
And a ControlMaster connection has been established, but only to mallory, not to 10.0.0.2 (even though both adresses point to the same host). With ControlMaster=no set in ~/.ssh/config, new connections will 1) not try to set up a new ControlMaster but 2) will try to use existing sockets in ControlPath.

And that's exactly what happened: the "ssh mallory" would use the existing socket, the "ssh 10.0.0.2" would create a completely new connection.

Now, some time after the ControlMaster has been created (after January 2), the group membership of the user has changed: the user got added to another group ("lp").

New SSH connections to the host are just that: "new". And will therefore see the real thing. Old SSH connections over the ControlMaster socket will be spawned as a child process off that already existing SSH process that has been in place from before the group membership changed and will have an old view of the system. This can be reproduced quite nicely, using two terminals:
1|dummy@fedora0$ ssh -o ControlMaster=yes localhost

2|dummy@fedora0$ ssh localhost id; ssh 127.0.0.1 id
uid=1000(dummy) gid=1000(dummy) groups=1000(dummy)
uid=1000(dummy) gid=1000(dummy) groups=1000(dummy)
Now let's add dummy to a group and try again:
fedora0# usermod -a -G lp dummy
fedora0# id dummy
uid=1000(dummy) gid=1000(dummy) groups=1000(dummy),7(lp)

2|dummy@fedora0$ ssh localhost id; ssh 127.0.0.1 id
uid=1000(dummy) gid=1000(dummy) groups=1000(dummy)
uid=1000(dummy) gid=1000(dummy) groups=1000(dummy),7(lp)
I still don't know if this is a feature or bug, but I found it interesting enough to document :-)