bug#52533: guix deploy breaks SSH access with a PAM error

[Maxim Cournoyer]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Mathieu Othacehe <othacehe@gnu.org> skribis:
>
>>> This sounds a lot like this:
>>>
>>>   https://issues.guix.gnu.org/32182#1
>>
>> I was just kicked out of my own server due to this PAM/SSH issue. It
>> happens quite frequently here. Time for a fix :).

Not a meaningful contribution to the discussion, but my workaround is to
disable PAM; as it is not enabled in OpenSSH by default, perhaps we
should also leave it off unless requested?  What are the advantages of
having it on?

> Note that ‘guix deploy’ now opens a single SSH session, starting from
> 7f20e59a13a6acc3331e04185b8f1ed2538dcd0a, which might help mitigate the
> problem.
>
>> Regarding the two potential solutions that you proposed in 2018, are
>> they still actual? If yes, I could maybe try to implement the second
>> suggestion: introducing service chain-loading.
>
> Service chain-loading was implemented in the Shepherd a few years ago.
> However, it doesn’t really help; consider these two scenario:
>
>   • You do ‘guix system reconfigure && herd restart term-tty1’.  In that
>     case, all is good: ‘term-tty1’, will run the new ‘mingetty’ process
>     (post-glibc upgrade, thanks to service chain-loading) and ‘login’
>     will happily load the .so files listed in /etc/pam.d/login (also
>     post-glibc upgrade).
>
>   • You run ‘guix system reconfigure’ but do not restart ‘term-tty1’,
>     ‘sshd’, and all the other services that depend on PAM: these
>     pre-glibc upgrade programs will try dlopening the post-glibc upgrade
>     PAM plugins, which will break.
>
> The crux of the problem rather is the global /etc/pam.d: it’s valid for
> pre-glibc upgrade programs, or for post-glibc upgrade programs, but not
> both.
>
> FHS distros have a similar problem though; how do they handle it?  Do
> they force services to be restarted when glibc is upgraded, or something
> along these lines?

I just asked this question in Debian's OFTC channel:

"how does debian handle glibc updates?  are services restarted when it
happens?  Or does it postpone updating glibc until the next reboot?"

And got for answer: "there is no magic postponing of updates"; the
external needrestart [0] program was also mentioned.

Researching some more, it seems this may be handled on Debian by the use
of postinst scripts (which is an arbitrary shell script run after a
package is installed); so the libc package of Debian for example
restarts the postgres service to avoid problems:

[0]  https://github.com/liske/needrestart
[1]  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=710275

> In our case, suppose libpam honors $PAM_DIRECTORY; we could tweak each
> PAM-using Shepherd service (login, sshd, etc.) so that it sets
> PAM_DIRECTORY… but how would we get the PAM_DIRECTORY value for the OS
> being configured?  Tricky!

Good question, but that seems a good path to pursue; old services would
be using their own old pam modules, allowing them to continue running
unimpacted, while new ones would get the updated pam modules.

> We could maybe sidestep the issue altogether with socket-activated
> services: they’d be started on-demand, so the second scenario above
> would be unlikely.  But getting there is quite a bit of work…

I fail to see how this would be a solution for openssh, which would
typically already be running unless you've never login ounce since the
machine was up (or am I missing something?).  Also, it seems to me inetd
can already do "socket activation", if this was somehow useful.

Thanks,

Maxim