Re: Savannah bug->email gateway problems affecting groff

[Bob Proulx]

Dave Kemper wrote:
> In the past couple days I've made several updates to groff Savannah
> tickets, which have always generated an immediate email to me and, not
> too long after, one in the lists.gnu.org archive of the bug-groff
> email list.  Neither of these has been getting the emails since August
> 12.  (See, for instance, http://savannah.gnu.org/bugs/?62901, which
> has three comments by longtime groff project members, requiring no
> moderation; none of these appear in the email archive at
> http://lists.gnu.org/r/bug-groff/2022-08/.)

This was due to a misconfiguration which took me way too long to
figure out.  Fixed now as of a few minutes ago.

Found the systemd config included NoNewPrivileges=yes which completely breaks
things.  It prevents all suid in child processes such as /usr/sbin/sendmail and
anything else too.  Completely breaking it.

I removed that restriction and all is working again.  Re-Upgraded all
to the latest security releases.  I had downgraded for testing if
that was the problem.  All seems okay now.

The file with this configuration was set up May 27 therefore presumably apache
had not been restarted since then.  That's about the time this new web
server was brought online.  So I presume the dust was still settling.

On the 12th I applied the recent Trisquel point release upgrades which
included apache and the restart of apache on the 12th got that config
setup from May 27th for the first time.  This added to the confusion
since from my perspective only the security patch release had happened
recently and the change from May 27th was over the horizon old.  But
it was just a long "hang-fire".

See also https://savannah.nongnu.org/support/?110692 where other
people were impacted too.

> Similarly, a comment I added to
> http://savannah.nongnu.org/support/?110692 never showed up in my inbox
> (past comments I've submitted here have always shown up immediately)
> and is not reflected in
> http://lists.gnu.org/r/savannah-hackers/2022-08/.

The application of NoNewPrivileges=yes "broke things good".  This
prevented apache from sending email since sending email requires suid
functionality to work.  Lots of the system requires set-uid to work.
The system needs setuid in order to remain secure.  Disabling it is
very much like the classic problem with premature optimization.  It's
probably not the root of *all* evil but definitely the root of this evil.

> > I know I have been busy in real life and haven't been
> > able to attend the keyboard much the past few days.  But others in the
> > team should be able to handle things.
>
> I'm happy to redirect this query, but I don't know who else to contact
> about this, and there seems little point in filing a Savannah ticket
> over it, since no one is getting notified about them while email
> notifications are down.

It is tragic that the other couple of tickets that were filed with the
ticket system could not send an email notification to the Savannah
team.  I only saw them after fixing the problem and then going looking
for new tickets.

For my part I feel that it took me much too long to debug the problem.
I was completely unaware that systemd could set NoNewPrivileges=yes
and was not thinking of looking there for problems.  It just isn't
something I would have ever guessed happening!  I can only say that
eventually I got there in the end.

In the debugging of this I found it interesting that apachectl calls
systemctl and systemctl calls apachectl.  They break the loop by using
an internal variable as a flag to stop calling the other.  A comment
says they do this otherwise systemd gets confused.  This only happens
during start and both stop and restart are normal.  Isn't it a
wonderful world these days?  Not!  Hopefully that was read with a
heavy level of sarcasm.

Bob