Re: Distribution delays on mailing lists for GNU LilyPond

[Ian Kelling]

Ian Kelling <iank@fsf.org> writes:

> Jean Abou Samra <jean@abou-samra.fr> writes:
>
>> Hello, dear administrators of GNU list servers,
>>
>> Sorry for the possible double post, I was not sure which address to
>> send this to. This is to let you know that mailing lists related
>> to the GNU LilyPond project have recently been experiencing abnormal
>> distribution delays.
> Hi, I work at FSF and am the main person there who works on the mailing
> lists.
>
> There are some common causes of delays.
>
> #1, most common: When someone posts to the list from an address that has
> never posted before, the message gets held until a moderator approves it
> to ensure it isn't spam. Once approved, their future posts are not held
> for moderation. The moderators are usually just the listhelper team,
> which is about 2 people who do this a few times a day. That is described
> here: http://listhelper.nongnu.org/. That can be done much more often if
> there are people who do moderation. All that needs to happen is that the
> list administrator/owner gives out the moderation password to people,
> and optionally add their address to the list moderator field so they get
> notified when a message is held for moderation. lilypond-devel has their
> list owner/admin listed as drl@gnu.org. So, I would ask them about that.
>
> #2, sometimes mailman gets hung up and delays sending out messages. We
> don't really know why, but we know that in order to debug the issue, we
> need to first move to mailman 3. Mailman 2 is no longer supported
> upstream. There is a good chance the issue will just go away when we
> move. Mailman 3 also has some nice features like allowing people to post
> directly from a web archive interface (the current archive interface
> will continue to exist in mailman 3, there will be 2 or more archive
> interfaces). We want to have a few lists try out mailman 3 to detect any
> issues before we move all the lists. Initially, the listhelpers won't be
> able to help remove spam, they will need a few months to port over their
> automation to mailman 3. There could be other bumps, but we will work
> hard to address them and if needed, move lists back to mailman 2 while
> we do. Does lilypond want one or more of their lists to be one of the
> initial group to move to mailman 3?
>
> #3, least common: sometimes there is an issue with a specific mail
> server which causes delays. This can often be fixed. I'm happy to check
> if that is the case, just email the details of the message to
> mailman@gnu.org. If it hasn't been posted, wait at least 3 hours from
> when you sent it. Please include the message-id header, from, to,
> subject, and as close to possible the exact time it was sent including
> a timezone.

I have an update. A lists.gnu.org user from emacs helped me identify
some occasional backlogs that were happening, causing 2 - 3 hour delays
of some messages every once in a while. I've done a deep dive into the
performance of the system, identified some bottlenecks, and figured out
how to eliminate them, making those tasks get done much more
efficiently. Unfortunately, there was some bumps in the road in that
process over the last week, leading to some long delays, but as of
yesterday things are stable. Previously, a post to lilypond-user would
often take 10 minutes to go out to ~1000 subscribers. Now, if there
aren't a lot of other messages being processed, that will go out to all
subscribers in about 2 minutes. Obviously, that will take longer if lots
of messages get posted at the same time, 10 messages will take 10
minutes or so, and messages from other lists can affect this, but there
are only a handful of lists on our server with lots of subscribers like
lilypond-user. The time to deliver scales linearly with the # of
subscribers.

The reasons for delayed messages I mentioned in my quote above still
exist, and I want to note again that lilypond can add moderators to
approve messages from new posters faster than the listhelpers, and this
should be even more helpful now that messages are processed faster.

The next major improvement I plan to work on is deploying public-inbox,
https://public-inbox.org/README.html, a new list archive which will
operate alongside the existing one. It has many nice features, one that
is not obvious is that it's search function should work better. An
example is here https://inbox.sourceware.org/binutils/ . Mailman 3 will
come after that.

-- 
Ian Kelling | Senior Systems Administrator, Free Software Foundation
GPG Key: B125 F60B 7B28 7FF6 A2B7  DF8F 170A F0E2 9542 95DF
https://fsf.org | https://gnu.org