bug-make
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Make losing jobserver tokens on Windows


From: Magnus Ihse Bursie
Subject: Re: Make losing jobserver tokens on Windows
Date: Mon, 21 Feb 2022 15:56:37 +0100
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.6.1

On 2022-02-18 16:04, Paul Smith wrote:
On Fri, 2022-02-18 at 10:18 +0100, Magnus Ihse Bursie wrote:
make[2]: INTERNAL: Exiting with 1 jobserver tokens available; should
be 24!

This effectively turns the highly parallelized builds into
single-threaded builds, and is absolutely detrimental for
performance. :-( On the flip side, this also makes for the perfect
testing environment to really get to the bottom of this issue.

So, my first question is: Is this a known problem? Are there any
workarounds/fixes to get around it?
Please be sure to tell us which version of GNU make you're using.
Right now it's:

GNU Make 4.3
Built for x86_64-pc-cygwin

but I've seen this issue as far back as I can remember (3.8x). But it is definitely likely to be a Cygwin-specific issue as you say. I'll try to bring this up with the Cygwin folks as well.

I am not aware of any issues like this.  However, I recommend that you
try using the Visual Studio build of GNU make and see if that works
better.  I'm not very familiar with the Cygwin version of GNU make;
it's not maintained by us but my understanding is that they've needed
to apply some patches (maybe this understanding is outdated though).
I'd definitely like to try to use the official version! Is there some pre-built binaries for these? I tried to look around but could not find anything newer than 3.81 at http://gnuwin32.sourceforge.net (and I'm not even sure if that's "official").

(In fact, prior to this discussion, I was not aware of any other option than Cygwin to run a decently modern version of GNU make on Windows.)

I'm trying to built it from source right now (and it does not seem complicated), but even so, an official binary makes sure I'm not introducing any issues from my local build environment.

I believe that the goal of the Cygwin port is to implement a POSIX
layer on top of Windows, so I'm assuming that when you compile GNU make
with Cygwin it uses the POSIX implementation of the jobserver.  This
implementation relies on certain POSIX behaviors of pipes and signals
and it's possible that the Cygwin wrappers here have an issue that is
leading to this problem.
Is there some way I can verify this hypothesis (e.g. debug output flags), other than looking at the Cygwin source on how they configure/build make?

/Magnus


The Windows jobserver uses Windows semaphores: it's a simpler
implementation than the POSIX one and may be more reliable since it's
using a native W32 implementation.

Note, all the above is just my personal noodling based on no actual
knowledge, as I'm not really a Windows person.

Maybe Eli will have some other thoughts.

Otherwise: Any suggestions on how to go on and debug this? I am
willing to build and test an instrumented debug build of make, but I
will need assistance to find my way around the source and spot likely
candidates for the source of the problem.
Well, the implementation is in the src/w32/s32os.c file in the make
source code.

But if the problem occurs reliably with a cygwin build and doesn't
happen with a Visual Studio build, then my suspicion is that it's an
issue with Cygwin rather than GNU make (since we don't see this kind of
problem on other POSIX-based systems).

Of course it could be a bug in GNU make where we are implicitly
assuming some kind of behavior which isn't actually guaranteed by
POSIX.




reply via email to

[Prev in Thread] Current Thread [Next in Thread]