Re: make 3.81 + i386-pc-interix* parallelism problem, and "write error"

[Tim Murphy]

Hi

2009/5/6 Markus Duft <address@hidden>:
> On Wed, 2009-05-06 at 09:02 +0100, Tim Murphy wrote:

>> Do you really need it? Do you use sub-makes (i.e. do you have
>> makefiles that call make again on other makefiles)?
>
> hmm. we're using automake, so i guess yes, every subdir is managed by a
> sub-make. does building parallel inside one dir work even without
> jobserver, or is there no capability for this anymore as soon as i
> disable it? (i thought so until now, but what you're saying sounds
> somewhat like beeing able to keep building parallel, and just loosing
> the capability to spread down in subdirs in parallel).

What the jobserver does is to control the maximum number of concurrent
tasks across sub-makes.  e.g. if you ran with -j4 and had 4
sub-invocations of make you could have 16 tasks running at the same
time which might make builds slower rather than faster on a
particualar computer.

With the jobserver enabled, all the parent make and all it's sub-makes
have to ask for tokens from one jobserver.  This means that for -j4 a
total of 4 tasks can run at any one time and no more.

>> I might be proposing a huge effort for you and you might not be
>> compiling code that you can change at will, but if not then you could
>> try using include to compose a big makefile rather than using
>> sub-makes.
>
> hehe. you're right, thats a big effort, and i guess not an option as
> long as automake doesn't have propper support for that...

Automake is neither here nor there - I have seen this approach done
before with automake.  It depends on how you write your .am files -
basically all targets must be specified with a path that is either
absolute or at least relative to the top level directory so that they
are guaranteed to be unique.

>> This brings its own complexities but it is pretty cool because it
>> represents the whole dependency tree properly and it sort of allows
>> more parallelism.
>
> yeah that's right, but again: does this work without jobserver?

It works because you only have one make process so you don't need a
jobserver to co-ordinate.

>> Anyhow the alternative is to debug job handling for interix and work
>> out what's happening.  It could be some kind of issue with make
>> noticing that some job has finished, or something to do with the job
>> server blocking or just a bit of odd logic for interix.  It's not easy
>> to debug this without having the os and a debugger in front of you.

printf will work fine. :-)

> mhm... and bad enought gdb on interix is badly broken, so the better
> debugger is printf :-/ ...
>> 2009/5/6 Markus Duft <address@hidden>:
>> > On Tue, 2009-03-24 at 07:06 +0100, Markus Duft wrote:
>> >> On Tue, 2009-03-17 at 17:56 +0100, Markus Duft wrote:
>> >> > Hi!
>> >>
>> >> ping...
>> >
>> > ping ping.... at least a comment on this would be cool... don't leave me
>> > in the dark! i think my report is quite detailed, and not one of the
>> > "make is broken - fix it" ...
>> >
>> >>
>> >> >
>> >> > I'm running make on x86 interix (SFU/SUA). All works like a charm, 
>> >> > except
>> >> > for occasional problems with multi-core machines (I'm not sure if the 
>> >> > same
>> >> > problems would occur on single cores too...).
>> >> >
>> >> > I'm seeing two different problems, the "bader" one being a jobserver 
>> >> > token
>> >> > leak. Sometimes make exits with messages like this:
>> >> >
>> >> >     make: INTERNAL: Exiting with 3 jobserver tokens available; should be
>> >> > 2!
>> >> >
>> >> > But much worse is the case where make simply hangs, and seems to be out 
>> >> > of
>> >> > jobserver tokens! Is this a known problem? I tried the patch from [1], 
>> >> > but
>> >> > that didn't make any difference.
>> >> >
>> >> > Also I found a patch against 3.80, adding the jobserver_tokens global
>> >> > variable, so I assume that one has been incorporated into 3.81, since 
>> >> > this
>> >> > variable already exists... (?).
>> >> >
>> >> > I attached the output of this command and the corresponding .diff file:
>> >> >     while ( cd tests && perl ./run_make_tests.pl features/parallelism
>> >> > -make ../make -debug -verbose ); do echo ...; done > xx 2>&1 &
>> >> >
>> >> > the test-1 log is seemingly another problem, which occurred while I 
>> >> > tried to
>> >> > faster provocate the error I'm reporting here by putting my 2 core AMD 
>> >> > CPU
>> >> > under heavy load. Ugh... another one :(
>> >> >
>> >> > FYI I attached the config.log too, so you can identify potential issues
>> >> > caused by assumptions which may be wrong for interix.
>> >> >
>> >> > [1] http://savannah.gnu.org/bugs/?18124
>> >> >
>> >> > ----
>> >> >
>> >> > Now for the second problem, which is far easier to work around: Again 
>> >> > only
>> >> > occasionally I'm getting "make[N]: write error". After
>> >> > R[eading]T[he]F[ine]S[source] I found out that ferror(stdout) triggers 
>> >> > this
>> >> > message. Strange... seems like there was an error writing to stdout..? 
>> >> > or
>> >> > could this point to an error writing to the jobserver fds?
>> >> >
>> >> > I worked around the problem for now by simply not exiting on this error,
>> >> > since it's not too critical for me if a line in the output is missing or
>> >> > something like that. Also since it's only occasionally, I can simple do 
>> >> > make
>> >> > again, If I really need all the output.
>> >> >
>> >> > BTW I never noticed missing output when I got "write error", but output
>> >> > tends to be kilometers long, so I could have overseen that :)
>> >> >
>> >> > Any ideas regarding those two?
>> >> >
>> >> > P.S.: I just noticed that the --enable-case-insensitive-file-system 
>> >> > (interix
>> >> > is insensitive, as it uses window's fs') is win32 only as-is, since it 
>> >> > uses
>> >> > strcmpi. Attached is a patch fixing this for interix. Maybe this fixes 
>> >> > for
>> >> > this feature for mac osx too (is there support for max osx yet?).
>> >> >
>> >> > Thanks in advance, Cheers,
>> >> > Markus Duft
>> >> >
>> >> > _______________________________________________
>> >> > Bug-make mailing list
>> >> > address@hidden
>> >> > http://lists.gnu.org/mailman/listinfo/bug-make
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Bug-make mailing list
>> >> address@hidden
>> >> http://lists.gnu.org/mailman/listinfo/bug-make
>> >
>> >
>> >
>> > _______________________________________________
>> > Bug-make mailing list
>> > address@hidden
>> > http://lists.gnu.org/mailman/listinfo/bug-make
>> >
>>
>>
>>
>
>



-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/