Re: [bug #33138] .PARLLELSYNC enhancement with patch

[Tim Murphy]

[sorry to Eli for forgetting to replay-all]

This is an awesome feature, especially for anyone trying to parse huge parallel build logs with errors in them - used to have this problem where you couldn't work out why X.cpp had that error at line 21 and realised that it was because that error didn't actually belong to that output.

IMHO, get it in and fix the details later because as it is it's exceedingly useful.

What would be super cool is being able to get make to expand some sort of variable at the start and another one at the end of the output so that there was a way to see where one rule ended and the next one began.

When most rules are a single job this doesn't seem important but when you're doing anything non trivial it becomes hard to see what is where.

Regards,

Tim

On 16 April 2013 09:56, Tim Murphy <address@hidden> wrote:

This is an awesome feature, especially for anyone trying to parse huge parallel build logs with errors in them - used to have this problem where you couldn't work out why X.cpp had that error at line 21 and realised that it was because that error didn't actually belong to that output.

IMHO, get it in and fix the details later because as it is it's exceedingly useful.

What would be super cool is being able to get make to expand some sort of variable at the start and another one at the end of the output so that there was a way to see where one rule ended and the next one began.

When most rules are a single job this doesn't seem important but when you're doing anything non trivial it becomes hard to see what is where.

Regards,

Tim

On 16 April 2013 09:30, Eli Zaretskii <address@hidden> wrote:

> Date: Tue, 16 Apr 2013 05:54:13 +0000
> From: "Paul D. Smith" <address@hidden>

>
> I did a little bit of code rearrangement, but I still think this code will not
> work on Windows and might possibly not compile on Windows.

Indeed, it will not.  Some cursory comments below.

> Hopefully we can fix that.

We shall see...

Here's what I see in the changes that is not friendly to Windows:

 . STREAM_OK uses fcntl and F_GETFD which both don't exist on Windows.

 . FD_NOT_EMPTY will only work if its argument descriptor is connected
   to a file, not the console.  is this condition always true?

 . open_tmpfd will need close examination on Windows, especially since
   it closes the stream (the issue is whether the file will still be
   automatically deleted when the dup'ed file descriptor is closed).

 . pump_from_tmp_fd will need to switch to_fd to binary mode, since
   this is how tmpfile opens the temporary file; we need output mode
   match the input mode.

 . acquire_semaphore uses fcntl and F_SETLKW, which don't exist on
   Windows.  the commentary to that function is not revealing its
   purpose, so I'm unsure how to implement its equivalent on Windows.
   can someone explain why is this needed and what should it do?  the
   name seems to imply that using fcntl is an implementation detail,
   as a crude semaphore -- is that right?  similarly for
   release_semaphore.  (see also the next item.)

 . is there any significance to the fact that sync_handle is either
   stdout or stderr?  is that important for the synchronization to
   work, or was that just a convenient handle around?  also, the
   documentation I have indicates that locking works only on files;
   is it guaranteed that these handles are redirected to files before
   acquire_semaphore is called?

 . calculation of combined_output in start_job_command will need to be
   reimplemented for Windows, since the reliance on st_dev and st_ino
   makes assumptions that are false on Windows.

Other notes:

 . outputting stdout and stderr separately is IMO a misfeature: it
   breaks the temporal proximity of messages written to these streams,
   and this makes it harder to understand failures.

 . why is "leaving directory FOO" displayed _before_ releasing the
   stderr output? it should be after, I think.

 . AFAIU, the call to fcntl inside sync_output can be interrupted by a
   signal, and will return -1.  shouldn't the code do something
   non-trivial to avoid a total failure in this case?

 . suggest to use "acquire_semaphore" and "release_semaphore" in the
   call to perror in those two functions, as "fcntl" is not specific
   enough, and also reveals implementation details.

 . likewise, "read()" and "write()" in diagnostics from
   pump_from_tmp_fd are less specific than they could be; how about
   "read from temporary file" instead?

 . btw, why an error in acquire_semaphore and release_semaphore causes
   Make to bail out?  isn't it better to fall back on the normal
   operation mode instead?

 . I don't understand why PARLLEL_SYNC is turned on unconditionally in
   job.h based on POSIX being defined.  it should IMO be in config.h
   instead, set at configure time.

 . the new argument to log_working_directory is not documented in the
   commentary to the function.

Thanks.

_______________________________________________
Bug-make mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/bug-make

--
You could help some brave and decent people to have access to uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/

--
You could help some brave and decent people to have access to uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/