[bug #22923] option to prevent "interspersed" output in parallel builds

[Timothy N Murphy]

Follow-up Comment #8, bug #22923 (project make):


Hi 

My team do this with GNU make at the moment by using a replacement shell.  

It was proprietary until fairly recently and now it has been released under
the EPL.  It is hosted by the Symbian Foundation and you'd need a login but
since it's EPLd now there's nothing stopping me from mailing it to someone.

It's called Talon and you use it by setting

SHELL:=talon
TALON_SHELL:=bash

So make executes recipes in the Talon shell and Talon spawns them in Bash. It
buffers the full command output and finally tries to grab a system-wide
semaphore (not on clusters) before printing the output. 

There are some tricks. e.g. if you have multiline recipes then  you need to
join them up into one line to prevent the iondividual steps from interleaving
e.g.

fred.exe: fred.o bob.o
    link $^ -o address@hidden
    elf2e32 address@hidden -o $@

. . .becomes . . .

fred.exe: fred.o bob.o
    link $^ -o address@hidden && 
    elf2e32 address@hidden -o $@

 


It has some other nice features like:

* timeouts - don't let a single badly behaved command prevent you from
finding all the other issues in your 10 hour build.

* retries - being able to retry build commands (in case of intermittent
compiler license server failures or network problems). 

* easily-parsed logs - It is also able (optionally) to wrap each bit of
output in delimiters (XML actually) and print the exit  code of commands.  You
basically don't have to have hundreds of programs different error regexps in a
table to know what went wrong in the build.  You can also see clearly where
commands from one recipe stop and commands for the next recipe begin.

* performance data - it can print out the time taken by each command and
commands can be given a type e.g. "link" so that you can easily find out at
the end of the build how much time linking takes as a proportion and you know
if it's more critical than other types of task.

We used to do most of this in bash but had problems with implementing timeout
and retry.  Bash 4 seems to have those so it might be worth another look.  At
the moment Talon is in plain C, works on Windows and Linux and someone got it
going on a Mac with a small effort.

In "Example Recipe 1", at the end of this post, I've listed the
de-interleaved output for one recipe. All most log parsers need to know is if
a recipe failed or not (in the status tag at the bottom) and which component
and project to blame for the build failure (in the start tag). The time is
also in the tag at the bottom,  




EXAMPLE RECIPE 1

<recipe name='postlink'
target='/home/tmurphy/test-epocroot/epoc32/release/armv5/urel/test_pp2.exe'
host='rhodes' layer='' component=''
bldinf='/home/tmurphy/hg/build/sbsv2/raptor/test/smoke_suite/test_resources/pp/pp2.inf'
mmp='/home/tmurphy/hg/build/sbsv2/raptor/test/smoke_suite/test_resources/pp/pp2.mmp'
config='arm.v5.urel.gcce4_4_1' platform='armv5' phase='ALL' source=''>
<![CDATA[
+ /home/tmurphy/test-epocroot/epoc32/tools/elf2e32 --sid=0x10003a5c
--version=10.0 --capability=TCB+ProtServ+DiskAdmin+AllFiles+PowerMgmt+CommDD
'--linkas=test_pp2{000a0000}[00000001].exe' --fpu=softvfp --targettype=EXE
--output=/home/tmurphy/test-epocroot/epoc32/release/armv5/urel/test_pp2.exe
--elfinput=/home/tmurphy/test-epocroot/epoc32/release/armv5/urel/test_pp2.exe.sym
--uid1=0x1000007a --uid2=0x100039ce --uid3=0x00000001 --priority=Low
--stack=0x2000 --heap=0x5000,0xffff --debuggable --paged
--compressionmethod=inflate
--libpath=/home/tmurphy/test-epocroot/epoc32/release/armv5/lib

]]><time start='1262514094.06189' elapsed='0.014' />
<status exit='ok' attempt='1' />
</recipe>



The various data about the target (e.g. which component the target belongs
to) is cuistomisable and can be added to the start tag through adding a
special prefix onto commands.

fred.exe: fred.o bob.o
    |NAME='linknpostlink';COMPONENT=mycomponent;PROJECT=fred.prj;|link $^ -o
address@hidden && 
    elf2e32 address@hidden -o $@

# then we can tell Talon how to use these parameters we sent it:

TALON_RECIPE_ATTRIBUTES:=name='$$NAME' component='$$COMPONENT'
project='$$PROJECT'

which gives you somethig roughly like:

<recipe name='linknpostlink' component='mycomponent' project='fred.prj'>
<![CDATA[
+ link fred.o bob.o -o fred.exe.sym
+ elf2e32 fred.exe.sym -o fred.exe

]]><time start='1262514064.04133' elapsed='0.023' />
<status exit='ok' attempt='1' />
</recipe>


I'll put it up somewhere if anyone is actually interested.

Regards,

Tim

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?22923>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/