Re: [PATCH] core-count: A new program to count the number of cpu cores

[Pádraig Brady]

On 31/05/11 01:28, James Youngman wrote:
> [ CC += bug-findutils, += Paolo, -= bug-coreutils ]
> 
> 2009/11/3 Pádraig Brady <address@hidden>:
>> Pádraig Brady wrote:
>>> Paolo Bonzini wrote:
>>>> Maybe we want a --parallel option (too bad -p is taken) for xargs that
>>>> forces the creation of the number of processes passed with -P or taken
>>>> from nproc (for example by starting "md5sum $1 $5 $9 ...", "md5sum $2 $6
>>>> $10 ...", etc.)?
>>>> That would be an interesting alternative to this core-count proposal...
>>>
>>> I'm not sure what you mean here.
>>> I already suggested to the xargs maintainer that `xargs -P`
>>> should be equivalent to xargs -P$(nproc).
>>> `nproc` as an external command would still be useful though.
>>
>> Here's a patch for that.
>> It needs to be updated to reference the new gnulib
>> when Bruno's nproc update hits gnulib.
> 
> I'm sorry, I totally missed this email because although it was sent to
> me it got buried under a ton of gnulib stuff.
> 
> As I understand things the intent of the patch is to (optionally I
> suppose) make xargs keener to keep the CPUs busy (or whatever else
> we're parallelising over I guess), even if it needs to launch
> processes with short command lines to do so?   That is With -P N, if
> there are less than N running children and xargs reads an argument it
> should launch a new child process?   If that's the approximate idea, I
> think that there could be use cases for this.
> 
> Could we discuss this aspect on bug-findutils instead?   Hence I
> changed the subject and the CC list...

The patch presented was flawed due to optional parameter processing.
It was just to do support `xargs -P $(nproc)` without needing `nproc`.

Also discussed was changing distribution of input to children.
I.E. by adding a --parallel option which on a machine with 4 available
cores would:

$ seq 1 13 | xargs --parallel
1 5 9 13
2 6 10
3 7 11
4 8 12

Note -P $(nproc) is implicit in the above.
I also discussed a mechanism using non blocking reading of input,
to more efficiently split the input parameters across children.

memory definitely hazy on this one :)

cheers,
Pádraig.