Re: New module nproc

[Jim Meyering]

Hi Glen, et al,

Thanks for working on this!

Glen Lenker wrote:
...
> When we first looked at parallelizing sort, it was actually our first
> choice to use glthreads. We were on the fence though between pthreads
> and glthreads because we were concerned that Pth might complicate our
> changes if glthreads defaulted to Pth in the absence of pthreads. We
> eventually decided to just use pthreads when we posted to coreutils
> and Jim Meyering said the he would prefer pthreads. You are right, Its

Just so we're clear, when I heard work was progressing on two
implementations, one pthreads-based and another openmp-based,
I said the former sounded more likely to be accepted.

> an easy change (we changed the two letters before), and to be honest I
> don't see anything wrong with using glthreads instead of just
> pthreads.
>
> We also considered using libgomp's omp_get_num_procs before, but
> decided to go along with just sysconf. I looked at the Linux
> implementation and from what I could understand, saw that libgomp
> would either use sysconf itself, or return the number of cpus
> available its affinity masks. Since we don't use libgomp anywhere else
> in the code, I didn't think it would make sense to use their use
> libgomp to detect the number of processors and then just create the
> threads myself, since even if it did whatever information it had on
> running tasks and processors, it probably wouldn't be different from
> the number returned from sysconf.

libgomp does handle one other case: that in which sysconf
can't do the job, yet sysctl's HW_NCPU can:

static int
get_num_procs (void)
{
#ifdef _SC_NPROCESSORS_ONLN
  return sysconf (_SC_NPROCESSORS_ONLN);
#elif defined HW_NCPU
  int ncpus = 1;
  size_t len = sizeof(ncpus);
  sysctl((int[2]) {CTL_HW, HW_NCPU}, 2, &ncpus, &len, NULL, 0);
  return ncpus;
#else
  return 0;
#endif
}

And according to xz's (git://ctrl.tukaani.org/xz.git) log history for
src/common/cpucores.h, that is useful at least for systems using dietlibc.

> As far as mingw is concerned, I was hoping that someone else could
> help with this (I don't have a Windows machine). How easy would it be
> to get the nproc module to work under Windows?

That's a fine approach.
Our first priority should be Linux and Unix.
If someone wants to spend time on mingw, afterwards, that'll be
icing on the cake.  IMHO, making mingw support a prerequisite would
be counterproductive.

>> Also, the omp_get_num_procs() function has the advantage that you can
>> influence it through an environment variable, so that users have the ability
>> to let multithreaded programs access only, say, 3 out of 4 CPU cores, if
>> a responsive machine is more important to them than fast execution.
>
> I like this. This would be a very useful ability, and I am sure there
> would be a lot of other perks for using OpenMP instead of using
> pthreads. I am just not sure that this should be the main reason for
> including OpenMP into the code.

The code is small.
Perhaps using its ideas is enough.