Re: "*** INTERNAL: readdir: Invalid argument" error

[Paul Smith]

On Wed, 2013-05-15 at 17:45 +0000, Edward Welbourne wrote:
> > Exactly.  Right now, the code already walks along PATH, and constructs
> > <dir>/sh.exe, but instead of calling 'stat' or 'access', it calls
> > 'file_exists_p', and the side effect of that is that Make reads the
> > entire directory <dir> into its internal storage, hashes it, etc.
> 
> Am I the only one finding it hard to believe that's anything but
> hugely inefficient ?  Testing for the existence of a file should not
> involve iterating the whole of its directory !  Well, at least, not in
> userspace code - if the O/S's implementation of the system function
> does that, that's the O/S's affair.

This function was not specifically designed for this use-case.  It's
mainly used to look up targets.  Part of what it does is cache the
results of the directory read so that subsequent lookups don't have to
use a system call, they can just check the cached contents of the
directory and determine whether a target exists or not.  In order to
make that cache it obviously has to use readdir() so it takes over some
aspect of the work that the OS would do anyway to look up the file.

Because of the amount of file and directory lookups make does (remember,
for every target that make might want to build, if it uses an implicit
rule there are a LOT of stat()-type calls to see if some variation of
that file exists, its prerequisites, etc.) this caching is has been
proven to be a big win in the past.

It's unlikely that the caching will be helpful when looking through your
%PATH%, in fact it may be counter-productive (adding useless directories
to make's internal cache).  Make will likely not be trying to build
targets in directories in %PATH%.  So really it's probably not the best
idea to use the caching readdir functionality to search for sh.exe.