RE: Simpler example of pathological behavior of directory caching

[Martin Dorey]

> If you put the makefile in a different directory
...
> Or if you list it explicitly but disable built-in rules

It was sadly unclear to me that you'd done one of those with your:

>> $ make
>> touch target1
..
>> glob: target3 target10 target9 target8 target7 target6 target5 target1 
>> target2 target4

> Clearly there's a bug here.

I don't think it's a recent regression or an implementation flaw so much as a 
hole in the design - some missing communication to dir.c, perhaps from file.c.  
Perhaps there's still some value in some of the mail I was working on:

Once an invocation of make with no arguments has populated its 
directory_contents cache of the "." directory, which it normally does here, if 
you call the makefile "Makefile" and have nothing else in the directory:

(gdb) bt
#0  __closedir (dirp=0x65f170) at ../sysdeps/posix/closedir.c:34
#1  0x000000000040a453 in dir_contents_file_exists_p (dir=0x65f080, 
filename=0x42a753 "GNUmakefile") at dir.c:748
#2  0x000000000040a76e in dir_file_exists_p (filename=0x42a753 "GNUmakefile", 
dirname=0x42aef5 ".") at dir.c:770
#3  file_exists_p (name=0x42a753 "GNUmakefile") at dir.c:819
#4  0x000000000041de31 in read_all_makefiles (makefiles=<optimized out>) at 
read.c:250
#5  0x0000000000406e0f in main (argc=<optimized out>, argv=<optimized out>, 
envp=<optimized out>) at main.c:1969
(gdb)

... or here if you call it GNUmakefile:

(gdb) bt
#0  __closedir (dirp=0x65f170) at ../sysdeps/posix/closedir.c:34
#1  0x000000000040a453 in dir_contents_file_exists_p (dir=0x65f080, 
filename=0x655906 "GNUmakefile.o") at dir.c:748
#2  0x000000000040a76e in dir_file_exists_p (filename=0x655906 "GNUmakefile.o", 
dirname=0x42aef5 ".") at dir.c:770
#3  file_exists_p (name=0x655906 "GNUmakefile.o") at dir.c:819
#4  0x0000000000412007 in pattern_search (address@hidden, address@hidden, 
address@hidden, address@hidden) at implicit.c:737
#5  0x0000000000412980 in try_implicit_rule (address@hidden, address@hidden) at 
implicit.c:45
#6  0x0000000000420241 in update_file_1 (depth=<optimized out>, file=0x667860) 
at remake.c:523
#7  update_file (address@hidden, depth=<optimized out>) at remake.c:336
#8  0x0000000000420c57 in update_goal_chain (address@hidden) at remake.c:151
#9  0x000000000040710a in main (argc=<optimized out>, argv=<optimized out>, 
envp=<optimized out>) at main.c:2234
(gdb)

... then I failed to find any source that invalidates that cache, be it the 
whole cache, just one directory or one entry in that directory.  The only thing 
I found that changes the cache after the directory reading has completed is 
file_impossible, and I don't see where that's used for targets that are 
possible.  $(wildcard target*) seems like it's always going to call glob and 
glob seems to always use the directory_contents cache.

I can engineer an intermediate result if I create a load of junk in the 
directory too.  It's missed out ~half the glob matches:

address@hidden:/tmp$ mkdir kyle-rose2
address@hidden:/tmp$ cd kyle-rose2
address@hidden:/tmp/kyle-rose2$ cp ../kyle-rose/Makefile .
address@hidden:/tmp/kyle-rose2$ cat GNUmakefile 
all: $(addprefix target,1 2 3 4 5 6 7 8 9 10)
        @echo glob: $(wildcard target*)

target%:
        touch $@
address@hidden:/tmp/kyle-rose2$ for ii in `seq 10000`; do touch junk$ii; done
address@hidden:/tmp/kyle-rose2$ ~/download/make-git/make --no-builtin-rules
touch target1
touch target2
touch target3
touch target4
touch target5
touch target6
touch target7
touch target8
touch target9
touch target10
glob: target3 target5 target6 target2 target10 target1
address@hidden:/tmp/kyle-rose2$

I think that's because it starts reading the directory to look for GNUmakefile, 
stops when it finds it, then resumes the readdir() calls after the "target" 
files have been created.  There's enough junk that glibc has to go back to the 
kernel, not just to its getdents results.

-----Original Message-----
From: Bug-make [mailto:address@hidden On Behalf Of Paul Smith
Sent: Thursday, October 06, 2016 16:09
To: Kyle Rose; address@hidden
Subject: Re: Simpler example of pathological behavior of directory caching

On Thu, 2016-10-06 at 10:26 -0700, Kyle Rose wrote:
> This is not a weird or contrived use case: this is wildcard not
> finding targets in a recipe executed after they've been built as
> explicit prerequisites.

I discovered the difference.  If you put the makefile in a different
directory, like this:

      cd $HOME
      cp Makefile /tmp
      make -f /tmp/Makefile

Or if you list it explicitly but disable built-in rules, like this:

     cd $HOME
     make -rf Makefile

then it works as expected.  Otherwise it fails as you see.

Clearly there's a bug here.  I'll look into it.

_______________________________________________
Bug-make mailing list
address@hidden
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_mailman_listinfo_bug-2Dmake&d=CwIGaQ&c=DZ-EF4pZfxGSU6MfABwx0g&r=oBMzc8Omr1YTgjig4n4076T3IKL7TuNH9HpVbojD-ms&m=oFb1D7dSSuB3pYbY84FFjZ3O6hYP8I3ueb-agC8x7bE&s=-9wJcI_K36Y7wVo6Vv1dzF8u8xe-euxFrNffIKIB0NM&e=