[patch v2] glob: resolve DT_UNKNOWN via is_dir

[DJ Delorie]

[v2: changed malloc failure from ignore to error; added support for
alloca; tested by copying to glibc and testing there]

The DT_* values returned by getdents (readdir) are only hints and
not required.  In fact, some Linux filesystems return DT_UNKNOWN
for most entries, regardless of actual type.  This causes make
to mis-match patterns with a trailing slash (via GLOB_ONLYDIR)
(see make's functions/wildcard test case).  Thus, this patch
detects that case and uses is_dir() to make the type known enough
for proper operation.

Performance in non-DT_UNKNOWN cases is not affected.

The lack of DT_* is a well known issue on older XFS installations
(for example, RHEL 7 and 8, Fedora 28) but can be recreated by
creating an XFS filesystem with flags that mimic older behavior:

$ fallocate -l 10G /xfs.fs
$ mkfs.xfs -n ftype=0 -m crc=0 -f /xfs.fs
$ mkdir /xfs
$ mount -o loop /xfs.fs /xfs

diff --git a/lib/glob.c b/lib/glob.c
index f8d8a306f2..c28c92a42c 100644
--- a/lib/glob.c
+++ b/lib/glob.c
@@ -1381,7 +1381,33 @@ glob_in_dir (const char *pattern, const char *directory, 
int flags,
               if (flags & GLOB_ONLYDIR)
                 switch (readdir_result_type (d))
                   {
-                  case DT_DIR: case DT_LNK: case DT_UNKNOWN: break;
+                  case DT_DIR: case DT_LNK: break;
+                 case DT_UNKNOWN:
+                   {
+                     /* The filesystem was too lazy to give us a hint,
+                        so we have to do it the hard way.  */
+                     char *fullpath, *p;
+                     bool isdir;
+                     int need = strlen (directory) + strlen (d.name) + 2;
+                     int use_alloca = glob_use_alloca (alloca_used, need);
+                     if (use_alloca)
+                       fullpath = alloca_account (need, alloca_used);
+                     else
+                       {
+                         fullpath = malloc (need);
+                         if (fullpath == NULL)
+                           goto memory_error;
+                       }
+                     p = stpcpy (fullpath, directory);
+                     *p++ = '/';
+                     strcpy (p, d.name);
+                     isdir = is_dir (fullpath, flags, pglob);
+                     if (!use_alloca)
+                       free (fullpath);
+                     if (isdir)
+                       break;
+                     continue;
+                   }
                   default: continue;
                   }