[PATCH 1/2] quotearg: avoid undefined and/or O(N**2)

[Paul Eggert]

Avoid undefined and O(N**2) behavior in some very unlikely cases.
* lib/quotearg.c (quotearg_n_options): Document that N must
be less than MIN (INT_MAX, IDX_MAX), and add this to the
abort test; this also avoids a conditional branch.
Use xpalloc instead of xrealloc, to avoid O(N**2) behavior in
very-unlikely cases.
---
 ChangeLog      |  8 ++++++++
 lib/quotearg.c | 18 +++++++++---------
 2 files changed, 17 insertions(+), 9 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 804e22897..d511911fd 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,13 @@
 2021-04-03  Paul Eggert  <eggert@cs.ucla.edu>
 
+       quotearg: avoid undefined and/or O(N**2)
+       Avoid undefined and O(N**2) behavior in some very unlikely cases.
+       * lib/quotearg.c (quotearg_n_options): Document that N must
+       be less than MIN (INT_MAX, IDX_MAX), and add this to the
+       abort test; this also avoids a conditional branch.
+       Use xpalloc instead of xrealloc, to avoid O(N**2) behavior in
+       very-unlikely cases.
+
        xgethostname: reorganize / simplify
        xgethostname and xgetdomainname were essentially copies long
        ago, but they’ve diverged.  Bring them back together again
diff --git a/lib/quotearg.c b/lib/quotearg.c
index 365d6d1dd..570468917 100644
--- a/lib/quotearg.c
+++ b/lib/quotearg.c
@@ -864,7 +864,8 @@ quotearg_free (void)
    OPTIONS specifies the quoting options.
    The returned value points to static storage that can be
    reused by the next call to this function with the same value of N.
-   N must be nonnegative.  N is deliberately declared with type "int"
+   N must be nonnegative; it is typically small, and must be
+   less than MIN (INT_MAX, IDX_MAX).  The type of N is signed
    to allow for future extensions (using negative values).  */
 static char *
 quotearg_n_options (int n, char const *arg, size_t argsize,
@@ -874,22 +875,21 @@ quotearg_n_options (int n, char const *arg, size_t 
argsize,
 
   struct slotvec *sv = slotvec;
 
-  if (n < 0)
+  int nslots_max = MIN (INT_MAX, IDX_MAX);
+  if (! (0 <= n && n < nslots_max))
     abort ();
 
   if (nslots <= n)
     {
       bool preallocated = (sv == &slotvec0);
-      int nmax = MIN (INT_MAX, MIN (PTRDIFF_MAX, SIZE_MAX) / sizeof *sv) - 1;
+      idx_t new_nslots = nslots;
 
-      if (nmax < n)
-        xalloc_die ();
-
-      slotvec = sv = xrealloc (preallocated ? NULL : sv, (n + 1) * sizeof *sv);
+      slotvec = sv = xpalloc (preallocated ? NULL : sv, &new_nslots,
+                              n - nslots + 1, nslots_max, sizeof *sv);
       if (preallocated)
         *sv = slotvec0;
-      memset (sv + nslots, 0, (n + 1 - nslots) * sizeof *sv);
-      nslots = n + 1;
+      memset (sv + nslots, 0, (new_nslots - nslots) * sizeof *sv);
+      nslots = new_nslots;
     }
 
   {
-- 
2.30.2