Re: ptrdiff_t overflow checks for malloc-posix etc.

[Bruno Haible]

> 2021-05-09  Bruno Haible  <bruno@clisp.org>
> 
>       {malloc,realloc,calloc}-gnu: Fix autoconf macro (regression 2021-04-18).

Even after this is fixed, I see a test failure (building the newest m4)
on AIX in 64-bit mode: test-calloc-gnu fails with exit code 1.

Looking at the details: The configure output has

  checking whether calloc (0, n) and calloc (n, 0) return nonnull... no

which indicates that the configure script has noticed that it should
enable the calloc() replacement. But config.status has

  S["REPLACE_MALLOC"]="1"
  ...
  S["REPLACE_CALLOC"]="0"
  ...
  S["REPLACE_REALLOC"]="1"

On AIX, all three functions ought to be replaced. Why is REPLACE_CALLOC zero?
Let's look at the assignments to REPLACE_CALLOC and the uses of REPLACE_MALLOC
in the configure script:

12111:  REPLACE_CALLOC=0;
12115:  REPLACE_MALLOC=0;
12192:  test "$gl_cv_malloc_ptrdiff" = yes || REPLACE_MALLOC=1
12226:    REPLACE_MALLOC=1
12232:  REPLACE_CALLOC=$REPLACE_MALLOC
32513:  if test $REPLACE_CALLOC = 0; then
32577:    REPLACE_CALLOC=1 ;;
32582:  if test $REPLACE_CALLOC = 1; then
32597:  REPLACE_CALLOC=$REPLACE_MALLOC
32599:  if test $REPLACE_CALLOC = 1; then
39364:  if test $REPLACE_MALLOC = 0; then
39419:    REPLACE_MALLOC=1 ;;
39424:  if test $REPLACE_MALLOC = 1; then
39437:  if test $REPLACE_MALLOC = 1; then

And when I execute the configure script with 'sh -x', I get these assignments
in sequence:

+ REPLACE_CALLOC=0
+ REPLACE_CALLOC=0
+ REPLACE_CALLOC=1
+ REPLACE_CALLOC=0

So, REPLACE_CALLOC gets set to 1 but then it gets set to 0 later.

There are two ways to fix this:
  (a) Never set REPLACE_CALLOC to 0, except in the gl_STDLIB_H_DEFAULTS macro.
  (b) Ensure that gl_FUNC_CALLOC_POSIX gets expanded only once, before
      gl_FUNC_CALLOC_GNU.

I prefer (a), because that is the common practice already: The REPLACE_*
variables are getting set to 0 early, and then any macro can set it to 1
depending on whatever conditions.

This patch does it, and fixes the AIX 64-bit issue.


2021-05-09  Bruno Haible  <bruno@clisp.org>

        {realloc,calloc}-gnu: Fix autoconf macro (regression 2021-04-18).
        * m4/realloc.m4 (gl_FUNC_REALLOC_POSIX): Don't reset REPLACE_REALLOC
        to 0 if it is already 1 after gl_FUNC_REALLOC_GNU was executed.
        * m4/calloc.m4 (gl_FUNC_CALLOC_POSIX): Don't reset REPLACE_CALLOC
        to 0 if it is already 1 after gl_FUNC_CALLOC_GNU was executed.

diff --git a/m4/calloc.m4 b/m4/calloc.m4
index 7575a69..fe12b15 100644
--- a/m4/calloc.m4
+++ b/m4/calloc.m4
@@ -1,4 +1,4 @@
-# calloc.m4 serial 26
+# calloc.m4 serial 27
 
 # Copyright (C) 2004-2021 Free Software Foundation, Inc.
 # This file is free software; the Free Software Foundation
@@ -72,7 +72,9 @@ AC_DEFUN([gl_FUNC_CALLOC_POSIX],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
   AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
-  REPLACE_CALLOC=$REPLACE_MALLOC
+  if test $REPLACE_MALLOC = 1; then
+    REPLACE_CALLOC=1
+  fi
   dnl Although in theory we should also test for size_t overflow,
   dnl in practice testing for ptrdiff_t overflow suffices
   dnl since PTRDIFF_MAX <= SIZE_MAX on all known Gnulib porting targets.
diff --git a/m4/realloc.m4 b/m4/realloc.m4
index 9925917..0abc418 100644
--- a/m4/realloc.m4
+++ b/m4/realloc.m4
@@ -1,4 +1,4 @@
-# realloc.m4 serial 23
+# realloc.m4 serial 24
 dnl Copyright (C) 2007, 2009-2021 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -57,5 +57,7 @@ AC_DEFUN([gl_FUNC_REALLOC_POSIX],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
   AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
-  REPLACE_REALLOC=$REPLACE_MALLOC
+  if test $REPLACE_MALLOC = 1; then
+    REPLACE_REALLOC=1
+  fi
 ])