Re: ptrdiff_t overflow checks for malloc-posix etc.

[Bruno Haible]

Paul Eggert wrote:
> I installed the attached patches into Gnulib to make its malloc 
> replacements ptrdiff_t safe.

When testing m4-1.4.18b on IRIX 6.5, I get a test failure:

FAIL: test-reallocarray

Let's look in detail:

$ ./test-reallocarray ; echo $?
2

There is a call
  p = realloc (NULL, 2*1073741824);
which returns NULL with errno being 0.

Since the 'reallocarray' module depends on 'realloc-gnu', and the
'realloc-gnu' and 'realloc-posix' documentation says:

  Portability problems fixed by Gnulib:
  @itemize
  @item
  Upon failure, the function does not set @code{errno} to @code{ENOMEM} on
  some platforms:
  mingw, MSVC 14.

  @item
  On some platforms, @code{realloc (p, n)} can succeed even if @code{n}
  exceeds @code{PTRDIFF_MAX}.  Although this behavior is arguably
  allowed by POSIX it can lead to behavior not defined by POSIX later,
  so @code{realloc-posix} does not allow going over the limit.
  @end itemize

So, what the documentation implies and what the reallocarray unit test
verifies is that
  realloc (NULL, n)  where n > PTRDIFF_MAX
1) returns NULL and
2) sets errno to ENOMEM.

On IRIX (in n32 ABI), expectation 1) is fulfilled but 2) is not.
Likewise for malloc and calloc.

I'm adding two patches
  - to make sure that the 'realloc-gnu' unit test already fails in this
    situation,
  - to fix 'realloc-gnu' on IRIX, so that it actually compiles the
    replacement code lib/realloc.c.


2021-05-09  Bruno Haible  <bruno@clisp.org>

        malloc-gnu, realloc-gnu, calloc-gnu: Ensure errno gets set on IRIX.
        * m4/malloc.m4 (gl_CHECK_MALLOC_POSIX): Require AC_CANONICAL_HOST. Set
        gl_cv_func_malloc_posix to 'no' also on IRIX.

        malloc-gnu, realloc-gnu, calloc-gnu tests: Verify errno is set.
        * tests/test-malloc-gnu.c: Include <errno.h>.
        (main): Verify that, when an allocation larger than PTRDIFF_MAX failed,
        errno is ENOMEM.
        * tests/test-realloc-gnu.c: Likewise.
        * tests/test-calloc-gnu.c: Likewise.

0001-malloc-gnu-realloc-gnu-calloc-gnu-tests-Verify-errno.patch
Description: Text Data

0002-malloc-gnu-realloc-gnu-calloc-gnu-Ensure-errno-gets-.patch
Description: Text Data