bug#55654: 28.1.50; HEAP Error with ImageMagick on Windows MSYS2

[Eli Zaretskii]

> From: Fabio Leimgruber <fabio.leimgruber@web.de>
> Date: Thu, 26 May 2022 12:06:46 +0200
> 
> Compiled Emacs with these options:
> 
> --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 
> --build=x86_64-w64-mingw32 --with-wide-int --with-jpeg --with-xpm --with-png 
> --with-tiff --with-rsvg --with-xml2 --with-gnutls --with-xft 
> --with-imagemagick --with-modules --with-json --with-native-compilation 
> LDFLAGS=-fstack-protector
> 
> The error is triggered when I do find-file, which uses helm and treemacs in 
> turn loading images via ImageMagick.
> 
> Running Emacs under GDB shows the backtrace below.
> 
> Any hints on debugging this further?
> 
> #+begin_example
> warning: HEAP[emacs.exe]:
> warning: Invalid address specified to RtlValidateHeap( 000001EE1E190000, 
> 000001EE25BC1B80 )
> 
> Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
> 0x00007ff99eaca313 in ntdll!RtlRegisterSecureMemoryCacheCallback () from 
> C:\WINDOWS\SYSTEM32\ntdll.dll
> (gdb) bt
> #0  0x00007ff99eaca313 in ntdll!RtlRegisterSecureMemoryCacheCallback () from 
> C:\WINDOWS\SYSTEM32\ntdll.dll
> #1  0x00007ff99ea8cd24 in ntdll!memset () from C:\WINDOWS\SYSTEM32\ntdll.dll
> #2  0x00007ff99ea2e115 in ntdll!RtlValidateHeap () from 
> C:\WINDOWS\SYSTEM32\ntdll.dll
> #3  0x00007ff92596750f in NotifyShims () from C:\WINDOWS\SYSTEM32\AcLayers.dll
> #4  0x00007ff99e269c9c in msvcrt!free () from C:\WINDOWS\System32\msvcrt.dll
> #5  0x00007ff92575b662 in ?? () from 
> C:\Users\LeimgruberF\opt\msys64\mingw64\bin\libMagickCore-7.Q16HDRI-7.dll

I think this happens because ImageMagick tries to free memory that
Emacs allocated.  Emacs on MS-Windows uses its own implementation of
malloc and free, so a scenario where Emacs allocates and some library
frees, or vice versa, will never work.

Does ImageMagick has a facility whereby an application can tell it
what functions to use for allocation and deallocation of memory?  If
so, we can use those facilities to force ImageMagick to use our
allocation functions.

If the above doesn't help, I think the way to debug this is to install
ImageMagick sources and debug info, or build it with debug information
on your machine, and see what memory is being free'd here, and where
was it allocated.