Signed integer overflow in ansicstr() when parsing \x{...}

[Jakub Wilk]

Machine: aarch64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -g -Og -fsanitize=undefined
uname output: Linux ubuntu 5.15.0-1013-oracle #17~20.04.1-Ubuntu SMP Mon Jul 4 
05:29:46 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Machine Type: aarch64-unknown-linux-gnu

Bash Version: 5.1
Patch Level: 16
Release Status: release

Description:

Parsing very long \x{...} sequences inside $''-strings triggers 
signed integer overflow, which is undefined behavior.

Repeat-By:

    $ ./configure CFLAGS='-g -Og -fsanitize=undefined'
    ...
    $ make
    ...
    $ ./bash -n <<< "\$'\\x{ffffffff}'"
    strtrans.c:149:14: runtime error: signed integer overflow: 268435455 * 16 
cannot be represented in type 'int'

Fix:

Use an unsigned variable for arithmetic, like when parsing \u.

--
Jakub Wilk