Re: [PATCH v3 09/49] semihosting: Adjust error checking in common_semi_cb

[Peter Maydell]

On Mon, 23 May 2022 at 16:35, Richard Henderson
<richard.henderson@linaro.org> wrote:
> On 5/23/22 05:13, Peter Maydell wrote:
> > The gdb implementation of the isatty call returns 0 or 1 on
> > success, and -1 on failure (though the only failure mode it has
> > is "you messed up the protocol packet format"):
> > https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/remote-fileio.c;h=fe191fb6069a53a3844656a81e77069afa781946;hb=HEAD#l1039
>
> Technically, isatty = 0 is failure not success and should also set ENOTTY.

There are multiple different APIs here with similar names but
not necessarily always the exact same behaviour in all cases:

Arm semihosting SYS_ISTTY:
https://github.com/ARM-software/abi-aa/blob/main/semihosting/semihosting.rst#sys-istty-0x09
 returns 1 if the handle identifies an interactive device.
         0 if the handle identifies a file.
         A value other than 1 or 0 if an error occurs
         (and implicitly sets errno, I assume)

GDB File-I/O isatty:
https://sourceware.org/gdb/onlinedocs/gdb/isatty.html#isatty
  Returns 1 if fd refers to the GDB console, 0 otherwise
Documentation doesn't say how it reports errors. Actual implementation
returns -1 and sets errno. (We should probably report some of these
spec issues as gdb bugs...)

isatty() POSIX function:
 returns 1 for a terminal
         0 with errno ENOTTY for not-a-terminal
         0 with some other errno for error cases

It looks like our 'host' implementation of the semihosting SYS_ISTTY
doesn't correctly do the matching up between the semihosting spec
and the isatty() function, so it will return the wrong value for
the error case.

thanks
-- PMM