Re: [RFC PATCH] test-c-stack2.sh: skip if the platform sent SIGILL on an invalid address.

[Ivan Zakharyaschev]

Hi Bruno,

On Sat, 29 Dec 2018, Bruno Haible wrote:

> > "system in development" is the one which suits 
> > Linux/E2k better. The port to E2K (MCST Elbrus general purpose hardware 
> > architecture) is quite mature, but not yet released publicly.
> 
> Thanks for the info. Based on it, I found a couple of other pointers as well:
> [1][2].

> [1] 
> https://linux.slashdot.org/story/99/03/31/2324218/linus-will-move-to-moscow-to-work-with-elbrus

[1] is fun. :)

> > As for the SIGILL peculiarity, it has a reason in the Elbrus architecture. 
> > ...
> > And it's not a segmentation fault.

Meanwhile, I have found out that my explanations about it being the 
consequence of tagged memory (at least, in this specific case of 
test-c-stack.c) were largely incorrect. I'm sorry for that misleading 
information. I've studied the assembler code and found the other true 
reason in this specific case: these are faults "hidden" in an explicitly 
"speculative" computation which utltimately result in SIGILL. (The E2K ISA 
is reminiscent of IA64; this can help get the idea.) The specific kind of 
the fault is "forgotten", unfortunately.

Bruno, this discovery makes your claims even more strong and relevant: 
this kind of fault is expected by all programs to be SIGSEGV normally, and 
they can't care whether the computation was done speculatively or not 
(i.e., with immediate effects).

> I believe you should make it signal a SIGSEGV or SIGBUS, not SIGILL, for
> the following reasons:
> 
> * Look at the second table in
>   http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/signal.h.html.
>   It defines a couple of signal codes for SIGILL, SIGSEGV, and SIGBUS.
>   It implies that SIGILL means an invalid instruction (and "illegal operand"
>   means an invalid operand that is in the instruction stream).
>   Whereas SIGSEGV and SIGBUS mean a problem with an instruction in combination
>   with a memory address.

Thanks for the explanation concerning "illegal operand"! This was a 
rebuttal more relevant given my first imagined explanation, but not the 
actual one. But anyway important to know.

> * The main users of SIGSEGV and SIGBUS are catching stack overflow, garbage
>   collection, and similar (e.g. by use of GNU libsigsegv). The fact that
>   you observe an incompatibility between your Linux adaptation and
>   application programs that work fine across Linux/BSD/AIX/Solaris is a sure
>   indication that you will encounter similar incompatibilities along the 
> lines,
>   until you fix that port, to produce SIGSEGV or SIGBUS instead of SIGILL.

That's what I'm feeling now, too. It only remains a question concerning 
the hardware: whether it can save the type of the fault that happened in a 
speculative computation to give it back when the result of the speculative 
computation is actually needed.

> This reminds the segmented architectures, such as the ones used by AIX
> and Linux/ia64. In these OSes, SIGSEGV is produced when a memory address
> is used that does not fit with the instruction.

Thanks for the information about similar conditions (to what I wrote about 
tagged memory) in other OSes!

Besides, in many aspects including the newly mentioned by me explicitly 
speculative instructions, E2K reminds IA64.

And it'd be interesting to have a look how they treat faults coming from 
speculative computations in Linux/ia64 to get an idea whether it can be 
done in a manner with better conformance to POSIX.

* * *

Here are the actual facts about what happens on E2k (and little bit on 
IA64) with a set of minimal contrasting examples:

Here are four example programs; the first two write to the memory, the latter
two first read from the memory. (There is an amazing difference
between the last two examples.) Probably, the demonstrated contrasts
do not cover all conditions under which SIGILL can occur.

 $ cc -Wall -xc - && ./a.out; echo $?
 int main(int argc, char ** argv) {
   *(char*)0 = 175;
   return 0;
 }
 Segmentation fault
 139
 $ cc -Wall -xc - && ./a.out; echo $?
 int main(int argc, char ** argv) {
   if (0 < argc)
     *(char*)0 = 175;
   return 0;
 }
 Segmentation fault
 139
 $ cc -Wall -xc - && ./a.out; echo $?
 int main(int argc, char ** argv) {
   if (0 < argc)
     ++*(char*)0;
   return 0;
 }
 Illegal instruction
 132
 $ cc -Wall -xc - && ./a.out; echo $?
 int main(int argc, char ** argv) {
   ++*(char*)0;
   return 0;
 }
 Segmentation fault
 139
 $ cc --version
 lcc:1.23.12:Aug--6-2018:e2k-v4-linux
 gcc (GCC) 5.5.0 compatible
 $

This leads to a suspicion that not only the direction of the memory
access matters (read or write), but also the speculative execution of
the memory access instruction (in the third example) -- for the sake
of optimization, something is done before the actual value of the
condition is computed. (Otherwise, without a speculative computation,
it's unclear how a redundant condition can affect anything.) The
speculative instructions are written explicitly in E2K ISA (and this
is also like this in IA64 AFAIR). Let's have a look at the assembler
code of these four examples:

$ cc -Wall -xc - -S -o a.s && cat a.s
int main(int argc, char ** argv) {
  *(char*)0xbad = 175;
  return 0xbeef;
}
        .file   "-"
        .ignore ld_st_style
        .ignore strict_delay
        .text
        .global main
        .type   main, #function
        .align  8
main:
        {
          setwd wsz = 0x4, nfx = 0x1, dbl = 0x0
          return        %ctpr3
          adds,0        0x0, _f16s,_lts1lo 0xaf, %g16
          addd,1        0x0, _f32s,_lts2 0xbeef, %r0
        }
        {
          nop 4
          stb,2 0x0, _f16s,_lts0lo 0xbad, %g16
        }
        {
          ct    %ctpr3
        }
        .size   main, .- main
        .weak   elbrus_optimizing_compiler_v1.23.12_Aug__6_2018
        elbrus_optimizing_compiler_v1.23.12_Aug__6_2018 = 0x0

The curly braces group instructions to be executed simultaneously in a
single VLIW.

"adds" is employed to put the constant 175 (0xaf) into register g16.

"stb" STores a Byte into memory address 0xbad from register g16.

$ cc -Wall -xc - -S -o b.s && cat b.s
int main(int argc, char ** argv) {
  if (0 < argc)
    *(char*)0xbad = 175;
  return 0xbeef;
}
        .file   "-"
        .ignore ld_st_style
        .ignore strict_delay
        .text
        .global main
        .type   main, #function
        .align  8
main:
        {
          setwd wsz = 0x4, nfx = 0x1, dbl = 0x0
          return        %ctpr3
          adds,0,sm     0x0, _f16s,_lts1lo 0xaf, %g16
        }
        {
          nop 1
          cmplsb,0      0x0, %r0, %pred0
          addd,1        0x0, _f32s,_lts0 0xbeef, %r0
        }
        {
          nop 2
          stb,2 0x0, _f16s,_lts0lo 0xbad, %g16 ? %pred0
        }
        {
          ct    %ctpr3
        }
        .size   main, .- main
        .weak   elbrus_optimizing_compiler_v1.23.12_Aug__6_2018
        elbrus_optimizing_compiler_v1.23.12_Aug__6_2018 = 0x0

"cmplsb" puts the comparison result between 0 and register r0 (the
first function argument must reside here) in to the predicate register
pred0.

The "stb" instruction to store a value into memory is like in the
previous example, but is executed "predicatively" (expressed by the
tail "? %pred0"), i.e., only under the given condition.

(The "adds,0,sm" instruction is employed to put the constant 175 (0xaf)
into register g16 and has the "speculative" flag: ",sm"; but in this
example, we don't perceive anything special because of this. This is
the way to mark instructions, which are executed before it is actually
known whether their execution is needed.)

$ cc -Wall -xc - -S -o c.s && cat c.s
int main(int argc, char ** argv) {
  if (0 < argc)
    ++*(char*)0xbad;
  return 0xbeef;
}
        .file   "-"
        .ignore ld_st_style
        .ignore strict_delay
        .text
        .global main
        .type   main, #function
        .align  8
main:
        {
          setwd wsz = 0x4, nfx = 0x1, dbl = 0x0
          return        %ctpr3
          ldb,2,sm      0x0, _f16s,_lts1lo 0xbad, %g16
        }
        {
          nop 1
          cmplsb,0      0x0, %r0, %pred0
          addd,1        0x0, _f32s,_lts0 0xbeef, %r0
        }
        {
          adds,0,sm     %g16, 0x1, %g16
        }
        {
          nop 1
          stb,2 0x0, _f16s,_lts0lo 0xbad, %g16 ? %pred0
        }
        {
          ct    %ctpr3
        }
        .size   main, .- main
        .weak   elbrus_optimizing_compiler_v1.23.12_Aug__6_2018
        elbrus_optimizing_compiler_v1.23.12_Aug__6_2018 = 0x0

"ldb,2,sm" LoaDs a Byte from memory address 0xbad into register g16;
the ",sm" flags mean that it is executed "speculatively", i.e.,
without immediate side effects (faults), because it's not known yet at
this moment whether these actions will be needed.

Then, in this register g16, there is a speculative computation
"adds,0,sm" (to increase by one; it also has the "speculative" flags).

Then, "stb" needs to store the result from g16 into the real memory
(non-speculatively). It is remembered that g16 holds a result of a
speculative computation, and hence ,if it was with a delayed fault, a
real fault occurs at this moment (but not a SIGSEGV, rather a SIGILL
-- a signal that the speculative computation is invalid).


$ cc -Wall -xc - -S -o d.s && cat d.s
int main(int argc, char ** argv) {
  ++*(char*)0xbad;
  return 0xbeef;
}
        .file   "-"
        .ignore ld_st_style
        .ignore strict_delay
        .text
        .global main
        .type   main, #function
        .align  8
main:
        {
          nop 2
          setwd wsz = 0x4, nfx = 0x1, dbl = 0x0
          return        %ctpr3
          ldb,0 0x0, _f16s,_lts1lo 0xbad, %g16
          addd,1        0x0, _f32s,_lts2 0xbeef, %r0
        }
        {
          adds,0        %g16, 0x1, %g16
        }
        {
          nop 1
          stb,2 0x0, _f16s,_lts0lo 0xbad, %g16
        }
        {
          ct    %ctpr3
        }
        .size   main, .- main
        .weak   elbrus_optimizing_compiler_v1.23.12_Aug__6_2018
        elbrus_optimizing_compiler_v1.23.12_Aug__6_2018 = 0x0

Here we see same things as in the previous example, but without the
comparison and without any speculative computations. The fault that
occurs is a SIGSEGV (immediately, not delayed).

Why is there a difference between the second and the third examples?

Well, it happened so that nothing faulty is done in the speculative
computation in the second example, and because of that, the execution
comes to the memory write and its fault. In contrast, in the third
example, the fault of the memory write can occur only after bringing
the result of the speculative computation into reality and this
already causes a fault.

We can think of it as if register g16 was marked as invalid during the
speculative computation, and this caused a SIGILL an an attempt to use
it; however, the reason why it was marked invalid has been forgotten.

It's interesting to test whether the type of the fault that happens in
a speculative computation is really forgotten and any type of fault
gives a SIGILL in case there is an attempt to use its result
non-speculatively. It would mean that Linux/e2k can hardly conform to
POSIX well, as Bruno said, because POSIX requires different signals for
different cases and incompatibilities can't be forgiven on the reason of
speculative computations in the CPU.

Here is a test demonstrating that SIGFPE may hide beside SIGILL:

 $ cc -Wall -xc - && ./a.out; echo $?
 int main(int argc, char ** argv) {
   if (0 < argc)
     *(char*)0xbad = 175 / (argc - 1);
   return 0xbeef;
 }
 Illegal instruction
 132
 $ cc -Wall -xc - && ./a.out; echo $?
 int main(int argc, char ** argv) {
   *(char*)0xbad = 175 / (argc - 1);
   return 0xbeef;
 }
 Floating point exception
 136

I've inspected the assembler code in these two examples, and can
confirm that the division is actually done speculatively in the second
program.

* * *

BTW, saving and forgetting the type of the original fault doesn't seem
to be something expensive to implement (after some thought): when a
register is marked as invalid, it shouldn't matter anymore what value
it holds. So, the same register can be used to save the information
about the type of the fault.

* * *

I wanted to see how Linux/ia64 handles these complications arising
from speculative computations possibly causing a fault; and powered on
such a machine, and had a look at the above examples with SIGILL on
E2K: the third one, and the fifth one (speculative division by zero).

The third example from above:

address@hidden:~/test-speculative-SIGSEGV$ cc -Wall -O3 -xc - -S -o c.s && cat 
c.s
int main(int argc, char ** argv) {
  if (0 < argc)
    ++*(char*)0xbad;
  return 0xbeef;
}
        .file   ""
        .pred.safe_across_calls p1-p5,p16-p63
        .section        .text.startup,"ax",@progbits
        .align 16
        .align 64
        .global main#
        .type   main#, @function
        .proc main#
main:
        .prologue
        .body
        .mmi
        cmp4.ge p6, p7 = 0, r32
        addl r14 = 2989, r0
        addl r8 = 48879, r0
        ;;
        .mmi
        (p7) ld1 r15 = [r14]
        ;;
        (p7) adds r15 = 1, r15
        nop 0
        ;;
        .mib
        (p7) st1 [r14] = r15
        nop 0
        br.ret.sptk.many b0
        .endp main#
        .ident  "GCC: (Debian 4.6.3-14) 4.6.3"
        .section        .note.GNU-stack,"",@progbits
address@hidden:~/test-speculative-SIGSEGV$ cc -Wall -O3 c.s && ./a.out; echo $?
Segmentation fault
139

The fifth example from above:

address@hidden:~/test-speculative-SIGSEGV$ cc -Wall -O3 -xc - -S -o e.s && cat 
e.s
int main(int argc, char ** argv) {
  if (0 < argc)
    *(char*)0xbad = 175 / (argc - 1);
  return 0xbeef;
}
        .file   ""
        .pred.safe_across_calls p1-p5,p16-p63
        .global __divdi3#
        .section        .text.startup,"ax",@progbits
        .align 16
        .align 64
        .global main#
        .type   main#, @function
        .proc main#
main:
        .prologue 12, 33
        .mib
        .save ar.pfs, r34
        alloc r34 = ar.pfs, 1, 3, 2, 0
        .save rp, r33
        mov r33 = b0
        nop 0
        .mmi
        adds r37 = -1, r32
        mov r35 = r1
        .body
        cmp4.ge p6, p7 = 0, r32
        ;;
        .mii
        nop 0
        sxt4 r37 = r37
        nop 0
        .mmb
        addl r36 = 175, r0
        addl r8 = 48879, r0
        (p6) br.cond.dpnt .L2
        ;;
        .mib
        nop 0
        nop 0
        br.call.sptk.many b0 = __divdi3#
        ;;
        .mmi
        addl r14 = 2989, r0
        nop 0
        mov r1 = r35
        ;;
        .mmi
        nop 0
        st1 [r14] = r8
        addl r8 = 48879, r0
.L2:
        .mii
        nop 0
        mov ar.pfs = r34
        nop 0
        ;;
        .mib
        nop 0
        mov b0 = r33
        br.ret.sptk.many b0
        .endp main#
        .ident  "GCC: (Debian 4.6.3-14) 4.6.3"
        .section        .note.GNU-stack,"",@progbits
address@hidden:~/test-speculative-SIGSEGV$ cc -Wall -O3 e.s && ./a.out; echo $?
Floating point exception
136

Notes on the assembler: the possible groupings into VLIWs are
separated by double semicolons (";;"). Predicative execution of
instructions is marked by a prefix with the corresponding predicate
register in parentheses, like "(p7)" in the code above:

        .mmi
        (p7) ld1 r15 = [r14]
        ;;
        (p7) adds r15 = 1, r15
        nop 0
        ;;
        .mib
        (p7) st1 [r14] = r15

These are the "load", "add", and "store" instructions corresponding to: 
++*(char*)0xbad

All this shows that gcc-4.6 on IA-64 doesn't generate speculative
computations for the same examples that had speculative computations
on E2K. Unfortunately, this means that we couldn't compare the
interesting bits of the behavior between Linux/e2k and Linux/ia64
quickly. Perhaps, editing the IA64 assembler code can give a desired
example.


-- 
Best regards,
Ivan