bug#8546: fix for Emacs pseudovector incompatibility with GCC 4.6.0

[Paul Eggert]

On 04/25/11 03:23, Eli Zaretskii wrote:
> Could you please tell more what is it in the Emacs macros that
> triggers this problem?  You say that "Emacs's pseudovector
> implementation dissembles about the types", but could you please point
> out where in the code this happens?

Sure.  First, let me briefly describe the optimization, which
is allowed by the C standard.  Here's an idealized example:

        struct a { int size; ...; };
        struct b { int size; ...; };
        struct a *p = ...;
        struct b *q = ...;
        p->size = 0;
        q->size = 1;
        return p->size;

ISO C allows a compiler to optimize the last statement
to "return 0;".  If P and Q point to the
same memory location, storing through P and loading through
Q results in undefined behavior, because P and Q are incompatible
types.  When the behavior is undefined, the compiler is allowed
to generate whatever code it likes, including the optimized code.

Most of Emacs is OK with this optimization.  However, the pseudovector
code currently does stuff like this when checking is enabled
(I am giving the preprocessor output of XSETPVECTYPE followed by
XSETBUFFER, and a simplified version of it assuming the x86
to make things clearer):

        struct buffer *b = ...;
        b->size |= 0x4020000;
        Lisp_Object o = (Lisp_Object) b;
        struct Lisp_Vector *v = (struct Lisp_Vector *) o;
        if ((v->size & 0x4020000) != 0x4020000)
          abort ();

It's the conversion of struct buffer * to struct Lisp_Vector * that
results in undefined behavior, for the same reason that the
earlier example does: the code is reading a word via a struct Lisp_Vector *
pointer, which means that the compiler is free to (and GCC does) delay
the earlier store of that word until after the check, which means the code
aborts.