lround() result usually bad in range [1048576,2097152)

[Richard Kelly]

>Submitter-Id:       net
>Originator: Rick Kelly
>Organization:  address@hidden
>
>Confidential:       no
>Synopsis:   lround() result usually bad in range [1048576,2097152)
>Severity:   serious
>Priority:   medium
>Category:   libc
>Class:              sw-bug
>Release:    libc-2.3.2
>Environment:
        
Host type: i386-redhat-linux-gnu
System: Linux schween.jpl.nasa.gov 2.4.20-28.9 #1 Thu Dec 18 13:45:22 
EST 2003 i686 i686 i386 GNU/Linux
Architecture: i686

Addons: linuxthreads c_stubs glibc-compat
Build CFLAGS: -march=i386 -DNDEBUG=1 -finline-limit=2000 -g -O3
Build CC: gcc
Compiler version: 3.2.2 20030222 (Red Hat Linux 3.2.2-5)
Kernel headers: 2.4.20
Symbol versioning: yes
Build static: yes
Build shared: yes
Build pic-default: no
Build profile: yes
Build omitfp: no
Build bounded: no
Build static-nss: no

>Description:
        lround() produces garbage for nearly all arguments greater
        than or equal to 2**20 but less than 2**21.  The problem
        occurs because lround() attempts to right shift a 32-bit
        unsigned integer by 32 bits, and that operation is undefined
        according to the standard.
        
        Within that range, some rare arguments, e.g. integers plus
        one half, do actually lead to correct results, at least on
        this machine.  That's because the erroneous shift's result
        just happens to be right for those bit patterns.
        
        The problem appears in sysdeps/ieee754/dbl-64/s_lround.c, on
        line 61, which is excerpted below:
        
            result = ((long int) i0 << (j0 - 20)) | (j >> (52 - j0));
                                                     ^^^^^^^^^^^^^^
                                                        BAD SHIFT
        
        In our case, j is a uint32_t, and j0 is 20, which makes the
        right shift undefined.

>How-To-Repeat:
        long int j = lround( 1071930.0008 );
        /* j should be 1071930, but instead is large neg value. */

>Fix:
        Change lround() code to handle this case, so it doesn't do
        the bad shift. There are several system-dependent variations
        of the s_lround.c file.  We don't know if the other variations
        are right or wrong.

Rick
--
Richard M Kelly                             address@hidden
[PGP  =>  D5 C3 CC D2 B7 D0 A9 B0  D5 90 B2 55 5A 80 23 B6  FC AB 67 B4]