[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

t-test gets NaN instead of 0 for significance on x86-64

From: Ben Pfaff
Subject: t-test gets NaN instead of 0 for significance on x86-64
Date: Tue, 01 Sep 2009 21:15:49 -0700
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux)

Some time ago, Matej Cepl <address@hidden> reported that in 0.6.2-pre5 failed on x86-64 with GCC 4.4:

> PASS: tests/bugs/
> 19c19
> < #Pair 0|A & B#3|      1.000| NaN#
> ---
>> #Pair 0|A & B#3|      1.000|.000#
> compare output
> FAIL: tests/bugs/

This evening, I've run the same test on an x86-64 machine
( with GCC 4.3.2 and I reproduce this test
failure.  I also get the same failure with 0.6.1, although I have
to run the test by hand there since this test was new in

After some fussing, I tracked the source of the NaN to this
calculation in pscbox() in src/language/stats/t-test.q:

      double correlation_t =
        pairs[i].correlation * sqrt (df) /
        sqrt (1 - pow2 (pairs[i].correlation));

In this particular test case, pairs[i].correlation is almost
exactly 1.0, such that 1 - pow2 (pairs[i].correlation) comes out
just slightly negative, making the square root yield NaN.

John, do you have a suggestion for the correct fix?  I don't know
enough about the math here to say.
Ben Pfaff

reply via email to

[Prev in Thread] Current Thread [Next in Thread]