[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
## t-test gets NaN instead of 0 for significance on x86-64

**From**: |
Ben Pfaff |

**Subject**: |
t-test gets NaN instead of 0 for significance on x86-64 |

**Date**: |
Tue, 01 Sep 2009 21:15:49 -0700 |

**User-agent**: |
Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux) |

Some time ago, Matej Cepl <address@hidden> reported that
t-test-alpha3.sh in 0.6.2-pre5 failed on x86-64 with GCC 4.4:
>* PASS: tests/bugs/t-test-alpha3.sh*
>* 19c19*
>* < #Pair 0|A & B#3| 1.000| NaN#*
>* ---*
>*> #Pair 0|A & B#3| 1.000|.000#*
>* compare output*
>* FAILED*
>* FAIL: tests/bugs/t-test-paired.sh*
This evening, I've run the same test on an x86-64 machine
(bellini.debian.org) with GCC 4.3.2 and I reproduce this test
failure. I also get the same failure with 0.6.1, although I have
to run the test by hand there since this test was new in
0.6.2-pre5.
After some fussing, I tracked the source of the NaN to this
calculation in pscbox() in src/language/stats/t-test.q:
double correlation_t =
pairs[i].correlation * sqrt (df) /
sqrt (1 - pow2 (pairs[i].correlation));
In this particular test case, pairs[i].correlation is almost
exactly 1.0, such that 1 - pow2 (pairs[i].correlation) comes out
just slightly negative, making the square root yield NaN.
John, do you have a suggestion for the correct fix? I don't know
enough about the math here to say.
--
Ben Pfaff
http://benpfaff.org

**t-test gets NaN instead of 0 for significance on x86-64**,
*Ben Pfaff* **<=**