Re: Test Failure

[Jason Stover]

My comments are interspersed below. I think the problem is one of
display, not any erroneus computation happening inside pspp. Though I
did not test this in a debugger, due to a couple of tests I ran with
pspp, I believe pspp is storing the correct double-precision
values. You can see a few more digits by increasing the number of
decimals printed in the output, though they are prematurely
truncated. (But I will check it with a debugger later to be sure.) A
small test script that shows two problems is at the end of this
message.

On Mon, Apr 17, 2006 at 11:41:28AM +0800, John Darrington wrote:
> On Sun, Apr 16, 2006 at 06:44:39PM -0700, Ben Pfaff wrote:
>      John Darrington <address@hidden> writes:
>      
>      > Suddenly a clean check-out gives me:
>      >
>      > 42c42
>      > <      .99     2.00     9.21      .99      .00      .01 
>      > ---
>      >>      .99     2.00     9.21      .99      .01      .01 
>      > compare output for chisq distribution
>      > FAILED
>      > FAIL: tests/expressions/randist.sh
>      
>      Hmm.  I can't reproduce that with a clean checkout, even with
>      `make -f Smake check'.
>      
>      I'm on x86 (32 bit) GNU/Linux.
> 
> Me too.
> 
> 
> What version of math library do you have?  One machine where the
> problem manifests itself has 
> 
> ls -l /lib/libm.so.6 
> lrwxrwxrwx 1 root root 13 2006-04-10 20:37 /lib/libm.so.6 -> libm-2.3.6.so
> 
> another machine which doesn't give this error has
> 
> ls -l /lib/libm.so.6 
> lrwxrwxrwx  1 root root 13 2005-12-01 09:12 /lib/libm.so.6 -> libm-2.3.5.so

The value shown in the fifth column is the value for the 
chi-square density with 2 degrees of freedom, at the point
9.21.

I have an x86 desktop with

$ ls -l /lib/libm.so.6 
lrwxrwxrwx  1 root root 13 2004-07-16 13:27 /lib/libm.so.6 -> libm-2.3.2.so

that does not show the problem. 

Anyway, the correct value to many decimal places is:

        5.000851002352739e-03

So the value stored in tests/expressions/randist/chisq.out is
rounded differently than that reported by running the test. The value 
should be reported as 0.01, as on the second line of John's test.

But on this machine:

> uname -ap
OpenBSD math.gcsu.edu 3.8 GENERIC#138 i386 Intel(R) Xeon(TM) CPU 2.80GHz 
("GenuineIntel" 686-class)

and

> ls -l /usr/lib/libm.so.2.0 
-r--r--r--  1 root  bin  90107 Sep 10  2005 /usr/lib/libm.so.2.0

I get a different failure, but not the one John reported:

2c2
<      .01     2.00      .02      .01      .49      .99 
---
>      .01     2.00      .02      .01      .50      .99 
compare output for chisq distribution
FAILED
FAIL: tests/expressions/randist.sh

Which is the same type of error as John's failed test: The second line
is rounded up as it should be, but the value stored in
tests/expressions/randist/chisq.out (shown on the first line) is
not. pspp seems to have the correct value stored, as can be seen by
telling the PRINT command to print more decimal places. The correct
value to many digits is 4.950249168745841e-01. The reported value in
this case should be rounded to 0.50, if the user wants only two decimal
places displayed.

I don't know where pspp does its rounding before displaying, so I'm
not sure what the problem is. But pspp does seem to truncate both
values prematurely, even with the PRINT FORMATS pdf (F9.6) in the
script below. The doc says that COMPUTE assigns a format of F8.2 to
any numeric target variable. Is there no way to specify a different
format?

> I'm not sure what all the columns mean in
> tests/expressions/randist/chisq.out.  Can we add an explanation somewhere?
>
> Is the 5th column the probability? If so, then my book of tables tells
> me that Chi-Squared for 2 degrees of freedom and p = 0.01 is indeed
> 9.21, in which case the test is wrong, and for some reason my machine
> is the only one computing the value correctly.

The fifth column isn't a probability, it's the value of the chi-square
density function.

The first column is the 'lower tail' probability:

    Prob (X < x)

where X is a chi-square random variable with 2.00 degrees of freedom (the
degrees of freedom are stored in the second column). The third column is
the lower case 'x' in that 'Prob' statement above; in the case of John's
failed test, x = 9.21. The fourth column is pspp's re-computation of 
Prob(X < x), so this value should be equal to that shown in the first column.
The fifth column is the value of the probability density function at x, i.e.,
the function f(t) such that 

    Prob (X < x) = integral_{0}^{x} f(t) dt

and the last column is Prob(X > x), which should equal 1-(first column).


Here is the test script that shows both problems:

     DATA LIST LIST/P df.
     COMPUTE x = IDF.chisq (P, df). 
     COMPUTE cdf = cdf.chisq (x, df).
     COMPUTE pdf = pdf.chisq (x, df). /* Format MUST be F8.2? */
     COMPUTE sig = sig.chisq (x, df).
     PRINT FORMATS pdf (F9.6).
     PRINT OUTFILE='chisq.out'/P df x cdf pdf sig.
     BEGIN DATA.
     0.01 2 /* Should show .495025 in column 5 when x=0.02. */
     0.99 2 /* Should show .005001 in column 5 when x=9.21.*/
     END DATA.

-Jason