Re: Covariance Matrices

[John Darrington]

On Wed, Aug 20, 2008 at 10:57:33AM -0400, Jason Stover wrote:
     
     I had planned to add a one-pass algorithm, but used a two-pass algorithm
     first because, "usually", two-pass algorithms have lower relative errors 
than
     one-pass algorithms, according to
     
     "Algorithms for Computing the Sample Variance: Analysis and 
Recommendations"
     TF chan, G. Golub, R. LeVeque. American Statistician, v37 n3, 1983, pp. 
242-247.
     
     So I had planned to add a one-pass algorithm, based on the algorithms in 
that
     paper, but never got around to it.
     

     My only suggestion is that the code in covariance-matrix.[ch] should have
     both one- and two-pass algorithms. So maybe add covariance_accumulate()
     and change covariance_pass_two () to incorporate your changes, but passing
     an argument like double *means to use the means.


My opinion is that we should prefer speed rather than precision.  So
all things being equal, I would use the single pass method.   However,
in cases where there is a compelling reason to need another pass, then
the more accurate method can be used.

However, in many PSPP commands, the logic required to determine how
many passes are necessary is quite nasty; it can depend on exactly
which options are selected.   For  some time now, I've been thinking
of a scheme where each statistic is aware of its own dependencies.
With such a scheme, it would be possible to specify a set of
statistics, then the minimum number of passes would be automatically
determined, and the most accurate method for that number would be
automatically selected.

This scheme would take a bit of thought, and a lot of recoding.  But
if the routines to calculate these statistics have a similar
interface, then that'll be the first step.

J'




-- 
PGP Public key ID: 1024D/2DE827B3 
fingerprint = 8797 A26D 0854 2EAB 0285  A290 8A67 719C 2DE8 27B3
See http://pgp.mit.edu or any PGP keyserver for public key.

signature.asc
Description: Digital signature