[Octave-bug-tracker] [bug #35827] The hygepdf and hygecdf functions are numerically unstable for large populations

[Rik]

Update of bug #35827 (project octave):

              Depends on:                         => bugs #34362            

    _______________________________________________________

Follow-up Comment #1:

Just to be clear, I get the following when running the sample code


hygecdf(10,10000,200,1000)
error: discrete_cdf: P must be a nonzero, non-negative vector


If I replace the cdf with pdf then Octave produces NaN.

What does Matlab return for


nchoosek (9800, 990)


I'm guessing it returns Inf which indicates that they are not using the
straightforward implementation of two binomial coefficients divided by a
third.

The algorithm for hygepdf is pretty straightforward with two binomial
coefficients divided by a third
(http://en.wikipedia.org/wiki/Hypergeometric_distribution).  Octave implements
this directly which you can see by looking at the code in hygepdf.m.

The actual binomial coefficient in bincoeff.m is the exponential of some
gammaln functions.  Thus, it would be pretty easy to pull the definition from
bincoeff into hygepdf.  You could then make use of the rules for multiplying
and dividing exponentials to add or subtract terms BEFORE applying the final
exponential.

I just did a quick test with the example you provided and it does not produce
NaN.  Instead the answer is 0.0043171.  This is a lot better but at this point
I'm not sure which is more accurate: Matlab or Octave?  How did you determine
that the correct answer was 0.0076?



    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?35827>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/