Re: [Bug-gnubg] Re: Rollout jsd, statsig etc. [LONG]

[Timothy Y. Chow]

On Tue, 17 Nov 2009, Massimiliano Maini wrote:
> 2009/11/16 Timothy Y. Chow <address@hidden>:
> > The multivariate tail probability, for
> > example, tells you only the probability that some strange event will occur
> > *under the assumption that the equities are equal to the estimated
> > equities*.  This is *not* the same as *the probability that the true
> > equities are different from their estimated values*.
> 
> Just for my understanding, the bayesian approach would be no different
> with respect to that, right ?
> 
> It will give you a probability that some strange event will occur under 
> the assumption that the estimated pdf are the ones you have on the 
> current trial/iteration (plus the initial assumption on those pdf), 
> right ?

No.  Under the Bayesian approach, the reported probability that (for 
example) Play A is the best is the probability that Play A is the best, 
*assuming* that the equities of the various plays were all chosen 
uniformly at random before you saw any rollout results.  (I'm assuming you 
pick a uniform prior.)

Here, the "strange event" you're measuring is, "the equity really was 
something very different from the estimated equity, yet somehow we managed 
to see these very non-representative results."  It should be clear that 
measuring this event is more meaningful than *assuming* that the true 
equity is the estimated equity, and computing the probability that you 
will get misleading results in the future.  If you're going to *assume* 
that the true equity is the estimated equity, why would you roll things 
out further?  What you're interested in is the possibility that the true 
equity is *not* the estimated equity, so you shouldn't *assume* that 
they're equal.

On Tue, 17 Nov 2009, Massimiliano Maini wrote:
> Accurately report everything gnubg is actually computing would be a
> tremendous task. There are so many things that are done and are
> unexplained and/or with blatantly arguable assumptions.
> But we'll try to avoid wrong statements.

Yes, that's what I meant to say; sorry if I phrased it badly.

Tim