Re: [Bug-gnubg] Re: Rollout jsd, statsig etc. [LONG]

[Timothy Y. Chow]

On Tue, 17 Nov 2009, Massimiliano Maini wrote:
> If the above is true, on top of assuming a uniform prior, you assume
> that the real pdf are the ones you have on your last step.
> This is not far from assuming that the real equities are the estimated
> ones. Duh, I must be missing something ...

Maybe a simplified example will help.

I fill a bag with 1000 marbles, some white and some black, and hand the 
bag to a group of four people: Martha the Mathematician, Bob the Bayesian, 
Helen the Hypothesis tester, and Stan the Statistics Student.  I invite 
them to sample 10 marbles at random from the bag.  They do so; 8 are white 
and 2 are black.  I ask, "What is the probability that I put more black
marbles than white marbles in the bag?"

Martha complains, "Unless we make some further assumptions, your question 
is unanswerable.  Either you put more black marbles than white marbles in 
the bag or you didn't.  There's no probability calculation to do here."

Bob says, "You're right, Martha; we need to assume something.  Let's 
assume that Tim picked a number from 0 to 1000 uniformly at random, and 
put that many white marbles in the bag, filling the rest of the bag with 
black marbles.  Given that assumption, and given that our random sample 
has produced 8 white marbles and 2 black marbles, can you now calculate, 
for each i from 0 to 1000, the probability P(i) that Tim put i white 
marbles in the bag?"  Martha brightens.  "Yes!" she says, and she happily 
cranks out a probability distribution P(i).  "Good," says Bob.  "Let's now 
compute the sum of P(i) from i = 0 to 499; that's our answer to Tim's 
question."

Helen objects, "Bob, I don't like your assumption; it seems unjustified to 
me.  For all we know, there are just as many white marbles as black 
marbles in the bag, and we got a skewed sample by random chance."  Bob 
replies, "That doesn't seem very likely to me."  Martha, eager to have 
something to do, starts to crank away.  She says, "In fact, the 
probability that, assuming there are 500 white marbles and 500 black 
marbles, we would get at least 8 white marbles in our random sample is..." 
and she states a figure.  Helen concedes, "Well, that's a pretty small 
number.  So we can rather confidently reject my suggestion that there are 
equally many white and black marbles."  I ask, "So what's your answer to 
my question?"  Helen says, "I can't answer your question without making 
assumptions that I'm not prepared to make.  I'll stick to what I can say 
for sure: the probability that we would see as many as 8 white marbles out 
of random sample of 10 marbles is very small, if there were equally many 
white and black marbles."

Stan the student now interjects, "Helen, I don't see why you can't answer 
Tim's question.  We got 8 white marbles and 2 black marbles.  So our best 
guess is that there are 800 white marbles and 200 black marbles in the 
bag.  Why don't we just assume that Tim filled the bag by repeating the 
following procedure 1000 times: with probability 0.8, he threw a white 
marble into the bag, and with probability 0.2, he threw a black marble 
into the bag.  Then we can calculate the probability that he would end up 
with more black marbles than white marbles in the bag.  Wouldn't that 
number answer Tim's question?"  Martha says, "I don't know if it would 
answer Tim's question, but I can compute your number.  It works out to 
be..." and she states a number.  "That's pretty small," says Stan.  "And 
that's my answer to Tim's question."

Now what do I say after hearing all this discussion?  I would say that 
Martha, Bob, and Helen have all taken defensible positions.  They have 
stated what assumptions they're prepared to make what assumptions they're 
not prepared to make, and have reasoned accordingly.  The one position I 
don't find to be defensible is Stan's.  His reasoning doesn't make sense 
to me.  Why should we think that the result of his calculation is a 
sensible answer to my question?  Even if Stan's number turns out to be 
close to Bob's number, does that do anything to legitimize Stan's 
reasoning?

Tim