There was a bunch of discussion over at Andrew Gelman's blog about "bet proof" interpretations of confidence intervals. The relevant paper is here.

Below is what I original wrote in grey. Nope. I was misreading. The actual definition of bet-proofness is also weird. Here it is in essence:

A confidence interval construction procedure is bet proof if for every betting scheme $b(x)$ there exists some value of the parameter $\Theta$ such that this scheme will lose in the long run.

This is kind of the converse of what I was thinking before. But here's my example of why that makes no sense (quoted here from what I wrote on Andrew's blog after I finally figured out what was going on).

Now, with the above reworded Definition 1, I can see how the game is about revealing X and not revealing Theta. But, I don’t see how it is interesting. Let me give you an example:

Our bet is to sample a random sample of every adult who lives within 2 blocks of me. We will measure their heights X, then you’ll construct a confidence interval CI(X), and I will bet whether after we measure all the rest of them, the population mean Theta will be in the CI.

Now, being a good Bayesian, I use Wald’s theorem and realize that any strategy to decide what my bet winnings will be that doesn’t use a prior and a loss function will be dominated by one that does…. So I’ll go out to Wikipedia and google up info about the population of the US and their heights, and I”ll construct a real world prior and then I’ll place my bet.

Now bet proofness says that because it’s the case that if the actual height of people in a 2 block radius is 1 Million Feet (ie. THERE EXISTS A THETA), my prior will bias my bets in the wrong direction and I will not be able to make money…. that this CI is all good, it’s bet proof.

And how is that relevant to any bet we will actually place?"

The basic principal of bet-proofness was essentially that if a sample of data X comes from a RNG with known distribution $D(\Theta)$ that has some parameter $\Theta$, then even if you know $\Theta$ exactly, so long as you don't know what the X values will be, you can't make money betting on whether the constructed CI will contain the $\Theta$ (the paper writes this in terms of $f(\Theta)$ but the principal is the same since f is a deterministic function).

The part that confused me, was that this was then taken to be a property of the individual realized interval... "Because an interval came from a bet-proof procedure it is a bet-proof realized interval" in essence. But, this defines a new term "bet-proof realized interval" which is meaningless when it comes to actual betting. The definition of "bet-proof procedure" explicitly uses averaging over the possible outcomes of the data collection procedure $X$ but after you've collected $X$ and told everyone what it is, if someone knows $\Theta$ and knows $X$ they can calculate exactly whether the confidence interval does or does not contain $\Theta$ and so they win every bet they make.

So "bet-proof realized confidence interval" is really just a technical term meaning "a realized confidence interval that came from a bet proof procedure" however it doesn't have any content for prediction of bets about that realized interval. The Bayesian with perfect knowledge of $\Theta$ and $X$ and the confidence construction procedure wins every bet! (there's nothing uncertain about these bets).

5 Responses leave one →
March 8, 2017

Personally I found that discussion a bit long-winded - thanks for summarizing your thoughts! At the end of the day, a frequentist CI is still a frequentist CI (i.e. can only really be understood as embedded in a long-run sequence)- and "bet-proofness" seems mainly to map to situations where the Bayesian CI with flat-ish priors makes sense. I still don't understand why the original poster thought this concept is somehow easier/more intuitive than teaching students the truth about frequentist CIs and Bayesian alternatives...

• March 8, 2017

Yes, I think it was so long winded because it WASN'T obvious, and also because there is this punning "bet proof procedure" vs "realized bet-proof interval"... There is nothing bet-proof about the realized interval except that it came from a class of procedures called bet-proof.

Yikes!