Bet-Proofness as a property of Confidence Interval Construction Procedures (Not realized intervals)

2017 March 8
by Daniel Lakeland

There was a bunch of discussion over at Andrew Gelman's blog about "bet proof" interpretations of confidence intervals. The relevant paper is here.

Below is what I original wrote in grey. Nope. I was misreading. The actual definition of bet-proofness is also weird. Here it is in essence:

A confidence interval construction procedure is bet proof if for every betting scheme b(x) there exists some value of the parameter \Theta such that this scheme will lose in the long run.

This is kind of the converse of what I was thinking before. But here's my example of why that makes no sense (quoted here from what I wrote on Andrew's blog after I finally figured out what was going on).

Now, with the above reworded Definition 1, I can see how the game is about revealing X and not revealing Theta. But, I don’t see how it is interesting. Let me give you an example:

Our bet is to sample a random sample of every adult who lives within 2 blocks of me. We will measure their heights X, then you’ll construct a confidence interval CI(X), and I will bet whether after we measure all the rest of them, the population mean Theta will be in the CI.

Now, being a good Bayesian, I use Wald’s theorem and realize that any strategy to decide what my bet winnings will be that doesn’t use a prior and a loss function will be dominated by one that does…. So I’ll go out to Wikipedia and google up info about the population of the US and their heights, and I”ll construct a real world prior and then I’ll place my bet.

Now bet proofness says that because it’s the case that if the actual height of people in a 2 block radius is 1 Million Feet (ie. THERE EXISTS A THETA), my prior will bias my bets in the wrong direction and I will not be able to make money…. that this CI is all good, it’s bet proof.

And how is that relevant to any bet we will actually place?"

The basic principal of bet-proofness was essentially that if a sample of data X comes from a RNG with known distribution D(\Theta) that has some parameter \Theta, then even if you know \Theta exactly, so long as you don't know what the X values will be, you can't make money betting on whether the constructed CI will contain the \Theta (the paper writes this in terms of f(\Theta) but the principal is the same since f is a deterministic function).

The part that confused me, was that this was then taken to be a property of the individual realized interval... "Because an interval came from a bet-proof procedure it is a bet-proof realized interval" in essence. But, this defines a new term "bet-proof realized interval" which is meaningless when it comes to actual betting. The definition of "bet-proof procedure" explicitly uses averaging over the possible outcomes of the data collection procedure X but after you've collected X and told everyone what it is, if someone knows \Theta and knows X they can calculate exactly whether the confidence interval does or does not contain \Theta and so they win every bet they make.

So "bet-proof realized confidence interval" is really just a technical term meaning "a realized confidence interval that came from a bet proof procedure" however it doesn't have any content for prediction of bets about that realized interval. The Bayesian with perfect knowledge of \Theta and X and the confidence construction procedure wins every bet! (there's nothing uncertain about these bets).

 

5 Responses leave one →
  1. Chris Wilson permalink
    March 8, 2017

    Personally I found that discussion a bit long-winded - thanks for summarizing your thoughts! At the end of the day, a frequentist CI is still a frequentist CI (i.e. can only really be understood as embedded in a long-run sequence)- and "bet-proofness" seems mainly to map to situations where the Bayesian CI with flat-ish priors makes sense. I still don't understand why the original poster thought this concept is somehow easier/more intuitive than teaching students the truth about frequentist CIs and Bayesian alternatives...

    • Daniel Lakeland
      March 8, 2017

      Yes, I think it was so long winded because it WASN'T obvious, and also because there is this punning "bet proof procedure" vs "realized bet-proof interval"... There is nothing bet-proof about the realized interval except that it came from a class of procedures called bet-proof.

      Yikes!

  2. Kit Joyce permalink
    March 8, 2017

    I think your argument had even more weight when you pointed out that the Bayesian with just *some* good prior information about theta (but not necessarily perfect knowledge) can have a betting advantage.

    Overall I would say that this bet-proofness criterion potentially fixes some pathologies with the concept of (frequentist) coverage, like picking the empty set with prob 0.05 and all of R with probability 0.95 for a real-valued parameter, but you still just have a statement about sampling distributions which doesn't really tell you anything useful about what you know or how confident you should be about it.

    • Daniel Lakeland
      March 9, 2017

      So the thing that I finally came to is that having *some* information about theta helps you after you see the X and the confidence interval. Then, you can bet on whether Theta is in that specific interval and win money. But, before you see the X, you won't know which interval you'll get, and even if you know Theta perfectly as long as on average *over repeated trials* the confidence procedure covers 95% of the posterior interval for your perfect delta-function prior... you can't bet ahead of time and win. That's really what bet-proofness is. It also seems irrelevant to anything real. In practice we go get some data and then want to find out what we know about the parameters. Confidence procedures that are bet proof don't help us learn anything about the parameter vs just doing a bayesian analysis with real prior info.

    • Daniel Lakeland
      March 9, 2017

      See my updated post. I was misinterpreting the definition of bet-proofness... But the real definition is in some sense even weirder, what is the relevance of the mathematical existence of a parameter value that breaks my betting system? A million foot tall person will break any betting scheme I have on the height of people... Still I'll take every bet you offer me about million foot tall people.

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS