In discussion at Gelman’s blog, I gave a kind of summary description of what I thought Frequentist statistics was about (that is, principled frequentist as opposed to just “what is done a lot”). One of the commenters “ojm” felt that I was really misstating the case.

So, I sat down to try to figure out what frequentist statistics was more carefully, and went to one of the “sources” I felt very comfortable with: Larry Wasserman. Larry defines a Frequentist Inference as one in which we “construct [a] procedure with frequency guarantees”

But in what sense is this “guarantee” correct? Every one of these guarantees is of the approximate form “Random Number Generators of the type \$\$R(i,q)\$\$ will produce data \$\$\{D_i\}\$\$ such that \$\$q^*(\{D_i\}) – q\$\$ is in interval \$\$I(R,q*)\$\$ 95% of the time that a sample is taken”

Notice how this doesn’t say anything about the world? It says something about the relationship between a mathematical function \$\$R(x,q)\$\$ and an estimator function \$\$q*\$\$

So, when can we apply Frequentist procedures to *real world* processes, and expect anything useful?

Here’s some distinguishing cases that I think are important:

1. Frequentist inference about a finite population at a given time with an approximately known distribution, using a random number generator to sample the population.
1. In this case, I think we’re good, since we’re relying on a random number generator there really is an \$\$R(i,q)\$\$ so to speak.
2. Frequentist inference about a finite population at a given time without an approximately known distribution using a random number generator to sample the population.
1. Again, we have a random number generator \$\$R(i,q)\$\$ but it gets mapped through an unknown distribution of population values, but we can still construct “distribution free” inferences. It’s possible! Things like Bootstrap and permutation tests and soforth have this flavor.
3. Frequentist inference about a finite population at a given time with an approximately known distribution and a sample where we have an approximate model of the sampling process.
1. Here we’re on shakier territory. We took a sample, but we don’t know exactly how that sample was produced, but we have an approximate model. Here we can at least get conditional on the model being good enough, something of use.
4. Frequentist inference about a finite population at a given time with an unknown distribution and an unknown sampling process for which we have no good model (Say, convenience sampling of Mechanical Turk respondents to a survey).
1. Here, I’m sorry, but we’re just sunk. You can make up whatever Frequentist “guarantees” you like about random number generators \$\$R(i,q)\$\$ but you need to work VERY HARD to convince me that they mean anything about the world. (and the kind of work required is going to more or less push you into the categories 1-3)
5. Frequentist inference about a process through time considered as an infinite set of potential samples (ie. the measurement error distribution of an electronic instrument used to make many repeated measurements) where we have scientific reasons to believe the frequency distribution is stable.
1. This might work, so long as that science holds. But the term “frequency guarantee” sounds a lot stronger than “provided the world acts like random number generator R then …”
6. Frequentist inference in which you use an IID sampling type likelihood that isn’t calibrated to a known distributional shape for the output of some process that isn’t RNG based sampling of a finite population, and you do maximum likelihood estimation.
1. This is a Bayesian calculation, but depending on how you actually construct a confidence interval, you might be adding something in that is Frequentist here… but most likely not based on what I typically see people do. I’d recommend accepting that you made assumptions to choose that Likelihood based on what you know (a fact in your head) and are calculating what that implies about other facts using probability theory as generalized logic. Oh, and just because you didn’t put in a prior doesn’t mean you don’t have one, you just have a flat one.