It's really a kind of logic or thinking or analysis which is in question here, not what a person "truly believes" etc. Does an analysis rely on the frequency properties of a distribution to define the mathematical operations that it entails? Then it has a different structure than if your analysis relies on a measure of plausibility. We need words to describe both things. The standard ones are Frequentist and Bayesian, but these also get used in more general contexts so there's ambiguity perhaps.

It's perfectly possible for a single person to be as you say pluralist, and they need not actually believe certain stuff, when I say "a frequentist has to believe ..." this is really short hand for "the logical consequences of doing an analysis in which a model is taken to represent a set of repeated samples with a stable frequency distribution and a test is performed to determine whether that assumption can be rejected are in part..." You can see why that gets abbreviated.

I think it's important to explore the logical consequences of taking a model in hand and trying to find out something in the world from it, and the logical consequences are different depending on whether you're testing the frequency properties of a functional of a distribution under a random sampling assumption, or determining the degree of reasonableness associated using Bayesian mathematics.

One question of interest to practitioners is "what the heck am I doing anyway?" because many people are taught procedures and formulas and soforth and don't think about what is the structure underlying those ideas. From that perspective, considering whether "something that is typically done" is "a restricted kind of Bayesian analysis" or "a test-based analysis assuming a frequency distribution" is a useful compare-and-contrast exercise.

]]>In particular though, this model doesn't reject anything based on Frequency grounds. The question is just can we find some parameter values in the high probability density region of the resulting posterior distribution and do they correspond approximately to reality?

A model is Frequentist when it uses the frequency properties of sampling from a distribution for inference, accept or reject at a certain p value to form a confidence interval, and you're doing Frequentist inference. Choosing to regularize by restricting your models to various families a la Laurie Davies' examples on Gelman's blog, and you're still fitting a random number generator to your data so that you can produce some "fake data" and see if it's "like" your real data when the parameters have various values.

If you believe that your goal in choosing a model is to approximate the frequency properties of your data to within a sufficiently good "epsilon" which need not be zero, but should be in some sense "small" then you can't agree with my orange juice example, because it was specifically chosen to FAIL to approximate the sampling distribution in a very obvious way:

http://models.street-artists.org/2014/03/21/the-bayesian-approach-to-frequentist-sampling-theory/

going in to the analysis we know that there's an upper bound on what an orange juice carton can hold that's in the vicinity of say 2.5 liters or less, and yet intentionally, I chose the exponential distribution which has density out to infinity and nontrivial density out to 4 or 5 TIMES the maximum that we know can fit. The math nevertheless works out and gives acceptable inference in a Bayesian sense. A whole variety of frequentist tests would reject this exponential distribution.

Is it possible to get good frequentist inference from the exponential distribution in this case? I'm sure you could post-hoc knowing that the Bayesian calculation produces OK inference, concoct some way to make it work sufficiently well for a Frequentist inference procedure.

To get "good" defensible Frequentist inference, you'd have to take advantage of the random selection and probably do something like bootstrapping, or kernel density estimation or choose a class of models and test the parameters to see if the sample was unusual under those parameters to get a confidence interval.

]]>"How do we know whether it's because "there's no use in doing that", or the model really is a bad one?"

Well, I hope we know what you use the model for, and what precision is required. I also hope we know that some violations of model assumptions are problematic for certain computations and others aren't (heavy tails will destroy a sample variance, light tails won't).

You apply the same thinking when doing Bayesian modelling. For the very same reason for which you understand that testing continuity of a frequentist model doesn't make sense, you'd also use a continuous Bayesian model despite the fact that you know that all data are discrete. These considerations are not so mysterious after all, only they are hardly ever spelled out.

"The fact is that the core philosophy of Frequentism is that there is a "real" distribution which describes what would happen after a large repeated sampling experiment." For me that's a thought construct, an idealization. Pretty much everyone knows this when pressed although admittedly many tend to forget it at times. The model should fit the process in some relevant respects, not in all respects. Again it's the same with Bayesian modelling.

"The testing in essence asks the question "is this data a low-frequency occurrence under some assumptions?" "

I'd say that the testing asks the question "Are the data compatible, in the sense defined by the test statistic, with the model?" The statement that the data are indeed a low-frequency occurrence under the model can never be confirmed. That's not what testing is about.

"because the Bayesian answer answers a real question: "How much do we know about the process?"" - well, in order to get Bayes started, you need the answer pre-data in numerical form already before you start observing and analysing data.

]]>But this is one of the major flaws in testing based inference. We reject a model with a test. How do we know whether it's because "there's no use in doing that", or the model really is a bad one? What tells us that "there's no use in doing this" vs "we should start again?" there is no logic to it, just guesswork. Typically the guesswork leads closer and closer to a Bayesian answer, because the Bayesian answer answers a real question: "How much do we know about the process?" not an imaginary question "how often would things happen if we continued to repeat this process indefinitely?"

The fact is that the core philosophy of Frequentism is that there is a "real" distribution which describes what would happen after a large repeated sampling experiment. All tests are based on the notion that we can reject stuff based on it not producing the right frequencies of some statistic.

The entirety of bootstrapping is predicated on my notion of "a moderate sample will fill up the real distribution" for example. If there's some region of space that hasn't occurred in your sample but could easily occur in the future at some point... Bootstraps will give horrendous results.

Typically, testing either corresponds to a Bayesian calculation, or it answer the wrong question. The main exceptions are where we're talking about trying to construct a reliable random number generator machine or computer program. The "die harder" tests are exactly the kind of thing that's needed for their special purpose.

Frequency based inference relies on testing. The testing in essence asks the question "is this data a low-frequency occurrence under some assumptions?"

Bayesian inference relies on likelihood, it asks "under what assumptions is this dataset relatively high probability?" of course it defines "relatively" in terms of a trade-off of prior probability and data likelihood, and it defines probability in terms of "isn't too surprising to us" not "how often stuff happens in an infinite future string of occurrences"

The part that gets confused is when people use likelihood based inference and think of it as Frequentist. It isn't. Likelihood ratio tests may be, I haven't thought too hard about that, but likelihood based inference on parameters *is* the Bayesian thing to do (when you have flat priors).

]]>There's no use to test a frequentist model for deviations that are not relevant for what we want he model to achieve. Obviously all observed data are discrete so you can design a test that rejects every continuous model (the issue is the same for Bayesians but doesn't play out in terms of tests there). But we don't do that, because this mismatch between model and reality is not normally relevant to the kind of conclusion that we want to draw from the model. Same if we have large amounts of data. You can easily find a test that rejects a parsimonious model but there's no use in doing that. We'd want to detect some deviations from the model that would lead us to misleading conclusions in case we continue using the model despite of them; other deviations can be tolerated.

"Bayesian modeling doesn't require that the data fill up the distribution, only that it not be in the extremely unusual portion of the distribution." But this can easily happen and happens regularly in cases where people assume exchangeability but in fact there is dependence . (I'm using the term "dependence" here in a rather intuitive manner; for the Bayesians I should probably add "conditionally on the parameter" but then the Bayesian may deny that there's any "true" distribution to which "in fact" could apply, so I mean something like "it behaves as if you're generating data from a frequentist model with dependence that is big enough for making a difference".)

]]>However, on Bayesian grounds it doesn't invalidate the model if there's moderate probability for some region x and yet x never occurs. That's the point of the statement you're quoting. A "test" is a way of checking whether some Frequency assumption is violated, they compare data to the output of random number generators. The logic of Bayesian modeling doesn't require that the data fill up the distribution, only that it not be in the extremely unusual portion of the distribution. Those are the fundamental organizing principles behind Frequentist and Bayesian models.

]]>So part of this blog, and I don't want it to be the only part, is to discuss the organizing principles behind different kinds of analyses.

I think it helps to figure out what is the principle that makes us think that a particular analysis works, and it could help people who do these "classical" analyses to realize that they're already basically doing Bayesian analyses, because then they can actually engage fully with that thought process, including using informed priors and choosing alternative likelihoods based on Bayesian principles. So that's more or less the reason why I spend time on that issue. I think there are lots of other good blog topics though so maybe I'll spend a little time on some other ones because that is actually a goal of mine, to keep the blog diverse.

]]>