# On models of the stopping process, informativeness and uninformativeness

I had a great conversation with Carlos Ungil over at Andrew Gelman’s blog where we delved deep into some confusions about stopping rules and their relevance to Bayesian Inference. So here I’m going to try to lay out what it is I discovered through that conversation, and I’m going to do it in the context of Cox/Jaynes probability with explicit background knowledge, translating from what I consider the much more nebulous terms like “deterministic” or “random”.

**Standard Textbook Stopping Problem:**

First off. Let’s talk about the “standard textbook” type stopping rule: “Flip a bernoulli coin with a constant p until you see 3 heads in a row and then stop” Suppose you get 8 observations HTTHTHHH. Now, the rule is completely known to you, and so you can determine immediately by looking at the data whether a person following the rule will stop or not. If someone hands you this data and say “given what you know about the rule, will they stop?” you will say P( STOP_HERE | Data, Rule) = 1. *for you* the rule is deterministic thanks to the textbook description. Under this scenario, knowing that they stopped at N=8 does not add anything that you didn’t already know. Therefore it can’t add any information to the inference. Basically

P(Parameters | Data, Rule, STOP) = P(Parameters | Data,Rule)

### Standard Real World Stopping problem:

In the real world, the reason why people stop is rarely so clearly known to you. The experimenter tells you “Our collaborators ran a few trials and read the protocols from another lab, and tried this thing out, and it seemed to work, and so we tried it in our lab, and after collecting 8 samples we saw that the results were consistent with what we’d been told to expect, and so we stopped.”

Or a survey sample of a population comes with a dataset of answers from 120 people, and a description of the protocol: “we ran a small preliminary study to test out our questions, and found in the preliminary study that the phrasing of the questions didn’t seem to matter, and so we used the observed variability in this previous study to determine that a sample of 120 people should give a cost effective dataset for answering the questions we asked, we then sampled a fixed 120 people” but… because the preliminary study was based on a mixture of different questions and soforth… it’s not included in the data set. Here, the information gleaned in the earlier trial study is expressed only through the choice of 120 people. The “fixed deterministic” rule “sample 120 people” is informative for a bigger model in which you include the earlier steps.

Or a slightly more sophisticated version: “we sampled until our Bayesian model for the probability of parameter q being in the range 0-0.1 was less than 0.01, ie. p(q < 0.1) < 0.01” To the people running the study, a Bayesian posterior is a deterministic function of the data and the model. Everyone who has the same model always calculates the same posterior from the given data. But note, *you* don’t know what their Bayesian model was, either priors or likelihood.

### Deterministic vs Random stopping rule vs Cox/Jaynes restatement

In the literature, a stopping rule is called “informative” if it is “a random stopping rule that is probabilistically dependent on a parameter of interest”. I personally think this is a terrible definition, because “random” is usually a meaningless word. It gave me a lot of trouble in my conversation with Carlos, because when I think random I pretty much exclusively use that in the context of generated with a random number generator… but that’s not what is meant here. So let’s rephrase it in Cox/Jaynes probability terminology.

**A stopping rule is informative to you if given your background knowledge, and the data up to the stopping point, you can not determine with certainty that the experiment would stop, and there is some parameter in your model which would cause you to assign different probabilities to stopping at this point for different values of that parameter.**

Under this scenario, the *fact of stopping* is itself data (since it can’t be inferred with 100% accuracy just from the other data).

Now in particular, notice that the informativeness of a stopping rule depends on the background data. This should be no surprise. To a person who doesn’t know the seed of a random number generator runif(1) returns a “random” number, whereas to a person who knows the seed, the entire future stream of calls to runif is known with 100% certainty. It’s best to not use the term random and replace it with “uncertain” or better yet “uncertain given the knowledge we have”.

### What can you usually infer from the stopping rule?

If you’re in a situation where the stopping rule is uncertain to you, then if you believe it is helpful for your modeling purposes, you can add into your model of the data a model for the stopping rule (or a model for the choice of the “fixed” sample size). This is particularly of interest in real world rules along the lines of “my biologist friends told me what I should be getting from this experiment so I tried about 8 experiments and they kind of seemed to be consistent with what I was told, so I stopped”. The actual rule for whether the experimenter would stop is very nebulous, but the fact of stopping might tell you something you could use to describe distributions over relevant real-world parameters. For example, suppose there’s an issue with the way a drug is delivered that can cause toxicity if you do it “wrong”. The fact that the biologist stopped after 8 experiments suggests that they believe p(DetectToxicity | DoingItWrong) is near 1, so that if you haven’t seen it in 8 tries then you are virtually certain you are doing it “right”.

So, eliciting information about the stopping rule is very useful because it can show you that there are potentially parameters you need to include in your model for which the fact of stopping informs those parameters, and particularly, parameters *that describe the nebulous uncertain rule*.

In the example above about sampling until a Bayesian posterior distribution excluded the range 0-0.1 with 99% probability, if someone tells you exactly the description of the Bayesian model, then if you plug in the data, you will immediately know whether the rule said stop or continue. But, if you know what the likelihood function was, but not the prior, then you could potentially infer something about the prior that was used from the fact that the sampling stopped at this point. If you think that prior was based on real background information, this lets you infer some of that real background info.

## Summary:

To a Cox/Jaynes Bayesian there is no such thing as “random” only “uncertain given the background information”. A stopping rule can teach you something about your parameters precisely when your model doesn’t predict the fact of stopping with probability = 1 *and* your model has a parameter which affects the probability of stopping conditional on data in other words, p(STOP_HERE | Data_so_Far, Background, Params) is a non-constant function of Params

Then, given the data, the fact that the experiment stopped is additional information about some of the Params

Stopping rules are not irrelevant to Bayesian models, but they are only relevant in those circumstances, if you feel that the stopping rule seems vague or the reasons for the choice of the sample size seem based on some information that you’re not privy to, then you might want to invent some parameters that might help you explain the connection between the fact of stopping, and the things you want to know such as “hidden” prior probabilities in use by the experimenter that inform their stopping.

The thing that becomes obvious in the restatement is that the fact of stopping is data just like any other data. And its informative precisely when the data is not redundant with other knowledge and whatever your model is predicts different probabilities for that data for different regions of parameter space.

Your model might even predict 100% probability to stop for some regions of parameter space. And then the stopping rule is a deterministic given the parameter value but perhaps only 10% given a different value. There’s no mysterious random powers inherent in the rule, there’s only stuff you know for certain and stuff you’re uncertain about.

When people talk about the relevance of stopping rules to Bayesian inference, the issue is whether the fact that data collection was stopped at some particular point does provide any additional information which is missed if the stopping rule is ignored and a fixed size is assumed. When the stopping rule used is indeed informative, it affects inference by changing the likelihood.

Many of the questions you discuss are about how you could (but not necessarily should!) use the assumptions implied by someone else’s stopping rule (or their experimental design in general) to establish the prior for your model.

Interesting as that may be, that’s a completely unrelated issue and it doesn’t even require that a stopping rule is used at all. If the original design included a stopping rule which was finally not put in place for some reason, you can still use it to shape your prior. You can also use the assumptions implied by the stopping rules used in other experiments or by alternative stopping rules that may have been discussed.

> Now in particular, notice that the informativeness of a stopping rule depends on the background data. This should be no surprise. To a person who doesn’t know the seed of a random number generator runif(1) returns a “random” number, whereas to a person who knows the seed, the entire future stream of calls to runif is known with 100% certainty.

However, both will agree on the non-informativeness of the stopping rule.

I don’t know about what “people” do, but I know where I am now. I have a logically consistent view of when I can expect to use information about the stopping mechanism that works within a conception of probability as a real valued logical quantity instead of some kind of mysterious “inherent randomness” (ie. an objective fact of randomness of the stopping rule about which all people will agree).

It is not the case that everyone will agree on the non-informativeness of the stopping rule.

You collect data until your Bayesian model excludes some region of the parameter space. You use a likelihood for your data that I’m aware of, but a prior that I’m not. Your posterior distribution is a deterministic function of the data and your model. Your stopping rule is uninformative *to you* (ie. you can’t learn anything from the fact that you stopped).

In order to analyze this data, I can build a model in which your prior is represented by an additional parameter or two. My model (likelihood *and* prior) now has a new parameter, and the information that you stopped at a certain point is itself informative in my model whereas in your model it’s totally uninformative. Why? because your background knowledge already has that prior as a 100% certain thing, you’re the one who chose it.

There are vast seas of application for this concept. If in my model there is a relationship between your choice of prior and the parameter that describes the actual data (such as I know that you’ve collected similar data in the past, and you have informed reasons to choose your prior). Then this process is not only informative to me about what your prior was, but also I can put correlations between your choice of prior for stopping and my parameter for the data. Something like:

myPriorLocation ~ normal(myPriorGuess,myPriorUncertainty);

yourPriorLocation ~ normal(myGuessForYou,myUncertainty);

myDataLocation ~ normal((yourPriorLocation+myPriorLocation)/2,myUncertainty);

StoppingN ~ someDistribution(yourPriorLocation);

Data ~ someDataDistribution(myDataLocation,…);

By inferring your prior from the stopping rule, it influences the high probability region of my estimate for the data location parameter, a kind of partial pooling between what I originally thought, and what I discern from your stopping. Plus, it adds a parameter. The likelihood in my model now has an extra StoppingN ~ someDistribution() factor, and it is a function of 2 parameters instead of 1. You’re right, this is a choice, it’s not something that I necessarily SHOULD do (esp. if I think you used a very poorly informed prior). But if your prior is based on something like a preliminary study, or studies in related animals, related regions of the world, related chemicals… whatever, then it’s a useful thing to do for me.

My likelihood is different even though your stopping rule is not only uninformative to you, but actually deterministic to you.

The literature is full of confusion. Some people are pure Frequentists when it comes to probability, some people are Cox/Jaynes Bayesians, and the vast majority of the Bayesian literature seems to be full of people who use the mathematics of Bayes but the probabilistic conception of Frequentism, to this latter group, whether you know the stopping rule or not is irrelevant it’s the “objective” fact of whether it had a “random” component or not, and whether the random component was or was not “objectively” dependent on the parameter that describes the “true” distribution of the data…

To that group, if you say “I used a bayesian model and stopped when it excluded a certain parameter region at a certain probability” they can look at this and say “objectively deterministic” even though they have no idea what prior you used, and then they can magically say “the true data distribution is normal with unknown mean” (Frequentist conception of the data generating process) and the stopping rule is “objectively deterministic” (Frequentist conception of what it means to be deterministic), and so I am required to analyze it with the likelihood

data ~ normal(location,scale);

That’s all nonsense in my opinion. Bayes is all about making choices based on your background knowledge. If when you look at the data, your background knowledge doesn’t give you the ability to determine whether the data collection process would stop, but you can think of a way to create a parameter that you can partially infer from the fact of stopping (ie. it would put different predictive probabilities of stopping for different parameter values), and you want to use that parameter in your model, then the stopping rule is informative to you.

Note, the truth is even I found it hard to escape the trap of “if you plug in the data you will always get the same answer therefore the stopping rule is deterministic” look how it seduces you and yet it’s clearly a frequentist conception (ie. repeated applications of the rule with the same data, by you, produce the same result).

Thinking in terms of uncertainty and logic: “If you told me you had collected all this data, but you didn’t tell me whether there was more to come or not, given what I know about your stopping rule, can I determine with certainty that you will or won’t stop?”

that’s the interpretation that is consistent with Cox

> It is not the case that everyone will agree on the non-informativeness of the stopping rule.

Everyone will agree in your example, the stopping rule is deterministic to the person who knows the internal state of the pseudo-random number generator runif(1) and it is non-deterministic to the person who doesn’t know but both will find the stopping rule non-informative.

> You collect data until your Bayesian model excludes some region of the parameter space. You use a likelihood for your data that I’m aware of, but a prior that I’m not. Your posterior distribution is a deterministic function of the data and your model. Your stopping rule is uninformative *to you* (ie. you can’t learn anything from the fact that you stopped).

The stopping rule is also uninformative to you because it is deterministic. If you know that it is deterministic, you know that it is non-informative. And if you don’t know that it is deterministic, it’s still non-informative given the data (which is the same for you and me). I don’t see how you’re going to extract much information from the stopping rule if your model doesn’t let you see it’s deterministic. At least this is what I think, but I would be interested in seeing a detailed example of your methods.

Let’s say I perform a coin-tossing experiment (trying to infer the value of theta) and get two heads.

I think the likelihood is L(theta)=theta^2, regardless of my stopping rule (as long as it is a deterministic function of the data).

If I understand what you wrote, you will consider the stopping rule non-informative if you know the details of the stopping rule but it will be informative if you know only partially the details of the stopping rule. This is remarkable, the less you know about the stopping rule the more informative it will be!

Maybe you could give an example, using whatever assumptions you see fit for the sake of the illustration, of what will the likelihood function L(theta) and how the inference will change in different scenarios:

(0) you don’t know if N was fixed or there was a different stopping rule of any kind

(1) you know that N was fixed (=2)

(2) you know that there was a deterministic stopping rule different from N fixed, but not the exact details

(3) you know that the stopping rule was to stop after two heads

From what I have understood, your likelihood will be L(theta)=theta^2 in scenarios (1) and (3).

Carlos: this is the problem I was having with the terminology “deterministic” which is why I felt really confused about the definition. This “deterministic” is a frequentist concept and is much more limited than the Bayesian probability as numerical plausibility or credence.

It makes perfect sense that information about a thing you know a lot about is uninformative. It’s like if you know you’re wearing blue jeans and I tell you “hey you’re wearing blue jeans” this is uninformative. It’s only if you put your pants on in the dark and haven’t looked at them yet that you will find “hey you’re wearing blue jeans” to be informative. The less you know about something, the more data about it is informative, that seems obvious to me once you look at it in this way. Of course, data is uninformative if you don’t know what it means. If I say “Hey grobnatzig freeblongis” unless you have some idea what those words might mean… still uninformative.

Now, if there are no parameters in my model which cause me to assign different probabilities to your stopping rule, then I also can’t determine much from the data. it’s still uninformative in the way “grobnatzig freeblongis” is.

I have already given an example in which the deterministic-to-you stopping rule is informative to me, the case where you stop after your bayesian model excludes a region of your parameter space, but I don’t know your Bayesian prior. The information I gain is something about your Bayesian prior, and I can also use it to infer something about the parameter if I think your Bayesian prior is informative about the parameter.

In your cases (just for simplicity let’s imagine we’re sampling measurements of length from something my background says might be normalish distributed, it’s maybe easier for me to describe than a bernoulli coin. One parameter of interest then is the mean of the normal)

0) I have no information to assign probabilities about stopping with, and I have no data about whether you did in fact stop sampling. I have only the data so far. In this case, I have a model p(Data | Params, Knowledge) p(Params | Knowledge) just based on whatever information I do have, let’s say I suspect data comes clustered around a central value with a mean squared error of 2 just for illustration, so Data ~ normal(mu,2); mu ~ my_prior_for_mu().

1) I know N was chosen to be 2, but I have no knowledge about how. I have no basis to infer anything from the fact that N = 2 so I have the same model as 0.

2) To make this little bit of extra knowledge helpful, I need to assume that I have some partial knowledge of how N was chosen (that is, that I can interpret the fact of stopping in some way, I’m not in the “grobnatzig freeblongis” type case. I create a model with partial information about how you selected N, with new parameters describing my understanding of that process, and I fit this new model. In this model “I Stopped at N=2” is data. And assuming I have some model over this data, then ParStop is the parameters of the stopping model. If I in addition think what I can infer about ParStop is also informative about the parameter Mean, then I should use these new parameters in my likelihood. p(Data | Mean) p(Mean | ParStop) p(ParStop | “Stop at N=2”, My Stopping Model) p(ParStop, Mean | Background)

3) In this case, if you tell me N=2 I can immediately infer HH without data. Or, if you tell me HH I can infer N without data, so my model is the same as 0 if I treat HH as the data. My model is independent of all data except N if I treat N as the data.

The interesting case is I think (2) because it’s the case where there’s some hidden information in the process of sampling that is partially revealed by the choice to get a certain N, *even if that choice would always be made the same way in repeated trials that got the same data*

In other words, the fact that the stopping rule repeats perfectly means its Frequentist probability is 1 but the fact that I don’t know everything about that rule means that its Bayesian Probability given my background is NOT 1.

Here’s an example where we can plug in some numbers and express the different inferences using Stan notation (ack… trying to do this a little too quickly with kids begging for popsicles in the heat…. let me come back to it, because you could argue that all I’m doing is “changing priors” here, but I can make it have a “likelihood” type term in which N acts as data for an inference… or you can fill in the blanks if you see where I’m going).

I know you’re collecting data on the length of some object, and I get from you the data set {1.04, 0.97, STOP After 2}

I also know that you have a lot of experience with this measurement apparatus, and that you’re motivated to measure this object with reasonably tight precision. (some way to partially interpret the stopping rule)

Also I have reason to believe that the size of this object is O(1) so that I could in the absence of any data put say a gamma(2,2) prior on the length.

Now, if I don’t use the information about the stopping rule, I get the inference

mu ~ gamma(2,2);

sigma ~ exponential(1.0/0.5); // I guess the measurement instrument isn't too inaccurate

Data ~ normal(mu,sigma);

If I do use the fact that I know you have knowledge of the measurement instrument, then together with the data N, I get the inference:

mu ~ gamma(2,2);

yourPosteriorSigma ~ exponential(1.0/.02); // information about how much error I know you're willing to tolerate yourMeasurementSigma ~ gamma(10.0,10.0/(yourPosteriorSigma*sqrt(N)); // inference about what you must think the measurement error in the machine is

`Data ~ normal(mu,yourMeasurementSigma);`

The fundamental quantity is the joint distribution, which here depends on data N and in the first case doesn’t.

I think you could make it more like what you expect to see in a prior*likelihood situation by

yourMeasurementSigma ~ MyPriorOverYourMeasurementSigma();

sqrt(N) ~ gamma(10.0,10.0/(yourMeasurementSigma/yourPosteriorSigma));

or something like that, where sqrt(N) is on the left hand of a sampling statment because it’s data, this is obviously an approximation, because N is discrete.

again just checking in here quickly to give hints between other sunday duties.

> The interesting case is I think (2)

Let’s stay with the other cases for a second then. If I understand correctly, you get always the same likelihood for theta. You said that your model in (3) and (1) is the same as in (0). Do you mean that you get the same posterior distribution for theta in all these cases?

Yes as far as I can see you get the same posterior for the parameter in 0 1 and 3.

The case 2 is where because you have partial background data that can be used when you know there was a stopping rule that was triggered you get to infer something.

In probability as logic this 2 is the equivalent of what random means to a frequentist. And the partial information allows you to invent a parameter which is then probabilistically related to the fact of stopping. It has the same “random and related to the parameters of the model” but the meaning of random is different for Bayesian interpretation. In Bayesian interpretation random means you can’t be sure stopping would happen without being told it did happen.

When the Bayesian knows that the stopping rule is deterministic in the sense of always coming out the same conditional on the data, then the likelihood for the inference about the parameter that informs stopping looks like 1 when given the parameter and the data the Bayesian predicts stopping, and 0 when given the parameter and the data the Bayesian predicts no-stopping. So, after being told that the data collection stopped, the posterior for the parameter that the Bayesian uses to predict stopping is the prior, restricted to the region where the value is consistent with stopping at the given N.

When there’s something else going on (like for example, the Bayesian doesn’t know enough about the rule to predict with certainty) then you’d get intermediate probabilities for stopping and a continuous likelihood.

(I apology for being reiterative, but I want to be sure I remain on track)

If you don’t have any information about a stopping rule, you get some posterior for theta.

If you have complete information about the stopping rule, you get the same posterior.

However, if you have partial information about the stopping rule you might be able to extract additional information from the data and get a more precise posterior distribution for theta.

Don’t you find that slightly disturbing? If you know precisely the stopping rule, should you ignore some of that knowledge to improve your inference about theta?

carlos, we’re out of reply depth, so I started over at the bottom.

Carlos, looks like we’re out of Reply depth. Here I am replying to your comment: http://models.street-artists.org/2017/06/24/on-models-of-the-stopping-process-informativeness-and-uninformativeness/#comment-217212

Thanks, you’ve pointed out a situation in which we are ambiguously talking about different situations.

Let’s reiterate your situations, and add another one which is directly related to my number 2:

your scenarios:

(0) you don’t know if N was fixed or there was a different stopping rule of any kind

(1) you know that N was fixed (=2)

(2) you know that there was a deterministic stopping rule different from N fixed, but not the exact details

(3) you know that the stopping rule was to stop after two heads

From what I have understood, your likelihood will be L(theta)=theta^2 in scenarios (1) and (3).

In scenario 0 I have my simple posterior from above using just my prior on the mean and the measurement error size.

In scenario 1 the fact of stopping tells me nothing and I have the same posterior as 0.

In scenario 2 I assumed I have some partial information about the stopping rule, for example that it relates somehow to an instrument precision/measurement error information that I think you have. a new parameter is born describing what I know about what you know about the measurement error, and the fact of stopping informs me about what you know about the measurement error. That also informs my posterior for the length, it would probably make it more concentrated since you are assumed to know something real about the instrument error.

In scenario 3 to map to my measurement error scenario, the rule was say to stop after two heads of a separate coinflip… this doesn’t help me infer anything, except what the coinflips are, a fact I don’t care about, but I do in fact learn about them if I care to.

In scenario 4) you tell me that you stopped after plugging in the data to your bayesian model, you give me the explicit prior and likelihood that you used. Now. rather than inferring partial information about what you know about the measurement error, I have already been handed complete information about what you know about the measurement error. So my posterior distribution is an even more concentrated version of (2), since I use your very good information abut the measurement error exactly. In fact, unless I have some reason to add something else to my model (like for example that I think there’s a slight consistent bias to your measurements or whatever) then my posterior becomes your posterior.

If you now pointed out that I think in scenario 4 that I should throw out the email you sent me with the exact measurement error prior, and go back to scenario (2) then that really would be something weird. However, I don’t think that. In scenario 4 I use your prior because you’re the one who knows the instrument.

Does that help us?

Ok, so instead of analysing the data using your model and your prior if I tell you my model and my prior you might like them better and use them instead, getting a different (maybe better) posterior distribution for the parameter of interest.

And if you know quite a lot but not everything about my experimetal design you can use the fact that there was a stopping event to infer a bit more and reverse engineer my model to some extent and use that to improve your model so you get a different (maybe better) posterior. I don’t really think that’s feasible in the real world but I guess it’s not logically impossible.

Yes!!! we’ve arrived at the same point. Thank you again Carlos.

Feasibility in the real world is of course very dependent on the problem. To me, knowing how to analyze the stopping rule in a fully Bayesian context thanks to your help with clarifications at least gives me the tool I need to argue to people whether or not the stopping has meaning.

There are enough cases that I am faced with whether the stopping rule was clearly pretty nebulous, so simply telling people “hey here’s the things I need to know about the experimental design to determine whether we need to think about the stopping” that’s really helpful.

Even if every time we say “looks like we should just ignore the stopping rule” at least we’ve actually done it based on some logically consistent Bayesian principle.

Please Note:Your comment is awaiting moderation.Хорошо смотрится реклама, где максимально используется окружающее пространство

https://clck.ru/LhYFs

https://clck.ru/LhWo7

https://clck.ru/LhVFU

http://bit.ly/2twMzjo

https://clck.ru/LhXCB

https://clck.ru/LhYVo

https://clck.ru/LhVGV

https://clck.ru/LhXEo

https://clck.ru/LhYHo

https://clck.ru/LhZ2y

https://clck.ru/LhYaL

https://clck.ru/LhZ34

https://clck.ru/LhYzk

https://clck.ru/LhVi3

https://clck.ru/LhX5c

Please Note:Your comment is awaiting moderation.Хорошо смотрится реклама, где максимально используется окружающее пространство

https://clck.ru/LhWVZ

https://clck.ru/LhZ49

https://clck.ru/LhZ4Z

https://clck.ru/LhZ8V

https://clck.ru/LhVWH

https://clck.ru/LhVyt

https://clck.ru/LhV3C

https://clck.ru/LhYsm

https://clck.ru/LhY2Y

https://clck.ru/LhXyA

https://clck.ru/LhZ8X

http://bit.ly/35iRJwQ

https://clck.ru/LhW7j

https://clck.ru/LhW66

https://clck.ru/LhW7Z

Please Note:Your comment is awaiting moderation.Хорошо смотрится реклама, где максимально используется окружающее пространство

http://trb-lbz.sch.b-edu.ru/?unapproved=40707&moderation-hash=149a8267291b26a3876f3e6d1a0fe66a#comment-40707

http://mobile.gamepower7.com/forum/member.php?75455-glodkix&vmid=12438#vmessage12438

https://www.educaedu-brasil.com/blog/bolsa-estudo-brasil/?unapproved=35976&moderation-hash=05148f771538e728cebaaa52a5b46dba#comment-35976

https://emergency.sandiegocountyplumber.com/privacy-policy/?unapproved=37990&moderation-hash=2be08799a3cbebc945b8e8513be1e9af#comment-37990

https://www.grosirsandalsurabaya.com/blog/cara-merawat-dan-membersihkan-sandal-sepatu-crocs?unapproved=770&moderation-hash=307264ee12fa543db04efb88f37733d1#comment-770

https://www.myfhc.org/blog/uncategorized/fhc-sees-healthy-growth/?unapproved=70631&moderation-hash=4c9a93d15f8ef3aac9f4a498de7c4213#comment-70631

https://www.resportes.com.br/como-controlar-o-estoque-de-produtos-de-clinica-e-consultorio/?unapproved=605&moderation-hash=a068aed8aec04b36d29fadc1f8acbe53#comment-605

https://ok-arenda.ru/arenda-zhilogo-pomescheniya-v-moskve/?unapproved=143&moderation-hash=cb1c471a9441ea572d241b2bada8f75f#comment-143

https://www.hellsehen-guilia.ch/gaestebuch/?unapproved=2847&moderation-hash=c3eeb415ab21b787456438d634c605ea#comment-2847

http://tecnociencia-sociedad.com/aterrizaje-en-marte-made-in-spain/?unapproved=18923&moderation-hash=5914e615d252f98657cf25a43a949754#comment-18923

http://www.0559444619.com/member.php?u=79053

http://sy-therapy.lu/120/?unapproved=167913&moderation-hash=3af9c077b82872594b2b731b5d3c5ca4#comment-167913

http://www.dentsuinnovak.com/?unapproved=55831&moderation-hash=678f68b2d0b2c521e1ae1228515342fc#comment-55831

http://benoitandco.fr/?unapproved=17574&moderation-hash=813beeb0a867a0918812852d75f325d6#comment-17574

https://criptofund.ru/satoshipay/?unapproved=5930&moderation-hash=a7e668423edd4e8b5d72980275e188ab#comment-5930

Please Note:Your comment is awaiting moderation.Хорошо смотрится реклама, где максимально используется окружающее пространство

http://smileandchew.com/test/?unapproved=58267&moderation-hash=31489887bf8e13f12884eaf5fd9ecd6f#comment-58267

https://www.estelle.edu/six-skin-care-trends-to-watch-for-in-2020/?unapproved=24293&moderation-hash=81a5e30832824f74c4016bc32f5096f2#comment-24293

https://youngpromusic.com/blog/?unapproved=588&moderation-hash=bb0c79318c48769397bb2d5cd0a14042#comment-588

http://absoluteseniorcare.com/find-a-good-care-center-for-your-self/?unapproved=90793&moderation-hash=51ad27e821879b593fcc8267a5993371#comment-90793

http://bordepolitico.com/cdmx/conoce-el-proyecto-de-la-constitucion-cdmx/?unapproved=363019&moderation-hash=6456c492ab61cfa68dd1471794238495#comment-363019

http://100pudov.kg/video-blog/206/v?id=206#comm111

https://robotics-forum.eu/viewtopic.php?f=2&t=284300

http://bugaroy.ru/2019/09/20/%d0%b2%d1%81%d0%b5%d1%80%d0%be%d1%81%d1%81%d0%b8%d0%b9%d1%81%d0%ba%d0%b8%d0%b9-%d1%81%d0%be%d0%b2%d0%b5%d1%82-%d0%bc%d0%b5%d1%81%d1%82%d0%bd%d0%be%d0%b3%d0%be-%d1%81%d0%b0%d0%bc%d0%be%d1%83%d0%bf/?unapproved=316&moderation-hash=6889e525833ec8d4cebaefd625732a61#comment-316

http://blog.patriottimber.com/independence-day-colonial-flag-project-part-3/?unapproved=52003&moderation-hash=465e5432aa48fa415fe87c21be72ffab#comment-52003

https://sale.guide/akcii/CHernaja-pjatnica-do–80-v-magazine-585-Zolotoj_81923

http://www.inowroclaw.w.szu.pl/forum/viewtopic.php?f=5&t=11502

https://cs-10rus.ru/hello-world/?unapproved=1751&moderation-hash=9449f1119b5eeb354f5cbd3ed97fa0ba#comment-1751

https://evakuators.kz/?unapproved=52494&moderation-hash=a1f2df723f0142518a5307d55759b3ed#comment-52494

http://evered.info/forum/viewtopic.php?f=2&t=220907&sid=e30d1d44edcc022e4ec2f015e3cd7949

https://italiancar.com.au/happy-birthday-topolino/fiat-500-topolino-2/?unapproved=49453&moderation-hash=b0a0ca95739b1dd563858954c4c81463#comment-49453

Please Note:Your comment is awaiting moderation.Хорошо смотрится реклама, где максимально используется окружающее пространство

https://otitoff.ru/preparaty/obezbolivayushhee-pri-otite.html?unapproved=55907&moderation-hash=746156dc8a0b0016dcde8e6f0c2a34eb#comment-55907

https://ayrshirewarhammerenthusiasts.com/forums/topic/%d1%85%d0%be%d1%80%d0%be%d1%88%d0%be-%d1%81%d0%bc%d0%be%d1%82%d1%80%d0%b8%d1%82%d1%81%d1%8f-%d1%80%d0%b5%d0%ba%d0%bb%d0%b0%d0%bc%d0%b0-2/

http://rab32.ru/?p=144&unapproved=23335&moderation-hash=c4980e98573ccb3f1588e4ba34f3adf6#comment-23335

http://bkorsar.ru/kak-vosstanovit-vindovs/?unapproved=69615&moderation-hash=b8d6353059ce1fe9abcad8528cb89c33#comment-69615

http://soportec.com.co/in-faucibus-orciluctus-et/mauris-posuere/?unapproved=49761&moderation-hash=d751b0c836550bb804b6cd729e95d0f5#comment-49761

https://lederzentrum.pl/czeste-pytania/kolory-tapicerek/saab.html?unapproved=22168&moderation-hash=b2be5aa0ee4fb1e04e01fef896ad1bdc#comment-22168

https://pinaken.com/blog/7-ways-to-thank-your-parents-for-what-they-have-done-for-you/?unapproved=4179&moderation-hash=2c06f625d015eddefcf9daac8a17e310#comment-4179

http://gmduo.com/home.php?mod=space&uid=46239

https://mpii.fhi360.org/issue-1-february-2016?unapproved=49625&moderation-hash=5b085848a7ba4d06c09814f8bcfcaf41#comment-49625

https://ronwhitetraining.com/speed-reading-2/speed-reading-2-3137.html?unapproved=18552&moderation-hash=6a2d02dda30328cf8ae31d5b61e507c1#comment-18552

http://www.muirbeachcsd.com/what-water-system-do-you-want/?unapproved=44045&moderation-hash=be9edfde939e1fe5c85015d593c7004e#comment-44045

https://www.nasatalk.com/global-climate-change/?unapproved=21645&moderation-hash=676fd1add4cd94db491363517c545847#comment-21645

http://umrayya.ishtar-gate.org/2010/07/11/hello-world/?unapproved=476109&moderation-hash=0529ee48af90936622ea13ba6525b12b#comment-476109

http://attraheramera.tarotguiderna.se/2017/10/12/187/?unapproved=222352&moderation-hash=771cb496147b9348c2d0a39055f401f5#comment-222352

https://www.peachlandview.com/2019/05/16/name-caller-kicked-from-council-chambers-during-discussion-on-expenses/?unapproved=98900&moderation-hash=069cef032d81b1ea8ab5803197b58988#comment-98900

Please Note:Your comment is awaiting moderation.Хорошо смотрится реклама, где максимально используется окружающее пространство

http://worldfossilsociety.org/2012/08/ammonites-dined-on-plankton/?unapproved=4501125&moderation-hash=6bff7047d577e25bf674435f49a4cd4c#comment-4501125

http://usjcfoot.fr/gallery2/?level=picture&id=425&PHPSESSID=

https://cyourad.com/author/charlessmunk/

http://talkingmakeup.com/beauty/smashbox-24-hour-photo-finish-shadow-primer/?unapproved=3646274&moderation-hash=9ecf06adf26dd696cc7a4b9937875e37#comment-3646274

http://brka-kumertau.ru/?unapproved=11655&moderation-hash=85a82352939eb6f064f567d270bdadef#comment-11655

http://myiowadefenselawyer.com/increasing-parolees-to-improve-public-safety/?unapproved=59109&moderation-hash=1b1f848af536bb4fc882ec476f244a04#comment-59109

https://pepitestroniques.com/keyboards-applications-mao/?unapproved=2278&moderation-hash=d092f8333e4fc4097a984cdbe4dcf2df#comment-2278

http://forum.opencart-destek.com/konu-%D0%B8%D0%BD%D1%82%D0%B5%D1%80%D0%BD%D0%B5%D1%82-%D0%BF%D1%80%D0%BE%D0%B4%D0%B2%D0%B8%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5-%D1%80%D0%B0%D1%81%D0%BA%D1%80%D1%83%D1%82%D0%BA%D0%B0-%D1%81%D0%B0%D0%B9%D1%82%D0%B0.html?pid=98386#pid98386

http://www.jpcunha.com/hino-da-neci-nucleo-especializado-para-o-cidadao-incluso/?unapproved=48374&moderation-hash=100d5335d8a8f43849ccc0660e4493d2#comment-48374

http://www.socialprotectionfloorscoalition.org/2015/01/world-social-forum/?unapproved=67865&moderation-hash=0fa00a0296dda836444aeb530ef341fc#comment-67865

http://www.dev.hydes.in/analytix1/kunena-2013-02-09/analytix-users-category/2559773#2560939

http://yemlattechnologies.com/?unapproved=6120&moderation-hash=3f702867a45033ae7a4d45e4c6f77443#comment-6120

https://www.htplaw.com/slider/backup-software-usb-backup-and-other-support-systems?unapproved=33419&moderation-hash=e5c5586c06d68b7761e828250e63a240#comment-33419

http://acturnov.com/2017/10/01/turnovsti-atleti-za-hranicemi/?unapproved=42864&moderation-hash=95478733b918172984cc55aa7f58c90e#comment-42864

http://howcam.com/author/charlesoptof/

Please Note:Your comment is awaiting moderation.Хорошо смотрится реклама, где максимально используется окружающее пространство

http://phillyroboticsexpo.com/prx-2018/?unapproved=7535&moderation-hash=769394bb0588a9a703b1629e24860fe9#comment-7535

http://dawndaispeaks.com/vitamin-d-being-pretty-hurts/?unapproved=42676&moderation-hash=d3d53d8f45c19a318b315d52a382ce07#comment-42676

http://www.kuk-buk.pl/2012/02/coq-au-vin-wedlug-magdy-gessler.html/comment-page-1?unapproved=40250&moderation-hash=b335f593028be66c125794fd476c253f#comment-40250

https://www.pogames.net/overwatch/?unapproved=37783&moderation-hash=ecb43c05db1294ee5c3be5602efbcadd#comment-37783

https://malepotency.ru/zaryadka-dlya-potentsii.html?unapproved=8506&moderation-hash=c04647c1ae34a58960eeef88106b229d#comment-8506

https://remont-ekspert.ru/194-kleim-oboi-svoimi-rukami-kak-pravilno-kleit-oboi.html?unapproved=33592&moderation-hash=5b4f128054a259577625f9c69e7b3b13#comment-33592

http://polunalku.fi/yleinen/miksi/?unapproved=348374&moderation-hash=e7aa3b203f24c26c370fe749100ef987#comment-348374

http://qpsychology.pl/few-words-about-job-crafting/?unapproved=52044&moderation-hash=3f8958577fcafcf08085d63057f0995d#comment-52044

https://www.datingsiterating.co.uk/datingdirect-review/?unapproved=8095&moderation-hash=59ccb350c3ddb3365a49a71f9da55136#comment-8095

https://www.olivomedspa.com/blog/uncategorized/another-sample-post/?unapproved=7606&moderation-hash=42c79afde718ac23e5bacd9aca62ba97#comment-7606

http://education.bankerstrust.com/articles/home-buying-down-payment-requirements/?unapproved=1206&moderation-hash=0a536d2b98ce9b3829f47d4730643312#comment-1206

http://quake.olympus.ru/forum/member.php?u=957271

http://rumorfix.com/2013/09/watch-miley-cyrus-debuts-new-video-for-23-with-mike-will-made-it-wiz-khalifa-juicy-j/?unapproved=782547&moderation-hash=13042cdf2111653ec772a939d12cf802#comment-782547

https://pissporno.net/outdoor-pisse-porno/geilem-weib-platzt-die-blase/?unapproved=34462&moderation-hash=caaaee30cbf3bda86fb10cb1311e6dc4#comment-34462

http://manalifeguide.com/2018/07/11/hello-world/?unapproved=282022&moderation-hash=b0e8071b83f33d02cc3656e8e4ced539#comment-282022

Please Note:Your comment is awaiting moderation.Хорошо смотрится реклама, где максимально используется окружающее пространство

http://fotografia-moreno.com/sesion-fotografica-en-exterior-snaut/?unapproved=40832&moderation-hash=5cbe129f97b32f5c9b74fa7d2b6e7dcd#comment-40832

http://forza.in.ua/member.php?49744-CharlesRit

https://www.shadeddog.com/discography/lsc-2300-2399/lsc-2354/comment-page-1/?unapproved=177746&moderation-hash=de3fd1e2a7cbb0b6c73642cfdabd24a6#comment-177746

http://kar-srtschool.edu.tomsk.ru/nedelya-estestvenno-nauchnogo-tsikla-3/?unapproved=11193&moderation-hash=e3a408b1589612e1b661b1f1afef8e0f#comment-11193

http://morusshop.com/decor-your-floor-with-indian-designer-carpets/?unapproved=38165&moderation-hash=920634aeebc17dde60f9e2ff929dd505#comment-38165

http://coloncleansenow.com/fiber-and-enema-colon-cleanse/?unapproved=24444&moderation-hash=ce44458a0bf2ea7582ce57fab8a6e016#comment-24444

https://www.nesquiikz.com/hacking/phishing/comment-page-9/?unapproved=18046&moderation-hash=7d6df3c47d32d22155f3bee9dc6138f5#comment-18046

http://49.231.238.163/board/viewtopic.php?f=3&t=191625

http://h-ap.ru/2019/12/05/%d0%b3%d0%b4%d0%b5-%d0%b7%d0%b0%d0%ba%d0%b0%d0%b7%d0%b0%d1%82%d1%8c-%d0%b1%d1%8b%d1%82%d0%be%d0%b2%d0%ba%d0%b8-%d0%b8-%d0%b4%d0%b5%d1%82%d1%81%d0%ba%d0%b8%d0%b5-%d0%b4%d0%be%d0%bc%d0%b8%d0%ba%d0%b8/?unapproved=3748&moderation-hash=dc0ad5a9535185774589105efc96bc3e#comment-3748

http://www.serendio.com/pre-modeling-routines-in-r/?unapproved=31885&moderation-hash=cd2bec55c800b16393a916eb0b230010#comment-31885

http://www.acerisk.com/?p=1172&unapproved=231&moderation-hash=ce60db99386f29c9df788fc4ad891daf#comment-231

https://dgsforum.com/showthread.php?tid=39291&pid=85019#pid85019

http://www.gebbiewilson.co.uk/survey-to-map-scotlands-tenant-farming-sector/?unapproved=40995&moderation-hash=2c0b43fe031a5fbd7d5bc39d1817bfc6#comment-40995

http://www.kuk-buk.pl/2012/02/coq-au-vin-wedlug-magdy-gessler.html/comment-page-1?unapproved=40250&moderation-hash=b335f593028be66c125794fd476c253f#comment-40250

http://vb.ghalaa.com/showthread.php?p=5075592#post5075592