I’m blue about this being published

2015 September 2
by Daniel Lakeland

rgybstudy

The above graph summarizes the situation that Andrew Gelman has been railing on about how chasing the noise with p values leads to a great career and top publications!

Published recently in Psychological Science, this study has Open Science Data. So I pulled the Study1 data and quickly plotted it in R. On the x axis is a measure of how manipulable the participant is times the manipulation that was done. More sad is to the right. On the y axis the relative blue-yellow accuracy as a fraction of red-green accuracy.

You may notice the two outstanding data points on the left at the top? Yeah, I think that’s their effect.  Basically, two people who were easy to manipulate and happened to get the amusement manipulation were outstanding blue-yellow discriminators. All the rest… well, look for yourself.

5 Responses
  1. September 2, 2015

    He! That is sort of what I imagine when I see a p-value, but no plot of the raw data. It is really cool of them to publish their data. An interesting thing here is, what could you do with your “find”? That article is going to be around for years, and surely be cited plenty. How could you ever get this plot to the attention of the many readers of the article? It’s not like you can comment on the article on Psychological Science’s site…

  2. September 6, 2015

    You can post this on PubMed Commons, the database of comments on all biomedical etc. literature. And/or you can post it on PubPeer, which also has a large audience.
    Several of us on twitter have related criticisms https://twitter.com/ptgoodbourn/status/639356659089838082 , some of them might be working on a formal letter.

  3. Nick Brown permalink
    September 8, 2015

    Another frustrating here is that the dataset contains only two significant digits for each of the colour perception tests (which, as I understand it, should all have lots of recurring decimals, as they are numbers out of 48 trials expressed as percentages). More importantly, only a small number of values are present for each colour axis. There are 26 different values for RG and only 15 for BY. Indeed, the value of 0.5 appears 53 times (out of 130 cases) for BY. This makes me wonder about how the measurements were conducted.

    I took the dataset for Study 2 and excluded just one participant (#2), with a high BY score and no recorded value for the sadness manipulation check. The t test went from t(128)=2.05, p=.043 to t(127)=1.836, p=.069. Excluding all (four) participants with no value for the manipulation check gave t(124)=1.818, p=.071.

Comments are closed.