The Dutch toilet cleaner ‘WC-EEND’ (literally: 'Toilet Duck')
aired a famous commercial
in 1989 that had the slogan ‘We from WC-EEND advise… WC-EEND’. It is now a
common saying in The Netherlands whenever someone gives an opinion that is
clearly aligned with their self-interest. In this blog, I will examine the
hypothesis that blogs are, on average, of higher quality than journal articles. Below, I present 5
arguments in favor of this hypothesis. [EDIT: I'm an experimental psychologist. Mileage of what you'll read below may vary in other disciplines].
1. Blogs have Open Data, Code, and Materials
When you want to evaluate scientific
claims, you need access to the raw data, the code, and the materials. Most journals
do not (yet) require authors to make their data publicly available (whenever
possible). The worst case example when it comes to data sharing is the American
Psychological Association. In the ‘Ethical
Principles of Psychologists and Code of Conduct’ of this professional
organization that supported
torture, point 8.14 says that psychologists only have to share data when
asked to by ‘competent professionals’ for the goal to ‘verify claims’, and that these
researchers can charge money to compensate any costs that are made when they
have to respond to a request for data. Despite empirical proof that most scientists do not
share their data
when asked, the APA considers this ‘ethical conduct’. It is not. It’s an
insult to science. But it’s the standard that many relatively low quality scientific
journals, such as the Journal of Experimental Psychology: General, hide behind
to practice closed science.
On blogs, the norm is to provide access to
the underlying data, code, and materials. For example, here is Hanne Watkins,
who uses data she collected to answer some questions about the
attitudes of early career researchers and researchers with tenure towards
replications. She links to the data and materials, which are all available
on the OSF. Most blogs on statistics will
link to the underlying code, such as this blog by Will Gervais on whether you
should run well-powered studies or many small-powered studies. On average,
it seems to me almost all blogs practice open science to a much higher extent
than scientific journals.
2. Blogs have Open Peer Review
Scientific journal articles use peer review
as quality control. The quality of the peer review process is as high as the
quality of the peers that were involved in the review process. The peer review process
was as biased as the biases of the peers that were involved in the review
process. For most scientific journal articles, I can not see who reviewed a
paper, or check the quality, or the presence of bias, because the reviews are
not open. Some of the highest quality journals in science, such as PeerJ and Royal Society Open Science,
have Open Peer Review, and journals like Frontiers
at least specify the names of the reviewers of a publication. Most low quality
journals (e.g., Science, Nature) have 100% closed peer review, and we don’t
even know the name the handling editor of a publication. It is often impossible
to know whether articles were peer reviewed to begin with, and what the quality
of the peer review process was.
Some blogs have Open pre-publication Peer Review. If you read the latest DataColada blog post, you can see the two
reviews of the post by experts in the field (Tom
Stanley and Joe
Hilgard) and several other people who shared thoughts before the post went
online. On my blog, I sometimes ask people for feedback before I put a blog
post online (and these people are thanked in the blog if they provided feedback),
but I also have a comment section. This allows people to point out errors and
add comments, and you can see how much support or criticism a blog has
received. For example, in this blog on why omega
squared is a better effect size to use than eta-squared, you can see why
Casper Albers disagreed by following a link to a blog
post he wrote in response. Overall, the peer review process in blog posts
is much more transparent. If you see
no comments on a blog post, you have the same information about the quality of
the peer review process as you’d have for the average Science article. Sure, you
may have subjective priors about the quality of the review process at Science
(ranging from ‘you get in if your friend is an editor’ to ‘it’s very rigorous’)
but you don’t have any data. But if a
blog has comments, at least you can see what peers thought about a blog post,
giving you some data, and often very important insights and alternative
viewpoints.
3. Blogs have no Eminence Filter
Everyone can say anything they want on a
blog, as long as it does not violate laws regarding freedom of speech. It is an
egalitarian and democratic medium. This aligns with the norms in science. As
Merton (1942) writes: “The acceptance or
rejection of claims entering the lists of science is not to depend on the
personal or social attributes of their protagonist; his race, nationality,
religion, class, and personal qualities are as such irrelevant.” We see
even Merton was a child of his times – he of course meant that his *or her* race,
etcetera, is irrelevant.
Everyone can write a blog, but not everyone
is allowed to publish in a scientific journal. As one example, criticism
recently arose about a special section in Perspectives on Psychological Science
about ‘eminence’ in which the only contribution from a woman was
about gender and eminence. It was then pointed out that this special
section only included the perspectives on eminence by old American men, and
that there might be an issue with diversity in viewpoints in this outlet.
I was personally not very impressed by the published
articles in this special section, probably because the views on how to do
science as expressed by this generation of old American men does not align with
my views on science. I have nothing against old (or dead) American men in
general (Meehl be praised), but I was glad to hear some of the most important
voices in my scientific life submitted responses to this special issue. Regrettably,
all these responses were rejected. Editors can make those choices, but I am
worried about the presence of an Eminence Filter in science, especially one
that in this specific case filters out some of the voices that have been most important in shaping me as a scientist.
Blogs allows these voices to be heard, which I think is closer to the desired
scientific norms discussed by Merton.
4. Blogs have Better Error Correction
In a 2014 article, we published a
Table 1 of sample sizes required to design informative studies for different statistical
approaches. We stated these are sample sizes per condition, but for 2
columns, these are actually the total sample sizes you need. We corrected this
in an erratum. I know this erratum was published, and I would love to link to
it, but honest to Meehl, I can not find it. I just spend 15 minutes searching
for it in any way I can think of, but there is no link to it on the journal
website, and I can’t find it in Google scholar. I don’t see how anyone will
become aware of this error when they download our article.
When I make an error in a blog post, I can
go in and update it. I am pretty confident that I make approximately as many
errors in my published articles as I make in my blog posts, but the latter are
much easier to fix, and thus, I would consider my blogs more error-free, and of
higher quality. There are some reasons why you can not just update scientific
articles (we need a stable scientific record), and there might be arguments for
better and more transparent version control of blog posts, but for the
consumer, it’s just very convenient that mistakes can easily be fixed in blogs,
and that you will always read the best version.
5. Blogs are Open Access (and might be read more).
It’s obvious that blogs are open access.
This is a desirable property of high quality science. It makes the content more
widely available, and I would not be surprised (but I have no data) that blog
posts are *on average* read more than scientific articles because they are more
accessible. Getting page views is not, per se, an indication of scientific quality.
A video on Pen Pineapple Apple
Pen gets close to 8 million views, and we don’t consider that high quality music
(I hope). But views are one way to measure how much impact blogs have on what
scientists think.
I only have data for page views from my own
blog. I’ve made a .csv
file with the page views of all my blog posts publicly available (so you
can check my claims below about page views of specific blog posts below, cf.
point 1 above). There is very little research on the impact of blogs on
science. They are not cited a lot (even
though you can formally cite them) but they can have clear impact, and it
would be interesting to study how big their impact is. I think it would be a
fun project to compare the impact of blogs with the impact of scientific
articles more formally. Should be a fun thesis project for someone studying
scientometrics.
Some blog posts that I wrote get more views
than the articles I comment on. One commentary blog post I wrote on a paper
which suggested there was ‘A surge of
p-values between 0.041 and 0.049 in recent decades’. The paper received 7147
view at the time of writing. My blog post received 11285 views so far. But it
is not universally true that my blogs get more pageviews than the articles I
comment on. A commentary
I wrote on a horribly flawed paper by Gilbert and colleagues
in Science, where they misunderstood how confidence intervals work, has
only received 12190 hits so far, but the article info
of their Science article tells me their article received three times as many
views for the abstract, 36334, and also more views for the full text (19124). On
the other hand, I do have blog posts that have gotten more views than this
specific Science article (e.g., this post on Welch’s
t-test which has 38127 hits so far). I guess the main point of these
anecdotes is not surprising, but nevertheless worthwhile to point out: Blog are
read, sometimes a lot.
Conclusion
I’ve tried to measure blogs and journal
articles on some dimensions that, I think, determine their scientific quality. It
is my opinion that blogs, on average, score better on some core scientific
values, such as open data and code, transparency of the peer review process, egalitarianism,
error correction, and open access. It is clear blogs impact the way we think
and how science works. For example, Sanjay Srivastava’s pottery
barn rule, proposed in a 2012 blog, will be implemented in the journal Royal
Society Open Science. This shows blogs can be an important source of
scientific communication. If the field agrees with me, we might want to more
seriously consider the curation of blogs, to make sure they won’t disappear in
the future, and maybe even facilitate assigning DOI’s to blogs, and the
citation of blog posts.
Before this turns into a ‘we who write
blogs recommend blogs’ post, I want to make clear that there is no intrinsic
reason why blogs should have higher scientific quality than journal articles. It’s
just that the authors of most blogs I read put some core scientific values into
practice to a greater extent than editorial boards at journals. I am not
recommending we stop publishing in journals, but I want to challenge the idea
that journal publications are the gold standard of scientific output. They fall
short on some important dimensions of scientific quality, where they are
outperformed by blog posts. Pointing this out might inspire some journals to improve
their current standards.