Peer reviews: Wikipedia references

I like Wikipedia, despite the flak it’s taken from those who say the information there is unreliable. It’s a form of controlled chaos that works surprisingly well most of the time. There’s vandalism, bias, unresearched and incorrect information, but there is a lot of oversight, both “official” and ad hoc, and Wikipedia is, in general, a valuable tool. Using Wikipedia to learn about things and as a springboard for further research, while understanding its limitations, is certainly not a problem.

Researchers submitting papers for peer review also like Wikipedia. As a reviewer, I’m seeing more and more papers that cite Wikipedia in their references. And that is a problem, precisely because of the “chaos” aspects.

For one thing, Wikipedia entries are constantly changing. When you cite an entry today, the bit of it that supports what your paper is saying might change tomorrow. The key paragraph might say something very different when I review your paper, compared with what it said when you cited it; indeed, the information you’re citing might not be there at all any more.

For another, even if the information you’re citing is still there and remains intact, as you read it... it might simply be wrong, or insufficiently supported to hold up to peer review.

Wikipedia itself encourages external references within its entries, and discourages unsubstantiated claims. Clearly, though, any extensive text can’t cite references for every fact — the references would outweigh the content, in the end — so a great deal goes into Wikipedia entries as “fact” with no cited support. That information might or might not be correct; it takes further research to determine whether it is.

And that is the job of the researcher: to do the research. As it stands, Wikipedia is so easy to use and so rife with mostly-correct information that citing it is a convenient shortcut. But it’s a too-convenient shortcut that this reviewer, at least, will not accept. You’ll notice that I rarely refer to Wikipedia even here, in these loose, informal, non-refereed pages, and that when I do, it’s for a very limited purpose, to provide a pointer to an informative explanation, and nothing more.

So I expect in research papers, and I will set that out here for anyone who thinks that I, or some like-minded reviewer, might one day be asked to review her work:

Do not use Wikipedia as a reference for any substantive or normative information. I will ding you for it in the review, and I will insist that such references be replaced by proper ones before recommending publication. I will accept references to Wikipedia for purely explanatory things that don’t matter to the substance of your paper — but even there, please don’t overdo it.

Here’s a contrived example of an unacceptable Wikipedia reference for a hypothetical paper that says something about inflation:

During periods of moderately high inflation, it can actually be easier to establish a startup business because investors are more willing to spend their money than to save it.[1]

---

[1] http://en.wikipedia.org/wiki/Inflation

The reference is to a section of the Wikipedia article, at the time that I write this, that says this:

Inflation is also viewed as a hidden risk pressure that provides an incentive for those with savings to invest them, rather than have the purchasing power of those savings erode through inflation. In investing, inflation risks often cause investors to take on more systematic risk, in order to gain returns that will stay ahead of expected inflation.

...but if the statement of that sentence in the paper is important, a stronger reference is needed.

Here’s a version of the example that I would consider acceptable:

During periods of moderately high inflation[1], it can actually be easier to establish a startup business because investors are more willing to spend their money than to save it.[2]

---

[1] http://en.wikipedia.org/wiki/Inflation

[2] ...a reference to a peer-reviewed paper on venture capital...

In this case, Wikipedia is used to give general information about what inflation is, in case the reader needs some background. I accept that Wikipedia is accurate enough for that. Meanwhile, the substantive reference points to a source that we consider to be more reliable.

Of course, this is all not to say that everything in peer-reviewed papers, news media, published books, and the like is reliable and correct; it clearly isn’t. Even “proper” citations can have bad information, and multiple independent sources are a good thing when that’s possible, and the reference is important enough.