Nature’s recent study comparing the accuracy of Wikipedia vs. Encyclopaedia Britannica in the sciences- the first such comparison carried out by a premier science journal- has generated significant interest in quite a few circles. Wikipedia is many things to many people, and there has been precious little data regarding its accuracy. In light of the recent controversy surrounding subtle vandalism in John Seigenthaler’s Wikipedia article, Nature’s conclusion doubtlessly surprised many:
However, an expert-led investigation carried out by Nature – the first to use peer review to compare Wikipedia and Britannica’s coverage of science – suggests that such high-profile examples are the exception rather than the rule. The exercise revealed numerous errors in both encyclopedias, but among 42 entries tested, the difference in accuracy was not particularly great: the average science entry in Wikipedia contained around four inaccuracies; Britannica’s, about three.
Aside from the notable results, which many took to vindicate Wikipedia’s editorial model, the public has been witness to an unusually fierce struggle between established academic institutions. 
Britannica, specifically, has taken out large, fiercely-worded ads in the New York Times and the London Times publicly criticizing Nature’s comparison methodology. The controversy centers around study transparency, article selection and how article length was normalized, major vs minor error counts, and the conclusion drawn from the 4:3 error ratio. Britannica’s take?
“The study was so poorly carried out and its findings so error-laden that it was completely without merit.”
There’s high-quality commentary on the current controversy at Arstechnica, on what this may mean for Britannica at Content Matters, and on some of the high-profile events involving Wikipedia that set the stage for this showdown at Plastic. As of today, a google search for Wikipedia Britannica Nature study nets over 300,000 results.
Nature’s summary of the study includes links to Britannica’s response and Nature’s counter-response. Most of the public correspondence (and media coverage on the disagreement) between Britannica and Nature has focused on disputing and supporting factual claims about the study. Such details are critically important to work out, but this emphasis has also obscured some of the broader issues surrounding the study. In that spirit, here are the questions I think should be explored more fully in parallel with the current methodological evaluation:
Why do the study? Assessing Wikipedia’s accuracy is important, but as I’m sure Nature realizes, this isn’t just about Wikipedia. This is about Wikipedia’s community-driven content and editorial model, and has implications for what roles such a model is qualified to play in the future of science.
What this study isn’t about: Simply comparing the number of mistakes per article in a specific article subset doesn’t touch on many of the central strengths and weaknesses of Wikipedia and Britannica. Importantly (though not exhaustively), Britannica’s editorial consistency and tradition of recruiting Nobel Prize winners to write articles are necessarily passed over, as are Wikipedia’s impressive breadth, depth, and agility, as are the potential institutional and organizational biases both encyclopedias bring to the table. This study is intriguing but necessarily limited.
Why have Nature do the study? This sort of comparative institutional analysis is new ground for Nature. There’s a lot of value to having this study done- but was Nature the right organization to do it? Nature has, in addition to its professional reputation, one of the best scientific peer-review networks in the world, which was easily re-purposed for this comparative analysis. But much of Nature’s great professional reputation arises from filling a certain ‘gatekeeper’ role in the academic community, and though some of that expertise will carry over into designing a methodology to reliably compare vastly different popular reference works, it’s unclear just how much will.
With Britannica responding in force (and from a business perspective, they have little choice but to do so), the stakes are high. The world, not to mention Nature, would be worse off if all that came of this situation was a flawed study and a tarnished reputation for a premier science journal. Then again, should Nature come out of this ambitious, unprecedented undertaking relatively unscathed, the journal may be able to fill a larger role (and one more secure from PLoS-style competition) in the future ecology of academic literature.
