Sunday, December 12, 2010

NASA discovery slips through the cracks


Felisa Wolfe-Simon explains her discovery at a news conference at NASA headquarters. Photo: AFP

The recent discovery of bacteria that can live on arsenic, and even incorporate the toxic chemical into the structure of its DNA, last week sent the global media into a frenzy (see here for an example). According to NASA, the research was a breakthrough for astrobiology, the search for extra-terrestrial life, because it suggests that there are alternative chemical pathways through which life can exist, broadening the possibilities for detecting life elsewhere in the universe. The research was published in the very prestigious journal Science.

Since the news broke a little more a week ago, many scientists have voiced their concerns about the validity of the research (see here and here for extensive critiques of the research by expert biochemists). The story quickly turned from triumphant discovery to an embarrassing case of research that slipped through the cracks of the peer review process.

Peer review is the system of quality control in science that ensures research is robust. Despite what some people may think, scientists cannot publish whatever they want. Potential articles are read and critiqued by other experts in the field, who advise the editor of a journal about whether the article is scientifically sound and whether it is of sufficient quality for publication.

While peer review is an effective form of quality control, it is system that relies heavily on people’s honesty and good will. Reviewing articles is an unpaid, time-consuming and rather thankless job that comes with the territory of being an academic. The integrity of published research depends on reviewers taking the time to properly check and question scientific claims. I am no expert in biochemistry, so I will not speculate about why or how the arsenic research was allowed to be published, but it is a pertinent example of how the peer review system springs the occasional leak.


Bacteria from Mono Lake in California can survive and grow in high concentrations of arsenic, but do they really incorporate arsenic into their DNA backbone? Photo: Felisa Wolfe-Simon

The reverse situation can also occur. Last year, a group of 14 stem cell researchers wrote an open letter about reviewers they thought were blocking quality research in an attempt to promote their own instead. While the vast majority of scientists would never deliberately reject someone else’s research for their own benefit, the cutthroat nature of academia may be driving some researchers to engage in unethical reviewing practices.

Instances such as these have prompted some to suggest that the system of peer review be changed. I recently attended a seminar by Cameron Neylon, an academic and a prolific blogger who dedicates much of his time to questioning the current ways that we publish our results in science. Among other things, Neylon champions the idea of open publishing and post-publication review. His view is that the current system of peer review is archaic and is a relic of an era in which journals were printed in hardcopy, leaving limited space for articles in each edition. Now, almost all journals publish their articles online in electronic form, which means there is no longer the issue of limited space. He suggests that the current system of pre-publication review is too slow, delaying the release of important research discoveries, and also acts as an unnecessary filter that blocks the publication of many research projects funded by tax-payers’ money.

Neylon proposes that articles be peer reviewed after they are published (post-publication review). In essence, this means any research, regardless of its quality and integrity, can be published online. The research is then publically scrutinised by other scientists in online forums that are linked to the online version of the article.

I certainly agree that post-publication review is an invaluable tool that can only enhance the quality of published research. Comments from biochemists in online forums and blogs about the supposed arsenic-based bacteria are a perfect example of how useful post-publication review can be. However, I support the idea of post-publication review as a supplement rather than as a substitute for pre-publication review.

The current peer review system encourages scientists to write the best possible articles prior to submission, and the process of addressing reviewer comments often significantly improves the quality of the finished product. It is my opinion that articles published online without prior scrutiny are likely to be more sloppily-written than those currently reviewed pre-publication. Furthermore, while it is nice to think that science research is a perfectly cost-effective and efficient endeavour, not all research is worthy of publication. Simply publishing everything regardless of its quality means that scientists must spend a lot more time searching online databases to sort good research from the rubbish.

Nevertheless, healthy debate about the value of current system of peer review continues with gusto. Many scientists discuss this issue in online forums and while support for post-publication review is increasing, the form it would take is hotly contested. I strongly encourage all scientists to think about this issue and engage in the debate, since any changes in the peer review process will have major consequences for the way we conduct and publish science in the future.