Single Versus Double Blind Reviewing at Wsdm 2017

Peer review is disquisitional to maintaining enquiry quality at conferences. But what are the implications of using single blind vs double blind review?

Peer review is the process of subjecting an author's research to the scrutiny of experts. Using a peer review process at your conference allows yous to easily evaluate every paper submitted to your event, and take only the most credible for publication. It also gives researchers the opportunity to improve the quality of their papers before publication, equally reviewers will frequently requite recommendations for improvement along with their assessment.

While peer review remains the dominant method for research validation , there are many ways yous tin facilitate your review process. The two main conditions of peer review are single-blind and double-blind. Forests have been felled in the pursuit of documenting the pros and cons of single  blind and double blind peer review. Which method meliorate upholds the quality of published enquiry remains an active area of debate, with many high-ranking conferences and journals falling on opposing sides of the statement.

In this commodity, I'll compare both reviewing types and outline their advantages and disadvantages, so you tin make up your mind for yourself. I'll as well look at what happened when a informatics briefing tested single vs double-blind peer review for reviewer bias.

Fix? Let'southward leap in.

Unmarried blind review

The "blind" in single blind review refers to what information authors tin see. In a single blind peer review, reviewers' identities are kept hidden from authors. This is the traditional form of peer review, and it's still the type that's most common.

Advantages

The advantage of single bullheaded reviewing is it allows your reviewers to critique papers without any influence being exerted by the authors. If a reviewer knows their proper name won't be visible to the author (or to the public), information technology gives them the freedom to give an honest critique – even when it's not favourable.

Disadvantages

Even though information technology's the most widely used form of peer review , single bullheaded isn't always looked upon in a positive light in the academic community. For case, reviewers could apply the fact that they are bearding equally an opportunity to be unusually critical of authors.

For instance, a survey by the Publishing Enquiry Consortium found that whilst 85% of respondents had experienced unmarried-blind review, only 52% described it as effective (and it was the preferred option for simply 25%).

Double blind review

In a double blind peer review, the identities of both the authors and reviewers are kept hidden. If you're collecting submissions for double blind review, you lot'll demand to ensure they've been prepared in a way that hides authors' identities fully i.east. no repeated citations of their own or colleagues piece of work throughout the submission. So it'south a good idea to brand sure that the peer review software you're using for your conference can facilitate both single and double blind review.

Advantages

While double bullheaded has all the benefits of unmarried blind, it'due south got some added ones to boot. Because both authors and reviewers are "blind", this guards against reviewers being influenced by an author'south prestige . This ways that even submissions from the stars in your field are considered on their individual merit rather than on the author's reputation.

Another benefit of double bullheaded is information technology reduces the possibility of reviewer bias . Bias tin can accept many forms, for example an writer's gender , country of origin, bookish status or previous publication history.

Disadvantages

In practice, however, double blind oftentimes doesn't actually mean double blind. It'southward likely that your research field is relatively small, so it volition often exist apparent to reviewers who the author is based on citation , subject field , or writing style.

The survey to a higher place was more positive in its assessment of double blind reviews (although only 45% of respondents had direct experienced information technology). Despite this, 71% of respondents described information technology every bit effective, and it was the preferred option for 56%.

Yet unsure which method is best for your event'southward peer review process ? read on.

What happened when a informatics conference tried both

The Web Search and Data Mining briefing (WSDM) conference tested unmarried vs double-blind review for reviewer bias. Here'due south what they found.

WSDM had historically used single blind weather condition in its rigorous review process which results in an credence rate of just 15.half-dozen%.

Two of the 2017 briefing's co-chairs, Andrew Tomkins of Google and Min Zhang from Tsinghua Academy, were asked to consider switching to double bullheaded review. Curious about whether review conditions can reduce reviewer bias, they discovered that no experiment had been washed on single vs double blind in computer science.

So, with the help of William D. Heavlin, also from Google, they decided to comport 1 out .

Setting upwardly the experiment

The researchers wanted to examine whether double blind review had an touch on implicit reviewer bias with respect to an author's gender, country, prestige and affiliation. And then they divide their conference reviewers into ii groups:

Group 1. Those who could access author information

Group ii. Those who had couldn't

Both groups then bid on papers to signal their interest in reviewing them. In one case behest was consummate, the organising commission assigned each submission two reviewers from each accomplice.

When the organisers analysed the review data, here's what they discovered…

Reviewer bias in unmarried vs double bullheaded conditions

There were three significant differences in the behaviour of the conference's single vs double-bullheaded review groups.

one. The reviewers from the single-blind cohort bid for papers less prolifically , bidding on 22% fewer, on boilerplate, than those in the double-blind cohort.

two. The bids entered by single-blind reviewers were weighted towards submissions from acme universities and companies , compared with the bids of double-bullheaded reviewers.

3. The single-blind reviewers were relatively more likely to submit a positive review for submissions from prestigious authors or loftier-quality organisations than were their double-blind counterparts.

Conferences should "seriously consider" advantages of double-blind review

The experiment institute that under unmarried blind review atmospheric condition, reviewers make employ of information about authors and their affiliations. And this disadvantages some authors.

Submissions from researchers at prominent organisations may also benefit from single blind conditions. This is because an increased number of bids makes them more likely to be allocated to the most appropriate reviewers.

The findings also suggest that, nether unmarried blind atmospheric condition, a submission written by an eminent author from a prestigious organisation might get a more positive review than the aforementioned paper written by an early-career researcher from an unknown arrangement.

"Conspicuously, our understanding of the implications of reviewing methodologies remains nascent," the authors said. "Nonetheless, we feel that program and general chairs of conferences should seriously consider the advantages of using double-bullheaded reviewing."

Read the total paper published in the Proceedings of the National University of Sciences of the United states of America.

Conclusion

Peer review is like commonwealth, the proverb goes, despite its flaws, it'due south the best system nosotros accept.

Whether you're using unmarried or double blind, brand certain your review software can allow you to add together additional measures like preventing nepotism based on email domain and organisation. This can help forestall a conflict of interest betwixt the reviewer and author, and when combined with double blind review, tin can make your conference'due south review very strong.

To see how this works in practice, bank check out our online conference platform.

westconsecter.blogspot.com

Source: https://www.exordo.com/blog/the-peer-review-process-single-versus-double-blind/

Share this post