Ars Technica has a nice summary of a recent paper in PLoS that attempted to assess the quality of the peer review process. From the Ars summary:
To examine what makes a good reviewer, they took advantage of the journal Annals of Emergency Medicine, which has maintained a detailed database of reviewers and post-review ratings (on a five-point scale) of their work, performed by the editors of the journal. The researchers contacted the reviewers and surveyed them about various factors that might contribute to skill in the process. A diverse set of 306 reviewers who had performed a total of nearly 3,000 reviews were used as the data set.
[…]
In news that may be disturbing for journal editors everywhere, very few factors leapt out as having a consistent and significant correlation with the quality of a review, although some factors did have strong correlations in individual tests. The only positive factors linked to quality of reviews were age (younger reviewers were better) and working at an academic hospital. Ironically, service on an Institutional Review Board, which evaluates and approves experiments on humans, consistently correlated with lower-quality peer reviews. Even these factors, however, were only slightly better than random at predicting review quality.
Ultimately the peer review process is always going to have a subjective component to it, since the processes of intuition and patterning that are fundamental to scientific insight and understanding are not really very deterministic. But there’s another possible reason why the study failed to find strong correlates of review quality; the very assessment of quality itself is equally subjective. I personally believe that the peer review system is like democracy – far from ideal but better than anything else out there. The best way to ensure general quality is to ensure that a maximum number of scientists in a given field participate in the process. Perhaps one way to achieve this would be to extend reviewer privileges to graduate students who have passed oral qualifiers?
Journal article citation: Callaham ML, Tercier J (2007) The Relationship of Previous Training and Experience of Journal Peer Reviewers to Subsequent Review Quality. PLoS Med 4(1): e40 doi:10.1371/journal.pmed.0040040
Related article: Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health peer review of clinical grant applications. J Investig Med. 2006 Jan;54(1):13-9. PMID: 16409886