We are planning to use UCount in ICST Transactions, which are being launched in 2011. The idea is that after each review all authors of the reviewed paper will evaluate the reviewer on several criteria and such information will be used in the future to help editors to select better reviewers. Moreover, certain indicators (such as how fast is reviewer in preparing the review, or reviewer’s bias) can be computed automatically, using the techniques for review analysis described in [Casati2010, Ragone2011].
The purpose of this blogpost is to get the feedback of the research community on what would be the appropriate criteria and to collect information about similar initiatives. Please let us know what you think about the criteria below.
Criteria to be evaluated by the author
- fairness, i.e., authors’ perception that their work got comments it deserved (e.g., I as an author know the work is so-so, and this is correctly seen by reviewer, or I know the work is of top quality, but the reviewer did not recognize it;
- helpfulness, i.e., authors’ perception that the reviewers’ comments help to improve the work;
- politeness, i.e., authors’ perception on the overall tone of the review.
Criteria to be computed automatically
- responsiveness, i.e., how quick the reviewer is in providing comments to the authors;
- bias, i.e., how different the reviewers’ marks are biased towards affiliation, country or gender of the authors;
- agreement, i.e., how often the reviewers’ marks are different from the marks of the other reviewers;
- prediction ability, i.e., how the reviewers’ marks correlate with the later impact (e.g., citation count) of the paper
P.S. A position paper about UCount is online and open for your comments at http://altmetrics.org/workshop2011/parra-v0/.
[Casati2010] F. Casati, M. Marchese, K. Mirylenka, A. Ragone. Reviewing peer review: a quantitative analysis of peer review, 2010.
[Ragone2011] Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, Maurizio Marchese. A Quantitative Analysis of Peer Review. 13th Conference of the International Society for Scientometrics and Informetrics (ISSI’2011) (to appear).
