Dear TC6 members,
Here is my contribution to action point AP 57/4 of the Zurich meeting: "A report on the use of software packages for paper reviewing was asked to A.Iyengar, V.Tschammer, G.Leduc, R.Boutaba and A.Danthine".
TC6 recommends two tools: - Confman http://confman.unik.no/%7Econfman/whats_new.html - CyberChair http://www.cyberchair.org/requestUse.html
Both seem quite adequate. Here are some more detailed comments.
It is a mixture of my personal viewpoint and some views I collected other WG members. It is by no means a fully objective assessment. Therefore, I would be a bit reluctant to put something like this on the IFIP web page for instance. If we intend to do it, we need to find a larger consensus on that.
Best regards, Guy
Confman -------
The ConfMan tool is good. It allows PC members to state their areas of expertise, to fill out the review form, to read other reviewers comments anonymously, to sort the papers according to several criteria, to vote for papers and discuss them.
The nice thing with ConfMan is that you can evaluate the tool by accessing their web pages, even before any actualisation.
Confman provides a convenient support for the PC discussion and ballot. PC members can read the abstract, all reviews and if they want they can print out the paper.
Each PC member can comment each paper and the reviews. These comments become immediately visible to other PC members. There is one important EXCEPTION: PC members cannot follow the discussion of their own papers. They also do not see their places in the paper ranking.
An important feature is the ballot. Usually there are two ballots: a preliminary ballot for reducing the number of papers to discuss and the final ballot. Each PC member can vote to accept, reject the paper or abstain.
There are different lists offered to get an overview of the papers and the state of the ballot. (Of course, access is only by personal authentification). Furthermore it is possible to rank the papers according to the conference chair decisions (overall or weighted marks).
Confman provides a manual for newcomers, but the system is simple to use.
BUT
PC members cannot bid for reviewing papers though ConfMan can be extended (and has been) with some facilities (such as the paper selection table for reviewers).
Other tools (e.g. Cyberchair) have more flexible functionality in this respect: potential reviewers are allowed to express strong, weak preferences, and exclusions. The program then allocates papers with a 'clever' algorithm.
Also, if the tool is badly configured, PC members can read comments on their own submitted papers during the selection process. This has happened several times.
Cyberchair ----------
Cyberchair is comparable to ConfMan in that it allows PC members to state their areas of expertise, to fill out the review form, to read other reviewers comments, and to discuss them.
An interesting point regarding Cyberchair is that it provides a more sophisticated algorithm to show up relations/contradictions between reviews compared to Confman. Colors are used to identified different categories of conflicts between reviewers.
An enhanced version of Cyberchair is also available at the official site at the University of Twente. This version, which mainly comprises a new chat feature. Using this feature the system manages a chat web page for each paper, containing all comments provided for that paper and allowing PC members to provide their comments, either anonymously or openly. This chat feature is also endowed with a notification service: the reviewers of a paper are immediatly notified by an e-mail containing the new comment. It is worth noting that with this feature we experienced a better involvement of the PC in the discussion phase.
BUT
Cyberchair does not preserve the anonymity of the reviewers completely. A PC member can see the names of the reviewers for all papers, during the discussion. Perhaps this can be hidden, but I'm not sure.
There is no voting scheme.
________________________________________________________________________ Prof. Guy Leduc Tel : +32 4 366 26 98 Universite de Liege Secr : +32 4 366 26 91 Reseaux Informatiques Fax : +32 4 366 29 89 Research Unit in Networking (RUN) leduc@montefiore.ulg.ac.be Institut d'Electricite Montefiore, B 28, B-4000 LIEGE 1, BELGIUM http://www-run.montefiore.ulg.ac.be/users/leduc.html