I just filled out the CHI reviewer pre-review questionnaire. The final question asks about general thoughts on the CHI review process. I have documented my answer below. None of the potential improvements I mention are really novel – they have been implemented at other conferences or journals before.

Edit (01.10.2015, 09:15 UTC+2): I would not expect such changes to be made for CHI 2016 or 2017. Instead, some of these changes could be tried out at smaller conferences first in order to work out a usable implementation.

(For the record: this is not about open access. While I am a fan of open source/science/access/…, I think that the ACM’s approach (much freedom for authors, affordable access to the Digital Library and individual papers, offering an OA option) is very reasonable.)

In general, I would like to see four changes to the current review process:

Post-Publication Peer Review

I find the currently practiced pre-publication peer review quite problematic. I have reviewed or otherwise seen plenty of papers that contained interesting findings but were rejected (sometimes rightfully) due to some flaw or another.
A huge share of these rejected papers was never published in another venue, the insight contained in them (and errors from which others could learn) lost to the community. Many other papers were only published one or more years later – thereby delaying all research that could build on it or try to replicate its findings.

I would very much like CHI to adopt a post-publication peer review or a similar approach that improves speed and visibility. For example, a process similar to alt.chi (in some ways) would be great, where all submitted papers are made public or semi-public and peer review then selects those that are to be presented at the conference. Ideally, reviews would also be published for all papers. This would also make it harder for authors to re-submit their paper unaltered to another conference without addressing the issues mentioned by the reviewers. I hate when I give extensive feedback to a paper and the authors do not even bother to fix the spelling errors that I pointed out when they submit it to another conference.

Open Peer Review

Quite often I read a paper which incorrectly quotes a paper of mine or which omits important related work. With the current review process, the quality of the reviews depends on the ACs ability and desire to find the right reviewers.  Allowing any researcher to provide a review for a submitted paper would make sure that domain experts can chime in and point out flaws or interesting use cases that the other reviewers missed. This has already been tried at alt.chi, too.

Modular Peer Review

The standard PCS review form asks the reviewer how they would rate their expertise on a four-point Likert scale. This is a rather simplistic measure. While I feel very competent to judge novelty or validity of a sensing technique, I can not honestly tell whether the correct statistical tests were used in the evaluation of this technique. Similarly, I won’t be able to tell whether all relevant related work has been cited in a paper on design techniques but can certainly offer my opinion on further applications or spelling errors.

I would prefer to describe my expertise more accurately and also focus only on certain aspects of a paper as a reviewer. By telling the AC that I do not know enough about statistics, I give them the opportunity to find another reviewer who does. Furthermore, the AC could also assign a subset of reviewers to individual aspects of a paper (e.g., related work, writing style, experimental setup, statistical tests, technical correctness, replicability). This could avoid duplicated work and simultaneously increase the quality of the reviews.

Author-Reviewer Collaboration

Reviewers point out weaknesses and suggest improvements in a paper. Given the significant amount of time some reviewers invest in reviewing a paper, and given that their suggestions may significantly improve a paper, it might be a good idea to offer a way of attributing their contributions. For example, reviewers could be mentioned by name in the Acknowledgements section (if they want) or they could become co-authors of the final submission. This would certainly change the character of the review process and introduce some problems. While the aforementioned changes could already be implemented for next year’s CHI conference, this final suggestion certainly needs more discussion and refinement to get it right. One option could be for reviewers to ‘fork’ a submission and submit ‘pull requests’ with proposed changes similar to software development processes on GitHub and similar platforms. (In general, a version-controlled approach to paper writing, including authorship attribution for each sentence sounds quite interesting.)

Given that the program chairs of recent CHI conferences have put much effort into evolving the conference series, I am optimistic that some of these proposed changes will be implemented in the near future. And frankly, it would be very fitting for the CHI community to be at the forefront of academic collaboration and publishing.