In a Hacker News discussion, I just stumbled upon two quotes which – in my view – beautifully capture the essence of the symbiosis of academic teaching and research:

“If you pay a man a salary for doing research, he and you will want to have something to point to at the end of the year to show that the money has not been wasted. In promising work of the highest class, however, results do not come in this regular fashion, in fact years may pass without any tangible result being obtained, and the position of the paid worker would be very embarrassing and he would naturally take to work on a lower, or at any rate a different plane where he could be sure of getting year by year tangible results which would justify his salary. The position is this: You want one kind of research, but, if you pay a man to do it, it will drive him to research of a different kind. The only thing to do is to pay him for doing something else and give him enough leisure to do research for the love of it. “

— J. J. Thompson

“I don’t believe I can really do without teaching. The reason is, I have to have something so that when I don’t have any ideas and I’m not getting anywhere I can say to myself, “At least I’m living; at least I’m doing something; I am making somecontribution” — it’s just psychological.

When I was at Princeton in the 1940s I could see what happened to those great minds at the Institute for Advanced Study, who had been specially selected for their tremendous brains and were now given this opportunity to sit in this lovely house by the woods there, with no classes to teach, with no obligations whatsoever. These poor bastards could now sit and think clearly all by themselves, OK? So they don’t get any ideas for a while: They have every opportunity to do something, and they are not getting any ideas. I believe that in a situation like this a kind of guilt or depression worms inside of you, and you begin to worry about not getting any ideas. And nothing happens. Still no ideas come.

Nothing happens because there’s not enough real activity and challenge: You’re not in contact with the experimental guys. You don’t have to think how to answer questions from the students. Nothing!

In any thinking process there are moments when everything is going good and you’ve got wonderful ideas. Teaching is an interruption, and so it’s the greatest pain in the neck in the world. And then there are the longer period of time when not much is coming to you. You’re not getting any ideas, and if you’re doing nothing at all, it drives you nuts! You can’t even say “I’m teaching my class.”

If you’re teaching a class, you can think about the elementary things that you know very well. These things are kind of fun and delightful. It doesn’t do any harm to think them over again. Is there a better way to present them? The elementary things are easy to think about; if you can’t think of a new thought, no harm done; what you thought about it before is good enough for the class. If you do think of something new, you’re rather pleased that you have a new way of looking at it.

The questions of the students are often the source of new research. They often ask profound questions that I’ve thought about at times and then given up on, so to speak, for a while. It wouldn’t do me any harm to think about them again and see if I can go any further now. The students may not be able to see the thing I want to answer, or the subtleties I want to think about, but they remind me of a problem by asking questions in the neighborhood of that problem. It’s not so easy to remindyourself of these things.

So I find that teaching and the students keep life going, and I would never accept any position in which somebody has invented a happy situation for me where I don’t have to teach. Never.”

— R.Feynman

I just filled out the CHI reviewer pre-review questionnaire. The final question asks about general thoughts on the CHI review process. I have documented my answer below. None of the potential improvements I mention are really novel – they have been implemented at other conferences or journals before.

Edit (01.10.2015, 09:15 UTC+2): I would not expect such changes to be made for CHI 2016 or 2017. Instead, some of these changes could be tried out at smaller conferences first in order to work out a usable implementation.

(For the record: this is not about open access. While I am a fan of open source/science/access/…, I think that the ACM’s approach (much freedom for authors, affordable access to the Digital Library and individual papers, offering an OA option) is very reasonable.)

In general, I would like to see four changes to the current review process:

Post-Publication Peer Review

I find the currently practiced pre-publication peer review quite problematic. I have reviewed or otherwise seen plenty of papers that contained interesting findings but were rejected (sometimes rightfully) due to some flaw or another.
A huge share of these rejected papers was never published in another venue, the insight contained in them (and errors from which others could learn) lost to the community. Many other papers were only published one or more years later – thereby delaying all research that could build on it or try to replicate its findings.

I would very much like CHI to adopt a post-publication peer review or a similar approach that improves speed and visibility. For example, a process similar to alt.chi (in some ways) would be great, where all submitted papers are made public or semi-public and peer review then selects those that are to be presented at the conference. Ideally, reviews would also be published for all papers. This would also make it harder for authors to re-submit their paper unaltered to another conference without addressing the issues mentioned by the reviewers. I hate when I give extensive feedback to a paper and the authors do not even bother to fix the spelling errors that I pointed out when they submit it to another conference.

Open Peer Review

Quite often I read a paper which incorrectly quotes a paper of mine or which omits important related work. With the current review process, the quality of the reviews depends on the ACs ability and desire to find the right reviewers.  Allowing any researcher to provide a review for a submitted paper would make sure that domain experts can chime in and point out flaws or interesting use cases that the other reviewers missed. This has already been tried at alt.chi, too.

Modular Peer Review

The standard PCS review form asks the reviewer how they would rate their expertise on a four-point Likert scale. This is a rather simplistic measure. While I feel very competent to judge novelty or validity of a sensing technique, I can not honestly tell whether the correct statistical tests were used in the evaluation of this technique. Similarly, I won’t be able to tell whether all relevant related work has been cited in a paper on design techniques but can certainly offer my opinion on further applications or spelling errors.

I would prefer to describe my expertise more accurately and also focus only on certain aspects of a paper as a reviewer. By telling the AC that I do not know enough about statistics, I give them the opportunity to find another reviewer who does. Furthermore, the AC could also assign a subset of reviewers to individual aspects of a paper (e.g., related work, writing style, experimental setup, statistical tests, technical correctness, replicability). This could avoid duplicated work and simultaneously increase the quality of the reviews.

Author-Reviewer Collaboration

Reviewers point out weaknesses and suggest improvements in a paper. Given the significant amount of time some reviewers invest in reviewing a paper, and given that their suggestions may significantly improve a paper, it might be a good idea to offer a way of attributing their contributions. For example, reviewers could be mentioned by name in the Acknowledgements section (if they want) or they could become co-authors of the final submission. This would certainly change the character of the review process and introduce some problems. While the aforementioned changes could already be implemented for next year’s CHI conference, this final suggestion certainly needs more discussion and refinement to get it right. One option could be for reviewers to ‘fork’ a submission and submit ‘pull requests’ with proposed changes similar to software development processes on GitHub and similar platforms. (In general, a version-controlled approach to paper writing, including authorship attribution for each sentence sounds quite interesting.)

Given that the program chairs of recent CHI conferences have put much effort into evolving the conference series, I am optimistic that some of these proposed changes will be implemented in the near future. And frankly, it would be very fitting for the CHI community to be at the forefront of academic collaboration and publishing.

Update (08. March 2012): Wiley has informed me that they just fixed the issue.

70 GB of digital content from Wiley’s Major Reference Works can be downloaded for free from Wiley’s web server. This is probably not intended, as Wiley still charges hundreds or even thousand of Euros for each one. I notified them of this fact several times during the last two months. They do not seem to mind. In this post I explain how I found out about this, how one can download the content, and why this is probably legal (at least in some jurisdictions).
Read the full post »