Members of the Program Committee (PC) assist a submission’s Senior Program Committee member (SPC) by writing a first-tier review for the paper and participating actively in the discussion phase.
Every paper has been assigned a Senior Program Committee member (SPC) and three Program Committee members (PCs). As a PC, your job is to provide critical assessments of the papers assigned to you based on the review form that we have prepared (see below). For each paper assigned to you, your review and the reviews by your fellow reviewers will form the basis for a discussion about the paper.
The SPC will lead the discussion, and if necessary provide an additional review to the paper. Based on the discussion, the SPC will arrive at a recommended decision. This recommended decision will be passed on to the Program Chairs and used as input to the Program Committee Meeting.
The review form has eight questions to answer for every submission:
- Paper Clarity: Is the paper clearly written? Is it well-organized? Is the result section or are the proofs clear enough so that expert readers will understand them and can reproduce the results (if applicable)?
- Interest to Audience: Will this paper attract the interest of ECIR 2020 attendees? Will it be intriguing and inspiring? Might it be highlighted afterwards in the press as particularly innovative?
- Paper Significance: Does the paper contribute a major breakthrough or an incremental advance? Are other people(practitioners or researchers) likely to use these ideas or build on them? Is it clear how this work differs from previous contributions and is related work adequately referenced?
- Strengths: Please list at least three strengths.
- Weaknesses: Please list at least three weaknesses or areas for improvement. Try to be specific and constructive.
- Overall Evaluation: In addition to a score, your narrative should provide the clear rationale for your recommendation of accept/reject.
- Reviewer’s confidence
- Nominate for Best Paper: whether this paper should be nominated as a best paper candidate.
Please provide a response for each of these questions.
Your review is not just a vote for whether the paper will be accepted; it is essential input to a discussion amongst the reviewing team and to the Program Chairs. It is also the input to the authors to guide them with the changes suggested, and to help them understand the outcome of the review process. You are assisting your fellow PC members, the SPCs, and the Program Chairs by providing arguments for or against acceptance.
In some cases, there will be divergence amongst reviewers’ numerical ratings of the paper; if you provide only a rating and terse summary, without an adequate rationale, it will not be helpful.
In the text area of “Overall Evaluation”, you should supply a detailed rationale for the responses you have provided across the first five questions. Start your review with an assessment of what you consider to be the main contribution of the paper. Please do not just repeat what the authors say they did. You should provide your own summary of what you gained by reading the paper.
Whether you like or dislike a paper, please say so in a manner that is helpful to the authors and informative to your area chairs. (You will be asked to rewrite reviews that do not meet this expectation.)
Please summarize your main points and refer back to the strengths and weaknesses you listed in the earlier sections. It is important to point out weaknesses and validity issues, but it is equally important to identify the contribution of a submission. Ultimately, a submission’s acceptance depends on its novel contribution, not perfection. Note again that we are looking for an evaluation of the paper, not just a recommendation.
One issue that may arise is that authors miss some of the prior research that has been published in the area. This should be regarded as being a fatal flaw only if the missing work critically affects their conclusions. If you do regard a paper as being unacceptable because of lack of reference to prior work, you should supply sufficient detail about this prior research in the form a complete reference ideally including its DOI, so that the authors can understand why you believe their paper should not be accepted. We encourage you to provide this level of detail for all references that you consider missing in the submission.
Finally, regarding anonymity, the CFP states: “Note however, that it is acceptable to explicitly refer in the paper to the companies or organizations that provided datasets, hosted experiments or deployed solutions. In other words, instead of stating for instance that an experiment ‘was conducted on the logs of a major search engine’, the authors should refer to the search engine by name. The reviewers will be informed that it does not necessarily imply that the authors are currently affiliated with the mentioned organization.” This was added in a previous year due to general community feedback that hiding particular product details and dataset details were being overly harmful to the description without adding much value for anonymity. While unusual, it does occur that external community members (e.g. academics) are allowed access to data or experimentation (either through sharing or temporary agreements). If you have questions about a particular submission, please review it in full and raise the issue of anonymity for discussion with the chairs.
Thank you for your contributions to creating an excellent program for ECIR 2020!
(This guideline is drafted based on the WSDM 2019 guideline, which was drafted based on previous IR conferences, so Thank you previous PC chairs.)