{"id":173,"date":"2018-11-09T13:38:09","date_gmt":"2018-11-09T13:38:09","guid":{"rendered":"http:\/\/ecir2020.novasearch.org\/?page_id=173"},"modified":"2018-11-09T13:38:09","modified_gmt":"2018-11-09T13:38:09","slug":"reviewing-guidelines","status":"publish","type":"page","link":"https:\/\/ecir2020.org\/reviewing-guidelines\/","title":{"rendered":"Reviewing Guidelines"},"content":{"rendered":"\n<p>Members of the Program Committee (PC) assist a submission\u2019s Senior Program Committee member (SPC) by writing a first-tier review for the paper and participating actively in the discussion phase.<\/p>\n\n\n\n<p>Every paper has been assigned a Senior Program Committee member (SPC) and three Program Committee members (PCs). As a PC, your job is to provide critical assessments of the papers assigned to you based on the review form that we have prepared (see below). For each paper assigned to you, your review and the reviews by your fellow reviewers will form the basis for a discussion about the paper.<\/p>\n\n\n\n<p>The SPC will lead the discussion, and if necessary provide an additional review to the paper. Based on the discussion, the SPC will arrive at a recommended decision. This recommended decision will be passed on to the Program Chairs and used as input to the Program Committee Meeting.<\/p>\n\n\n\n<p>The review form has eight questions to answer for every submission:<\/p>\n\n\n\n<ul><li><strong>Paper Clarity:<\/strong>\u00a0Is the paper clearly written? Is it well-organized? Is the result section or are the proofs clear enough so that expert readers will understand them and can reproduce the results (if applicable)?<\/li><li><strong>Interest to Audience:<\/strong>\u00a0Will this paper attract the interest of ECIR 2020 attendees? Will it be intriguing and inspiring? Might it be highlighted afterwards in the press as particularly innovative?<\/li><li><strong>Paper Significance:<\/strong>\u00a0Does the paper contribute a major breakthrough or an incremental advance? Are other people(practitioners or researchers) likely to use these ideas or build on them? Is it clear how this work differs from previous contributions and is related work adequately referenced?<\/li><li><strong>Strengths:<\/strong>\u00a0Please list at least three strengths.<\/li><li><strong>Weaknesses:<\/strong>\u00a0Please list at least three weaknesses or areas for improvement. Try to be specific and constructive.<\/li><li><strong>Overall Evaluation:<\/strong>\u00a0In addition to a score, your narrative should provide the clear rationale for your recommendation of accept\/reject.<\/li><li><strong>Reviewer\u2019s confidence<\/strong><\/li><li><strong>Nominate for Best Paper:<\/strong>\u00a0whether this paper should be nominated as a best paper candidate.<\/li><\/ul>\n\n\n\n<p>Please provide a response for each of these questions.<\/p>\n\n\n\n<p>Your review is not just a vote for whether the paper will be accepted; it is essential input to a discussion amongst the reviewing team and to the Program Chairs. It is also the input to the authors to guide them with the changes suggested, and to help them understand the outcome of the review process. You are assisting your fellow PC members, the SPCs, and the Program Chairs by providing arguments for or against acceptance.<\/p>\n\n\n\n<p>In some cases, there will be divergence amongst reviewers\u2019 numerical ratings of the paper; if you provide only a rating and terse summary, without an adequate rationale, it will not be helpful.<\/p>\n\n\n\n<p>In the text area of \u201cOverall Evaluation\u201d, you should supply a detailed rationale for the responses you have provided across the first five questions. Start your review with an assessment of what you consider to be the main contribution of the paper. Please do not just repeat what the authors say they did. You should provide your own summary of what you gained by reading the paper.<\/p>\n\n\n\n<p>Whether you like or dislike a paper, please say so in a manner that is helpful to the authors and informative to your area chairs. (You will be asked to rewrite reviews that do not meet this expectation.)<\/p>\n\n\n\n<p>Please summarize your main points and refer back to the strengths and weaknesses you listed in the earlier sections. It is important to point out weaknesses and validity issues, but it is equally important to identify the contribution of a submission. Ultimately, a submission\u2019s acceptance depends on its novel contribution, not perfection. Note again that we are looking for an evaluation of the paper, not just a recommendation.<\/p>\n\n\n\n<p>One issue that may arise is that authors miss some of the prior research that has been published in the area. This should be regarded as being a fatal flaw only if the missing work critically affects their conclusions. If you do regard a paper as being unacceptable because of lack of reference to prior work, you should supply sufficient detail about this prior research in the form a complete reference ideally including its DOI, so that the authors can understand why you believe their paper should not be accepted. We encourage you to provide this level of detail for all references that you consider missing in the submission.<\/p>\n\n\n\n<p>Finally, regarding anonymity, the CFP states: \u201cNote however, that it is acceptable to explicitly refer in the paper to the companies or organizations that provided datasets, hosted experiments or deployed solutions. In other words, instead of stating for instance that an experiment \u2018was conducted on the logs of a major search engine\u2019, the authors should refer to the search engine by name. The reviewers will be informed that it does not necessarily imply that the authors are currently affiliated with the mentioned organization.\u201d This was added in a previous year due to general community feedback that hiding particular product details and dataset details were being overly harmful to the description without adding much value for anonymity. While unusual, it does occur that external community members (e.g. academics) are allowed access to data or experimentation (either through sharing or temporary agreements). If you have questions about a particular submission, please review it in full and raise the issue of anonymity for discussion with the chairs.<\/p>\n\n\n\n<p>Thank you for your contributions to creating an excellent program for ECIR 2020!<\/p>\n\n\n\n<p><em>(This guideline is drafted based on the WSDM 2019 guideline, which was drafted based on previous IR conferences, so Thank you previous PC chairs.)<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Members of the Program Committee (PC) assist a submission\u2019s Senior Program Committee member (SPC) by writing a first-tier review for the paper and participating actively in the discussion phase. Every paper has been assigned a Senior Program Committee member (SPC) and three Program Committee members (PCs). As a PC, your job is to provide critical &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/ecir2020.org\/reviewing-guidelines\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Reviewing Guidelines&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"spay_email":""},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v18.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Reviewing Guidelines - ECIR 2020 | Online | 14-17 April 2020<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ecir2020.org\/reviewing-guidelines\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Reviewing Guidelines - ECIR 2020 | Online | 14-17 April 2020\" \/>\n<meta property=\"og:description\" content=\"Members of the Program Committee (PC) assist a submission\u2019s Senior Program Committee member (SPC) by writing a first-tier review for the paper and participating actively in the discussion phase. Every paper has been assigned a Senior Program Committee member (SPC) and three Program Committee members (PCs). As a PC, your job is to provide critical &hellip; Continue reading &quot;Reviewing Guidelines&quot;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ecir2020.org\/reviewing-guidelines\/\" \/>\n<meta property=\"og:site_name\" content=\"ECIR 2020 | Online | 14-17 April 2020\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/ecir2020\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@ecir2020\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https:\/\/ecir2020.org\/#organization\",\"name\":\"ECIR 2020\",\"url\":\"https:\/\/ecir2020.org\/\",\"sameAs\":[\"https:\/\/facebook.com\/ecir2020\",\"https:\/\/twitter.com\/ecir2020\"],\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ecir2020.org\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ecir2020.org\/wordpress\/wp-content\/uploads\/2019\/01\/favicon.png\",\"contentUrl\":\"https:\/\/ecir2020.org\/wordpress\/wp-content\/uploads\/2019\/01\/favicon.png\",\"width\":512,\"height\":512,\"caption\":\"ECIR 2020\"},\"image\":{\"@id\":\"https:\/\/ecir2020.org\/#\/schema\/logo\/image\/\"}},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ecir2020.org\/#website\",\"url\":\"https:\/\/ecir2020.org\/\",\"name\":\"ECIR 2020 | Online | 14-17 April 2020\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/ecir2020.org\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ecir2020.org\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ecir2020.org\/reviewing-guidelines\/#webpage\",\"url\":\"https:\/\/ecir2020.org\/reviewing-guidelines\/\",\"name\":\"Reviewing Guidelines - ECIR 2020 | Online | 14-17 April 2020\",\"isPartOf\":{\"@id\":\"https:\/\/ecir2020.org\/#website\"},\"datePublished\":\"2018-11-09T13:38:09+00:00\",\"dateModified\":\"2018-11-09T13:38:09+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ecir2020.org\/reviewing-guidelines\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ecir2020.org\/reviewing-guidelines\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ecir2020.org\/reviewing-guidelines\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ecir2020.org\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Reviewing Guidelines\"}]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Reviewing Guidelines - ECIR 2020 | Online | 14-17 April 2020","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ecir2020.org\/reviewing-guidelines\/","og_locale":"en_US","og_type":"article","og_title":"Reviewing Guidelines - ECIR 2020 | Online | 14-17 April 2020","og_description":"Members of the Program Committee (PC) assist a submission\u2019s Senior Program Committee member (SPC) by writing a first-tier review for the paper and participating actively in the discussion phase. Every paper has been assigned a Senior Program Committee member (SPC) and three Program Committee members (PCs). As a PC, your job is to provide critical &hellip; Continue reading \"Reviewing Guidelines\"","og_url":"https:\/\/ecir2020.org\/reviewing-guidelines\/","og_site_name":"ECIR 2020 | Online | 14-17 April 2020","article_publisher":"https:\/\/facebook.com\/ecir2020","twitter_card":"summary_large_image","twitter_site":"@ecir2020","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Organization","@id":"https:\/\/ecir2020.org\/#organization","name":"ECIR 2020","url":"https:\/\/ecir2020.org\/","sameAs":["https:\/\/facebook.com\/ecir2020","https:\/\/twitter.com\/ecir2020"],"logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ecir2020.org\/#\/schema\/logo\/image\/","url":"https:\/\/ecir2020.org\/wordpress\/wp-content\/uploads\/2019\/01\/favicon.png","contentUrl":"https:\/\/ecir2020.org\/wordpress\/wp-content\/uploads\/2019\/01\/favicon.png","width":512,"height":512,"caption":"ECIR 2020"},"image":{"@id":"https:\/\/ecir2020.org\/#\/schema\/logo\/image\/"}},{"@type":"WebSite","@id":"https:\/\/ecir2020.org\/#website","url":"https:\/\/ecir2020.org\/","name":"ECIR 2020 | Online | 14-17 April 2020","description":"","publisher":{"@id":"https:\/\/ecir2020.org\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ecir2020.org\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ecir2020.org\/reviewing-guidelines\/#webpage","url":"https:\/\/ecir2020.org\/reviewing-guidelines\/","name":"Reviewing Guidelines - ECIR 2020 | Online | 14-17 April 2020","isPartOf":{"@id":"https:\/\/ecir2020.org\/#website"},"datePublished":"2018-11-09T13:38:09+00:00","dateModified":"2018-11-09T13:38:09+00:00","breadcrumb":{"@id":"https:\/\/ecir2020.org\/reviewing-guidelines\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ecir2020.org\/reviewing-guidelines\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ecir2020.org\/reviewing-guidelines\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ecir2020.org\/"},{"@type":"ListItem","position":2,"name":"Reviewing Guidelines"}]}]}},"_links":{"self":[{"href":"https:\/\/ecir2020.org\/wp-json\/wp\/v2\/pages\/173"}],"collection":[{"href":"https:\/\/ecir2020.org\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ecir2020.org\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ecir2020.org\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ecir2020.org\/wp-json\/wp\/v2\/comments?post=173"}],"version-history":[{"count":0,"href":"https:\/\/ecir2020.org\/wp-json\/wp\/v2\/pages\/173\/revisions"}],"wp:attachment":[{"href":"https:\/\/ecir2020.org\/wp-json\/wp\/v2\/media?parent=173"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}