For Authors Important Dates Call for Papers Subject Areas Submission Instructions Camera-ready Instructions
Program Schedule Accepted Papers Tutorials Online format Code of Conduct Registration
Organization Organizing Committee Senior Program Committee Program Committee Reviewer Instructions For Sponsors Past Conferences AUAI Contact Us

Reviewing instructions UAI 2020


Thank you for reviewing for UAI! Your help is vital to our community: the technical content of the program is largely determined by the efforts and comments of the reviewers. Below are instructions and guidelines that will help you write reviews efficiently and effectively.

If you prefer to read these instructions as a pdf, please see here.

Conference system access

All reviews must be entered electronically into the CMT system. Reviewers may visit this site multiple times and revise their reviews as often as necessary before the reviewing deadline. Please make sure emails from CMT are not snagged by your spam filter, and if using Gmail consider adding a filter to make sure these emails show up in your Primary inbox.

Confidentiality

By viewing the papers, you agree that the UAI review process is confidential. Specifically, you agree not to use ideas and results from submitted papers in your work, research or grant proposals, unless and until that material appears in other publicly available formats, such as a technical report or as a published work. You also agree not to distribute submitted papers or the ideas in them to anyone unless approved by the program chairs.

Double blind reviewing

We use double blind reviewing. The authors do not know the identity of the reviewers; this also holds for authors who are on the program committee. In addition, the reviewers do not know the identity of the authors. The SPC members (area chairs), however, do know the authors’ and reviewer identities; this helps avoid accidental conflicts of interest. During the discussion period, reviewers see the other reviewers’ identities; this allows for a more informed discussion among reviewers.

Double blind reviewing is not perfect: by searching the Internet, a reviewer may discover (or think he/she may have discovered) the identity of an author. We encourage you not to actively attempt to discover the identities of the authors. If you have good reason to suspect that a paper has been published in the past, you can go and search on the Internet, but we ask that you first completely read the paper. Also, based on the experience of other double-blind conferences, we caution reviewers that the assumed authors may not be the actual authors; multiple independent invention is common and different groups build on each others’ work.

If you believe that you have discovered the identity of the author, please let us know in the “Confidential comments to PC members” in your review (see below).

Conflicts of interest

In reviewing, you may accidentally discover the identity of the authors. If this happens, and you believe that you may have a conflict of interest, please contact your SPC member (area chair) immediately.

Supplementary material

Some papers include supplementary material. Because of ambiguity in the original submission instructions, some submissions include the supplementary material in the original PDF file, following the references. Other papers have separately uploaded supplementary material which must be downloaded from CMT.

Your responsibility as a reviewer is to read and judge the main paper. Reading the supplementary material is optional. However, keeping in mind the space constraints of an UAI paper, you may want to consider looking at the supplementary material before stating that the authors did not provide a fully rigorous proof of their theorems, or only demonstrated qualitative results on a small number of examples.

Content of the Review

For each paper you will provide written comments under each of the headings below. Your review should address both the strengths and weaknesses of the paper - identify the areas where you believe the paper is particularly strong and particularly weak - this will be very valuable to the program chairs and the SPC.

Please be as precise as you can in your comments to the authors and avoid vague statements. Your criticism should be constructive where possible - if you are giving a low score to a paper then try to be clear in explaining to the authors the types of actions they could take to improve their paper in the future. If you think that this work is incremental relative to prior work, please cite the specific relevant prior work you are referring to. If you think the experiments are not realistic or useful, let the author(s) know what they could do to improve them (e.g., more realistic data sets, larger data sets, different evaluation metrics, sensitivity analyses, etc.).

Paper summary & broader context (Question 1 in CMT)

Summary

In this section please explain in your own words what problem the paper addresses and what it contributes to solving it.

Relevance and Impact

Is this paper a significant advance in the state of the art? Is this a paper that people are likely to read and cite in later years? Does the paper address an important problem? What is the potential impact inside the UAI community? Is this paper likely to generate impact outside the UAI community, e.g., in the natural or social sciences (given that the paper will be read)? Does it raise new research issues for the community? Is it a paper that is likely to have any lasting impact? Is this a paper that researchers and/or practitioners might find useful 5 or 10 years from now? Is this work that can be built on by other researchers?

Detailed Comments (Question 2 in CMT)

In this section, you should provide detailed comments on the paper’s novelty, technical quality and reproducibility, clarity of writing, and improvements that would be necessary to improve the score. We encourage reviewers to explicitly structure the detailed comments using these sections.

Novelty

Reviewers should reward papers that propose genuinely new ideas, papers that truly depart from the "natural" next step in a given problem or application. We recognize that novelty can sometimes be relative, and we ask the reviews to assess it in the context of the respective problem or application area. Although authors are expected to emphasize the connections of their work to previous literature, we expect that reviewers additionally contextualize the work with other past literature, both inside and outside of the UAI community; in some cases this may require doing a small literature search. Point out in your review if the contribution is an adaptation or extension of existing ideas.

Technical Quality and Reproducibility

Are the results technically sound? Are there obvious flaws in the conceptual approach? Is there a simpler approach that you think would work just as well? Are claims well-supported by theoretical analysis or experimental results? Did the authors ignore (or appear unaware of) highly relevant prior work? Are the experiments well thought out and convincing? Are there obvious experiments that were not carried out? Will it be possible for other researchers to replicate these results? Are the data sets and/or code publicly available? Is the evaluation appropriate? Did the authors discuss sensitivity of their algorithm/method/procedure to parameter settings? Did the authors clearly assess both the strengths and weaknesses of their approach?

Quality of Writing

Is the paper clearly written? Does it adequately inform the reader? Are there good use of examples and figures? Is it well organized? Are there problems with style and grammar? Are there issues with typos, formatting, references, etc.? It is the responsibility of the authors of a paper to write clearly, rather than it being the duty of the reviewers to try to extract information from a poorly written paper. Do not assume that the authors will fix problems before a final camera-ready version is published - unlike journal publications, there will not be time to carefully check that accepted papers are properly written. Think of future readers trying to extract information from the paper -- it may be better to advise the authors to revise a paper and submit to a later conference, than to accept and publish a poorly-written version. However if the paper is likely to be accepted, please do make suggestions to improve the clarity of the paper, and provide details of any typos that you have found.

Improvements Necessary to Improve the Score

Which improvements would make you give a better score to this paper? Here, we are interested in the points that are highest on your priority list. You do not need to comment on how difficult these changes would be to implement or whether these changes could be implemented by the authors for a camera-ready version. Please also include in this section clarifying questions for the authors that they could answer in the rebuttal period.

Confidential Comments that are only for the SPC and program chairs (optional)

This is an optional section. If there are any comments that you would like to communicate to the SPC (area chairs) and program chairs, but that you do not wish to be seen by the authors or by the other reviewers, they can go in this section. For example, you can include rankings or comparisons among the papers you have reviewed for UAI.

Writing Reviews: Numeric Scoring

For UAI 2020 we are using a 6-point scoring system. We strongly encourage you to use the full range of scores, if appropriate for your papers. Try to use sparingly the scores in the middle of the scale, i.e., 3’s and 4’s. Don’t be afraid to assign 1 or 6 if papers deserve them. If you are new to the UAI conference (or have not attended for a number of years) you may find it useful to take a look at online proceedings from recent UAI conferences to help calibrate your scores. The scoring system is as follows:

  1. Strong Reject: Wrong or known results.
  2. Reject: Clearly below the acceptance threshold.
  3. Weak Reject: Borderline, tending to reject.
  4. Accept: Good paper.
  5. Strong Accept: Top 15% of accepted papers.
  6. Award Candidate: Top 5% of accepted papers.

You will also be asked to score the confidence of your evaluation as follows:

  1. The reviewer's evaluation is an educated guess (least confident).
  2. The reviewer is fairly confident that the evaluation is correct.
  3. The reviewer is confident but not absolutely certain that the evaluation is correct.
  4. The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature (most confident).

The numeric scores are just a vehicle to let you express your evaluation of the paper, in addition to the main one, which is your in-depth, specific and carefully thought of review, and not to supplant it.

Author feedback and reviewer consensus

Between April 13 and April 20, authors will have a chance to submit feedback on their reviews. This is an opportunity to correct possible misunderstandings about the contents of the paper, or about previous work. Authors may point out aspects of the paper that you missed, or disagree with your review.

It is important to convey to the authors that their comments were read, even if they do not change the final evaluation of the paper. Therefore, please read each rebuttal carefully and keep an open mind. Do the authors’ comments make you change your mind about your review? Have you overlooked something? There is a separate box for reviewers to write responses to the author feedback. Please use this to let the authors know that you read and absorbed their rebuttal, and whether it swayed you one way or the other.

From April 21 to April 30, the area chairs will, where necessary, lead a discussion via the website and try to come to a consensus amongst the reviewers. The discussion will involve both marginal papers, trying to reach a decision on which side of the bar they should fall, and controversial papers, where the reviewers disagree. Many papers fall into these categories, and therefore this phase is important. While engaging in the discussion, recall that different people have somewhat different points of view, and may come to different conclusions about a paper. It may be helpful to ask yourself “do the other reviewers’ comments make sense?”, and “should I change my mind given what the others are saying?”