UAI 2021 - Reviewing Instructions

Thank you for reviewing for UAI! Your assessments largely determine the content and quality of the conference and your help is vital for our community. Please carefully read the instructions below - they explain the reviewing process and will help you to write your reviews efficiently and effectively.

General remarks

The timeline for the reviewing process is:

• February 20, 2021 (23:59 UTC): Paper submission deadline
• February 21-26, 2021: Bidding period
• March 9-April 8, 2021: Reviewing period
• April 21-29, 2021: Discussion period
• May 12, 2021: Author notification

Please mark these dates in your calendar. Our schedule is tight, so it is key that you respect the deadlines and respond to emails promptly.

The entire review process takes place within the CMT system (see below). In CMT, Program Committee members are called reviewers, and Senior Program Committee members (also called Area Chairs in some conferences) are called meta-reviewers. To prevent confusion, we will use the CMT terminology in this document.

Reviewers typically assess around 5-7 papers. For each assigned paper, their main responsibilities consist of writing constructive and informative reviews, reading the author responses, participating in the discussions, and providing recommendations regarding acceptance or rejection.

Meta-reviewers typically oversee around 10-15 papers. For each assigned paper, their main responsibilities consist of finalizing the list of reviewers, facilitating the reviewing and discussion process, summarizing it in a meta-review, and providing a recommendation regarding paper acceptance or rejection to the program chairs. In some cases, this may require a close reading of the paper. Meta-reviewers will also be asked to evaluate the quality of each review using three scores: “exceeds expectations”, “meets expectations”, and “fails to meet expectations”.

Reviewing is an essential part of making UAI a great conference. As a token of our appreciation for the work of the Program Committee, we do three new things this year: top reviewers will be awarded free UAI registrations, a list of reviewers who obtained very good reviewing scores will be posted on the conference website, and we will provide a certificate of your participation on request.

By reviewing for UAI 2021, you agree to abide by the UAI 2021 Code of Conduct.

Conference system

The entire review process (submission of papers, entering reviews, author feedback and discussions) takes place within the CMT system. In addition, we use the Toronto Paper Matching System (TPMS) to assign submissions to program committee members. Please note the following:

• If you have several CMT accounts, please make sure to log in with the email address that we used for the Program Committee invitation. If you are both an author and a reviewer, please use this same email address for both roles in CMT. Your TPMS profile should also be under this email address.
• Please log into TPMS and CMT before February 17th to make sure that your profiles are up to date. Using stale profiles cause poor assignments that may be outside your area of expertise.
• Please make sure that emails from CMT are not caught by your spam filter. If possible, we recommend that you create a filter to ensure that emails from CMT (e.g. email@msr-cmt.org) are delivered in your primary inbox.
• Login and other CMT system issues can only be resolved by the CMT support team. Please send an email to support@msr-cmt.org.
• You should not reply directly to emails sent by CMT. Instead, you should log in to CMT and reply from within this system.

Confidentiality

By reviewing for UAI 2021, you agree to keep all material and information confidential (including but not limited to papers, meta-data, bidding data, messages and discussions related to reviewing, names of committee members involved, etc) and use them only for the purposes of the reviewing process of UAI 2021. Particularly, you may not use ideas and results from submitted papers for your own work, research or grant proposals, or distribute them to others, unless and until the material appears in publicly available formats, such as a technical report or a published manuscript. You should delete all reviewing material, including (but not limited to) any submitted code, at the end of the reviewing cycle.

You should not talk about or distribute (parts of) submissions and/or content related to the reviewing of submissions to anyone without prior approval from the program chairs. This includes other reviewers or meta-reviewers, as they may have conflicts with these submissions. In general, your primary point of contact for any discussions regarding a submission is the assigned meta-reviewer for that submission.

You should also not talk to other reviewers or meta-reviewers about your own submissions (i.e., submissions you are an author on) or about submissions with which you have a conflict.

Double blind reviewing

The reviewing process is double blind at the level of reviewers and meta-reviewers (i.e., reviewers and meta-reviewers cannot see author identities and vice versa). To allow for a good discussion, the reviewers and the meta-reviewer of a paper can see each others’ identities.

Authors are responsible for anonymizing their submissions. In particular, they must make sure that the submission (including all supplementary material) does not disclose author identities or affiliations. Hence, among other precautions, they should use the third person to refer to their own work, they should not include names in the acknowledgements, and links to external resources that may reveal their identity or institution are forbidden. If you are assigned a submission that is not adequately anonymized, please contact the corresponding meta-reviewer by email.

Under no circumstances should you attempt to find out the identities of the authors for any of your assigned submissions (e.g., by searching on Google or arXiv). If you accidentally discover author identities, please do not disclose them to anyone else and do not let them influence your decision in any way.

Conflicts of interest

Authors, reviewers and meta-reviewers will be asked to separately enter two types of conflicts into CMT: domain conflicts and individual conflicts. Please find more details here.

The declared conflicts will be used to make sure that you are not assigned any papers you have a conflict with. If, despite these efforts, you suspect that you may have a conflict of interest with an assigned submission, then please contact the meta-reviewer (or, if needed, the program chairs) immediately.

Supplementary material

Some papers contain supplementary material. For your convenience, we require that supplementary text material is included in the same PDF as the main paper, after the references. A few submissions may include other types of supplementary material (e.g., code, data) that must be separately downloaded from CMT.

Your responsibility as a reviewer is to read and review the submission (main paper) itself. Looking at supplementary material is at your discretion. That said, you may want to look at supplementary material before criticizing a submission for insufficient details, proofs, or experimental results.

Executing code and clicking on links

We cannot provide any guarantees about submitted code and links. Hence, if you are planning to run supplementary source code, please make sure you are doing this in a secure environment, for example inside a Docker container, on a Virtual Machine image (using VirtualBox or VMWare), or on a network-isolated cloud instance. Please note that any links that are provided in a submission may contain vulnerabilities or may log visitor IP addresses (thereby revealing your identity).

Violations of formatting instructions

Submissions must be formatted using the UAI latex template and formatting instructions. Papers must be submitted as a PDF file and are limited to 8 pages in length, including all figures and tables. At most two additional pages containing only references are allowed. Authors must not change the template. Supplementary text material should be included after the main paper, in the same PDF (after all references, starting on a new page). There is no page limit nor a fixed template for supplementary material.

If you are assigned a paper that is overlength or appears to violate the UAI proceedings format (e.g., by decreasing margins or font size, by removing some pre-fixed spaces, etc), please notify the meta-reviewer immediately.

Dual submissions

Papers that are currently under review or have been accepted or published in a refereed venue with proceedings may not be submitted. This also applies to papers that are substantially equal. Moreover, it is not allowed to submit the same (or substantially equal) work to another refereed venue with proceedings while it is still under review at UAI. Papers in conferences or workshops without proceedings, technical report repositories and arXiv are not considered dual submissions.

If you suspect that a submission that has been assigned to you is a dual submission or if you require further clarification, please contact the corresponding meta-reviewer.

Author response

After the initial review period, the authors may submit responses to their reviews. This is an opportunity for them to correct possible misunderstandings about the contents of the paper, or about previous work. They may also point out aspects of the paper that you missed, or they may disagree with certain aspects of your review.

It is important that you read each rebuttal carefully and with an open mind. Do the authors’ comments change your opinion about the paper? Have you overlooked or misunderstood something?

Discussion period

After the author response period, the discussion period starts. This is a very important part of the reviewing process and all reviewers should actively participate. These discussions are especially important for borderline papers and for papers with widely varying assessments.

The discussions are led by the meta-reviewers. At the start of the discussion period, please read the other reviews and the authors’ response. Please try to understand their points of view. Do they bring up aspects that you viewed differently or missed? While engaging in the discussion, please be professional, polite and keep an open mind. You may change your opinion, but should not give in to undue influence. It is not required that there is full consensus about a paper; different people may come to different valid conclusions about a paper.

Contacting the program chairs

If you encounter any problem that you cannot solve with the assigned meta-reviewer, please contact the program chairs: uai2021programchairs@gmail.com.

Content of the Review

For each paper you should answer the questions on the review form in CMT (see below). You may revise your review as often as necessary before the reviewing deadline.

Q1: Summary and contributions

Please summarize the paper’s motivation and key contributions in a few lines.

Although this part of the review may not provide much new information to authors, it is very valuable to the meta-reviewer and the program chairs.

Please note that there are many examples of contributions that warrant publication at UAI. These contributions may be theoretical, methodological, algorithmic, applied, empirical, connecting ideas in separate fields (“bridge papers”), or providing a critical analysis (e.g., principled justifications of why the community is going after the wrong outcome or using the wrong types of approaches).

(Max. 500 characters.)

Q2: Main strengths

Please describe the main strengths of the work, considering primarily the following axes: originality/novelty, significance/impact, soundness/technical quality, reproducibility and clarity of writing. Please mention as many main strengths as there are, but avoid minor points here.

More detailed information regarding each of these aspects is given below:

• Originality/novelty: How do you rate the originality of the ideas given existing approaches? Do the authors propose an adaptation or extension of existing ideas, or are these genuinely new ideas that depart from the “natural” next step in the given problem or application? Do the authors consider a new problem formulation that has not been considered before?
• Significance/impact: Does the paper address an important problem? How relevant are the results for the UAI community? To what extent does the paper advance the state of the art? Do you expect that researchers or practitioners will be influenced by the proposed ideas? Is this a paper that people are likely to read and cite in later years? Is this work that can be built on by other researchers? Is the paper likely to generate impact outside of the UAI community, e.g., in the natural or social sciences?
• Soundness/technical quality: Depending on the type of paper, you can think of the following questions: Do the authors put their work properly into context? Is the conceptual approach sound? Is there a simpler approach that you think would work just as well? Are claims substantiated by theoretical analysis or experimental results? Are the proofs correct? Is the empirical evaluation well thought out and convincing? Does it include state-of-the-art competitors? Are there obvious experiments that were not carried out? Do the authors discuss the sensitivity of their procedure to parameter settings? Do the authors assess both the strengths and weaknesses of their approach?
• Reproducibility: Are there enough details so that other researchers can replicate the results? Are the data sets and/or code publicly available?
• Clarity of writing: Is the paper clearly written and well-structured? Can future readers easily extract information from the paper? Do the authors make good use of examples and figures to help readers understand the important concepts? Are there issues with style, grammar, typos, etc? Are there parts of the paper that need revision to improve clarity? We emphasize that it is the responsibility of the authors to write clearly. Also, please keep in mind that, unlike for journal papers, we do not have time to evaluate a revision. So you need to base your assessment on the current version (modulo small changes).

(Max. 1000 characters.)

Q3: Main weaknesses

Please describe the main weaknesses of the work, considering the same axes as in Q2: novelty/originality, impact/significance, soundness/technical quality, reproducibility, and clarity of writing. Please mention as many main weaknesses as there are, but avoid minor points here.

Your comments should be detailed, specific, and polite. Please avoid vague, subjective complaints. Think about the times when you received an unfair, unjustified, short, or dismissive review. Try not to be that reviewer! Always be constructive and help the authors understand your viewpoint, without being dismissive or using inappropriate language. Remember that you are not reviewing your level of interest in the submission, but its scientific contribution to the field.

For example, if you think that the work is incremental relative to prior work, please cite the specific relevant prior work with a complete citation. Or if you think the experiments are not realistic or useful, let the authors know what they could do to improve them (e.g., more realistic data sets, larger data sets, different evaluation metrics, sensitivity analyses, etc.).

(Max. 1000 characters.)

Q4: Detailed comments to the authors

Please provide constructive criticism and feedback that could help improve the work or its presentation (e.g., presentation suggestions, missing references, minor mistakes and typos or grammar improvements). You may also include questions to the author here.

If you would like the authors to clarify something during the author response phase, please articulate this clearly (e.g., “I would like to see results of experiment X” or “Can you please include details about the parameter settings used for experiment Y”). We suggest that you directly ask for clarifications only on points that are important for reaching a decision, since the authors’ response has a limited length.

(Unlimited number of characters.)

Q5: Overall score

Please select the category that best describes your overall assessment of the paper.

We use a 6-point scoring system for the overall assessment. We encourage you to use the full range of scores for your papers, if appropriate. You should not assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. If you are new to the UAI conference (or have not attended for a number of years) you may find it useful to take a look at online proceedings from recent UAI conferences to help calibrate your scores.

The scoring system is as follows:

1. Strong Reject: Wrong or known results.
2. Reject: Clearly below the acceptance threshold.
3. Weak reject: Borderline paper, tending to reject.
4. Weak Accept: Borderline paper, tending to accept.
5. Accept: Good paper
6. Strong Accept: Outstanding paper

Q6: Justification for your score

Please explain in a few lines how you arrived at your overall assessment. Which aspects mentioned under main strengths and main weaknesses (Q2 & Q3) did you weigh most heavily and why?

(Max. 500 characters.)

Q7: Confidence in your score

Please rate your confidence in your assessment.

The scoring system is as follows:

1. The reviewer's evaluation is an educated guess (least confident).
2. The reviewer is fairly confident that the evaluation is correct.
3. The reviewer is confident but not absolutely certain that the evaluation is correct.
4. The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature (most confident).

Q8: Ethical concerns

Does the submission raise potential ethical concerns? This includes methods, applications, or data that can create or reinforce unfair bias or that have a primary purpose of harm or injury. Please answer yes/no, and provide a brief explanation if your answer is yes.

Note that your overall rating should be independent of your answer to this question. Your duty here is only to flag papers that might need further evaluation regarding ethical aspects.

(Max. 500 characters.)

Q9: Complying with reviewing instructions

I have read the UAI reviewing instructions and certify that I comply with them. In particular, I keep the paper and supplementary materials (including code submissions) confidential, and delete all reviewing material at the end of the review cycle. (https://auai.org/uai2021/reviewing_instructions)

Q10: Considerations after author rebuttal and discussion period

Please leave this question blank when submitting your initial review, and only fill it out towards the end of the discussion period. Please indicate to the authors that you have read their response, and let them know if/how their rebuttal and the discussion period changed your opinion on the paper.

(Max. 2000 characters. To be used only during/after discussion period.)

Q11: Confidential comments to the meta-reviewer and the program chairs

This is an optional section that can be used for comments that you want to communicate to the meta-reviewer and program chairs, but not to the authors or other reviewers.

If you have previously reviewed (a version of) this work for another archival venue, please let us know here. Please be aware that meta-reviewers may also submit papers, so we suggest that you avoid writing about other papers you have reviewed for UAI (unless they have the same assigned meta-reviewer).

(Max. 2000 characters.)

Updated February 18th 2021 (Information on number of characters per question included on 9 March 2021.)
UAI 2021 Program Chairs

(Similar reviewing instructions have been used by other conferences, such as UAI 2020, ICLR 2020, ICML 2020 and NeurIPS 2020. Parts of the content have been taken from these documents.)

Sponsors