UAI 2022 - Reviewing Instructions

Thank you for reviewing for UAI! Your assessments largely determine the content and quality of the conference and your help is vital for our community. Please carefully read the instructions below - they explain the reviewing process and will help you to write your reviews efficiently and effectively.

General remarks

The timeline for the reviewing process is:

• February 25, 2022 (23:59 UTC): Paper submission deadline
• February 28-March 4, 2022: Bidding period
• March 15-April 12, 2022: Reviewing period
• April 19-28, 2022: Discussion Stage 1
• Discussions among Reviewers/Authors/AC.
• ACs encourage reviewers to acknowledge and respond to author responses.
• April 29-May 4, 2022: Discussion Stage 2
• Discussions among reviewers and AC (without authors).
• May 16, 2022: Author notification

Please mark these dates in your calendar. Our schedule is tight, so it is key that you respect the deadlines and respond to emails promptly.

The entire review process takes place within the OpenReview system (see below).

Reviewers typically assess around 4-7 papers. For each assigned paper, their main responsibilities consist of writing constructive and informative reviews, reading the author responses, participating in the discussions, and providing recommendations regarding acceptance or rejection.

Area Chairs typically oversee no more than 15 papers. For each assigned paper, their main responsibilities consist of finalizing the list of reviewers, facilitating the reviewing and discussion process, summarizing it in a meta-review, and providing a recommendation regarding paper acceptance or rejection to the program chairs. In some cases, this may require a close reading of the paper. Area Chairs will also be asked to evaluate the quality of each review using three scores: “exceeds expectations”, “meets expectations”, and “fails to meet expectations”.

Reviewing is an essential part of making UAI a great conference. As a token of our appreciation for the work of the Program Committee, we do three new things this year: top reviewers will be awarded free UAI registrations, a list of reviewers who obtained very good reviewing scores will be posted on the conference website, and we will provide a certificate of your participation on request.

By reviewing for UAI 2022, you agree to abide by the UAI 2022 Code of Conduct.

Confidentiality

By reviewing for UAI 2022, you agree to keep all material and information confidential (including but not limited to papers, meta-data, bidding data, messages and discussions related to reviewing, names of committee members involved, etc) and use them only for the purposes of the reviewing process of UAI 2022. Particularly, you may not use ideas and results from submitted papers for your own work, research or grant proposals, or distribute them to others, unless and until the material appears in publicly available formats, such as a technical report or a published manuscript. You should delete all reviewing material, including (but not limited to) any submitted code, at the end of the reviewing cycle.

You should not talk about or distribute (parts of) submissions and/or content related to the reviewing of submissions to anyone without prior approval from the program chairs. This includes other reviewers or area chairs, as they may have conflicts with these submissions. In general, your primary point of contact for any discussions regarding a submission is the assigned area chair for that submission.

You should also not talk to other reviewers or area chairs about your own submissions (i.e., submissions you are an author on) or about submissions with which you have a conflict.

Double blind reviewing

The reviewing process is double blind at the level of reviewers and area chairs (i.e., reviewers and area chairs cannot see author identities and vice versa). Authors are responsible for anonymizing their submissions. In particular, they must make sure that the submission (including all supplementary material) does not disclose author identities or affiliations. Hence, among other precautions, they should use the third person to refer to their own work, they should not include names in the acknowledgements, and links to external resources that may reveal their identity or institution are forbidden. If you are assigned a submission that is not adequately anonymized, please contact the corresponding area chair by email.

Under no circumstances should you attempt to find out the identities of the authors for any of your assigned submissions (e.g., by searching on Google or arXiv). If you accidentally discover author identities, please do not disclose them to anyone else and do not let them influence your decision in any way.

Conflicts of interest

Authors, Reviewers and Area Chairs will be asked to separately enter two types of conflicts into OpenReview: domain conflicts and individual conflicts. Please find more details here.

The declared conflicts will be used to make sure that you are not assigned any papers you have a conflict with. If, despite these efforts, you suspect that you may have a conflict of interest with an assigned submission, then please contact the area chair (or, if needed, the program chairs) immediately.

Supplementary material

Some papers contain supplementary material. For your convenience, we require that supplementary text material is included in the same PDF as the main paper, after the references. A few submissions may include other types of supplementary material (e.g., code, data) that must be separately downloaded from OpenReview.

Your responsibility as a reviewer is to read and review the submission (main paper) itself. Looking at supplementary material is at your discretion. That said, you may want to look at supplementary material before criticizing a submission for insufficient details, proofs, or experimental results.

Executing code and clicking on links

We cannot provide any guarantees about submitted code and links. Hence, if you are planning to run supplementary source code, please make sure you are doing this in a secure environment, for example inside a Docker container, on a Virtual Machine image (using VirtualBox or VMWare), or on a network-isolated cloud instance. Please note that any links that are provided in a submission may contain vulnerabilities or may log visitor IP addresses (thereby revealing your identity).

Violations of formatting instructions

Submissions must be formatted using the UAI latex template and formatting instructions. Papers must be submitted as a PDF file and are limited to 8 pages in length, including all figures and tables. At most two additional pages containing only references are allowed. Authors must not change the template. Supplementary text material should be included after the main paper, in the same PDF (after all references, starting on a new page). There is no page limit nor a fixed template for supplementary material.

If you are assigned a paper that is overlength or appears to violate the UAI proceedings format (e.g., by decreasing margins or font size, by removing some pre-fixed spaces, etc), please notify the meta-reviewer immediately.

Dual submissions

Papers that are currently under review or have been accepted or published in a refereed venue with proceedings may not be submitted. This also applies to papers that are substantially equal. Moreover, it is not allowed to submit the same (or substantially equal) work to another refereed venue with proceedings while it is still under review at UAI. Papers in conferences or workshops without proceedings, technical report repositories and arXiv are not considered dual submissions.

If you suspect that a submission that has been assigned to you is a dual submission or if you require further clarification, please contact the corresponding area chair.

Author response

After the initial review period, the authors may submit responses to their reviews. This is an opportunity for them to correct possible misunderstandings about the contents of the paper, or about previous work. They may also point out aspects of the paper that you missed, or they may disagree with certain aspects of your review.

It is important that you read each rebuttal carefully and with an open mind. Do the authors’ comments change your opinion about the paper? Have you overlooked or misunderstood something?

Discussion period

After the author response period, the discussion period starts. This is a very important part of the reviewing process and all reviewers should actively participate. These discussions are especially important for borderline papers and for papers with widely varying assessments.

The discussions are led by the area chairs. At the start of the discussion period, please read the other reviews and the authors’ response. Please try to understand their points of view. Do they bring up aspects that you viewed differently or missed? While engaging in the discussion, please be professional, polite and keep an open mind. You may change your opinion, but should not give in to undue influence. It is not required that there is full consensus about a paper; different people may come to different valid conclusions about a paper.

Contacting the program chairs

If you encounter any problem that you cannot solve with the assigned area chair, please contact the program chairs: uai2022programchairs@gmail.com.

Content of the Review

For each paper, please answer the questions on the review form in OpenReview (see below). You may revise your review as often as necessary by the reviewing deadline.

Q1: Summary and contributions

Please summarize the paper’s motivation and key contributions in a few lines.

Although this part of the review may not provide much new information to authors, it is very valuable to the area chairs and the program chairs.

Please note that there are many examples of contributions that warrant publication at UAI. These contributions may be theoretical, methodological, algorithmic, applied, empirical, connecting ideas in separate fields (“bridge papers”), or providing a critical analysis (e.g., principled justifications of why the community is going after the wrong outcome or using the wrong types of approaches).

(Max. 500 characters.)

Q2: Assessment of the paper

Please assess the work along the following axes: originality/novelty, significance/impact, soundness/technical quality, quality of experiments, reproducibility, clarity of writing, ethical considerations and resource contribution.

More detailed information regarding each of these aspects is given below:

1. Originality/novelty: How novel are the concepts, problems addressed, or methods introduced in the paper?
   • 4: Excellent: The main ideas of the paper are ground-breaking.
   • 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
   • 2: Fair: The paper contributes some new ideas.
   • 1: Poor: The main ideas of the paper are not novel or represent incremental
2. Significance/impact: How do you rate the likely impact of the paper on the UAI research community?
   • 4: Excellent: The paper is likely to have a high impact across more than one subfield of AI.
   • 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
   • 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
   • 1: Poor: The paper is likely to have minimal impact on AI.
3. Correctness/technical quality: Is the paper technically sound?
   • 4: Excellent: I am confident that the paper is technically sound, and I have carefully checked the details.
   • 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
   • 2: Fair: The paper has minor, easily fixable, technical flaws that do not impact the validity of the main results.
   • 1: Poor: The paper has major technical flaws.
4. Quality of experiments: If applicable, are the main claims well supported by experiments? (optional)
   • 4: Excellent: The experimental evaluation is comprehensive and the results are compelling.
   • 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
   • 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
   • 1: Poor: The experimental evaluation is flawed or the results fail to adequately support the main claims.
5. Reproducibility: Are the results (e.g., theorems, experimental results) in the paper easily reproducible?
   • 4: Excellent: key resources (e.g., proofs, code, data) are available and key details (e.g., proof sketches, experimental setup) are comprehensively described for competent researchers to confidently and easily reproduce the main results.
   • 3: Good: key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
   • 2: Fair: key resources (e.g., proofs, code, data) are unavailable but key details (e.g., proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
   • 1: Poor: key details (e.g., proof sketches, experimental setup) are incomplete/unclear, or key resources (e.g., proofs, code, data) are unavailable.
6. Clarity of writing: Is the paper well-organized and clearly written?
   • 4: Excellent: The paper is well-organized and clearly written.
   • 3: Good: The paper is well organized but the presentation could be improved.
   • 2: Fair: The paper is somewhat clear, but some important details are missing or unclear.
   • 1: Poor: The paper is unclear and very hard to understand.

Q3: Main strengths

Please describe the main strengths of the work, considering the same axes as in Q2. Please mention as many main strengths as there are, but avoid minor points here.

(Max. 1000 characters.)

Q4: Main weaknesses

Please describe the main weaknesses of the work, considering the same axes as in Q2. Please mention as many main weaknesses as there are, but avoid minor points here.

Your comments should be detailed, specific, and polite. Please avoid vague, subjective complaints. Think about the times when you received an unfair, unjustified, short, or dismissive review. Try not to be that reviewer! Always be constructive and help the authors understand your viewpoint, without being dismissive or using inappropriate language. Remember that you are not reviewing your level of interest in the submission, but its scientific contribution to the field.

For example, if you think that the work is incremental relative to prior work, please cite the specific relevant prior work with a complete citation. Or if you think the experiments are not realistic or useful, let the authors know what they could do to improve them (e.g., more realistic data sets, larger data sets, different evaluation metrics, sensitivity analyses, etc.).

(Max. 1000 characters.)

Q5: Detailed comments to the authors

Please provide constructive criticism and feedback that could help improve the work or its presentation (e.g., presentation suggestions, missing references, minor mistakes and typos or grammar improvements). You may also include questions to the author here.

If you would like the authors to clarify something during the author response phase, please articulate this clearly (e.g., “I would like to see results of experiment X” or “Can you please include details about the parameter settings used for experiment Y”). We suggest that you directly ask for clarifications only on points that are important for reaching a decision, since the authors’ response has a limited length.

(Max. 200000 characters.)

Q6: Overall score

Please provide your overall evaluation of the paper, carefully weighing the reasons to accept and the reasons to reject the paper.

We encourage you to use the full range of scores for your papers, if appropriate. You should not assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. If you are new to the UAI conference (or have not attended for a number of years) you may find it useful to take a look at online proceedings from recent UAI conferences to help calibrate your scores.

Ideally, we should have:

• No more than 25% of the submitted papers in (Accept + Strong Accept + Very Strong Accept + Award Quality) categories;
• No more than 20% of the submitted papers in (Strong Accept + Very Strong Accept + Award Quality) categories;
• No more than 10% of the submitted papers in (Very Strong Accept + Award Quality) categories;
• No more than 1% of the submitted papers in the Award Quality category.

Choices:

• 10: Award quality: Technically flawless paper with groundbreaking impact on one or more areas of AI, with exceptionally strong evaluation, reproducibility, and resources, and no unaddressed ethical considerations.
• 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI and excellent impact on multiple areas of AI, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
• 8: Strong Accept: Technically strong paper with, with novel ideas, excellent impact on at least one area of AI or high to excellent impact on multiple areas of AI, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
• 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
• 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
• 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
• 4: Borderline reject: Technically solid paper where reasons to reject, e.g., lack of novelty, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
• 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility, incompletely addressed ethical considerations.
• 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility, mostly unaddressed ethical considerations.
• 1: Very Strong Reject: For instance, a paper with trivial results, limited novelty, poor impact, or unaddressed ethical considerations.

Q7: Justification for your score

Please explain in a few lines how you arrived at your overall assessment. Which aspects mentioned under main strengths and main weaknesses (Q3 & Q4) did you weigh most heavily and why?

(Max. 500 characters.)

Q8: Confidence in your score

Please rate your confidence in your assessment.

The scoring system is as follows:

• 5: Very confident. I have checked all points of the paper carefully. I am certain I did not miss any aspects that could otherwise have impacted my evaluation.
• 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
• 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
• 2: Not very confident. I am able to defend my evaluation of some aspects of the paper, but it is quite likely that I missed or did not understand some key details, or can't be sure about the novelty of the work.
• 1: Not confident. My evaluation is an educated guess.

Q9: Complying with reviewing instructions

I have read the UAI reviewing instructions and certify that I comply with them. In particular, I keep the paper and supplementary materials (including code submissions) confidential, and delete all reviewing material at the end of the review cycle. (https://auai.org/uai2022/reviewing_instructions)

Q10: Ethical concerns

Does the submission raise potential ethical concerns? This includes methods, applications, or data that can create or reinforce unfair bias or that have a primary purpose of harm or injury. Please answer yes/no, and provide a brief explanation if your answer is yes. (optional)

Note that your overall rating should be independent of your answer to this question. Your duty here is only to flag papers that might need further evaluation regarding ethical aspects.

(Max. 500 characters.)

Updated March 15th 2022

UAI 2022 Program Chairs

(Similar reviewing instructions have been used by other conferences, such as UAI 2021, AAAI 2022, ICLR 2021, ICML 2021 and NeurIPS 2021. Parts of the content have been taken from these documents.)

Sponsors