UAI 2024 - Reviewing Instructions

Thank you for reviewing for UAI! Your assessments largely determine the content and quality of the conference and your help is vital for our community. Please carefully read the instructions below - they explain the reviewing process and will help you to write your reviews efficiently and effectively.

General remarks

The timeline for the reviewing process is:

• February 9th, 2024 (23:59 AOE): Paper submission deadline
• February 12th-16th, 2024: Bidding period
• February 26th-March 22nd, 2024: Reviewing period
• April 1st-8th, 2024: Discussion Stage 1
• Discussions among Reviewers/Authors/AC.
• ACs encourage reviewers to acknowledge and respond to author responses.
• April 9th-15th, 2024: Discussion Stage 2
• Discussions among reviewers and AC (without authors).
• April 16th, 2024: Meta-reviews due
• April 25th, 2024: Author notification

Please mark these dates in your calendar. Our schedule is tight, so it is key that you respect the deadlines and respond to emails promptly.

The entire review process takes place within the OpenReview system (see below).

Reviewers typically assess at most 6 papers. For each assigned paper, their main responsibilities consist of writing constructive and informative reviews, reading the author responses, participating in the discussions, and providing recommendations regarding acceptance or rejection.

Area Chairs typically oversee no more than 10 papers. For each assigned paper, their main responsibilities consist of finalizing the list of reviewers, facilitating the reviewing and discussion process, summarizing it in a meta-review, and providing a recommendation regarding paper acceptance or rejection to the program chairs. In some cases, this may require a close reading of the paper. Area Chairs will also be asked to evaluate the quality of each review using three scores: “exceeds expectations”, “meets expectations”, and “fails to meet expectations”.

Reviewing is an essential part of making UAI a great conference. As a token of our appreciation for the work of the program committee, the top scoring reviewers will get an honourable mention on the conference website.

By reviewing for UAI 2024, you agree to abide by the UAI 2024 Code of Conduct.

Confidentiality

By reviewing for UAI 2024, you agree to keep all material and information confidential (including but not limited to papers, meta-data, bidding data, messages and discussions related to reviewing, names of committee members involved, etc) and use them only for the purposes of reviewing your assigned manuscripts. Particularly, you may not use ideas and results from submitted papers for your own work, research or grant proposals, or distribute them to others, unless and until the material appears in publicly available formats, such as a technical report or a published manuscript. You should delete all reviewing material, including (but not limited to) any submitted code, at the end of the reviewing cycle.

You should not talk about or distribute (parts of) submissions and/or content related to the reviewing of submissions to anyone without prior approval from the program chairs. This includes other reviewers or area chairs, as they may have conflicts with these submissions. In general, your primary point of contact for any discussions regarding a submission is the assigned area chair for that submission.

You should also not talk to other reviewers or area chairs about submissions for which you are an author or have a conflict of interest.

Double blind reviewing

The reviewing process is double blind at the level of reviewers and area chairs (i.e. reviewers and area chairs cannot see author identities and vice versa). Authors are responsible for anonymizing their submissions. In particular, they must make sure that the submission (including all supplementary material) does not disclose author identities or affiliations. Hence, among other precautions, they should use the third person to refer to their own work, they should not include names in the acknowledgements, and links to external resources that may reveal their identity or institution are forbidden. If you are assigned a submission that is not adequately anonymized, please inform the corresponding area chair as soon as possible.

Under no circumstances should you attempt to find out the identities of the authors for any of your assigned submissions (e.g. by searching on Google or arXiv). If you accidentally discover author identities, please do not disclose them to anyone else and do not let them influence your decision in any way.

Conflicts of interest

Authors, Reviewers and Area Chairs will be asked to separately enter two types of conflicts into OpenReview: domain conflicts and individual conflicts. Please find more details here.

The declared conflicts will be used to make sure that you are not assigned any papers you have a conflict with. If, despite these efforts, you suspect that you may have a conflict of interest with an assigned submission, then please contact the area chair (or, if needed, the program chairs) immediately.

Supplementary material

This year, in addition to the main part of the submitted manuscript (of at most 8 pages), the manuscript may contain unlimited additional pages with references and appendices (for proofs, experimental protocols, and any further details required to substantiate the claims made in the paper). Submission may also include further supplementary material, such as source code or data, in a ZIP file of up to 50MB in size that must be separately downloaded from OpenReview.

Your responsibility as a reviewer is to read and review the main part of the submission, and in addition, to pay close attention to the additional appendices. Looking at the further supplementary material is at your discretion.

Executing code and clicking on links

We cannot provide any guarantees about submitted code and links. Hence, if you are planning to run supplementary source code, please make sure you are doing this in a secure environment, for example inside a Docker container, on a Virtual Machine image (using VirtualBox or VMWare), or on a network-isolated cloud instance. Please note that any links that are provided in a submission may contain vulnerabilities or may log visitor IP addresses (thereby revealing your identity).

Violations of formatting instructions

Submissions must be formatted using the UAI latex template and formatting instructions. Papers must be submitted as a PDF file. The main part of the manuscript is limited to at most 8 pages in length, including all figures and table. Additionally, the manuscript may contain unlimited additional pages with references and appendices (for proofs, experimental protocols, and any further details required to substantiate the claims made in the paper). Authors must not change the template. Supplementary text material should be included after the main paper, in the same PDF (after all references, starting on a new page). There is no page limit nor a fixed template for supplementary material.

If you are assigned a paper that is overlength or appears to violate the UAI proceedings format (for example, by decreasing margins or font size, or by removing some pre-fixed spaces), please notify the area chair immediately.

Dual submissions

Papers that are currently under review or have been accepted or published in a refereed venue with proceedings may not be submitted. This also applies to papers that are substantially identical. Moreover, authors are not allowed to submit the same (or substantially similar) work to another refereed venue with proceedings while it is still under review at UAI. Papers in conferences or workshops without proceedings, technical report repositories and arXiv are not considered dual submissions.

If you suspect that a submission that has been assigned to you is a dual submission or if you require further clarification, please contact the corresponding area chair.

Author response

After the initial review period, the authors may submit responses to their reviews. This is an opportunity for them to correct possible misunderstandings about the contents of the paper, or about previous work. They may also point out aspects of the paper that you missed, or they may disagree with certain aspects of your review.

It is important that you read each rebuttal carefully and with an open mind. Do the authors’ comments change your opinion about the paper? Have you overlooked or misunderstood something?

Discussion period

After the author response period, the discussion period starts. This is a very important part of the reviewing process and all reviewers should actively participate. These discussions are especially important for borderline papers and for papers with widely varying assessments.

The discussions are led by the area chairs. At the start of the discussion period, please read the other reviews and the authors’ response. Please try to understand their points of view. Do they bring up aspects that you viewed differently or missed? While engaging in the discussion, please be professional, polite and keep an open mind. You may change your opinion, but should not give in to undue influence. It is not required that there is full consensus about a paper; different people may come to different valid conclusions about a paper.

Contacting the program chairs

If you encounter any problem that you cannot solve with the assigned area chair, please contact the program chairs: uai2024pcs@gmail.com.

Content of the Review

For each paper, please answer the questions on the review form in OpenReview (see below). You may revise your review as often as necessary by the reviewing deadline.

Q1: Summary and contributions

Please summarize the paper’s motivation and key contributions in a few lines.

Although this part of the review may not provide much new information to authors, it is very valuable to the area chairs and the program chairs.

Please note that there are many examples of contributions that warrant publication at UAI. These contributions may be theoretical, methodological, algorithmic, applied, empirical, connecting ideas in separate fields (“bridge papers”), or providing a critical analysis (e.g. principled justifications of why the community is going after the wrong outcome or using the wrong types of approaches).

(Max. 500 characters.)

Q2: Assessment of the paper

Please assess the work along the following axes: originality/novelty, soundness/technical quality, whether claims are supported by convincing evidence, clarity of writing, reproducibility, ethical aspects.

More detailed information regarding each of these aspects is given below:

1. Originality/novelty: How novel are the concepts, problems addressed, methods introduced, or insights obtained in the paper?
   • 4: Excellent: The main ideas of the paper are ground-breaking.
   • 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
   • 2: Fair: The paper contributes some new ideas.
   • 1: Poor: The main ideas of the paper are not novel or represent incremental progress.
2. Correctness/technical quality: Is the paper technically sound?
   • 4: Excellent: I am confident that the paper is technically sound, and I have carefully checked the details.
   • 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
   • 2: Fair: The paper has minor, easily fixable, technical flaws that do not impact the validity of the main results.
   • 1: Poor: The paper has major technical flaws.
3. Extent to which claims are supported by evidence:
   • 4: Excellent: all claims are supported by very convincing evidence (in the form of comprehensive experimental evaluation, rigorous mathematical proofs, detailed (pseudo-)code, precise references, well-motivated and realistic assumptions) and the authors deliver what they promise.
   • 3: Good: the main claims are supported by convincing evidence (in the form of adequate experimental evaluation, proofs, (pseudo-)code, references, assumptions).
   • 2: Fair: the main claims are somewhat supported by evidence (but the experimental evaluation may be weak, or does not match entirely with the claims, important baselines may be missing, proofs contain important ideas but lack rigor, algorithmic details are only discussed superficially, references are imprecise, assumptions are not sufficiently motivated or explicated, etc.).
   • 1: Poor: the authors fail to convincingly backup their main claims (e.g., if the experimental evaluation is flawed, proofs are lacking or invalid, references are missing, assumptions are not realistic, not specified, or not motivated).
4. Reproducibility: Are the results (e.g. theorems, experimental results) in the paper easily reproducible?
   • 4: Excellent: key resources (e.g. proofs, code, data) are available and key details (e.g. proof sketches, experimental setup) are comprehensively described for competent researchers to confidently and easily reproduce the main results.
   • 3: Good: key resources (e.g. proofs, code, data) are available and key details (e.g. proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
   • 2: Fair: key resources (e.g. proofs, code, data) are unavailable but key details (e.g. proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
   • 1: Poor: key details (e.g. proof sketches, experimental setup) are incomplete/unclear, or key resources (e.g. proofs, code, data) are unavailable.
5. Clarity of writing: Is the paper well-organized and clearly written?
   • 4: Excellent: The paper is well-organized and clearly written.
   • 3: Good: The paper is well organized but the presentation could be improved.
   • 2: Fair: The paper is somewhat clear, but some important details are missing or unclear.
   • 1: Poor: The paper is unclear and very hard to understand.

Q3: Main strengths

Please describe the main strengths of the work, considering the same axes as in Q2. Please mention as many main strengths as there are, but avoid minor points here.

(Max. 3000 characters.)

Q4: Main weaknesses

Please describe the main weaknesses of the work, considering the same axes as in Q2. Please mention as many main weaknesses as there are, but avoid minor points here.

Your comments should be detailed, specific, and polite. Please avoid vague, subjective complaints. Think about the times when you received an unfair, unjustified, short, or disparaging review. Try not to be that reviewer! Always be constructive and help the authors understand your viewpoint, without being dismissive or using inappropriate language. Remember that you are not reviewing your level of interest in the submission, but its scientific contribution to the field.

For example, if you think that the work is incremental relative to prior work, please cite the specific relevant prior work with a complete citation. Or if you think the experiments are not realistic or useful, let the authors know what they could do to improve them; this might involve suggesting more realistic or larger data sets, different evaluation metrics, or a sensitivity analysis.

(Max. 3000 characters.)

Q5: Detailed comments to the authors

Please provide constructive criticism and feedback that could help improve the work or its presentation (e.g. presentation suggestions, missing references, minor mistakes and typos or grammar improvements). You may also include questions to the author here.

If you would like the authors to clarify something during the author response phase, please articulate this clearly (for example, “I would like to see results of experiment X” or “Can you please include details about the parameter settings used for experiment Y”). We suggest that you directly ask for clarifications only on points that are important for reaching a decision, since the authors’ response has a limited length.

(Max. 200000 characters.)

Q6: Overall score

Please provide your overall evaluation of the paper, carefully weighing the reasons to accept and the reasons to reject the paper.

We encourage you to use the full range of scores for your papers, if appropriate. You should not assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. If you are new to the UAI conference (or have not attended for a number of years) you may find it useful to take a look at online proceedings from recent UAI conferences to help calibrate your scores.

Ideally, we should have:

• No more than 25% of the submitted papers in (Accept + Strong Accept + Very Strong Accept + Award Quality) categories;
• No more than 20% of the submitted papers in (Strong Accept + Very Strong Accept + Award Quality) categories;
• No more than 10% of the submitted papers in (Very Strong Accept + Award Quality) categories;
• No more than 1% of the submitted papers in the Award Quality category.

Choices:

• 10: Award quality: Technically flawless, highly original paper, with exceptionally strong evidence for claims, excellent reproducibility and resources, no unaddressed ethical considerations, and the potential for a large impact in the field.
• 9: Very Strong Accept: Technically flawless and original paper, with very convincing evidence for claims, very good resources and reproducibility, no unaddressed ethical considerations, and the potential for significant impact in the field.
• 8: Strong Accept: Technically strong paper with novel ideas, with convincing evidence for claims, good resources and reproducibility, no unaddressed ethical considerations, and the potential for impact in the field.
• 7: Accept: Technically solid paper, with convincing evidence for claims, innovative aspects, good resources and reproducibility, no unaddressed ethical considerations, and the potential for impact in the field.
• 6: Weak Accept: Technically solid paper, with no major concerns with respect to provided evidence, resources, reproducibility, and ethical considerations.
• 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject. Please use sparingly.
• 4: Borderline reject: Technically solid paper where reasons to reject outweigh reasons to accept. Please use sparingly.
• 3: Reject: For instance, a paper with technical flaws, limited novelty, weak experimental evaluation, inadequate reproducibility, incompletely addressed ethical considerations.
• 2: Strong Reject: For instance, a paper with major technical flaws, no novelty, insufficient evidence to back up claims, poor experimental evaluation, poor reproducibility, mostly unaddressed ethical considerations.
• 1: Very Strong Reject: For instance, a paper with trivial results, no novelty, or serious ethical concerns.

Q7: Justification for your score

Please explain in a few lines how you arrived at your overall assessment. Which aspects mentioned under main strengths and main weaknesses (Q3 & Q4) did you weigh most heavily and why?

(Max. 500 characters.)

Q8: Confidence in your score

Please rate your confidence in your assessment.

The scoring system is as follows:

• 5: Very confident. I have checked all points of the paper carefully. I am certain I did not miss any aspects that could otherwise have impacted my evaluation. I am an expert on the subject matter and familiar with all related work.
• 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation. I am familiar with the research topic and most of the related work.
• 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g. novelty, proof of a theorem, experimental design, or statistical validity of conclusions. I am somewhat familiar with the topic but may not know all related work.
• 2: Not very confident. I am able to defend my evaluation of some aspects of the paper, but it is quite likely that I missed or did not understand some key details, or can't be sure about the novelty of the work. I don't know much about the topic.
• 1: Not confident. My evaluation is an educated guess, or the topic is outside my area of expertise.

Q9: Complying with reviewing instructions

I have read the UAI reviewing instructions and certify that I comply with them. In particular, I keep the paper and supplementary materials (including code submissions) confidential, and delete all reviewing material at the end of the review cycle.

Q10: Ethical concerns

Does the submission raise potential ethical concerns? This includes methods, applications, or data that can create or reinforce unfair bias or that have a primary purpose of harm or injury. Please answer yes/no, and provide a brief explanation if your answer is yes. (optional)

Note that your overall rating should be independent of your answer to this question. Your duty here is only to flag papers that might need further evaluation regarding ethical aspects.

UAI 2024 Program Chairs

Sponsors