Instructions for Reviewers and Area Chairs

Thank you for reviewing/area chairing for UAI 2025! Your service largely determines the content and quality of the conference and your help is vital for our community. Please carefully read the instructions below to ensure you are equipped with all information to provide a timely and good quality service.

Key Dates

  • February 10th, 2025 (23:59 AoE): Paper submission deadline
  • February 12th -19th, 2025: Area Chairs (ACs) and Reviewers bid on submissions
  • March 3rd - 24th, 2025: Reviewers write reviews
  • April 3rd, 2025 Reviews are released to Authors
  • April 3rd-12th, 2025: Author Response/Discussion Period
    • Authors can submit responses and discuss with Reviewers and ACs.
  • April 12th-18th, 2025: Reviewers+ACs Discussion Period
    • Discussions among Reviewers and ACs (without Authors).
  • April 18th-28th: ACs write meta-reviews
  • May 6, 2025 (23:59 AoE): Author Notification

Please mark these dates in your calendar. Our schedule is tight, so it is key that you respect the deadlines and respond to emails promptly.

General Information

  • OpenReview: The entire review process takes place within the OpenReview system (see below). You should make sure that your OpenReview profile is up to date, since it will be used for assigning submissions to you and to avoid conflicts of interest. If you have changed or plan to change your email address, please update OpenReview accordingly. We have no way of knowing whether an email sent to you from OpenReview has bounced, so it is crucial that you make sure that OpenReview has the correct email address for you at all times.
  • Conflict of Interest: Authors, Reviewers and ACs will be asked to separately enter two types of conflicts into OpenReview: domain conflicts and individual conflicts. Please find more details here. The declared conflicts will be used to make sure that you are not assigned any papers you have a conflict with. If, despite these efforts, you suspect that you may have a conflict of interest with an assigned submission, then please contact the AC immediately (if you are a Reviewer) or the Program Chairs (if you are an AC).
  • Double-blind Reviewing: The reviewing process is double blind at the level of Reviewers and ACs (i.e. Reviewers and ACs cannot see Authors’ identities and vice-versa), but not at the level of the Program Chairs. Reviewers' identities are not visible to the other Reviewers but are visible to the assigned ACs. ACs identities are not visible to the Reviewers. Under no circumstances should you attempt to find out the identities of the Authors for any of your assigned submissions (e.g. by searching on Google or arXiv). If you accidentally discover the Authors’ identities, please do not disclose them to anyone else and do not let them influence your decision in any way.
  • Code of Conduct: By serving as a Reviewer/AC for UAI 2025, you agree to abide by the UAI 2025 Code of Conduct.
  • Bidding: Bidding on papers that are relevant to your expertise and that you would like to review/area chair is crucial to our process. This ensures that you get assigned papers that you have the expertise to review and that match your interests. We will ask you to bid on at least 40 papers with a positive bid, so that we can match enough Reviewers/ACs for each paper.
  • Contacting the Program Chairs: If you encounter any problem that you cannot solve (with the assigned AC if you are a Reviewer), contact the Program Chairs at uai2025chairs+program@gmail.com.

Responsibilities, Guidelines, Best Practices

  • Confidentiality: As a Reviewer/AC for UAI 2025, you agree to keep all material and information confidential (including but not limited to papers, meta-data, bidding data, messages and discussions related to reviewing, names of committee members involved, etc.), and use them only for the purposes of reviewing/area chairing your assigned manuscripts. Particularly, you may not use ideas and results from submissions for your own work, research or grant proposals, or distribute them to others, unless and until the material appears in publicly available formats, such as a technical report or a published manuscript. You should delete all reviewing material, including (but not limited to) any submitted code, at the end of the reviewing cycle. You should not talk about or distribute (parts of) submissions and/or content related to the reviewing of submissions to anyone without prior approval from the Program Chairs. This includes Reviewers and ACs, as they may have conflicts with these submissions. In general, the primary point of contact for any discussions regarding a submission is the assigned AC for that submission (if you are a Reviewer) or the Program Chairs (if you are an AC). You should also not talk to Reviewers or ACs about submissions for which you are an Author or have a conflict of interest.
  • Violations of Double-blind Policy: Authors must make sure that the submission (including all supplementary material) does not disclose their identities or affiliations. Hence, among other precautions, Authors should use the third person to refer to their own work, should not include names in the acknowledgements, detailed information about grants, and links to external resources that may reveal the Authors' identities or affiliations are forbidden. If you are assigned a submission that is not adequately anonymized, please inform the corresponding AC (if you are a Reviewer) or the Program Chairs (if you are an AC) as soon as possible.
  • Violation of Formatting Instructions: Submissions must be formatted using the UAI latex template and formatting instructions. Manuscripts must be submitted as a single PDF file of at most 15 MB in size. The main part of the manuscript must not exceed 8 pages in length. Unlimited additional pages containing references and appendices (with proofs, experimental protocols, and any required details) are allowed and should be included after the main part in the same file. The appendices should start on a new page. If you are assigned a paper that is overlength or appears to violate the UAI proceedings format (for example, by decreasing margins or font size, or by removing some pre-fixed spaces), please notify the AC (if you are a Reviewer) or the Program Chairs (if you are an AC) as soon as possible.
  • Dual Submissions: Manuscripts that are currently under review or have been accepted or published in a refereed venue with proceedings may not be submitted. This also applies to manuscripts that are substantially identical. Moreover, submitting the same (or substantially similar) work to another refereed venue with proceedings while under review at UAI is not permitted. Manuscripts in conferences or workshops without proceedings, technical report repositories, and arXiv are not considered dual submissions. If you suspect that a submission that has been assigned to you is a dual submission or if you require further clarification, please contact the corresponding AC (if you are a Reviewer) or the Program Chairs (if you are an AC).
  • LLMs Policy: We do not allow the use of large language models (LLMs) to write reviews or meta-reviews, except as assistants for checking grammatical issues. As a Reviewer or AC, the responsibility of the text that you submit to OpenReview is on you, which includes any content that you write with LLMs.
  • Author Response/Discussion Period: In this period, Authors may submit responses to their reviews to address the reviewers concerns and to correct possible misunderstandings about the contents of the submission. They may also point out aspects of the submission that the Reviewers have missed, or they may disagree with certain aspects of the reviews. In this period Authors also have the possibility to interact with the Reviewers and ACs.

Specific to Area Chairs

  • Area Chairs’ responsibility: ACs typically oversee around 10 submissions. For each assigned submission, their main responsibilities consist of finalizing the list of Reviewers, facilitating the reviewing and discussion process, summarizing it in a meta-review, and providing a recommendation regarding paper acceptance or rejection to the Program Chairs. In some cases, this may require a close reading of the manuscript. ACs will also be asked to evaluate the quality of each review using three scores: “exceeds expectations”, “meets expectations”, and “fails to meet expectations”. The top-scoring Reviewers will get an honourable mention on the conference website. The lowest-scoring Reviewers may not be invited to review at future conferences.

Specific to Reviewers

  • Reviewers’ responsibility: Reviewers typically assess at most 6-7 papers. For each assigned paper, their main responsibilities consist of writing constructive and informative reviews, reading the Authors’ responses, participating in the discussions, and providing recommendations regarding acceptance or rejection.
  • Supplementary material: In addition to the main part of at most 8 pages, a manuscript may contain unlimited additional pages with references and appendices. Submissions may also include further supplementary material, such as source code or data, in a ZIP file of up to 50MB in size that must be separately downloaded from OpenReview. Reviewers are responsible for reading and reviewing the main part of the submission and, in addition, for paying close attention to the additional appendices. Looking at any additional supplementary material is at their discretion.
  • Executing Code and Clicking on Links: We cannot provide any guarantees about submitted code and links. Hence, if planning to run supplementary source code, Reviewers should make sure to do this in a secure environment, for example inside a Docker container, on a Virtual Machine image (using VirtualBox or VMWare), or on a network-isolated cloud instance. Please note that any links that are provided in a submission may contain vulnerabilities or may log visitor IP addresses (thereby revealing your identity).
  • Author Response/Discussion Period: It is important to read Authors’ responses carefully and with an open mind. Do the Authors’ comments change your opinion about the manuscript? Have you overlooked or misunderstood something? In this period the Authors have also the possibility to interact with the Reviewers and ACs. This is a very important part of the reviewing process and all Reviewers should actively participate. These discussions are especially important for borderline papers and for papers with widely varying assessments. The discussions are led by the ACs. While engaging in the discussion, please be professional, polite and keep an open mind. You may change your opinion, but should not give in to undue influence. It is not required that there is full consensus about a manuscript; different people may come to different valid conclusions.

Content of the Review

Q1: Summary and contributions

Please summarize the paper’s motivation and key contributions in a few lines.

Although this part of the review may not provide much new information to Authors, it is very valuable to the ACs and the Program Chairs.

There are many examples of contributions that warrant publication at UAI. These contributions may be theoretical, methodological, algorithmic, applied, empirical, connecting ideas in separate fields (“bridge papers”), or providing a critical analysis (e.g. principled justifications of why the community is going after the wrong outcome or using the wrong types of approaches).

(Max. 500 characters.)

Q2: Assessment of the paper

Please assess the work along the following axes: originality/novelty, soundness/technical quality, whether claims are supported by convincing evidence, clarity of writing, reproducibility, ethical aspects.

More detailed information regarding each of these aspects is given below:

  1. Originality/novelty: How novel are the concepts, problems addressed, methods introduced, or insights obtained in the paper?
    • 4: Excellent: The main ideas of the paper are ground-breaking.
    • 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
    • 2: Fair: The paper contributes some new ideas.
    • 1: Poor: The main ideas of the paper are not novel or represent incremental progress.
  2. Correctness/technical quality: Is the paper technically sound?
    • 4: Excellent: I am confident that the paper is technically sound, and I have carefully checked the details.
    • 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
    • 2: Fair: The paper has minor, easily fixable, technical flaws that do not impact the validity of the main results.
    • 1: Poor: The paper has major technical flaws.
  3. Extent to which claims are supported by evidence:
    • 4: Excellent: all claims are supported by very convincing evidence (in the form of comprehensive experimental evaluation, rigorous mathematical proofs, detailed (pseudo-)code, precise references, well-motivated and realistic assumptions) and the authors deliver what they promise.
    • 3: Good: the main claims are supported by convincing evidence (in the form of adequate experimental evaluation, proofs, (pseudo-)code, references, assumptions).
    • 2: Fair: the main claims are somewhat supported by evidence (but the experimental evaluation may be weak, or does not match entirely with the claims, important baselines may be missing, proofs contain important ideas but lack rigor, algorithmic details are only discussed superficially, references are imprecise, assumptions are not sufficiently motivated or explicated, etc.).
    • 1: Poor: the authors fail to convincingly back up their main claims (e.g., if the experimental evaluation is flawed, proofs are lacking or invalid, references are missing, assumptions are not realistic, not specified, or not motivated).
  4. Reproducibility: Are the results (e.g. theorems, experimental results) in the paper easily reproducible?
    • 4: Excellent: key resources (e.g. proofs, code, data) are available and key details (e.g. proof sketches, experimental setup) are comprehensively described for competent researchers to confidently and easily reproduce the main results.
    • 3: Good: key resources (e.g. proofs, code, data) are available and key details (e.g. proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
    • 2: Fair: key resources (e.g. proofs, code, data) are unavailable but key details (e.g. proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
    • 1: Poor: key details (e.g. proof sketches, experimental setup) are incomplete/unclear, or key resources (e.g. proofs, code, data) are unavailable.
  5. Clarity of writing: Is the paper well-organized and clearly written?
    • 4: Excellent: The paper is well-organized and clearly written.
    • 3: Good: The paper is well organized but the presentation could be improved.
    • 2: Fair: The paper is somewhat clear, but some important details are missing or unclear.
    • 1: Poor: The paper is unclear and very hard to understand.

Q3: Main strengths

Please describe the main strengths of the work, considering the same axes as in Q2. Please mention as many main strengths as there are, but avoid minor points here.

(Max. 3000 characters.)

Q4: Main weaknesses

Please describe the main weaknesses of the work, considering the same axes as in Q2. Please mention as many main weaknesses as there are, but avoid minor points here.

Your comments should be detailed, specific, and polite. Please avoid vague, subjective complaints. Think about the times when you received an unfair, unjustified, short, or disparaging review. Try not to be that reviewer! Always be constructive and help the authors understand your viewpoint, without being dismissive or using inappropriate language. Remember that you are not reviewing your level of interest in the submission, but its scientific contribution to the field.

For example, if you think that the work is incremental relative to prior work, please cite the specific relevant prior work with a complete citation. Or if you think the experiments are not realistic or useful, let the authors know what they could do to improve them; this might involve suggesting more realistic or larger data sets, different evaluation metrics, or a sensitivity analysis.

(Max. 3000 characters.)

Q5: Detailed comments to the authors

Please provide constructive criticism and feedback that could help improve the work or its presentation (e.g. presentation suggestions, missing references, minor mistakes and typos or grammar improvements). You may also include questions to the author here.

If you would like the Authors to clarify something during the author response phase, please articulate this clearly (for example, “I would like to see results of experiment X” or “Can you please include details about the parameter settings used for experiment Y”). We suggest that you directly ask for clarifications only on points that are important for reaching a decision, since the authors’ response has a limited length.

(Max. 200000 characters.)

Q6: Overall score

Please provide your overall evaluation of the paper, carefully weighing the reasons to accept and the reasons to reject the paper.

We encourage you to use the full range of scores for your papers, if appropriate. You should not assume that you were assigned a representative sample of submissions, nor should you adjust your scores to match the overall conference acceptance rates. If you are new to the UAI conference (or have not attended for a number of years) you may find it useful to take a look at online proceedings from recent UAI conferences to help calibrate your scores.

Ideally, we should have:

  • No more than 25% of the submitted papers in (Accept + Strong Accept + Very Strong Accept + Award Quality) categories;
  • No more than 20% of the submitted papers in (Strong Accept + Very Strong Accept + Award Quality) categories;
  • No more than 10% of the submitted papers in (Very Strong Accept + Award Quality) categories;
  • No more than 1% of the submitted papers in the Award Quality category.

Choices:

  • 10: Award quality: Technically flawless, highly original paper, with exceptionally strong evidence for claims, excellent reproducibility and resources, no unaddressed ethical considerations, and the potential for a large impact in the field.
  • 9: Very Strong Accept: Technically flawless and original paper, with very convincing evidence for claims, very good resources and reproducibility, no unaddressed ethical considerations, and the potential for significant impact in the field.
  • 8: Strong Accept: Technically strong paper with novel ideas, with convincing evidence for claims, good resources and reproducibility, no unaddressed ethical considerations, and the potential for impact in the field.
  • 7: Accept: Technically solid paper, with convincing evidence for claims, innovative aspects, good resources and reproducibility, no unaddressed ethical considerations, and the potential for impact in the field.
  • 6: Weak Accept: Technically solid paper, with no major concerns with respect to provided evidence, resources, reproducibility, and ethical considerations.
  • 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject. Please use sparingly.
  • 4: Borderline reject: Technically solid paper where reasons to reject outweigh reasons to accept. Please use sparingly.
  • 3: Reject: For instance, a paper with technical flaws, limited novelty, weak experimental evaluation, inadequate reproducibility, incompletely addressed ethical considerations.
  • 2: Strong Reject: For instance, a paper with major technical flaws, no novelty, insufficient evidence to back up claims, poor experimental evaluation, poor reproducibility, mostly unaddressed ethical considerations.
  • 1: Very Strong Reject: For instance, a paper with trivial results, no novelty, or serious ethical concerns.

Q7: Justification for your score

Please explain in a few lines how you arrived at your overall assessment. Which aspects mentioned under main strengths and main weaknesses (Q3 & Q4) did you weigh most heavily and why?

(Max. 500 characters.)

Q8: Confidence in your score

Please rate your confidence in your assessment.

The scoring system is as follows:

  • 5: Very confident. I have checked all points of the paper carefully. I am certain I did not miss any aspects that could otherwise have impacted my evaluation. I am an expert on the subject matter and familiar with all related work.
  • 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation. I am familiar with the research topic and most of the related work.
  • 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g. novelty, proof of a theorem, experimental design, or statistical validity of conclusions. I am somewhat familiar with the topic but may not know all related work.
  • 2: Not very confident. I am able to defend my evaluation of some aspects of the paper, but it is quite likely that I missed or did not understand some key details, or can't be sure about the novelty of the work. I don't know much about the topic.
  • 1: Not confident. My evaluation is an educated guess, or the topic is outside my area of expertise.

Q9: Complying with reviewing instructions

I have read the UAI reviewing instructions and certify that I comply with them. In particular, I keep the paper and supplementary materials (including code submissions) confidential, and delete all reviewing material at the end of the review cycle.

Q10: Ethical concerns

Does the submission raise potential ethical concerns? This includes methods, applications, or data that can create or reinforce unfair bias or that have a primary purpose of harm or injury. Please answer yes/no, and provide a brief explanation if your answer is yes. (optional)

Note that your overall rating should be independent of your answer to this question. Your duty here is only to flag papers that might need further evaluation regarding ethical aspects.

Last updated: February 18, 2025 08:28 (UTC)


Sponsors: TBA