UAI 2014 - Reviewing Criteria

Below we provide some guidelines to reviewers on how to write reviews, both the content of reviews and how the numerical scoring system works.

Writing Reviews: Content

For each paper you will provide written comments under each of the headings below. Your review should address both the strengths and weaknesses of the paper - identify the areas where you believe the paper is particularly strong and particularly weak - this will be very valuable to the program chairs and the SPC.

Novelty

This is arguably the single most important criterion for selecting papers for the conference. Reviewers should reward papers that propose genuinely new ideas or novel adaptations/applications of existing methods. In the case of application papers, novelty should be assessed more with respect to application domain than methodology. It is not the duty of the reviewer to infer what aspects of a paper are novel - the authors should explicitly point out how their work is novel relative to prior work. Assessment of novelty is obviously a subjective process, but as a reviewer you should try to assess whether the ideas are truly new, or are novel combinations or adaptations or extensions of existing ideas, or minor extensions of existing ideas, and so on.

Technical Quality

Are the results technically sound? Are there obvious flaws in the conceptual approach? Are claims well-supported by theoretical analysis or experimental results? Did the authors ignore (or appear unaware of) highly relevant prior work? Are the experiments well thought out and convincing? Are there obvious experiments that were not carried out? Will it be possible for later researchers to replicate these results? Are the data sets and/or code publicly available? Is the evaluation appropriate? Did the authors discuss sensitivity of their algorithm/method/procedure to parameter settings? Did the authors clearly assess both the strengths and weaknesses of their approach?

Potential Impact and Significance

Is this really a significant advance in the state of the art? Is this a paper that people are likely to read and cite in later years? Does the paper address an important problem (e.g., one that people outside UAI are aware of)? Does it raise new research issue for the community? Is it a paper that is likely to have any lasting impact? Is this a paper that researchers and/or practitioners might find useful 5 or 10 years from now? Is this work that can be built on by other researchers?

Quality of Writing

Please make full use of the range of scores for this category so that we can identify poorly-written papers early in the process. Is the paper clearly written? Does it adequately inform the reader? Is there a good use of examples and figures? Is it well organized? Are there problems with style and grammar? Are there issues with typos, formatting, references, etc.? It is the responsibility of the authors of a paper to write clearly, rather than it being the duty of the reviewers to try to extract information from a poorly written paper. Do not assume that the authors will fix problems before a final camera-ready version is published - unlike journal publications, there will not be time to carefully check that accepted papers are properly written. Think of future readers trying to extract information from the paper - it may be better to advise the authors to revise a paper and submit to a later conference, than to accept and publish a poorly-written version. However if the paper is likely to be accepted, feel free to make suggestions to improve the clarity of the paper, and provide details of any typos that you have found.

Additional Comments to the Authors (optional)

This is an optional section on the review form used to add additional comments for the authors that don’t naturally fit into any of the sections above.

Confidential Comments that are only for the SPC and program chairs (optional)

Again this is another optional section. If there are any comments that you would like to communicate to the SPC and program chairs, but that you do not wish to be seen by the authors, they can go in this section.

General Advice on Review Writing

Please be as precise as you can in your comments to the authors and avoid vague statements. Your criticism should be constructive where possible - if you are giving a low score to a paper then try to be clear in explaining to the authors the types of actions they could take to improve their paper in the future. For example, if you think that this work is incremental relative to prior work, please cite the specific relevant prior work you are referring to. Or if you think the experiments are not very realistic or useful, let the author(s) know what they could do to improve them (e.g., more realistic data sets, larger data sets, different evaluation metrics, sensitivity analyses, etc.).

Writing Reviews: Numeric Scoring

For UAI 2014 we are using a 7-point scoring system. We strongly encourage you to use the full range of scores, if appropriate for your papers. Try not to put all of your papers in a narrow range of scores in the middle of the scale, e.g., 3’s, 4’s, and 5’s. Don’t be afraid to assign 1’s/2’s, or 6’s/7’s, if papers deserve them. If you are new to the UAI conference (or have not attended for a number of years) you may find it useful to take a look at online proceedings from recent UAI conferences to help calibrate your scores. The scoring system is as follows:

• 7: An excellent paper, a very strong acceptance. I will fight for acceptance, this is potentially best-paper material.
• 6: A very good paper, should be accepted. I vote and argue for acceptance, clearly belongs in the conference.
• 5: A good paper overall, accept if possible. I vote for acceptance, although would not be upset if it were rejected because of the low acceptance rate.
• 4: Decent paper, but may be below the UAI threshold. I tend to vote for rejecting it, although would not be upset if it were accepted.
• 3: A weak paper, just not good enough. I vote for rejecting it, but could be persuaded otherwise.
• 2: A clear rejection. I vote and argue for rejection. It is clearly below the standards of the conference.
• 1: A strong rejection. I'm surprised it was submitted. I will actively fight for rejection.