How I Write Reviews: A Pragmatic and Practical Guide for My Mentees

This guide is not a set of universal rules; it outlines the approach that works for me, though it may not suit everyone. When collaborating on a review, it’s essential to agree on a common process. As a mentee, part of your professional responsibility is to communicate if any part of the process feels misaligned or unproductive. However, note that mentors may ultimately overrule certain suggestions to ensure consistency and coherence in the review approach.

Key Mindset Principles for Reviewers

Constructive Approach:

• Approach the review as a guide, not an enforcer—your role is to help authors improve their work.
• Be constructive and provide actionable feedback that authors can use effectively.
• Focus on finding reasons to accept the paper and list strengths, not just weaknesses.

Respect and Humility:

• Maintain humility; you may misunderstand parts of the paper, and that’s okay.
• Remember, your review is a tool for authors and editors, not a platform for expressing authority.
• Respect the privacy of the review process and protect the confidentiality of the paper.

Engagement and Contribution:

• Your input is valuable, even as a junior reviewer; your perspective adds to the discussion.
• Don’t be afraid to speak up during discussions—your questions and insights matter.

Objectivity and Transparency:

• Always justify your feedback with objective properties of the paper, not personal opinion alone.
• Acknowledge your limitations; it’s okay to be unsure and express that in your review.

Goal of the Reviews

In scientific communities, reviews play a critical role in assessing the rigor, validity, and relevance of research. They serve not only the authors, who receive constructive feedback to refine their work, but also the editors, who rely on reviews to make informed publication decisions. For authors, a well-crafted review highlights strengths, points out limitations, and suggests improvements, guiding them toward higher standards of research. For editors, reviews ensure that published work meets the field’s quality benchmarks, ultimately advancing knowledge and maintaining integrity in the literature.

Before the Review

Before the review process formally begins, a series of preliminary steps set the stage for reviewer involvement. Generally, authors submit their manuscripts to journals or conferences, where editors assess their suitability and identify potential reviewers. Reviewers may be directly invited based on expertise, or in some fields, they might have the option to ‘bid’ on papers or highlight their specialized knowledge.

PC Member vs. Reviewer

A PC Member (Program Committee Member) is responsible for overseeing the review process, making decisions, and guiding discussions, often for conferences. A Reviewer is tasked with evaluating papers for quality, rigor, and relevance but may not be involved in decision-making.

Desk Reject
A Desk Reject occurs when an editor decides that a submission is unsuitable for review and rejects it outright, usually due to a lack of fit with the journal’s scope, poor quality, or non-adherence to submission guidelines.

In the field of Software Engineering (SE), the process includes additional specific steps. Reviewers often first commit to serving as potential reviewers for a particular journal or conference cycle. Once committed, SE reviewers can indicate interest in specific papers by bidding on topics that match their research expertise. They also mark areas of special expertise to guide editors in assigning the most relevant papers. Importantly, reviewers in SE must identify any conflicts of interest to maintain objectivity, ensuring that reviews remain fair and unbiased.

Bidding Phase
The Bidding Phase allows reviewers to express interest in specific papers based on their expertise and preferences, helping editors match reviewers with the most relevant submissions.

Conflict of Interest
A Conflict of Interest arises when a reviewer has a relationship or bias—such as a personal, financial, or professional connection—that could affect their objectivity in reviewing a paper.

How to Choose a Paper to Review?

Selecting the right paper to review is a balance of expertise, interest, and professional responsibility. First, prioritize your expertise over passion; to assess a paper’s quality accurately, you need a solid foundation in its subject area. However, don’t disregard your interest entirely. Reviewing is far more engaging and effective when driven by motivation, making the process easier and more rewarding.

Each review is also an opportunity to learn. Approaching a paper with curiosity allows you to gain insight into others’ research methods, improve your own writing and analysis, and explore new tools, methods, or theories—sometimes even from marginal elements in the paper. Remember that reviewing isn’t always purely voluntary; at times, you may need to review papers that don’t align perfectly with your expertise or interests. Embrace these cases as chances to broaden your perspective and sharpen your adaptability in evaluation.

As researchers, we have a duty to contribute to the peer-review process, but also to be honest when a paper isn’t a good match for our skills. If you feel unable to assess a paper’s quality, for any reason, it’s essential to notify the appropriate stakeholder—whether that’s your mentor or the editor. An absent review is better than an invalid one that could mislead the evaluation process.

Preparing for the Review

Proper preparation is essential for an effective and aligned review process. Start by thoroughly checking the guidelines provided by the journal or conference. These guidelines outline specific expectations for reviewers, including criteria for assessment and standards of quality. Reading the call for papers can also provide insight into the focus and scope of the event or publication, helping you align your review with its objectives.

Here’s the revised alert box that includes definitions and groups the paper types by page limits:

Common Paper Types in SE Conferences & Page Limits

• 4–6 Pages:
  • Short Papers: Preliminary or smaller contributions.
  • ERA Papers: Early-stage or ongoing work.
  • NIER Papers: Novel or speculative ideas.
  • Tool Demonstrations: Showcasing developed tools.
  • Positional Papers: Arguing for a specific viewpoint.
  • Workshop Papers: For focused discussions.
• 6–10 Pages:
  • Industrial Track Papers: Addressing industry-specific problems.
  • Negative Results Papers: Highlighting non-confirmatory findings.
  • Experience Reports: Practical industry applications (can vary).
• 8–10 Pages:
  • Replication Studies: Validating previous research.
  • Experience Reports: Practical industry applications. (Can overlap with 6–10 pages depending on depth).
• 10–12 Pages:
  • Full Research Papers: Comprehensive new findings.
  • SLR/Mapping Studies: Comprehensive literature analysis.

Note: Always refer to specific conference or journal guidelines for precise page limits.

Understand the type of paper you are reviewing, whether it is a short communication, Early Research Achievement (ERA) paper, New Ideas and Emerging Results (NIER) submission, or a positional paper. It’s equally important to be aware of the track to which the paper was submitted or if it is part of a workshop, special event, or a special issue of a journal. These details influence the lens through which you should assess the work, as different submission types have different expectations and evaluation criteria.

Reread any emails sent to you by the authors, editors, or mentors to ensure you capture any additional information or instructions related to the review. This is especially important for understanding whether the review process is single-blind or double-blind, which affects the level of anonymity and how you should approach your feedback.

Single vs. Double-Blind Review

• Single-Blind Review: The reviewers know the identity of the authors, but the authors do not know who the reviewers are. This helps reviewers provide objective feedback but may introduce bias if the reviewers recognize the authors’ names.

• Double-Blind Review: Both the reviewers and the authors remain anonymous to each other. This aims to minimize bias based on the authors’ identity, fostering a fairer review process.

Note: Always confirm the review type in the guidelines provided by the journal or conference.

Reviewing

Read the Paper, Make Your Notes

The first step in reviewing a paper is to read it thoroughly and take detailed notes. Start by reading independently to form an unbiased perspective, especially if you are part of a review team. This helps ensure that your initial thoughts are your own before discussing with colleagues or mentors.

Overcoming Impostor Syndrome as a Reviewer

• Guidance and Safety Nets: If you’re a junior reviewer, remember that your mentor is there to guide you and ensure your feedback is constructive and appropriate. Don’t hesitate to take initiative and share your thoughts—your mentor won’t let you harm yourself or others in the process.
• Valuing Your Perspective: It’s natural to feel unsure about your expertise, but if a section of the paper is difficult for you to understand, it could also be unclear to other researchers. Your perspective is valuable, and you can always ask for guidance while still contributing your own unique insights.
• Accessibility of Content: Keep in mind that a well-written paper should be understandable to a general SE researcher, not just core experts in the specific subject area. Your feedback can help ensure clarity for a broader audience.

You must read the entire paper, but this doesn’t have to be done in a linear fashion. A strategic reading order can enhance understanding: begin with the abstract to get an overview, move to the conclusion or results to grasp the main findings, and then go back to the introduction if you’re unfamiliar with the topic. Follow this by reviewing the detailed description of the research, related works, and finally, the threats to validity. This method allows you to contextualize the study’s impact and rigor more effectively.

Suggested Reading Order for Reviewing a Paper

• Abstract: Get an overview of the paper.
• Conclusion/Results: Understand the main findings.
• Introduction: Gain context, especially if the topic is new to you.
• Description of the Research: Dive into the methodology and details.
• Related Works: See how the paper fits within the existing body of research.
• Threats to Validity: Assess the limitations and potential biases.
• Supplementary Materials/Online Appendices: Check for additional data or clarifications.

Don’t forget to check any online appendices and supplementary materials, as they often contain additional data, code, or clarifications that support the paper’s claims and methodology. Taking comprehensive notes throughout this process helps highlight strengths, gaps, or inconsistencies that you can refer back to when drafting the formal review.

How to Write Notes?

The note-taking method I use is one that has been refined through practical trial and error, rather than being a predefined approach. This method has proven effective not only for my own understanding but also for ensuring that others can easily follow my thoughts. It’s essential to avoid relying solely on memory—your notes should be detailed and self-contained. This way, you can refer back to them without losing context and keep your mind uncluttered, which supports better mental health. Write your notes in the same language as the paper or the review (these are typically the same) to avoid confusion during translation and to more effectively use technical terms.

How I Approach Note-Taking

• Highlight or Underline: As you read, highlight or underline key sections or areas that need attention.
• Numbering Highlights: Assign a number to each highlighted part in the margin, ensuring clarity and easy reference.
• Detailed Justifications: Write explanations or justifications for each highlight on a note page adjacent to the page under review, or on the back of the page if printed. Add extra note pages as needed for more space.
• Restart Numbering: Begin numbering from 1 on each new page to keep your notes organized.
• Circled Note Numbers: Circle each note number to distinguish them clearly.
• Cross-Referencing Notes: You can reference any note from any page using the format (p.n), where p is the page number and n is the note number. This helps maintain a cohesive flow of references throughout your review.

Ask Advice from Colleagues

During the process of reading and note-taking, it is valuable and often necessary to seek advice from colleagues or experts about related topics. This can provide clarity on certain points and enhance your understanding of the paper’s context. Don’t hesitate to look up or ask about unfamiliar technical terms, methods, or theories; being a reviewer doesn’t mean you need to know everything.

However, it’s crucial to respect the confidentiality of the paper. Never share specific details about the paper without prior permission. For example, asking a colleague what CodeBERT is and how it is generally used is appropriate, but discussing how the authors of the paper applied it in their research is not.

Lastly, avoid giving spoilers to any co-reviewers on your team during this phase. Keep your insights and findings contained until formal discussions begin, ensuring an unbiased and independent review process for everyone involved.

Revise Your Notes

After completing your initial read-through and note-taking, go back and review your notes and highlights. You may find that some points are no longer valid or relevant as your understanding of the paper deepens. Keep in mind that early sections, such as the introduction, should be understandable without requiring knowledge contained in later parts of the paper.

A helpful practice is to label each note according to specific aspects of the paper: Soundness, Significance, Novelty, Verifiability, and Presentation. For added clarity, you can even grade each note based on these aspects using symbols such as: :D (excellent), :) (good), :| (fishy), :( (bad), and :’( (awfull).

Key Aspects for Labeling Your Review Notes

• Soundness: Assess the logical and methodological rigor of the research.
  • Is the methodology appropriate for the research question?
  • Are the conclusions supported by the data presented?
• Significance: Determine the importance and impact of the work within the field.
  • Does the study address a relevant and current problem in the field?
  • Could the findings influence future research or practice?
• Novelty: Evaluate the originality and new contributions of the paper.
  • Is this work distinct from existing literature?
  • Does it offer a new perspective, method, or application?
• Verifiability: Check the transparency and reproducibility of the results.
  • Are the data and methods detailed enough for replication?
  • Does the paper provide links to code, data, or supplementary material?
• Presentation: Review the clarity and organization of the paper’s content.
  • Is the paper well-structured and easy to follow?
  • Are figures, tables, and explanations clear and relevant?

Note: The questions listed above are examples to illustrate each aspect and are not an exhaustive checklist. Use them as guidance to develop your own comprehensive assessment tailored to each review.

Tip: Use symbols like :D (excellent), :) (good), :| (neutral), :( (weak), and :’( (poor) to grade each note for easy reference.

Apply as many aspect labels as are relevant to each note. For example, a note labeled for Presentation and Verifiability could raise questions about Soundness if the issues are severe enough. This systematic labeling not only helps organize your critique more effectively but also prepares your mind to discuss your notes with others in a structured and clear manner.

Discuss Your Notes with a Mentor or Colleague

Engaging in a discussion about your notes is essential when reviewing in a team or tandem. This collaborative process deepens insights and helps create a more balanced evaluation. Start by bringing your well-organized notes to the meeting; while they should be clear for others to read, don’t force yourself into adopting another person’s mindset. Your unique perspective is valuable and adds depth to the discussion.

During these discussions, respect differing opinions. A colleague’s view might contrast with yours, but different doesn’t necessarily mean incorrect. Embracing this diversity of thought enriches the review process. Make sure to participate actively—this isn’t just a listening exercise or a lecture. Your role is to contribute meaningfully, so speak up and share your thoughts. Remember, there’s no such thing as a bad question except the one left unasked.

If you’re a junior reviewer or a mentee, acknowledge your role without feeling diminished. Being new is simply a fact, not a flaw, and your input is still valuable. After all, you’re participating in this process because you have potential and a perspective worth sharing.

Planned Stages for Review Discussion

1. Initial Impressions: Each member shares a brief summary of the paper’s overall quality and main takeaways.
2. Page-by-Page Review: Go through the paper, allowing each reviewer to explain their notes for each page before moving on to the next.
3. Aspect-Based Opinions: Discuss each aspect of the paper; Soundness, Significance, Novelty, Verifiability, and Presentation; with input from all members.
4. Consensus Building: Engage in dialogue to reconcile differing opinions and arrive at a common judgment or evaluation.

An effective discussion often starts with each member sharing their general impressions of the paper’s quality. Then, reviewing notes page by page ensures thoroughness—each person explains their notes, followed by moving to the next page. This helps ensure no detail is overlooked. After this, team members share their opinions on key aspects like Soundness, Significance, Novelty, Verifiability, and Presentation. These discussions should continue until a common judgment is reached, with space for exploring different viewpoints and reaching consensus collaboratively.

Turning Aspect Evaluations into a Final Judgment

When translating your assessments into a single judgment of acceptance or rejection, several factors should be considered to make an informed decision.

First, take into account the ranking of the conference or journal, which often indicates its prestige and expected quality standards. High-ranking venues typically have more rigorous standards and a lower acceptance rate. It’s important to be aware of the usual acceptance rate of the venue, as this can give you an idea of how selective the process is. However, never reject a paper simply to meet a venue’s acceptance metrics; base your judgment on whether the paper meets the quality standards of previous publications in the same venue.

Conference Ranking Example: CORE

The CORE Ranking system categorizes conferences based on their quality and impact in the research community, using a combination of expert assessments and historical metrics like citation counts and attendee feedback. These rankings help reviewers gauge the expected quality and rigor of submissions:

• A*: Top-tier, highly prestigious conferences with very selective acceptance rates.
• A: High-quality conferences with significant recognition.
• B: Good quality conferences that are well-regarded but less selective.
• C: Reputable conferences with broader acceptance rates.

Note: While CORE is valuable for identifying conference prestige, it may not always accurately capture the quality of interdisciplinary conferences (based on my experience), such as those focused on education or human-centric software engineering.

Journal Ranking Example: Q1–Q4 and D1–D10

Journals are ranked using quartiles or deciles, often based on impact metrics like the Journal Impact Factor (JIF) or Scopus CiteScore. These measures consider citation rates and publication influence:

• Q1: Top 25% of journals in a specific field, highly reputable and selective.
• Q2: Journals in the 26%–50% range, still respected but less competitive.
• Q3: Journals in the 51%–75% range, moderate impact.
• Q4: Bottom 25%, generally less prestigious.
• D1–D10: Decile rankings, with D1 representing the top 10% of journals.

While quartile and decile rankings provide a useful indication of journal prestige and influence, they may not capture the nuanced quality of niche or emerging research areas. Additionally, metrics used for ranking can vary, so it’s essential to consider them as one part of a broader quality assessment.

If all aspects of the paper are adequately addressed, you should recommend acceptance. If the paper excels in every aspect, a strong recommendation for acceptance is warranted, as awards and recognitions are often chosen from these top-tier submissions. On the other hand, if at least one aspect is critically inadequate, a rejection should be suggested. Similarly, if more than two of the five key aspects—Soundness, Significance, Novelty, Verifiability, and Presentation—are consistently lacking, the paper may not meet the publication standards and should be rejected.

Synonyms for Review Aspects

Different venues or mentors might use these terms interchangeably, so it’s helpful to recognize their equivalence when assessing and discussing a paper.

• Soundness: Rigor, Validity, Robustness, Reliability
• Significance: Importance, Impact, Relevance, Contribution
• Novelty: Originality, Innovation, Freshness, Uniqueness
• Verifiability: Transparency, Reproducibility, Traceability, Accountability
• Presentation: Clarity, Organization, Coherence, Readability

Your expertise level should also influence your final judgment. If you find yourself unsure about certain aspects, it’s appropriate to note this in your recommendation or opt for a “weak accept” or “weak reject” to communicate your uncertainty effectively.

Ultimately, if you’re working with a mentor, remember that they have the right to conclude the discussion and make the final decision, as they are responsible for the review’s quality and accountability to the editors. A good mentor will listen and carefully weigh the opinions and votes of all team members before making their final judgment.

Willing to Fight for It? Distinguishing Weak vs. Strong Accept/Reject

When making your final decision, consider your professional pride and responsibility. Ensure your judgment upholds the value of your field and maintains the quality of the venue, preventing any dilution of its standards.

• Strong Accept/Reject: Choose this when you are willing to fight for your opinion and serve as the champion or spokesperson for the decision. This reflects a high level of confidence and conviction. Often, these decisions are referred to simply as “accept” or “reject” without an adjective.
• Weak Accept/Reject: Use this when you support the decision but aren’t passionate enough to defend it vigorously. This indicates that while you believe in your judgment, you are open to considering other perspectives.

Writing and Phrasing the Review in the Expected Format

Drafting the final review is a valuable exercise, especially for juniors and mentees, as it offers hands-on practice in articulating evaluations and honing academic writing skills. Begin by using all the notes you have taken and incorporating input from any co-reviewers to ensure a balanced and thorough review. This collaborative synthesis will strengthen your final assessment.

Style and Format of the Review

Your review serves a dual purpose: providing constructive feedback for the authors to improve their paper and offering insight to editors for making informed decisions. Maintaining a formal tone by using third person plural (e.g., they, their, the authors, the paper) helps uphold professionalism.

When structuring your review, use different text formats as needed:

• 1-2 Sentence Bullet Points:
  • “The methodology section lacks clarity in explaining the data collection process.”
  • “The results are compelling, but additional statistical tests would strengthen the findings.”
• 1-2 Phrase Bullet Points:
  • “Minor typographical error in Table 2.”
  • “Inconsistent citation format on page 5.”
• Running/Fluid Texts:
  • “The introduction sets a strong foundation, providing necessary background and context. However, the transition into the problem statement could be more seamless to improve reader engagement. Additionally, while the results are presented effectively, the discussion lacks sufficient connection to related work, which would better highlight the study’s contribution.”

Ensure each list item is self-contained and understandable without vague references (e.g., avoid using “this” or “that” to refer to other points). This practice makes your feedback clearer and more actionable for authors.

Keep in mind that your review should be both informative and precise, helping authors refine their work and enabling editors to make confident decisions.

Before starting, take a moment to reread the Call for Papers (CFP) and any emails from the editors or conference organizers. This will help align your review with the expectations and criteria specific to the venue. Keeping these guidelines in mind will ensure that your feedback is both comprehensive and relevant.

While the specific fields or parts of the review that need to be completed may be named differently across venues or forms, the essential sections are usually present and expected. Understanding this will help you approach each section confidently, regardless of the format.

Review Fields as a Recipe Metaphor

Think of the different sections of a formal review as parts of a recipe:

• Strengths and Weaknesses: These are like a shopping list for the authors. They show what the authors are doing well and what needs improvement. Just like when shopping for ingredients, authors will first note these items without immediately addressing them in detail.
• Detailed Field: This section acts as the cooking instructions, explaining how to use the items from the shopping list. It provides the authors with clear guidance on how to improve their paper using the feedback you’ve highlighted.
• Summary of the Paper: This is the recipe overview, giving the authors a general idea of what their paper “cooks up” and allowing them to decide whether it aligns with their goals or expectations.

Using this metaphor, your review becomes a helpful guide, allowing the authors to understand what they need, how to use it, and the final result they should aim for.

Summarize the Paper

The first step in writing your review is to provide a summary of the paper, typically in 2-4 paragraphs of running text and written in your own words. This section serves as a sanity check and feedback for the authors, ensuring that their intended main message is clear and that you, as the reviewer, have understood what they aimed to convey.

Summarizing the paper helps confirm that the core ideas, methodologies, and results have been communicated effectively. It also demonstrates to both the authors and editors that you have thoroughly engaged with the content. Focus on the main objectives, key methods, primary results, and conclusions of the paper without delving into your critique just yet. This way, your summary provides authors with insight into how their paper is perceived by someone familiar with the field.

Collect Issues and Notes

Before including these issues in the review, ensure that you clean up your notes and those from your co-reviewers. Phrase each item into coherent sentences or structured phrases, making them easy for the authors to understand and address. This step helps organize feedback effectively and ensures clarity in your review.

Start by collecting all notes and issues raised by you and your co-reviewers, then categorize them into minor, non-minor, and major issues:

• Minor Issues: These include typographical errors, grammatical mistakes, and other small corrections that do not affect the paper’s overall validity or clarity. Use a concise list format with 1-2 phrases per item for these types of issues.
  Examples:
  • “Typo on page 3: ‘intial’ should be ‘initial.’”
  • “Inconsistent verb tense in the conclusion section.”
• Non-Minor Issues: These are more significant and may impact the quality or comprehensiveness of the paper but are not necessarily grounds for rejection. Present these in a list with 1-2 sentences per item to provide more context.
  Examples:
  • “The discussion lacks a comparison with recent work in human-centric SE, which limits the context of the findings.”
  • “The data analysis section needs more detail on the statistical methods used to ensure reproducibility.”
• Major Issues: These are serious shortcomings that could independently justify a rejection. Highlighting these issues clearly in your detailed comments is critical, ensuring that they are well-justified and explained.

Typically, these lists of issues and notes are placed in the strengths or weaknesses fields of the review form. However, in cases where no dedicated fields exist for listing these points, they should be included in the detailed review field. This ensures that all critical feedback is documented and visible for both authors and editors.

Review with a Mindset for Acceptance, Not Rejection

As a reviewer, your primary goal is to help improve the paper and guide the authors in perfecting their research. Focus on identifying reasons to accept rather than solely looking for reasons to reject. Listing the strengths of the paper is just as important as pointing out its weaknesses. Your role is to offer constructive feedback, not to express anger or wield your authority.

Remember, you’re not inherently superior to the authors; in fact, they may be more specialized experts in their field than you are. Approach the review with humility, collaboration, and the aim of contributing to the scientific conversation.

Understanding the difference between phrase and sentence formats is essential. Use phrases for minor issues, where a brief mention is sufficient, while sentences are necessary for non-minor issues, providing the context and clarity needed for effective revisions.

Grouping the Listed Issues and Strengths into the Aspects

Once you’ve collected and cleaned up all the issues and strengths, the next step is to organize them according to the main aspects of the paper: Soundness, Significance, Novelty, Verifiability, and Presentation. This grouping helps structure your review more effectively and ensures that your feedback is comprehensive and focused.

Soundness: This aspect evaluates the logical and methodological rigor of the paper. Issues like poorly explained methods, gaps in data analysis, or unsupported conclusions should be grouped here. Strengths such as a well-justified methodology or robust data analysis also fall under this category. Significance: Focuses on the importance and potential impact of the work. Group points that assess whether the research addresses a meaningful problem or contributes significantly to the field. Mention strengths like addressing a relevant gap or potential applications, and weaknesses such as limited real-world applicability or niche interest. Novelty: Assesses the originality of the research. Group issues or strengths that discuss how unique or innovative the work is. If the paper introduces a new approach or method, highlight that as a strength. Conversely, if it rehashes known ideas without adding value, that would be a point to mention under this aspect. Verifiability: This aspect covers the transparency and reproducibility of the research. Issues related to unclear or incomplete data and methods should be grouped here, as well as strengths like detailed supplementary material or well-documented experimental procedures. Presentation: Concerns the overall readability, clarity, and organization of the paper. Group issues such as poor structure, unclear figures, or grammatical errors here. Strengths like well-organized sections or effective visual aids should also be noted in this category.

Understanding the Interrelationships Among Review Aspects

The aspects of Soundness, Significance, Novelty, Verifiability, and Presentation are interconnected and often influence each other. For instance, Soundness in methodology can directly impact Verifiability, as a well-explained and logical approach makes replication possible. Similarly, a paper’s Significance might be undermined if Presentation is poor, making key contributions difficult to understand.

It’s important to recognize that you may sometimes be unable to assess one or more aspects due to gaps in the paper or your own expertise. This inability to evaluate an aspect is itself an issue and should be noted in your review. It highlights potential limitations in the paper’s clarity or scope that could affect other readers and reviewers as well.

Approach each aspect with an open mind and consider how weaknesses or strengths in one area might impact others. Acknowledging these connections will provide a more holistic and accurate evaluation.

Remember that some points can belong to more than one aspect. For instance, a poorly explained method could affect Soundness and Verifiability. Avoid copying and pasting the same point into multiple sections; instead, refer to the point by mentioning it within another aspect when needed (e.g., “As noted under Soundness, the unclear explanation of the method also impacts Verifiability”).

Justify the Items per Aspect in the Detailed Field

After grouping issues and strengths into their respective aspects, the next step is to justify these items in the detailed comments field, using running text. This section is vital for providing constructive feedback that is actionable for the authors and informative for the editors.

Start your detailed review with an overall judgment in 1-2 paragraphs, summarizing your general view of the paper. This initial overview sets the stage for the more detailed assessments that follow. Then, dedicate 2-3 paragraphs to each aspect: Soundness, Significance, Novelty, Verifiability, and Presentation. Use fluid, running text to maintain a professional and readable tone.

When justifying your opinions, reflect on the major issues and strengths you identified. Support your assessment with objective observations and properties of the paper—your aim is not to defend your opinion as an attorney would, but to guide the authors in improving their work. For example, if a method is unclear, mention what specific details are missing and how their inclusion could enhance clarity. Always provide enough context so that, if someone were to follow your suggestions, they could reasonably act on them.

Be respectful in your tone; remember that you are not perfect and may misunderstand parts of the paper. State your perspective humbly, acknowledging that your opinion is just one viewpoint and not an absolute truth. For instance, if a complex argument seems unclear, express it as, “This section could be clarified to help readers interpret it accurately,” rather than as a definitive failure of the paper.

Lastly, take time to reflect on your own reactions to the paper. Identify why you might be having issues with certain sections—is it due to unusual grammatical structures, your status as a junior reviewer, or multiple plausible interpretations due to insufficient explanations? Understanding these triggers can help you offer more balanced and constructive feedback.

By combining objective observations, respectful language, and actionable advice, you position yourself as a guide for the authors and maintain the integrity of your review process.