A broad interest becomes academically concrete when it is attached to a research question, a method and a scale of observation. The result should not depend on the abstract’s wording, the prestige of a journal or the confidence of a conclusion.
The working order is:
question → design → sample and reporting → analysis → limitations
At each stage, record what is stated, what is missing and what the paper can reasonably support.
Start at the Question, Not the Abstract
Begin with the problem in a neutral sentence: what relationship, effect, experience or decision is being investigated? Keep that question separate from the contribution the authors claim to have made.
For a review, the Cochrane Handbook, chapter 3 defines its scope through the population, the interventions and comparisons, and the outcomes of interest. Make those fields explicit:
- Population: Who or what is under investigation?
- Intervention and comparison: What is being compared, contrasted or deliberately left unchanged?
- Outcomes: Which observations would answer the question?
- Design: Why is the proposed design appropriate for this question, and what forms of bias might it permit?
The Cochrane Handbook also says that included study designs should be justified in relation to both appropriateness to the review question and potential for bias. That justification is more useful than a design label on its own.
If the proposed design cannot answer the stated question, polishing the abstract will not repair the mismatch.
Match the Design to a Real Checklist
Identify what the researchers actually did, not merely the label used in the title. Then choose a checklist that corresponds to that design.
The Critical Appraisal Skills Programme (CASP) offers free downloadable checklists for appraisal across different study types. Its published menu includes:
- Systematic Reviews with Meta-Analysis of Observational Studies
- Systematic Reviews with Meta-Analysis of Randomised Controlled Trials (RCTs)
- Randomised Controlled Trial (RCT) Checklist
- Systematic Review Checklist
- Qualitative Studies Checklist
- Cohort Study Checklist
- Diagnostic Study Checklist
- Case Control Checklist
- Economic Evaluation Checklist
- Clinical Prediction Rule Checklist
- Cross-Sectional Studies Checklist
Select the closest match and use it to ask concrete questions. If the paper does not provide enough information to complete part of the checklist, record that absence rather than silently supplying the expected answer.
- Does the checklist match the design actually used?
- Does the design address the question posed?
- Which required information is absent?
- Which answers concern the design itself, and which concern reporting?
- Can later sections resolve any of the apparent gaps?
A checklist is useful here as a map of questions, not as a substitute for judgment.
Read the Sample and the Reporting
The sample establishes the paper’s scale: who or what was observed, under what conditions, in which comparisons and how much of the material remained available for analysis. Trace the path from eligibility to recruitment, group formation, observation and analysis.
- Do the analysed participants or cases resemble the population named in the question?
- How were participants or cases selected?
- What happened between recruitment and analysis?
- Which exclusions, missing observations or incomplete outcomes affected the available evidence?
- Are the groups described clearly enough for the comparison to be inspected?
Reporting is not separate from methodological appraisal. The Cochrane Handbook finds that the completeness of published reports is generally quite poor and that essential information for assessing risk of bias is frequently missing.
Drawing on Dechartres and colleagues, the Cochrane Handbook reports findings across 20,920 trials drawn from Cochrane reviews. The percentage judged at unclear risk of bias was 49% for random sequence generation, 57% for allocation sequence concealment, 31% for blinding and 25% for incomplete outcome data.
These figures make the appraisal problem concrete. Look for the methods needed to judge sequence generation, concealment, blinding and outcome completeness. An “unclear” judgement is not proof that the trial was definitely biased; it may indicate that the report does not provide enough information to make the judgement.
The same Cochrane Handbook source provides an important counterweight: more recent trials were less likely to be judged at unclear risk of bias, suggesting that reporting has been improving. That trend does not establish that the individual paper is complete. Apply the reporting information to that paper rather than to the research field as a whole.
Read the Analysis for What It Can Actually Support
Now ask whether the analysis realises the comparison promised by the question. Read how the evidence was assembled, which observations entered the analysis, and whether the conclusion remains within the population, comparison and outcomes actually studied.
Randomised designs deserve particular attention at this point. The Cochrane Handbook states that randomisation is the only way to prevent systematic differences between intervention groups in baseline characteristics involving both known and unknown factors. Randomisation therefore supports a particular kind of comparability; it does not remove the need to inspect allocation, concealment, blinding, incomplete outcomes or the handling of missing information.
Non-randomised designs require a different appraisal. The Cochrane Handbook’s guidance on non-randomised studies directs attention to bias domains associated with those designs. Adjustment or statistical adjustment should be read as part of that appraisal, not treated as an automatic exemption from bias.
- Does the analysed population match the sample described?
- Does the comparison answer the original question?
- Are design-relevant bias domains addressed?
- Are missing information and incomplete outcomes visible?
- Does the conclusion claim more than the design and reported evidence can support?
Write the conclusion in the narrowest defensible form. If the question, comparison or outcome is unclear, uncertainty should remain in the reading note rather than disappear into a stronger summary sentence.
Read the Limitations Last, Using a Completeness Standard
Read the limitations section after working through the question, design, sample and analysis. By then, the checklist will have produced an independent record of unresolved concerns. Compare the authors’ stated limitations with those omissions.
For a systematic review, the PRISMA 2020 statement paper provides a concrete reporting reference. It presents a checklist of 27 items, an expanded checklist with recommendations for each item, and an abstract checklist. A complete PRISMA-shaped report makes it easier to see what was done, what was not done and what cannot be evaluated. A short limitations paragraph cannot substitute for that visibility.
For other study types, use the EQUATOR Network’s reporting-guideline index before declaring a paper incomplete. The network indexes guidelines by study type, including artificial intelligence and machine learning studies. A relevant guideline helps distinguish a genuine reporting omission from a difference between report formats.
- Does an appropriate reporting guideline exist for this study type?
- Which required details are absent?
- Do the stated limitations cover the concerns found during appraisal?
- Which missing details limit interpretation of the result?
- Is the report sufficiently transparent to show the gap, even if it does not close it?
Completeness improves inspectability; it does not by itself prove that a study is unbiased or correct.
Judge Evidence Strength from What the Paper Reports
Assess the chain rather than an isolated feature. Ask whether the question fits the design, the sample supports the intended scale, the reporting permits appraisal, the analysis respects the comparison and the limitations disclose what remains unresolved.
A potentially appropriate design can still provide weak support when essential information is missing. A clear report can still describe a design that does not answer the stated question. The appropriate next step follows from the unresolved link: refine the question, seek a fuller report, narrow the claim, consult the matching checklist, or regard the paper as insufficiently interpretable.
This sequence turns a broad academic interest into checkable directions: a research object, a method, a scale and a defensible reading of the available evidence.