How to rate up the quality of evidence

Modified on Thu, 22 Aug at 8:23 AM

See this topic in the GRADE handbook: Factors that can increase the quality of the evidence

The text below is taken from the GRADE workinggroup official JCE series, article number 9:
GRADE guidelines: 9. Rating up the quality of evidence

Factors that may increase the quality of evidence

Large magnitude of effect (direct evidence, relative risk [RR]=2–5 or RR=0.5–0.2 with no plausible confounders); very large with RR>5 or RR<0.2 and no serious problems with risk of bias or precision (sufficiently narrow confidence intervals); more likely to rate up if effect rapid and out of keeping with prior trajectory; usually supported by indirect evidence.
Dose-response gradient.
All plausible residual confounders or biases would reduce a demonstrated effect, or suggest a spurious effect when results show no effect.

We have noted previously that GRADE is relevant to rating evidence regarding the impact of interventions on patient-important outcomes—rather than, for instance, prognostic studies that identify patient characteristics associated with desirable or adverse outcomes. Using the GRADE framework, evidence from observational studies is generally classified as low. Unsystematic clinical observations are usually at a high risk of bias and therefore generally receive a rating of very low quality evidence. There are times, however, when we have high confidence in the estimate of effect from such studies. GRADE has therefore suggested mechanisms for rating up the quality of evidence from observational studies.

The circumstances under which we may wish to rate up the quality of evidence for intervention studies will likely occur infrequently and are primarily relevant to observational studies (including cohort, case–control, before–after, and time series studies) and to nonrandomized experimental or interventional studies (e.g., providing treatment to one of the two matched groups). Indeed, although it is theoretically possible to rate up results from randomized control trials (RCTs), we have yet to find a compelling example of such an instance.

Dose–response gradient

The presence of a dose–response gradient has long been recognized as an important criterion for believing a putative cause–effect relationship [16]. Such a gradient may increase our confidence in the findings of observational studies and thus enhance the assigned quality of evidence (Table 1).

For example, our confidence in the results of observational studies that show an increased risk of bleeding in patients who have supra-therapeutic anticoagulation levels is increased by the finding that there is a dose–response gradient between higher levels of the international normalized ratio and the increased risk of bleeding [17]. Similarly, infant growth is slowest in infants fed exclusively with breast milk, accelerated to some extent in infants fed in part with breast milk and part formula, and further accelerated in infants fed exclusively with formula [18]. A systematic review of observational studies investigating the effect of cyclooxygenase-2 inhibitors on cardiovascular events found an RR with rofecoxib of 1.33 (95% CI: 1.00, 1.79) with doses less than 25mg/d and an RR of 2.19 (95% CI: 1.64, 2.91) with doses more than 25mg/d [19].

A final example is the striking dose–response gradient associated with the rapidity of antibiotic administration in patients presenting with sepsis and hypotension (Fig. 1) [20]. This dose–response relationship increases our confidence that the effect on mortality (large absolute increases in mortality with each hour's delay) is real and substantial.

Go to the original article for full text, or go to a specific chapter in the article:

Large magnitude of effect

Dose–response gradient

Plausible confounding can increase confidence in estimated effects
Other considerations

A final note of caution

See the Assessing Other factors and upgrading Training video from McMaster CE&B GRADE site:
http://cebgrade.mcmaster.ca/upgrading/index.html