Are the results of this single preventive or therapeutic trial valid?

Was the assignment of patients to treatments randomised?  

-and was the randomisation list concealed?

Were all patients who entered the trial accounted for at its conclusion? -and were they analysed in the groups to which they were randomised?

Were patients and clinicians kept “blind”

to which treatment was being received?

Aside from the experimental treatment,

were the groups treated equally?

Were the groups similar at the start of the trial?

Occurrence of diabetic neuropathy

Relative Risk Reduction

RRR

Absolute Risk Reduction

ARR

Number Needed to Treat

NNT

Usual Insulin

Control Event Rate

CER

Intensive Insulin

Experimental Event Rate

EER

CER - EER

CER

CER - EER

1/ARR

9.6%

2.8%

9.6% - 2.8%       =  71%

        9.6%

9.6% - 2.8% = 6.8%

(4.3% to 9.3%)

1/6.8% =  15 pts,

(11 to 23)

Relative Risk Reduction

RRR

Absolute Risk Reduction

ARR

Number Needed to Treat

NNT

CER

EER

CER - EER

CER

CER - EER

1/ARR

Can you apply this valid, important evidence about a treatment in caring for your patient?

Do these results apply to your patient?

Is your patient so different from those in the trial that its results can’t help you?

How great would the potential benefit of therapy actually be for your individual patient?

Method I: f

Risk of the outcome in your patient, relative to patients in the trial. expressed as a decimal: ______

NNT/F =  _____/_____  =  _________

(NNT for patients like yours)

Method II: 1 / (PEER x RRR)

Your patient’s expected event rate if they received the control treatment: PEER:______

1 / (PEER x RRR) = 1/________ = _______

(NNT for patients like yours)

Are  your  patient’s  values  and preferences satisfied by the regimen and its consequences?

Do your patient and you have a clear assessment of their values and preferences?

Are they met by this regimen and its consequences?

Are the results of this systematic review of therapy valid?

Is it a systematic review of randomised trials of the treatment you’re interested in?

Does it include a methods section that describes:

finding and including all the relevant trials?

assessing their individual validity?

Were the results consistent from study to study?

Odds Ratios  (OR)

0.9

0.85

0.8

0.75

0.7

0.65

0.6

0.55

0.5

.05

209[1]

139

104

83

69

59

52

46

41[2]

.10

110

73

54

43

36

31

27

24

21

Control

.20

61

40

30

24

20

17

14

13

11

Event

.30

46

30

22

18

14

12

10

9

8

Rate

.40

40

26

19

15

12

10

9

8

7

(CER)

.50[3]

38

25

18

14

11

9

8

7

6

.70

44

28

20

16

13

10

9

7

6

.90

101[4]

64

46

34

27

22

18

15

12[5]

Can you apply this valid, important evidence from a systematic review in caring for your patient?

Do these results apply to your patient?

Is your patient so different from those in the overview that its results can’t help you?

How great would the potential benefit of therapy actually be for your individual patient?

Method I: In the table on page 1, find the intersection of the closest odds ratio from the overview and the CER that is closest to your patient’s expected event rate if they received the control treatment (PEER):

Method II: To calculate the NNT for any OR and PEER:

                            ___1 - {PEER x (1 - OR)}____

 NNT    =             (1 - PEER) x PEER x  (1 - OR)

Are  your  patient’s  values  and preferences satisfied  by   the   regimen   and  its consequences?

Do your patient and you have a clear assessment of their values and preferences?

Are they met by this regimen and its consequences?

Should you believe apparent qualitative differences in the efficacy of therapy in some subgroups of patients?  Only if you can say “yes” to all of the following:

1.    Do they really make biologic and clinical sense?

2.    Is the qualitative difference both clinically (beneficial for some but useless or harmful for others) and statistically significant?

3.    Was this difference hypothesised before the study began (rather than the product of dredging the data), and has it been confirmed in other, independent studies?

4.    Was this one of just a few subgroup analyses carried out in this study?

Are the results of this diagnostic study valid?

1.    Was there an independent, blind comparison with a reference (“gold”) standard of diagnosis?

2.    Was the diagnostic test evaluated in an appropriate spectrum of patients (like those in whom it would be used in practice)?

3.    Was the reference standard applied regardless of the diagnostic test result?

Target Disorder

(iron deficiency anaemia)

Totals

Present

Absent

Diagnostic

Test Result

Positive

(<65 mmol/L)

           731          a

b        270

a+b             1001

(serum ferritin)

Negative

(>65 mmol/L)

            78           c

d       1500

c+d             1578

Totals

            809     a+c

b+d   1770

a+b+c+d     2579

Target Disorder

Totals

Present

Absent

Diagnostic

Test Result

Positive

          a

b

a+b

Negative

                      c

d

c+d

Totals

a+c

b+d   1770

a+b+c+d

Is the diagnostic test available, affordable, accurate, and precise in your setting?

Can you generate a clinically sensible estimate of your patient’s pre-test probability (from practice data, from personal experience,  from the report itself, or from clinical speculation)

Will the resulting post-test probabilities affect your management and help your patient? (Could it move you across a test-treatment threshold?; Would your patient be a willing partner in carrying it out?)

Would the consequences of the test help your patient?

Are the results of this prognosis study valid?

1.    Was a defined, representative sample of patients assembled at a common (usually early) point in the course of their disease?

2.    Was patient follow-up sufficiently long and complete?

3.    Were objective outcome criteria applied in a “blind” fashion?

4.    If subgroups with different prognoses are identified, was there adjustment for important prognostic factors?

5.    Was there validation in an independent group (“test-set”) of patients?

Are the valid results of this prognosis study important?

1.    How likely are the outcomes over time?

2.    How precise are the prognostic estimates?

Clinical Measure

Standard Error (SE)

Typical calculation of CI

Proportion (as in the rate of some prognostic event, etc) where:

the number of patients = n

the proportion of these patients who experience the event = p

Ö {p x (1-p) / n}

where p is proportion and n is number of patients

If p = 24/60 = 0.4 (or 40%) & n=60

SE=Ö{0.4 x (1-0.4) / 60} = 0.063 (or 6.3%)

95% CI is 40% +/- 1.96 x 6.3% or 27.6% to 52.4%

n from your evidence: ________

p from your evidence: ________

Ö {p x (1-p) / n}

where p is proportion and n is number of patients

Your calculation:

SE: ____________

95% CI:

Can you apply this valid, important evidence about prognosis in caring for your patient?

1.    Were the study patients similar to your own?

2.    Will this evidence make a clinically important impact on your conclusions about what to offer or tell your patient?

Are the results of this harm study valid?

1.    Were there clearly defined groups of patients, similar in all important ways other than exposure to the treatment or other cause?

2.    Were treatment exposures and clinical outcomes measured the same ways in both groups (e.g., was the assessment of outcomes either objective (e.g., death) or blinded to exposure)?

3.    Was the follow-up of study patients complete and long enough?

Do the results satisfy some “diagnostic tests for causation”?

·       Is it clear that the exposure preceded the onset of the outcome?

·       Is there a dose-response gradient?

·       Is there positive evidence from a “dechallenge-rechallenge” study?

·       Is the association consistent from study to study?

·       Does the association make biological sense?

Adverse Outcome

Totals

Present

(Case)

Absent

(Control)

Exposed to the

Yes

(Cohort)

a

b

a+b

Treatment

No

(Cohort)

c

d

c+d

Totals

a+c

b+d

a+b+c+d

Should these valid, potentially important results of a critical appraisal about a harmful treatment change the treatment of your patient?

1.    Can the study results be extrapolated to your patient?

2.    What are your patient’s risks of the adverse outcome?

To calculate the NNH[6] for any Odds Ratio (OR) and your Patient’s Expected Event Rate for this adverse event if they were NOT exposed to this treatment (PEER):

NNH =          PEER (OR - 1) + 1      .            

             PEER (OR - 1) x (1 - PEER)

3.    What are your patient’s preferences, concerns and expectations from this treatment?

4.    What alternative treatments are available?

Are the results of this economic analysis valid?

1.    Is this report really asking an economic question:

·       comparing well-defined alternative courses of action?

·       with a specified point-of-view (a hospital, a ministry of health, or preferably society as a whole) from which the costs and effects are being viewed?

·       With clinically useful expressions of the costs and consequences of the alternative courses of clinical action?

·       Effects equal, and a simple comparison of costs: “cost-minimisation” analysis.

·       Effects unequal but measured in the same common unit of health: “cost-effectiveness analysis.”

·       Effects both unequal and measured in more than one kind of unit of health.

·       Converted into monetary units: “cost-benefit analysis.” 

·       Converted into personal preferences or utilities (QALYs): “cost-utility analysis.”

2.    Does it cite good evidence (that would meet the Therapy, Diagnosis, or Overview Guides) on the efficacy/accuracy of the alternatives?

3.    Does it  identify all the costs and effects you think it should, and did it select credible measures for them?

Are the valid results from this economic analysis important?

1.      Are the resulting costs or costs/unit of health gained impressive?

2.      Are the conclusions unlikely to change with sensible changes in costs and outcomes?

Treatment

           Cost/QALY

           (£ Aug. 1990)

Cholesterol testing and diet therapy (all adults aged 40-69)

              220

Neurosurgical intervention for head injury

              240

 Advice to stop smoking from general practitioner

              270

 Neurosurgical intervention for subarachnoid haemorrhage

              490

 Antihypertensive treatment to prevent stroke (ages 45-64)

              940

 Pacemaker implantation

            1100

 Hip replacement

            1180

 Valve replacement for aortic stenosis

            1140

 Coronary artery bypass graft (left main vessel disease, severe angina)

            2090

 Kidney transplant

            4710

 Breast cancer screening

            5780

 Heart transplantation

            7840

 Cholesterol testing and treatment (incrementally) of all adults aged 25-39

          14,150

 Home haemodialysis

           17,260

 Coronary artery bypass graft (one vessel disease, moderate angina)

           18,830

 Continuous ambulatory periotoneal dialysis

           19,870

 Hospital haemodialysis

           21,970

 Erythropoietin treatment for anaemia in dialysis patients (assuming

     10% reduction in mortality)

           54,380

  Neurosurgical intervention for malignant intracranial tumours

          107,780

 Erythropoietin treatment for anaemia in dialysis patients (assuming

      no increase in survival)

           126,290

Should this economic analysis be applied in your practice?

1.    Do the costs in it apply in your own setting?

2.    Are the treatments likely to be as effective in your setting?

3.    Is it worth it?

·       If a cost-minimisation analysis, is the  difference in costs big enough to warrant switching over to the cheaper one?

·       If a cost-effectiveness analysis, is the difference in effectiveness great enough for you to want to spend the difference?

·       If a cost-utility analysis, where does it lie in your local, current league table?

Are the results of this clinical decision analysis valid?

1.    Were all the important clinical strategies and outcomes included?

2.    Are the probabilities credible? (Was an explicit and sensible process used to identify, select, and combine the best external evidence into probabilities?)

3.    Are the utilities credible? (Were the utilities obtained in an explicit and sensible way from credible sources?)

4.    Was the robustness of the conclusion tested?  (Was the impact of clinically sensible differences in probabilities and utilities determined?)

Are the valid results from this decision analysis important?

1.    Did one course of action lead to clinically important gains in life-expectancy or other utility measure?

2.    Was the same course of action preferred despite clinically sensible changes in probabilities and utilities?

Should this decision analysis be applied in your practice?

Do the probabilities apply to your patient?

If not, can you adjust them appropriately?

Can your patient state their utilities in a usable and stable form?

Are the recommendations in this guideline valid?

1.    Were all important decision options and outcomes clearly specified?

2.    Was the evidence relevant to each decision option identified, validated, and combined in a sensible and explicit way?

3.    Are the relative preferences that key stakeholders attach to the outcomes of decisions (including benefits, risks and costs) identified and explicitly considered?

4.    Is the guideline resistant to clinically sensible variations in practice?

Is this valid guideline or strategy potentially useful?

1.    Does this guideline offer an opportunity for significant improvement in the quality of health care practice?

·       Is there a large variation in current practice?

·       Does the guideline contain new evidence (or old evidence not yet acted upon) that could have an important impact on management?

·       Would the guideline affect the management of so many people, or concern individuals at such high risk, or involve such high costs that even small changes in practice could have major impacts on health outcomes or resources (including opportunity costs)?

Should this guideline or strategy be applied in your practice?

1.    What barriers exist to its implementation? 

Can they be overcome?

2.    Can you enlist the collaboration of key colleagues?

3.    Can you meet the educational, administrative, and economic conditions that are likely to determine the success or failure of implementing the strategy?

·       credible synthesis of the evidence by a respected body

·       respected, influential local exemplars already implementing the strategy

·       consistent information from all relevant sources

·       opportunity for individual discussions about the strategy with an authority

·       user-friendly format for guidelines

·       implementable within target group of clinicians (without the need for extensive outside collaboration)

·       freedom from conflict with economic incentives, administrative incentives, patient expectations, and community expectations.