CRITICAL APPRAISAL

by Dr PHUA Kai Lit
School of Medicine and Health Sciences
Monash University Sunway Campus
Bandar Sunway, Malaysia

Click for Interview with Prof Michael Marmot on social determinants of bad health

WHAT IS IT?

The ability to read and evaluate scientific and scholarly material (especially articles published in scientific journals) in a sophisticated manner. To do so, you need the following skills:

1. Know where and how to search for scientific information
2. Know about the main types of research design (methodology)
3. Able to detect flaws in the research design of published scientific papers
4. Know how to interprete data published in research papers (including results of statistical tests)

WHY BOTHER?

It is important to possess critical appraisal skills and to read scientific literature because of the following:

a. Constant advances in diagnosis and treatment of diseases (new drugs, new medical procedures etc)
b. Important to know about latest news on adverse effects of drugs and procedures
c. Need to know about emerging and re-emerging diseases
d. Need to know about emerging drug-resistant infections
e. Be able to distinguish between high quality and low quality research
f. Be aware of existence of scientific fraud

SCIENTIFIC FRAUD

Short article on scientific fraud

Read long article on scientific fraud

Fraudulent letters sent to academic journals!

Public Citizen's problematic pharmaceutical drugs website

Drug safety and recalls (article by Phua Kai Lit)
Another version of this article will be appearing in the British journal Clinical Ethics soon

1. Outright fabrication of data
2. Falsification of data ("massaging of data")
3. Suppression of negative findings (suppression of data that does not support the research hypothesis)
4. Plagiarism (Example 1: using other people's ideas without acknowledgement. Example 2: using other people's written text and passing it off as your own i.e. without proper quotation)

WHERE TO LOOK FOR INFORMATION?

General Journals

New England Journal of Medicine
Journal of the American Medical Association
BMJ (British Medical Journal)
Lancet
Medical Journal of Malaysia
Singapore Medical Journal

American Journal of Public Health
Asia-Pacific Journal of Public Health

* Focus especially on "review articles" and "concensus recommendations of expert panels". These will give you information on current knowledge and the state of the art.

Literature Review Journals

Journal Watch

ACP Journal Club

Medical Letter on Drugs and Therapeutics

Books

Summary-type books e.g. "Current Medical Diagnosis and Treatment"
Classic textbooks e.g. Cecil's, Harrison's, Harvey's textbooks (medicine)
e.g. Maxcy-Rosenau-Last (public health)

Review volumes e.g.
Annual Review of Medicine

The Internet

General information

Google
Enclose your phrases (can be more than 1 phrase) in quotation marks
e.g. To search for info on HIV in Malaysia, type in "HIV" "Malaysia" simultaneously

Local websites

Malaysian Medical Resources

Foreign websites

Medscape

Clinical Practice Guidelines

U.S. Agency for Healthcare Research and Quality
Clinical Practice Guidelines Online

Ministry of Health, Singapore
Clinical Practice Guidelines Online

Computerised Literature Searches

Pub Med

Other HIGH QUALITY Sites

World Health Organization

Centers for Disease Control (CDC), U.S. Public Health Service

Cochrane Reviews

RESEARCH METHODOLOGY

1. Experiments
2. Quasi-experiments
3. Correlational Studies
4. Prospective Studies (Cohort Studies or Longitudinal Studies)
5. Retrospective Studies (Case-Control Studies)
6. Other e.g. meta-analysis

EXPERIMENTS

Researcher has control over the treatment/intervention e.g. RCT (Randomized Controlled Clinical Trial)

There should be at least one Experimental Group and one Control Group
Randomization i.e. reseach subjects are equally likely to end up either in the Experimental Group or in the Control Group
Blinding e.g. double blind experiments where research subjects do not know which group they are in (Expt or Control Group) and researchers do not know which group an individual is in either
Placebo effect (to correct for this problem, the Control Group given inert substance/sham treatment)

QUASI-EXPERIMENTS

This method is used when experiments cannot be done. Quasi-experiments are common in public health research.

Researcher cannot control characteristics of subjects. Randomization is not possible e.g. a public health campaign is held in City A (experimental group) but not in City B (the "control group")

CORRELATIONAL STUDIES

Study relationships or associations between variables.

CAUTION: Correlation does not necessarily mean causation!

PROSPECTIVE STUDIES (COHORT STUDIES OR LONGITUDINAL STUDIES) e.g. observe 2 groups of people over time (Group 1 is exposed to the suspected disease-causing agent while Group 2 is not exposed).
Are people in Group 1 more likely to get the disease than people in Group 2?

RETROSPECTIVE STUDIES (CASE-CONTROL STUDIES)

"Retrospective" = backward-looking e.g. Individuals with the disease (Group 1 of "cases") are compared to those without the disease (Group 2 of "controls"). Try to determine what could have caused the disease in Group 1 (identify possible risk factors).
Calculate the Odds Ratio. Is this statistically significant?

OTHER e.g. META-ANALYSIS

Combine the findings of several RCT to arrive at a definite conclusion

a. Identify studies on a similar research question
b. Use statistical techniques to get one conclusion

CRITICALLY APPRAISING A SCIENTIFIC PAPER

IMRAD Format

Introduction
Methodology
Results
And
Discussion

(1) ABSTRACT AND KEYWORDS

Scan the abstract (summary). It should summarise the contents of the paper properly.
Note the keywords

(2) INTRODUCTION

What is the stated goal of the article?
What is the rationale (why was the study done)?
Read the literature review. Is it done properly?

Is the research question clearly stated?
Are the research hypotheses very clear and explicitly laid out?
Is the research question important?
Is the research original or a replication of work done by other reseachers?

(3) METHODOLOGY

What is the research design?

Randomised controlled clinical trial?
Quasi-experiment?
Observational study e.g. cohort study, case-control study?

Subject Selection
What is the sampling method? What is the response rate? Any bias? What is the control group?

What is the dependent variable (outcome variable)?
What are the independent variables (predictor variables)?

(4) RESULTS

Presented in the form of graphs, charts, tables
What are the statistical tests used?
Is the statistical test misused (i.e. are the assumptions of the statistical test violated?
What are the results of the statistical tests?

(5) DISCUSSION

Evaluate the summary of findings
What are the limitations of the study?
Do the conclusions match the research question declared at the beginning of the article?
How do the findings compare with other research?
What further research is needed?

(6) REFERENCES

The references section should be carefully prepared in a well-written scientific paper. Different journals may use different formats BUT if you submit a paper to a particular journal, you should follow its format for refences STRICTLY!

(7) APPENDIX

The appendix often contains highly technical details on the research methodology

INTERPRETING RESULTS OF STATISTICAL TESTS

1. The p-value and "Statistically Significant" research findings

The p-value is the probability that the observed relationship (e.g., between variables) or a difference (e.g., between means) in a sample occurred purely by chance, and that in the population from which the sample was drawn, no such relationship or differences actually exist.

"Statistically significant" means the probability that an observed result has occurred by chance is very low. Because the probability is so low, the result seen is very likely to be real (not due to chance) and meaningful (significant).

If the p-value is > 0.05, it is NOT CONSIDERED statistically significant (strictly speaking, "there is not enough evidence to reject the null hypothesis H₀"). This is because the probability is more than 0.05 that it could have occurred by chance.

If p < 0.05, it is considered statistically significant (written as --> p < .05 *)
If p < 0.01, it is considered highly significant (written as --> p < .01 **)

Decision Rule:
Accept ("fail to reject")H₀if p > .05
Reject H₀ if p< .05

2. Null Hypothesis and Research Hypothesis

For the chi-square test:

H₀ : there is no association between Variable X and Variable Y. Any association seen is due to chance alone
H₁ : there is a statistically significant association between X and Y

For the t-test:

H₀ : there is no difference between the two underlying population means. Any difference seen is due to chance alone
H₁ : there is a statistically significant difference between the two underlying population means

3. Relative Risk (used for prospective studies)

How the Relative Risk is Calculated

	Becomes Diseased	No Disease
Exposed to risk factor	a	b
Not exposed to risk factor	c	d

Risk of Developing Disease for Those Exposed to Risk Factor over Time = a divided by (a+b)
Risk of Developing Disease for Those Not Exposed to Risk Factor = c divided by (c+d)

The Relative Risk is a/(a+b) divided by c/(c+d) i.e. risk of exposed group is compared to risk of unexposed group. Hence the term relative risk.
If the Relative Risk = 1, this means that the risk of developing disease for the exposed group is the same as the risk for the unexposed group. Therefore, there is no association between a risk factor and a disease
If the Relative Risk > > 1, there is a strong positive association
If the Relative Risk < < 1, there is a strong negative association

e.g. In a study on smoking and lung cancer, the Relative Risk (estimated from a sample) is 1.62
This means roughly that "People who smoke are 1.62 times more likely to develop lung cancer than those who do not smoke"
This means strictly that "The risk of developing lung cancer for people who smoke is 1.62 times the risk for people who do not smoke"

The next step is to determine if this estimated Relative Risk is statistically significant.

3b. Testing if a Relative Risk is Statistically Significant

To do this, we need to construct a 95% Confidence Interval.

(A 95% CI means roughly that the probability is 0.95 that the real value of a parameter lies within the interval calculated)
Strictly, it means that if you take 100 samples and calculate the statistic 100 times, 95 of them will fall within the 95% CI)
The estimated RR is used to construct the 95% CI for the true RR

Example:

H₀ : Relative Risk = 1 (no association between exposure to risk factor and Disease X)
H₁ : Relative Risk is not equal to 1 (association between exposure to risk factor and Disease X)

If 95% CI does not include 1, reject H₀
(prob. is 0.95 that true RR falls within the interval i.e. 95% sure that true RR is not equal to 1)

If 95% CI includes 1, accept ("fail to reject") H₀

Example 1

Estimated Relative Risk = 1.62
95% CI for true Relative Risk = 1.15 to 2.28
H₀ : RR = 1
H₁ : RR is not equal to 1

What is your conclusion?

Since the 95% CI does not include 1, we are 95% sure that the real Relative Risk is not equal to 1.
Therefore, we accept H₁ and conclude that (1)there is an assoc between exposure to the risk factor and chances of getting a Disease X and (2)People who are exposed to the risk factor are 1.62 times more likely to develop the Disease than those who are not exposed.

4. Odds Ratio (Often used for retrospective or case-control studies)

What is meant by the term "Odds"?

An example will clarify:

The probability of getting a 6 when a dice is thrown is 1/6

The odds of getting a 6 when a dice is thrown is 1:5 or 1/5
That is, when you throw a dice, there are 6 possible outcomes (1,2,3,4,5,6) and one outcome is the desired outcome while the other five outcomes are what you don't want to get.
Thus, there is: 1 possible "success" compared to 5 possible "failures" (1:5 or 1/5)

How the Odds Ratio is Calculated

	Has the disease now (CASES)	No disease (CONTROLS)
Exposed to risk factor in the past	a	b
Not exposed to risk factor in the past	c	d

Odds of Exposure for Cases = a:c or (a/c)
Odds of Exposure for Controls = b:d or (b/d)

The Odds Ratio is (a/c) divided by (b/d) --> This is the same as (a*d) divided by (b*c)

The Odds for the Cases are compared to the Odds for the Controls. Hence the term Odds Ratio

If the Odds Ratio = 1, there is no association between a risk factor and a disease (i.e. if cases and controls are compared, there is no difference in odds of exposure to a risk factor in the past.
If the Odds Ratio > > 1, there is a strong positive association
If the Odds Ratio < < 1, there is a strong negative association

e.g. In a study on lung cancer presence now and smoking in the past, the estimated OR is 2.14
This means roughly that "Lung cancer victims are 2.14 times more likely to be smokers than people without lung cancer"
(This means strictly that "Lung cancer victims' odds of exposure to smoking are 2.14 times the odds for those without lung cancer")

The next step is to determine if this estimated OR is statistically significant.

4b. Testing if an Odds Ratio is Statistically Significant

To do this, we need to construct a 95% Confidence Interval.

(A 95% CI means the probability is 0.95 that the real value of a parameter lies within the interval calculated)
The estimated OR is used to construct the 95% CI for the true OR

Example:

H₀ : Odds Ratio = 1 (no association between exposure to risk factor and Disease X)
H₁ : Odds Ratio is not equal to 1 (association between exposure to risk factor and Disease X)

If 95% CI does not include 1, reject H₀
(prob. is 0.95 that true OR falls within the interval i.e. 95% sure that true OR is not equal to 1)

If 95% CI includes 1, accept H₀

Example 1

Estimated Odds ratio = 2.5
95% CI for true Odds ratio = 2.0 to 3.0
H₀ : OR = 1
H₁ : OR is not equal to 1

What is your conclusion?

Since the 95% CI does not include 1, we are 95% sure that the real odds ratio is not equal to 1.
Therefore, we accept H₁ and conclude 1) that there is an assoc between exposure to the risk factor and chances of getting a Disease X and 2) People with Disease X are 2.5 times more likley to have been exposed to the risk factor in the past than people without the Disease X

Example 2

Odds Ratio = 1.3
95% CI = 0.9 to 1.7
H₀ : OR = 1
H₁ : OR is not equal to 1

What is your conclusion?

Since the 95% CI includes 1, accept ("fail to reject") H₀
Conclude that 1) there is no assoc between exposure to suspected risk factor and chances of getting Disease X and 2) People with the Disease are no more likely (equally likely) to have been exposed to the risk factor in the past compared to people without the Disease (hence we conclude there is no association between exposure and chances of getting the Disease. Therefore the suspected risk factor can probably be ruled out as an actual risk factor).

Interpreting the Relative Risk versus Interpreting the Odds Ratio

If Relative Risk = 2.1
Interprete roughly as "People who are exposed to the risk factor are 2.1 times more likely to develop the disease than people who have not been exposed"
Interprete strictly as "For people exposed to the risk factor, their risk of developing the disease is 2.1 times that of people who have not been exposed"

If Odds Ratio = 2.1 (in a case-control study)
Interprete roughly as "People who have the disease are 2.1 times more likely to have been exposed to the risk factor in the past compared to people who do not have the disease"
Interprete strictly as "For people who have the disease, their odds of exposure to the risk factor in the past are 2.1 times the odds of exposure for people who do not have the disease"

Thus, Relative Risk concerns the statement (as used in prospective studies i.e. looking forward in time)
"If people are exposed to a risk factor, how likely are they to develop a disease
(compared to those not exposed)?"

Odds Ratio concerns the statement (if used for retrospective, case-control studies i.e. looking backward in time)
"If people have a disease now, how likely are they to have been exposed to a risk factor in the past
(compared to those without the disease now)?"

NOTE: Odds Ratio analysis CAN be used for prospective studies too. But since they are harder to interprete, it is better to use Relative Risk analysis for prospective studies.

Sensitivity versus Specificity (of Screening Test)

	Truly has the disease	Does not actually have the disease
Tested positive	a	b
Tested negative	c	d
Total	a + c	b + d

Sensitivity (%) = a/(a+c) X 100
Sensitivity = ability of test to correctly identify persons who HAVE the disease

Specificity (%) = d/(b+d) X 100
Specificity = ability of test to correctly identify persons who DO NOT HAVE the disease

PPV versus NPV (of Screening Test)

	Truly has the disease	Does not actually have the disease	Total
Tested positive	a	b	a+b
Tested negative	c	d	c+d

Positive Predictive Value (PPV) (%) = a/(a+b) X 100

Negative Predictive Value (NPV) (%) = d/(c+d) X 100
* Using the same test in a population with higher prevalence increases positive predictive value. Conversely, increased prevalence results in decreased negative predictive value.
(Sensitivity and specificity of the clinical test are unaffected by prevalence of the disease).

**

**