Lecture 8: The Truth Check – Validity of Epidemiological Studies

The main goal of any study is to get an accurate estimate of the relationship between an exposure (like using SuperPaste) and an outcome (like getting dental caries). Accuracy involves both precision and validity.

• Precision (Reliability): Getting the similar result every time the study is done.
• Validity (Accuracy/Truth): The result is close to the true value in the real world.

1. Types of Validity

Epidemiological studies must satisfy two types of validity:

Type of Validity	Question Asked	Significance
Internal Validity	Are the methods used correct, so that the results obtained in the study sample are true for that sample?	This is the minimum requirement. If a study lacks internal validity, the results are meaningless.
External Validity (Generalizability)	Can the results from the study sample be extrapolated (generalized) to the target population (e.g., all 6-year-olds in India)?	A study must have internal validity before it can have external validity.

2. Threats to Validity: Systematic Errors (Biases)

Errors in a study fall into two categories:

• Random Errors (Chance): Due to uncontrollable causes (e.g., sampling variability). This is a threat to precision.
• Systematic Errors (Biases): Errors in the study’s methodology that cause the estimated association to differ from the true causal association. This is the primary threat to validity.

There are three major types of biases that threaten the validity of observational studies (Case-Control and Cohort):

---

3. Selection Bias

Selection Bias occurs when the procedures used to select study participants lead to an estimate that is systematically distorted. The way the study participants are chosen does not accurately represent the target population.

• How it Happens (The Problem):
  • Survival Bias: Selecting only prevalent cases (people who survived the disease long enough to be in the study). If SuperPaste helps people survive longer with caries, the study might only select SuperPaste users, biasing the results.
  • Loss to Follow-up (in Cohort Studies): If SuperPaste users are more likely to drop out of the study than regular paste users, the final groups being compared are no longer representative of the starting groups, creating bias.
• How to Deal With It (Prevention):
  • Design Stage: Use incident cases (new cases) rather than prevalent cases.
  • Data Collection Stage: Minimize non-response and loss to follow-up. Use blinding when diagnosing the outcome so the investigator doesn’t know the exposure status.

---

4. Information Bias

Information Bias occurs when there are systematic errors in measuring the characteristics of study participants (exposure, outcome, or confounders).

• How it Happens (The Problem):
  • Recall Bias (in Case-Control): People who have the outcome (caries) may be more likely to recall or report past exposures (e.g., sugary snacks) than people who are healthy (controls).
  • Interviewer Bias: If the interviewer knows the child is in the SuperPaste group, they might question the parent more rigorously about their diet, leading to different quality data.
• How to Deal With It (Prevention):
  • Set up precise operational definitions for variables (e.g., exactly how a “cup of coffee” or a “carious lesion” is measured).
  • Use detailed measurement protocols and trained, certified investigators.
  • Use repeated measurements (e.g., taking three blood pressure readings).

---

5. Confounding

Confounding means confusion of effects. This happens when the observed association between the Exposure and the Outcome is falsely influenced by a third factor (Confounder) that is associated with both the exposure and the outcome, but is not an intermediate step.

• Confounder Criteria: A factor must satisfy three conditions:
  1. It is associated with the Exposure (e.g., SuperPaste use).
  2. It is associated with the Outcome (e.g., dental caries).
  3. It is not caused by the Exposure.
• SuperPaste Example: If we find that SuperPaste users have less caries, is that the SuperPaste, or is it Income?
  • Children with higher Income (Confounder) are more likely to use expensive, new SuperPaste (Exposure).
  • Children with higher Income also have better nutrition and dental care (which prevents caries) (Outcome).
  • The association between SuperPaste and less caries is confounded by the higher income.
• Dangers of Confounding: It can simulate (create) an association that doesn’t exist, hide one that does, or even change the direction of an effect.
• How to Deal With It (Prevention):
  • Design Stage:
    • Restriction: Only study one income group.
    • Matching: Pair SuperPaste users with regular paste users who have the same income level.
    • Randomization (in Trials): Automatically balances known and unknown confounders.
  • Analysis Stage:
    • Stratified Analysis: Analyze the data separately within each income group.
    • Multivariate Analysis: Use regression models ($\text{logistic regression}$) to adjust for the effect of the confounder.

---

6. Evaluating the Results

Whenever you see a crude measure of association (RR or OR), you must follow this internal “spiral of truth”:

1. Is it due to Chance? (Check the confidence interval/p-value)
2. Is there Selection Bias?
3. Is there Information Bias?
4. Is there Confounding?

Only after ruling out chance and the three main biases can you state that the crude association is likely a true causal association.