AP Statistics Lectures

by Arnold Kling

Hypothesis Testing

People use statistical analysis to make various kinds of decisions. Some decisions are continuous, meaning that you can adjust a quantity over a range. Other decisions are discrete, meaning that it is an either-or decision.

An example of a continuous decision is the amount charged by auto insurance companies for different drivers and automobiles. When you buy auto insurance, the price that you pay depends on the company's statistical estimate of the amount it is likely to have to pay in claims.

An example of a discrete decision is a decision by the Food and Drug Administration (FDA) to approve or disapprove the use of a particular drug to treat a disease. This discrete decision-making context helps to motivate the concept of hypothesis testing.

Type I and Type II error

The FDA's goal is to approve drugs that are effective and to reject drugs that are ineffective or harmful. However, it is impossible for the FDA to be perfect at distinguishing effective drugs from ineffective or harmful drugs.

The FDA approved a diet drug called Fen-phen that later turned out to be harmful. The FDA has been cautious about recommending the use of cholesterol-reducing drugs, even though some lives might have been saved had more people been taking them earlier.

The FDA starts out with a presumption that the drug should not be approved. There are four possible outcomes of the decision-making process.

If the FDA refuses to approve the drug, and in fact the drug is harmful, then this is a correct decision.

If the FDA approves the drug and the drug is more helpful than harmful, then this is a correct decision.

If the FDA approves a drug that turns out to be bad (such as Fen-phen), then this is called Type I error.

If the FDA refuses to approve a new drug that turns out to be more helpful than harmful, then this is called Type II error.

The FDA starts with a null hypothesis, or H_{0}, that a new drug will not have a helpful effect. It tests an alternative hypothesis, or H_{a}, that the drug does have a helpful effect.

For the FDA, incorrectly rejecting the null hypothesis is the same as committing type I error. It would mean approving a bad drug.

Incorrectly failing to reject the null hypothesis is the the same as Type II error. It would mean failing to approve a good drug.

Presumed Innocent

In law, we say that someone is innocent until proven guilty.

- What is the null hypothesis?
- What is the alternative hypothesis?
- Would sending an innocent man to jail be Type I error or Type II error?
- Would letting a guilty man go free be Type I error or Type II error?