Chapter 7 Hypothesis Testing, One-Sample Inference

• In a one-sample problem, hypotheses are specified about a single distribution.
• In a two-sample problem, two different distributions are compared.

The null hypothesis, denoted by $H_0$, is the hypothesis that is to be tested. The alternative hypothesis, denoted by $H_1$ is the hypothesis that in some sense contradicts the null hypothesis.

Four possible outcomes in hypothesis testing

	Truth: $H_0$	Truth: $H_1$
Decision: Accept $H_0$	$H_0$ is true and $H_0$ is accepted	$H_1$ is true and $H_0$ is accepted (type II error, $\beta$)
Decision: Reject $H_0$	$H_0$ is true and $H_0$ is rejected (type I error, $\alpha$)	$H_1$ is true and $H_0$ is rejected (power, $1-\beta$)

• The probability of a type I error is the probability of rejecting the $H_0$ when $H_0$ is true, usually denoted by $\alpha$ and is commonly referred to as the significance level of a test.
• The probability of a type II error is the probability of accepting $H_0$ when $H_1$ is true, usually denoted by $\beta$.
• The power of a test is
  $$1 - \beta = 1 - \text{The probability of a type II error} = P(\text{rejecting } H_0|H_1 \text{ is true})$$
  The general aim in hypothesis testing is to use statistical tests that make $\alpha$ and $\beta$ as small as possible.

One-sample $t$ test for the mean of a normal distribution: One-Sided Alternatives

Critical-value method

Under $H_0$, For random variable $(\bar{X}-\mu_0) / (S/ \sqrt n) \sim t_{n-1}$, thus $P(t_{n-1} < t_{n-1, \alpha}) = \alpha$. To test $H_0: \mu = \mu_0$ ($\sigma$ unknown) vs. $H_1: \mu < \mu_0$ ($\sigma$ unknown) with a specified significance level $\alpha$, compute the value

$$t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}$$

• If $t < t_{n-1,\, \alpha}$, reject $H_0$. Which means the sample results is extremely rare under $H_0$, less than $\alpha$, it's unusual.
• If $t \geq t_{n-1,\, \alpha}$, accept $H_0$.

Some concepts

• The value $t$ in Equation 7.2 is a test statistic; the test procedure is based on this statistic.
• The value $t_{n-1,\, \alpha}$ is a critical value; if $t < t_{n-1,\, \alpha}$, reject $H_0$; if $t \geq t_{n-1,\, \alpha}$, accept $H_0$.
• Computing a test statistic and comparing it with a critical value determined by the type I error is the critical-value method of hypothesis testing.
• Traditionally, an $\alpha$ level of exactly .05 is used most frequently.

p-value method

We can also compute $p$-value: the $\alpha$ level at which the given value of the test statistic (such as $t$) is on the borderline between the acceptance and rejection regions:

$$p = P(t_{n-1} \leq t)$$

Thus $p$ is the area to the left of $t$ under a $t_{n−1}$ distribution. Given the above formula, The $p$-value can also be thought of as the probability of obtaining a test statistic as extreme as or more extreme than the actual test statistic obtained, given that the null hypothesis is true. For the above figure, we know $t$ statistic follows a $t_{n-1}$ distribution. Hence the probability of obtaining a $t$ statistic that is no larger than the actual $t$ under $H_0$ is $P(t_{n-1} \leq t)=p$-value.

$p$-value tells us exactly how significant our results are without performing repeated significance tests at different $\alpha$ levels (like $\alpha = 0.05, 0.1, ...$)

Guidelines for judging the significance of a $p$-value

• If $.01 \le p < .05$, the results are significant.
• If $.001 \le p < .01$, the results are highly significant.
• If $p < .001$, the results are very highly significant.
• If $p > .05$, the results are not statistically significant (sometimes denoted NS).
• If $.05 \le p < .10$, a trend toward statistical significance is sometimes noted.

One-sample $t$ test ($H_1: \mu > \mu_0$)

In many situations, we wish to use an alternative hypothesis that $\mu > \mu_0$, To test $H_0: \mu = \mu_0$ ($\sigma$ unknown) vs. $H_1: \mu > \mu_0$ ($\sigma$ unknown) with significance level $\alpha$, compute

$$t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}$$

• If $t > t_{n-1,\, 1-\alpha}$, reject $H_0$.
• If $t \le t_{n-1,\, 1-\alpha}$, accept $H_0$.

The $p$-value is

$$p = P(t_{n-1} > t)$$

One-sample $t$ test for the mean of a normal distribution: Two-Sided Alternatives

In the previous section the alternative hypothesis was assumed to be in a specific direction relative to the null hypothesis ($\mu > \mu_0$ or $\mu < \mu_0$). In most instances this prior knowledge is unavailable.

A two-tailed test is a test in which the values of the parameter being studied (in this case $\mu$) under the alternative hypothesis are allowed to be either greater than or less than the value of the parameter under the null hypothesis ($\mu_0$).

For a two-sided alternative, a reasonable decision rule is to reject $H_0$ if the test statistic $t$ is either too small or too large. That is, reject $H_0$ if $t < c_1$ or $t > c_2$, and accept $H_0$ if $c_1 \le t \le c_2$.

The constants $c_1$ and $c_2$ are chosen so that the total type I error is $\alpha$:

$$P(\text{reject } H_0 \mid H_0 \text{ true}) = P(t_{n-1} < c_1 \mid H_0 \text{ true}) + P(t_{n-1} > c_2 \mid H_0 \text{ true}) = \alpha$$

For a two-tailed test, half of the type I error is placed in each tail:

$$P(t_{n-1} < c_1 \mid H_0 \text{ true}) = P(t_{n-1} > c_2 \mid H_0 \text{ true}) = \alpha/2$$

Under $H_0$,

$$P(t_{n-1} < t_{n-1,\, \alpha/2}) = P(t_{n-1} > t_{n-1,\, 1-\alpha/2}) = \alpha/2$$

Therefore,

$$c_1 = t_{n-1,\, \alpha/2} = -t_{n-1,\, 1-\alpha/2}, \quad c_2 = t_{n-1,\, 1-\alpha/2}$$

One-sample $t$ test for a two-sided alternative: to test $H_0: \mu = \mu_0$ vs. $H_1: \mu \ne \mu_0$ with significance level $\alpha$, compute

$$t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}$$

• If $|t| > t_{n-1,\, 1-\alpha/2}$, reject $H_0$.
• If $|t| \le t_{n-1,\, 1-\alpha/2}$, accept $H_0$.

Alternatively, we might want to compute a $p$-value as we did in the one-sided case.

$p$-value for the two-sided one-sample $t$ test: let

$$t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}$$

Then

$$p = \begin{cases} 2P(t_{n-1} \le t), & \text{if } t \le 0 \\ 2\left[1 - P(t_{n-1} \le t)\right], & \text{if } t > 0 \end{cases}$$

In words, if $t \le 0$, the $p$-value is two times the area to the left of $t$ under a $t_{n-1}$ distribution. If $t > 0$, the $p$-value is two times the area to the right of $t$.

For a two-sided alternative, the $p$-value is the probability under $H_0$ of obtaining a test statistic as extreme as or more extreme than the observed test statistic. Because the alternative is two-sided, extremeness is measured by the absolute value of the test statistic.

The power of a test

One-Sided Alternatives: $\mathbf{\mu_1}$ less than $\mathbf{\mu_0}$

Suppose $H_0: \mu = \mu_0$ vs. $H_1: \mu = \mu_1$. ($\mu_1 < \mu_0$) where the underlying distribution was assumed to be normal and the population variance was assumed to be known. Under $H_0$,

$$Z = (\bar{X}-\mu_0) / (\sigma / \sqrt{n}) \sim N(0,1)$$

We can calculate test statistic $z$, and for a type I error of $\alpha$, $H_0$ is rejected if $z < z_{\alpha}$ and $H_0$ is accepted if $z ≥ z_{\alpha}$.

The power of the test is the probability of rejecting $H_0$ when $H_1$ is true. If the true mean under $H_1$ is $\mu_1$, then

$$\begin{aligned} \text{Power} &= P(\text{reject } H_0 \mid H_1 \text{ True}) \\ &= P(Z < z_{\alpha} \mid \mu = \mu_1) \\ &= P\left(\frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} < z_{\alpha} \mid \mu = \mu_1\right) \\ &= P\left(\bar{X} < \mu_0 + z_{\alpha}\frac{\sigma}{\sqrt{n}} \mid \mu = \mu_1\right) \end{aligned}$$

Under $H_1$, $\bar{X} \sim N\left(\mu_1, \frac{\sigma^2}{n}\right)$. After standardizing the limit,

$$\text{Power} = \Phi\left(\frac{\mu_0 + z_{\alpha}\sigma/\sqrt{n} - \mu_1}{\sigma/\sqrt{n}}\right) = \Phi\left(z_{\alpha} + \frac{\mu_0 - \mu_1}{\sigma}\sqrt{n}\right)$$

IMPORTANT: The power of a test tells us how likely it is that a statistically significant difference will be detected based on a finite sample size $n$. If the power is too low, then there is little chance of finding a significant difference and nonsignificant results are likely even if real differences exist between the true mean $\mu$ of the group being studied and the null mean $\mu_0$.

Example: Birthweight data Compute the power of the test for the birthweight data with an alternative mean of $115$ oz and $\alpha = .05$, assuming the true standard deviation is $\sigma = 24$ oz. We have $\mu_0 = 120$ oz, $\mu_1 = 115$ oz, $\alpha = .05$, $\sigma = 24$, and $n = 100$. Thus,

$$\begin{aligned} \text{Power} &= \Phi\left[z_{.05} + \frac{(120 - 115)\sqrt{100}}{24}\right] \\ &= \Phi\left[-1.645 + \frac{5(10)}{24}\right] \\ &= \Phi(0.438) \\ &= .669 \end{aligned}$$

Therefore, there is about a $67\%$ chance of detecting a significant difference using a $5\%$ significance level with this sample size.

One-Sided Alternatives: $\mathbf{\mu_1}$ greater than $\mathbf{\mu_0}$

Suppose $H_0: \mu = \mu_0$ vs. $H_1: \mu = \mu_1$. ($\mu_1 > \mu_0$) $H_0$ is rejected if $z > z_{1 - \alpha}$ and $H_0$ is accepted if $z \leq z_{1-\alpha}$.

$$\begin{aligned} \text{Power} &= P(\text{reject } H_0 \mid H_1 \text{ True}) \\ &= P(Z > z_{1-\alpha} \mid \mu = \mu_1) \\ &= P\left(\frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} > z_{1-\alpha} \mid \mu = \mu_1\right) \\ &= P\left(\bar{X} > \mu_0 + z_{1-\alpha}\frac{\sigma}{\sqrt{n}} \mid \mu = \mu_1\right) \\ &= 1-P\left(\bar{X} < \mu_0 + z_{1-\alpha}\frac{\sigma}{\sqrt{n}} \mid \mu = \mu_1\right) \\ &= 1 - \Phi\left(\frac{\mu_0 + z_{1-\alpha}\sigma/\sqrt{n} - \mu_1}{\sigma/\sqrt{n}}\right) \\ &= 1 - \Phi\left(z_{1-\alpha} + \frac{(\mu_0 - \mu_1)\sqrt{n}}{\sigma}\right) \end{aligned}$$

Using the relationships $\Phi(-x) = 1 - \Phi(x)$ and $z_{\alpha} = -z_{1-\alpha}$, the power expression can also be rewritten as

$$\Phi\left[-z_{1-\alpha} + \frac{(\mu_1 - \mu_0)\sqrt{n}}{\sigma}\right] = \Phi\left[z_{\alpha} + \frac{(\mu_1 - \mu_0)\sqrt{n}}{\sigma}\right], \quad \text{if } \mu_1 > \mu_0$$

Summery: Power for the one-sample $z$ test with known variance

For the hypothesis $H_0: \mu = \mu_0$ vs. $H_1: \mu = \mu_1$, where the underlying distribution is normal and the population variance $\sigma^2$ is known, the power is

$$\text{Power} = \Phi\left(z_{\alpha} + \frac{|\mu_0 - \mu_1|\sqrt{n}}{\sigma}\right) = \Phi\left(-z_{1-\alpha} + \frac{|\mu_0 - \mu_1|\sqrt{n}}{\sigma}\right)$$

The power depends on four factors: significance level $\alpha$, effect size $|\mu_0 - \mu_1|$, sample size $n$, and standard deviation $\sigma$.

Factors affecting the power

1. If the significance level decreases ($\alpha$ decreases), then $z_{\alpha}$ increases and the power decreases.
2. If the alternative mean is farther from the null mean ($|\mu_0 - \mu_1|$ increases), then the power increases.
3. If the standard deviation increases ($\sigma$ increases), then the power decreases.
4. If the sample size increases ($n$ increases), then the power increases.

Power Curve

For given levels of $\alpha = .05$, $\sigma = 24$ oz, $n = 100$, and $\mu_0 = 120$ oz, a power curve can be drawn for the power of the test under various alternative means $\mu_1$.

For the birthweight data in Example 7.2, the power ranges from 99% when $\mu = 110$ oz to about 20% when $\mu = 118$ oz.

Two-Sided Alternatives

Power for the one-sample $z$ test for the mean of a normal distribution

For the two-sided test

$$H_0: \mu = \mu_0 \quad \text{vs.} \quad H_1: \mu \neq \mu_0$$

with a specific alternative $\mu = \mu_1$, where the underlying distribution is normal and the population variance $\sigma^2$ is known, the exact power is

$$\text{Power} = \Phi\left[-z_{1-\alpha/2} + \frac{(\mu_0 - \mu_1)\sqrt{n}}{\sigma}\right] + \Phi\left[-z_{1-\alpha/2} + \frac{(\mu_1 - \mu_0)\sqrt{n}}{\sigma}\right]$$

This is approximately

$$\text{Power} \approx \Phi\left[-z_{1-\alpha/2} + \frac{|\mu_0 - \mu_1|\sqrt{n}}{\sigma}\right]$$

Derivation To see this, for a two-sided $z$ test we reject $H_0$ if

$$z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} < z_{\alpha/2} \quad \text{or} \quad z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} > z_{1-\alpha/2}$$

Multiplying each inequality by $\sigma/\sqrt{n}$ and adding $\mu_0$, the rejection criteria can be written in terms of $\bar{x}$ as

$$\bar{x} < \mu_0 + z_{\alpha/2}\frac{\sigma}{\sqrt{n}} \quad \text{or} \quad \bar{x} > \mu_0 + z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}}$$

The power of the test against the specific alternative $\mu = \mu_1$ is

$$\begin{aligned} \text{Power} &= P\left(\bar{X} < \mu_0 + z_{\alpha/2}\frac{\sigma}{\sqrt{n}} \mid \mu = \mu_1\right) + P\left(\bar{X} > \mu_0 + z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} \mid \mu = \mu_1\right) \\ &= \Phi\left(\frac{\mu_0 + z_{\alpha/2}\sigma/\sqrt{n} - \mu_1}{\sigma/\sqrt{n}}\right) + 1 - \Phi\left(\frac{\mu_0 + z_{1-\alpha/2}\sigma/\sqrt{n} - \mu_1}{\sigma/\sqrt{n}}\right) \\ &= \Phi\left[z_{\alpha/2} + \frac{(\mu_0 - \mu_1)\sqrt{n}}{\sigma}\right] + 1 - \Phi\left[z_{1-\alpha/2} + \frac{(\mu_0 - \mu_1)\sqrt{n}}{\sigma}\right] \end{aligned}$$

Using the relationship $1 - \Phi(x) = \Phi(-x)$, the last two terms can be combined as

$$\text{Power} = \Phi\left[z_{\alpha/2} + \frac{(\mu_0 - \mu_1)\sqrt{n}}{\sigma}\right] + \Phi\left[-z_{1-\alpha/2} + \frac{(\mu_1 - \mu_0)\sqrt{n}}{\sigma}\right]$$

Finally, because $z_{\alpha/2} = -z_{1-\alpha/2}$, we have

$$\text{Power} = \Phi\left[-z_{1-\alpha/2} + \frac{(\mu_0 - \mu_1)\sqrt{n}}{\sigma}\right] + \Phi\left[-z_{1-\alpha/2} + \frac{(\mu_1 - \mu_0)\sqrt{n}}{\sigma}\right]$$

Note: The exact two-sided power formula is more tedious to use than is usually necessary. Specifically, if $\mu_1 < \mu_0$, then the second term is usually negligible relative to the first term. However, if $\mu_1 > \mu_0$, then the first term is usually negligible relative to the second term. Therefore, the approximate power formula is usually used for a two-sided test because it represents the dominant term in the exact formula.

Illustration of power for a two-sided test for the mean of a normal distribution with known variance

Sample-Size Determination

One-Sided Alternatives

The problem of determining sample size can be summarized as follows: Given that a one-sided significance test will be conducted at level $\alpha$ and that the true alternative mean is expected to be $\mu_1$, what sample size is needed to be able to detect a significant difference with probability $1 - \beta$? Typical values for the desired power are 80%, 90%, ..., and so forth.

For a one-sided test, we set the desired power equal to $1 - \beta$:

$$\text{Power} = \Phi\left(z_{\alpha} + \frac{|\mu_0 - \mu_1|\sqrt{n}}{\sigma}\right) = 1 - \beta$$

We want to solve for $n$ in terms of $\alpha$, $\beta$, $|\mu_0 - \mu_1|$, and $\sigma$. Since

$$\Phi(z_{1-\beta}) = 1 - \beta$$

we have

$$z_{\alpha} + \frac{|\mu_0 - \mu_1|\sqrt{n}}{\sigma} = z_{1-\beta}$$

Subtract $z_{\alpha}$ from both sides and multiply by $\sigma/|\mu_0 - \mu_1|$:

$$\sqrt{n} = \frac{(-z_{\alpha} + z_{1-\beta})\sigma}{|\mu_0 - \mu_1|}$$

Since $-z_{\alpha} = z_{1-\alpha}$, squaring both sides gives

$$n = \frac{(z_{1-\alpha} + z_{1-\beta})^2\sigma^2}{(\mu_0 - \mu_1)^2}$$

Factors affecting the sample size

1. The sample size increases as $\sigma^2$ increases.
2. The sample size increases as the significance level is made smaller ($\alpha$ decreases).
3. The sample size increases as the required power increases ($1 - \beta$ increases).
4. The sample size decreases as the absolute distance between the null and alternative means, $|\mu_0 - \mu_1|$, increases.

Note: Sample-size estimates are usually only ballpark estimates because $\mu_1$ and $\sigma^2$ are often unknown. These values may be estimated from previous studies, similar experiments, prior knowledge, or a small pilot study. In practice, sample-size calculations are often used to check whether the proposed sample size is reasonably close to what is needed, rather than to determine an exact number.

Two-Sided Alternatives

The one-sided sample-size formula is appropriate when the direction of the alternative mean is specified in advance. If it is not known whether the alternative mean $\mu_1$ is greater than or less than the null mean $\mu_0$, then a two-sided test is appropriate.

For testing

$$H_0: \mu = \mu_0 \quad \text{vs.} \quad H_1: \mu \neq \mu_0$$

with significance level $\alpha$ and desired power $1 - \beta$, the sample size needed is

$$n = \frac{\sigma^2\left(z_{1-\beta} + z_{1-\alpha/2}\right)^2}{(\mu_0 - \mu_1)^2}$$

To see this, use the approximate power formula for a two-sided test and solve for $n$:

$$\Phi\left(-z_{1-\alpha/2} + \frac{|\mu_0 - \mu_1|\sqrt{n}}{\sigma}\right) = 1 - \beta$$

Since $\Phi(z_{1-\beta}) = 1 - \beta$, we have

$$-z_{1-\alpha/2} + \frac{|\mu_0 - \mu_1|\sqrt{n}}{\sigma} = z_{1-\beta}$$

Add $z_{1-\alpha/2}$ to both sides and multiply by $\sigma/|\mu_0 - \mu_1|$:

$$\sqrt{n} = \frac{\left(z_{1-\beta} + z_{1-\alpha/2}\right)\sigma}{|\mu_0 - \mu_1|}$$

Squaring both sides gives

$$n = \frac{\left(z_{1-\beta} + z_{1-\alpha/2}\right)^2\sigma^2}{(\mu_0 - \mu_1)^2}$$

This required sample size is always larger than the corresponding sample size for a one-sided test, because $z_{1-\alpha/2} > z_{1-\alpha}$.

Based on CI Width

Sometimes the treatment effect is already believed to exist, so the main goal is not hypothesis testing but estimating the effect with a desired degree of precision. In this case, sample size can be chosen based on the desired width of a confidence interval.

Example: Cardiology Suppose propranolol is known to lower heart rate over 48 hours in patients with angina at standard dosage levels. A new study uses a higher dose, and investigators want to estimate the drop in heart rate with high precision.

If precision is measured by the width of a two-sided $100(1-\alpha)\%$ confidence interval, then the CI for the true mean decline $\mu$ is

$$\bar{x} \pm t_{n-1,1-\alpha/2}\frac{s}{\sqrt{n}}$$

The width of this interval is

$$2t_{n-1,1-\alpha/2}\frac{s}{\sqrt{n}}$$

If we want the CI width to be no wider than $L$, then

$$2t_{n-1,1-\alpha/2}\frac{s}{\sqrt{n}} = L$$

so

$$n = \frac{4t_{n-1,1-\alpha/2}^2s^2}{L^2}$$

Because $t_{n-1,1-\alpha/2}$ is often approximated by $z_{1-\alpha/2}$, the required sample size is approximately

$$n = \frac{4z_{1-\alpha/2}^2s^2}{L^2}$$

Example: Cardiology

Find the minimum sample size needed to estimate the change in heart rate $\mu$ if the two-sided $95\%$ CI for $\mu$ must be no wider than $5$ beats per minute and the sample standard deviation for change in heart rate is $s = 10$ beats per minute.

We have $\alpha = .05$, $s = 10$, and $L = 5$. Therefore,

$$\begin{aligned} n &= \frac{4z_{.975}^2(10)^2}{(5)^2} \\ &= \frac{4(1.96)^2(100)}{25} \\ &= 61.5 \end{aligned}$$

Thus, at least 62 patients need to be studied.

The Relationship Between Hypothesis Testing and Confidence Intervals

Two-Sided Case

Suppose we are testing

$$H_0: \mu = \mu_0 \quad \text{vs.} \quad H_1: \mu \neq \mu_0$$

For a two-sided level $\alpha$ test:

• $H_0$ is rejected if and only if the two-sided $100(1-\alpha)\%$ confidence interval for $\mu$ does not contain $\mu_0$.
• $H_0$ is accepted if and only if the two-sided $100(1-\alpha)\%$ confidence interval for $\mu$ does contain $\mu_0$.

Note: The hypothesis-testing approach and the confidence-interval approach often lead to the same conclusion, but they emphasize different information. A p-value tells us how statistically significant the result is, while a confidence interval shows the likely range of the parameter and helps judge the practical size of the effect. Therefore, it is good practice to report both the p-value and the confidence interval whenever possible.

Example: Cardiovascular Disease For the cholesterol data, the p-value of $.037$ tells us how statistically significant the result is. The $95\%$ confidence interval for $\mu$, $(173.58, 189.46)$, gives a range of likely values that $\mu$ might assume. These two types of information are complementary.