Pooled variance

In statistics, pooled variance (also known as combined, composite, or overall variance) is a method for estimating variance of several different populations when the mean of each population may be different, but one may assume that the variance of each population is the same.

Under the assumption of equal population variances, the pooled sample variance provides a higher precision estimate of variance than the individual sample variances. This higher precision can lead to increased statistical power when used in statistical tests that compare the populations, such as the t-test.

The square-root of a pooled variance estimator is known as a pooled standard deviation (also known as combined, composite, or overall standard deviation).

Motivation

In statistics, many times, data are collected for a dependent variable, y, over a range of values for the independent variable, x. For example, the observation of fuel consumption might be studied as a function of engine speed while the engine load is held constant. If, in order to achieve a small variance in y, numerous repeated tests are required at each value of x, the expense of testing may become prohibitive. Reasonable estimates of variance can be determined by using the principle of pooled variance after repeating each test at a particular x only a few times.

Definition

If the populations are indexed $i=1,\ldots ,k$ , then the pooled variance $s_{p}^{2}$ (or $s_{c}^{2}$ ) can be estimated by the weighted average:

s_{p}^{2}={\frac {\sum _{{i=1}}^{k}(n_{i}-1)s_{i}^{2}}{\sum _{{i=1}}^{k}(n_{i}-1)}}={\frac {(n_{1}-1)s_{1}^{2}+(n_{2}-1)s_{2}^{2}+\cdots +(n_{k}-1)s_{k}^{2}}{n_{1}+n_{2}+\cdots +n_{k}-k}}

where $n_{i}$ is the sample size of population $i$ and the sample variances are:

s_{i}^{2}

{\frac {1}{n_{i}-1}}\sum _{{j=1}}^{{n_{i}}}\left(y_{j}-\overline {y_{j}}\right)^{2}

Use of $(n_{i}-1)$ weighting factors instead of $n_{i}$ comes from Bessel's correction.

Variants

The unbiased least square estimate:

s_{p}^{2}={\frac {\sum _{{i=1}}^{k}(n_{i}-1)s_{i}^{2}}{\sum _{{i=1}}^{k}(n_{i}-1)}}

and the biased maximum likelihood estimate:

s_{p}^{2}={\frac {\sum _{{i=1}}^{k}(n_{i}-1)s_{i}^{2}}{\sum _{{i=1}}^{k}n_{i}}}

are used in different contexts. The former can give an unbiased $s_{p}^{2}$ to estimate $\sigma ^{2}$ when the two groups share an equal population variance. The latter one can give a more efficient $s_{p}^{2}$ to estimate $\sigma ^{2}$ biasedly. Note that the quantities $s_{i}^{2}$ in the right hand sides of both equations are the unbiased estimates.

Example

Consider the following set of data for y obtained at various levels of the independent variable x.

x	y
1	31, 30, 29
2	42, 41, 40, 39
3	31, 28
4	23, 22, 21, 19, 18
5	21, 20, 19, 18,17

The number of trials, mean, variance and standard deviation are presented in the next table.

x	n	y_mean	S_y²	S
1	3	30.0	1.0	1.0
2	4	40.5	1.67	1.29
3	2	29.5	4.5	2.12
4	5	20.6	4.3	2.07
5	5	19.0	2.5	1.58

These statistics represent the variance and standard deviation for each subset of data at the various levels of x. If we can assume that the same phenomena are generating random error at every level of x, the above data can be “pooled” to express a single estimate of variance and standard deviation. In a sense, this suggests finding a mean variance or standard deviation among the five results above. This mean variance is calculated by weighting the individual values with the size of the subset for each level of x. Thus, the pooled variance is defined by

S_{P}^{2}={\frac {(n_{1}-1)S_{1}^{2}+(n_{2}-1)S_{2}^{2}+\cdots +(n_{k}-1)S_{k}^{2}}{(n_{1}-1)+(n_{2}-1)+\cdots +(n_{k}-1)}}

where n₁, n₂, . . . n_k are the sizes of the data subsets at each level of the variable x, and S₁², S₂², . . ., S_k² are their respective variances.

The pooled variance of the data shown above is therefore:

S_{P}^{2}=2.765\,

Pooled standard deviation

The pooled standard deviation follows from the square-root of a pooled variance.

Population-based statistics

The populations of sets, which may overlap, can be calculated simply as follows:

{\begin{aligned}&&N_{X\cup Y}&=N_{X}+N_{Y}-N_{X\cap Y}\\X\cap Y=\varnothing &\Rightarrow &N_{X\cap Y}&=0\\&\Rightarrow &N_{X\cup Y}&=N_{X}+N_{Y}\end{aligned}}

Standard deviations of non-overlapping (X ∩ Y = ∅) sub-populations can be aggregated as follows if the size (actual or relative to one another) and means of each are known:

{\begin{aligned}\mu _{X\cup Y}&={\frac {N_{X}\mu _{X}+N_{Y}\mu _{Y}}{N_{X}+N_{Y}}}\\[3pt]\sigma _{X\cup Y}&={\sqrt {{\frac {N_{X}\sigma _{X}^{2}+N_{Y}\sigma _{Y}^{2}}{N_{X}+N_{Y}}}+{\frac {N_{X}N_{Y}}{(N_{X}+N_{Y})^{2}}}(\mu _{X}-\mu _{Y})^{2}}}\end{aligned}}

For example, suppose it is known that the average American man has a mean height of 70 inches with a standard deviation of three inches and that the average American woman has a mean height of 65 inches with a standard deviation of two inches. Also assume that the number of men, N, is equal to the number of women. Then the mean and standard deviation of heights of American adults could be calculated as:

{\begin{aligned}\mu &={\frac {N\cdot 70+N\cdot 65}{N+N}}={\frac {70+65}{2}}=67.5\\[3pt]\sigma &={\sqrt {{\frac {3^{2}+2^{2}}{2}}+{\frac {(70-65)^{2}}{2^{2}}}}}={\sqrt {12.75}}\approx 3.57\end{aligned}}

For the more general case of M non-overlapping populations, X₁ through X_M, and the aggregate population $\scriptstyle X\,=\,\bigcup _{i}X_{i}$ :

{\begin{aligned}\mu _{X}&={\frac {\sum _{i}N_{X_{i}}\mu _{X_{i}}}{\sum _{i}N_{X_{i}}}}\\[3pt]\sigma _{X}&={\sqrt {{\frac {\sum _{i}N_{X_{i}}\sigma _{X_{i}}^{2}}{\sum _{i}N_{X_{i}}}}+{\frac {\sum _{i<j}N_{X_{i}}N_{X_{j}}(\mu _{X_{i}}-\mu _{X_{j}})^{2}}{{\big (}\sum _{i}N_{X_{i}}{\big )}^{2}}}}}\end{aligned}}

where

X_{i}\cap X_{j}=\varnothing ,\quad \forall \ i<j.

If the size (actual or relative to one another), mean, and standard deviation of two overlapping populations are known for the populations as well as their intersection, then the standard deviation of the overall population can still be calculated as follows:

{\begin{aligned}\mu _{X\cup Y}&={\frac {1}{N_{X\cup Y}}}\left(N_{X}\mu _{X}+N_{Y}\mu _{Y}-N_{X\cap Y}\mu _{X\cap Y}\right)\\[3pt]\sigma _{X\cup Y}&={\sqrt {{\frac {1}{N_{X\cup Y}}}\left(N_{X}[\sigma _{X}^{2}+\mu _{X}^{2}]+N_{Y}[\sigma _{Y}^{2}+\mu _{Y}^{2}]-N_{X\cap Y}[\sigma _{X\cap Y}^{2}+\mu _{X\cap Y}^{2}]\right)-\mu _{X\cup Y}^{2}}}\end{aligned}}

If two or more sets of data are being added together datapoint by datapoint, the standard deviation of the result can be calculated if the standard deviation of each data set and the covariance between each pair of data sets is known:

\sigma _{X}={\sqrt {\sum _{i}{\sigma _{X_{i}}^{2}}+\sum _{i,j}\operatorname {cov} (X_{i},X_{j})}}

For the special case where no correlation exists between any pair of data sets, then the relation reduces to the root-mean-square:

{\begin{aligned}&\operatorname {cov} (X_{i},X_{j})=0,\quad \forall i<j\\\Rightarrow &\;\sigma _{X}={\sqrt {\sum _{i}{\sigma _{X_{i}}^{2}}}}.\end{aligned}}

Sample-based statistics

Standard deviations of non-overlapping (X ∩ Y = ∅) sub-samples can be aggregated as follows if the actual size and means of each are known:

{\begin{aligned}\mu _{X\cup Y}&={\frac {1}{N_{X\cup Y}}}\left(N_{X}\mu _{X}+N_{Y}\mu _{Y}\right)\\[3pt]\sigma _{X\cup Y}&={\sqrt {{\frac {1}{N_{X\cup Y}-1}}\left([N_{X}-1]\sigma _{X}^{2}+N_{X}\mu _{X}^{2}+[N_{Y}-1]\sigma _{Y}^{2}+N_{Y}\mu _{Y}^{2}-[N_{X}+N_{Y}]\mu _{X\cup Y}^{2}\right)}}\end{aligned}}

For the more general case of M non-overlapping data sets, X₁ through X_M, and the aggregate data set $\scriptstyle X\,=\,\bigcup _{i}X_{i}$ :

{\begin{aligned}\mu _{X}&={\frac {1}{\sum _{i}{N_{X_{i}}}}}\left(\sum _{i}{N_{X_{i}}\mu _{X_{i}}}\right)\\[3pt]\sigma _{X}&={\sqrt {{\frac {1}{\sum _{i}{N_{X_{i}}-1}}}\left(\sum _{i}{\left[(N_{X_{i}}-1)\sigma _{X_{i}}^{2}+N_{X_{i}}\mu _{X_{i}}^{2}\right]}-\left[\sum _{i}{N_{X_{i}}}\right]\mu _{X}^{2}\right)}}\end{aligned}}

where:

X_{i}\cap X_{j}=\varnothing ,\quad \forall i<j.

If the size, mean, and standard deviation of two overlapping samples are known for the samples as well as their intersection, then the standard deviation of the aggregated sample can still be calculated. In general:

{\begin{aligned}\mu _{X\cup Y}&={\frac {1}{N_{X\cup Y}}}\left(N_{X}\mu _{X}+N_{Y}\mu _{Y}-N_{X\cap Y}\mu _{X\cap Y}\right)\\[3pt]\sigma _{X\cup Y}&={\sqrt {\frac {[N_{X}-1]\sigma _{X}^{2}+N_{X}\mu _{X}^{2}+[N_{Y}-1]\sigma _{Y}^{2}+N_{Y}\mu _{Y}^{2}-[N_{X\cap Y}-1]\sigma _{X\cap Y}^{2}-N_{X\cap Y}\mu _{X\cap Y}^{2}-[N_{X}+N_{Y}-N_{X\cap Y}]\mu _{X\cup Y}^{2}}{N_{X\cup Y}-1}}}\end{aligned}}

References

Killeen PR (May 2005). "An alternative to null-hypothesis significance tests". Psychol Sci. 16 (5): 345–53. doi:10.1111/j.0956-7976.2005.01538.x. PMC 1473027. PMID 15869691.

External links

This article is issued from Wikipedia - version of the 11/2/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.