Design effect

In survey methodology, the design effect (generally denoted as ${\displaystyle D_{\mathrm {eff} }}$ or ${\displaystyle D_{\mathrm {eft} }^{2}}$) is a measure of the expected impact of a sampling design on the variance of an estimator for some parameter. It is calculated as the ratio of the variance of an estimator based on a sample from an (often) complex sampling design, to the variance of an alternative estimator based on a simple random sample (SRS) of the same number of elements.^: 258 The ${\displaystyle D_{\mathrm {eff} }}$ (be it estimated, or known a priori) can be used to adjust the variance of an estimator in cases where the sample is not drawn using simple random sampling. It may also be useful in sample size calculations and for quantifying the representativeness of a sample. The term "design effect" was coined by Leslie Kish in 1965.

The design effect is a positive real number that indicates an inflation (${\displaystyle D_{\mathrm {eff} }>1}$), or deflation (${\displaystyle D_{\mathrm {eff} }<1}$) in the variance of an estimator for some parameter, that is due to the study not using SRS (with ${\displaystyle D_{\mathrm {eff} }=1}$, when the variances are identical).^: 53, 54

Some potential complex sampling that could introduce ${\displaystyle D_{\mathrm {eff} }}$ that is different than 1 include: cluster sampling (such as when there is correlation between observations), stratified sampling, cluster randomized controlled trial, disproportional (unequal probability) sample, non-coverage, non-response, statistical adjustments of the data, etc..

${\displaystyle D_{\mathrm {eff} }}$ can be used in sample size calculations, quantifying the representative of a sample (to a target population), as well as for adjusting (often inflating) the variance of some estimator (in cases when we can calculate that estimator's variance assuming SRS).

The term "Design effect" was coined by Leslie Kish in 1965.^{: 88, 258} Ever since, many calculations (and estimators) have been proposed, in the literature, for describing the effect of known sampling design on the increase/decrease in the variance of estimators of interest. In general, the design effect varies between statistics of interests, such as the total or ratio mean; it also matters if the design (e.g.: selection probabilities) are correlated with the outcome of interest. And lastly, it is influenced by the distribution of the outcome itself. All of these should be considered when estimating and using design effect in practice.^: 13