Design Effect · Tae Hyun Kim (Lowell)

Definition

The Design Effect (DEFF) measures the impact of a complex sampling design on variance relative to simple random sampling.

In cluster randomization: $DEFF = 1 + (m - 1) \times ICC$

where:

$m$ : average number of samples per cluster
$ICC$ : Intraclass Correlation Coefficient

Effective sample size: $n_{eff} = \frac{n}{DEFF}$

Intuitive Understanding

In cluster randomization, observations within the same cluster are similar to one another (e.g., people in the same city).

If everyone in a cluster were exactly identical, then observing 100 people in that cluster would yield only one person’s worth of information. The ICC measures this “similarity,” and the DEFF quantifies the resulting loss of information.

Key Properties

ICC (Intraclass Correlation Coefficient)

$ICC = \frac{\sigma^2_{between}}{\sigma^2_{between} + \sigma^2_{within}}$

$ICC = 0$ : no within-cluster correlation → DEFF = 1
$ICC = 1$ : perfect within-cluster correlation → DEFF = m

Impact of the DEFF

ICC	m=10	m=50	m=100
0.01	1.09	1.49	1.99
0.05	1.45	3.45	5.95
0.10	1.90	5.90	10.90

With an ICC of 0.05 and 100 people per cluster, the effective sample shrinks to roughly 1/6.

Why Use Cluster Randomization?

When interference arises in a pricing experiment, individual-level randomization becomes infeasible:

A friend gets a discount, so I demand one too
Price differences are shared over social media
Resale (arbitrage)

Cluster randomization reduces interference, but at the cost of statistical power.

Example

Sample Size Calculation

import numpy as np

def sample_size_clustered(baseline_rate, mde_relative, icc, cluster_size,
                          alpha=0.05, power=0.80):
    """
    클러스터 랜덤화를 위한 샘플 사이즈 계산
    """
    from scipy import stats

    # 개인 수준 필요 샘플
    p1 = baseline_rate
    p2 = baseline_rate * (1 + mde_relative)

    z_alpha = stats.norm.ppf(1 - alpha/2)
    z_beta = stats.norm.ppf(power)

    n_individual = 2 * ((z_alpha + z_beta)**2 * p1 * (1-p1)) / (p2 - p1)**2

    # 설계 효과
    deff = 1 + (cluster_size - 1) * icc

    # 클러스터 수준 필요 샘플
    n_clustered = n_individual * deff

    # 필요 클러스터 수
    n_clusters = np.ceil(n_clustered / cluster_size)

    return {
        'n_individual': int(n_individual),
        'deff': deff,
        'n_clustered': int(n_clustered),
        'n_clusters_per_arm': int(n_clusters / 2),
        'total_sample': int(n_clusters * cluster_size)
    }

# 예: 도시 수준 클러스터, ICC=0.05, 도시당 1000명
result = sample_size_clustered(
    baseline_rate=0.05,
    mde_relative=0.10,
    icc=0.05,
    cluster_size=1000
)

print(f"개인 랜덤화 필요 샘플: {result['n_individual']:,}")
print(f"설계 효과: {result['deff']:.2f}")
print(f"클러스터 랜덤화 필요 샘플: {result['n_clustered']:,}")
print(f"필요 클러스터 수 (그룹당): {result['n_clusters_per_arm']}")

Cluster Randomization Implementation

import hashlib

def cluster_randomize(cluster_id, experiment_name, treatment_prob=0.5):
    """클러스터 수준 무작위 할당"""
    hash_input = f"{cluster_id}_{experiment_name}"
    hash_value = int(hashlib.md5(hash_input.encode()).hexdigest(), 16)
    return 'treatment' if (hash_value % 100) / 100 < treatment_prob else 'control'

# 도시 수준 클러스터
cities = ['Seoul', 'Busan', 'Incheon', 'Daegu', 'Daejeon']
for city in cities:
    group = cluster_randomize(city, 'price_exp_2024')
    print(f"{city}: {group}")

ICC Estimation

import statsmodels.formula.api as smf

def estimate_icc(data, outcome_col, cluster_col):
    """
    혼합 효과 모델로 ICC 추정
    """
    # 널 모델 (무작위 절편만)
    model = smf.mixedlm(
        f"{outcome_col} ~ 1",
        data,
        groups=data[cluster_col]
    )
    result = model.fit()

    # 분산 성분 추출
    var_between = result.cov_re.iloc[0, 0]  # 클러스터 간 분산
    var_within = result.scale              # 클러스터 내 분산

    icc = var_between / (var_between + var_within)

    return icc

# ICC 추정
icc = estimate_icc(data, 'purchase_amount', 'city')
print(f"추정 ICC: {icc:.4f}")

A-B Testing - the context in which DEFF applies
Statistical Power - what DEFF affects
CUPED - an alternative approach to variance reduction

References

Kish, L. (1965). Survey Sampling.
Donner, A., & Klar, N. (2000). Design and Analysis of Cluster Randomization Trials.
Comprehensive Personalized Pricing Guide, Part V, §15.3

Local graph