OPE Deployment-Gate Playbook — Ruling "Trust / Distrust / A/B" Without Ground Truth

Estimating what a new policy is worth from logs alone — before it ever rides an A/B test — is off-policy evaluation (OPE). But the real question in practice is not “what is $\hat V(\pi_e)$?” It is “may I trust this estimate, right now, in a deployment decision?” — because nobody knows the true value. This note curates the operating rules of the deployment gate built by the ope-to-decision benchmark.

Status. The gate and battery rules are the benchmark repo’s proposal — a systematization of scattered folklore practice, not an established standard. Every number is an abbreviated transcription of values registered in the repo’s canonical number ledger (docs/LEDGER.md); the repo is canonical for raw values, precision, and generating experiments.

1. The Verdict Procedure — Two Stages

The procedure a practitioner actually walks (the frontstage) requires nothing beyond the logged layer. The grounds for trusting that procedure (the backstage scoring) were measured on stages that know the truth — a synthetic DGP, fully-labeled classification data, and the ZOZO production log’s approximate ground truth — via blind-then-reveal: commit the verdict first; open the truth only for scoring.

2. Gate v1 — a 3-Way Verdict over Three Diagnostics

Three diagnostics computed from logs alone — ESS (effective sample size)/n · max importance weight · support proxy — are held against pre-registered thresholds and ruled three ways:

• trust — use the estimate in the decision (comparative-first, with the confounding disclaimer attached).
• distrust — reject the estimate for this log × policy combination and route it to a correction path.
• ab_fallback — the estimate itself does not stand; fall back to an online A/B test.

Backstage scoring: the large-error rate (relative error >10%) is 4.6% on trust verdicts vs 44.4% on A/B-fallback verdicts — the diagnostics genuinely forecast variance-side risk. The support proxy, however, was fully blind to structural deficiency (signal 0 vs true deficiency 0.143) — it is kept for form and explicitly not trusted.

3. The Validity Battery — a Necessary-Condition Screen (gate v2, proposal)

Four screens joining gate v1 in parallel. All of them take only the logged layer as input, and their thresholds were committed before the experiments (pre-registered) and then only evaluated (no tuning):

arm	What it catches	Principle
E[w]	multiplicative misrecording (firing rate 1.0) · structural support deficiency (recovered at the expectation level)	the Horvitz–Thompson identity: if the recorded propensities are true, E[w]=1
harmonic	action-level mismatch between recorded and empirical frequencies — recovers even local deficiency where every global signal dies	per-action T(a*)=mean(1[a=a*]/pscore), E[T]=1
placebo	whether the weighting machinery manufactures signal out of nothing	an analyst-generated independent noise reward — true value 0 by construction
disagreement	the spread of weighting-family point estimates	caution: agreement is not evidence of validity — it is smallest when everything is consistently wrong

Combination rule (pre-registered): harmonic fail → ab_fallback; E[w] · placebo · disagreement fail → distrust; otherwise the gate v1 verdict stands. The decision itself is GO/NO-GO/A/B via a SNIPS confidence interval vs same-log mean(r) comparison.

The screens themselves are not this repo’s invention — the harmonic check is folklore that Li et al. (WWW 2015) put into print with the attribution “Bottou et al., private communication”; E[w]≈1 is Lefortier et al. (2016)‘s sanity check; the placebo is a transplant of epidemiology’s negative controls. The contribution is the combination: systematizing them into a pre-registered battery, making their forecasting power itself a falsification target via blind-then-reveal, and constructively exhibiting the boundary of what they cannot see.

4. Measured — What the Battery Catches, and What It Misses

• It catches: propensity noise (firing 1.0 — direction consistent with the theoretical prediction $e^{s^2/2}$), and structural support deficiency (per-action harmonic covers at 1.0 even where every global statistic stays silent).
• Real-data replication: the same screens replicate on real covariates (injection experiments on the 4 UCI datasets) — including one pre-registered expectation refuted and logged as-is: in-sample estimated propensities, quasi-null in the synthetic world, fired on every dataset under the real data’s double-softmax geometry.
• It cannot catch (in principle): confounding whose recorded pscores are the true marginals — by observational equivalence, no log statistic can tell the worlds apart. Measured: battery firings 0/240 while the bias alone grows to −0.073.
• The cost: over-caution that fires even on harmless injections (A/B deferral), and small-sample harmonic false alarms (0.275) — the price of safety is paid in deferral.

5. The Λ-Sensitivity Certificate — an Honest Exit for What It Cannot Catch

For the confounding the battery cannot see in principle, we compute instead: “how distorted would the recorded propensities have to be to flip this conclusion?” (the marginal sensitivity model’s worst-case interval — a tool demonstration of Kallus & Zhou 2018, not this repo’s proposal). The breakdown $\Lambda^*$ has median ≈1.07/≈1.04 (synthetic demonstration) — the stronger the confounding, the smaller the sensitivity assumption at which ranking assertions collapsed. Because $\Lambda$ is an assumption scale not identifiable from data, $\Lambda^*$ is a fragility report, not a robustness certificate: in the observational-equivalence world $\Lambda^*$ contracts from 1.31 to 1.05, reporting only that “the conclusion is turning fragile without knowing why.”

6. Decision Value — Against a Naive Point Estimate

An end-to-end experiment in which even the candidate policies are built from the log (including a genuinely bad one): on a contaminated log, naive IPS point estimation hands the bad candidate a false-go of 0.9 (mean regret 0.21), while the protocol’s rate is 0.0 — the battery fires and everything falls back to A/B. On healthy logs it stays decisive — go 0.95 on the good candidate, no-go 0.9 on the bad one, zero errors. In the real round (the ZOZO production log), the harmonic arm genuinely fired and the verdict landed on AB_FALLBACK · fragile — and no answer sheet exists to check whether that verdict was right or wrong. That is the definition of practice.

7. Operating Rules in Brief

1. Comparative-first — prefer same-log Δ comparisons (common errors cancel) over absolute thresholds (V̂ > T). No mixed comparisons; treat the near-boundary region with uncertainty intervals.
2. Read battery forecasting power per failure family only — never cite a pooled firing rate on its own (it depends on the family mix).
3. Attach the disclaimer to every “pass” — every recorded-propensity-based signal (ESS · E[w] · harmonic) is jointly blind to calibrated confounding. Logs whose records cannot be trusted (manual interventions, suspected unrecorded personalization) are treated as ab_fallback regardless of the verdict.
4. “I don’t know” is a legitimate output — inconclusive verdicts and A/B deferrals are a feature of the protocol, not a failure.
5. Do not port the thresholds — they are pre-registered, untuned values conditional on this benchmark family. Using them in another domain without recalibration is unfounded.

The canonical source for the full rules, supporting numbers, and the list of limitations: docs/PLAYBOOK.md · the working process lives in the six notebooks (frontstage 00→03→05 / backstage 01→02→04).