Skip to content

References: A/B Testing Methodology for LLMs

  1. A/B testing - Wikipedia - Comprehensive coverage of controlled online experiments including hypothesis design, traffic splitting, and statistical analysis that frame this chapter's LLM-specific guidance.

  2. Statistical hypothesis testing - Wikipedia - Foundational coverage of null hypothesis, p-value, and significance level used throughout this chapter's worked examples.

  3. Statistical power - Wikipedia - Explanation of the false-negative concept central to sample-size calculation; complements the chapter's emphasis on designing tests that can detect real effects.

  4. Trustworthy Online Controlled Experiments - Ron Kohavi, Diane Tang, Ya Xu - Cambridge University Press - The definitive academic-practitioner reference on A/B testing co-authored by leaders of the Microsoft, Airbnb, and LinkedIn experimentation programs; the structure of this chapter borrows heavily from their framework.

  5. Statistics (4th Edition) - David Freedman, Robert Pisani, Roger Purves - W. W. Norton - The accessible undergraduate statistics text that grounds the formal probability-theory chapters in real examples; ideal for engineers who want to deepen the statistical intuition this chapter builds on.

  6. Microsoft Experimentation Platform: KDD Tutorials - Ron Kohavi et al. - Decade-plus archive of A/B testing tutorials, papers, and case studies from one of the largest experimentation programs in industry; rich source of advanced material beyond this chapter.

  7. Evan Miller's A/B Test Calculators - Evan Miller - Free web calculators for sample size, sequential testing, and Bayesian inference that this chapter references for hands-on exercises.

  8. GrowthBook Documentation - GrowthBook - Reference for the open-source experimentation platform used in this chapter's examples for traffic splitting and effect-size reporting.

  9. Eppo Blog on Experimentation - Eppo - Practitioner posts on CUPED variance reduction, sequential testing, and metric design that extend this chapter's foundational treatment.

  10. Lilian Weng: A/B Testing for ML - Lilian Weng - Working notes from an OpenAI researcher on experimentation patterns specific to ML-driven products; relevant for adapting traditional A/B methodology to LLM-quality measurement.