Eytan Bakshy, eytan@fb.com | Sean J. Taylor, sjt@fb.com
Sunday, June 1, 9:00am - 12:00pm
Room 1255, North Quad Complex, University of Michigan, 105 South State Street, Ann Arbor
This tutorial teaches attendees how to design, plan, implement, and analyze online experiments. First, we review basic concepts in causal inference and motivate the need for experiments. Then we will discuss basic statistical tools to help plan experiments: exploratory analysis, power calculations, and the use of simulation in R. We then discuss statistical methods to estimate causal quantities of interest and construct appropriate confidence intervals. Particular attention will be given to scalable methods suitable for “big data”, including working with weighted data and clustered bootstrapping. We then discuss how to design and implement online experiments using PlanOut, an open-source toolkit for advanced online experimentation used at Facebook. We will show how basic “A/B tests”, within-subjects designs, as well as more sophisticated experiments can be implemented. We demonstrate how experimental designs from social computing literature can be implemented, and also review in detail two very large field experiments conducted at Facebook using PlanOut. Finally, we will discuss issues with logging and common errors in the deployment and analysis of experiments. Attendees will be given code examples and participate in the planning, implementation, and analysis of a Web application using Python, PlanOut, and R.
Basic knowledge of statistics and probability theory, and some familiarity with programming. We will be using R to do exercises involving power calculations and data analysis, so we recommend that attendees either have experience with R or come with a buddy who knows R.
pip install planout
at your terminal to install)pip install flask
at your terminal to install)Slides and code will be posted shortly before the tutorial.
Part 1: Experiments, causal inference, and planning.
1. Introduction.
2. Planning Experiments
Part 2: Implementing and analyzing experiments.
3. Designing and implementing experiments with PlanOut
4. Analyzing experimental data
Here are a few links to papers and books that we discuss or draw upon in the tutorial.
Field Experiments: Design, Analysis, and Interpretation. Alan Gerber & Donald Green. 2012.
Counterfactuals for Causal Inference: Methods and Principles for Social Research. Stephen L. Morgan & Christopher Winship. 2007.
Data Analysis Using Regression and Multilevel/Hierarchical Models. Andrew Gelman & Jennifer Hill. 2006.
Designing and Deploying Online Field Experiments. Eytan Bakshy, Dean Eckles, Michael Bernstein. ACM WWW 2014.
Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods. Eytan Bakshy & Dean Eckles. ACM KDD 2013.
On the near impossibility of advertising experiments. Randall A. Lewis & Justin M. Rao.