Michael Berk
Oct 3, 2022

--

Typically heavy hitters will lead to higher variance in your estimates. That's a problem for most statistical signifigance tests and is solved by getting a larger sample size i.e. running the test for longer.

Switchbacks have lower power than user-split tests, so switchbacks are more sensitive to heavy hitters. The optimization discussed in the post is not impacted by heavy hitters any more than switchbacks in general.

There's a couple more nuanced points here, but TLDR is switchback tests need to run longer than user-split no matter what. Heavy hitters don't help. The optimization is unaffected by heavy hitters.

--

--

Michael Berk
Michael Berk

No responses yet