Hey Vicky thanks for the great post! Just wanted to understand the rationale a bit better. If you kill a treatment before it reaches statistical significance how do you know it actually is performing poorly?

What I thought you referring to with the "Check KPIs are Not Harmed" comment was to closely monitoring experiments for bugs/harmful behavior. Practically, this takes the form of a real-time monitoring dashboard or auto-kill conditions for experiments.