Hi Leihua,

Yeah looks good to me. And thanks for sharing each step.

First, the reason you see negative adjusted lift is because your observed lift is not greater than the critical threshold b. This method limits the graduation criteria to be 1) stat sig and 2) positive lift experiments. Because 1.348 < 1.96, the assumption is not met.

Second, if your standard deviation is not equal for both groups (which usually the case) you need to use pooled standard deviation - article and video.

Third, just an FYI - a standard error is an estimate of the standard deviation of a given statistic. In this case, we're looking for the standard error of the lift. The steps are simple - 1) get the standard deviation of the statistic, 2) divide by sqrt(n).

Hope this helps anyone else looking to implement the method. And thanks again for the thorough comment.

Michael