Statistical Approaches in Online Testing
The talk would be devoted to the overview of statistical methodologies for online tests results analysis and problems arising during their application. We will consider mainstream statistical approaches, e.g. frequentist and Bayesian, and analyze their pros and cons for application in online testing. Then we will highlight the challenges that arise in online testing, e.g. continuous monitoring, multiple testing, Simpson’s paradox, delayed responses, etc, and describe existing modern ways for their solution. Also, we will explain how these problems are solved in Oracle Maxymiser. In the end of the talk we will briefly cover new advanced challenges in online testing.