ИСТИНА

Modern Internet companies improve their web services by means of data-driven decisions that are based on online controlled experiments also known as A/B tests. The scale of use of this state-of-the-art technique is impressive: for instance, the largest search engines report on more than hundreds concurrent experiments per day. An A/B test compares two variants of a service at a time, usually its current version and a new one, by exposing them to two groups of users. The aim of controlled experiments is to detect the causal effect of service updates on its performance relying on an Overall Acceptance Criterion (OAC). A typical OAC consists of a key metric, an evaluation statistic, and a statistical significance test. The most challenging problem is to define an appropriate OAC that is both interpretable and sensitive. This talk presents a brief overview of classical and novel approaches to obtain an OAC.