Orthogonal Traffic Assignment in Online Overlapping A/B Tests

EasyChair Preprint no. 3110

Abstract

Online controlled experiments, aka A/B tests, are widely used in data-driven decision-making of many companies, including Google, Amazon, Facebook, Microsoft, Yahoo etc. With hundreds of experiments running at the same time in Wechat, we employ an overlapping infrastructure to assign traffic. Traffic assignment in an improper way will cause unreliable conclusions to experiments, which may causes very significant business degradation. Even a little fractions of one percent key metric degradation will brings severe consequences. In this paper we propose a novel approach for traffic assignment in online overlapping A/B tests, this approach aims to reduce interactions between experiments with traffic overlap by a orthogonal traffic assignment design. We illustrated how this technique can reduce interactions between experiments, and evaluated the effects of using this technique compared to regular method. This technique, along with some other mechanisms, formed a reliable traffic assignment infrastructure for trustworthy overlapping A/B tests.

Keyphrases: A/B testing, Galois Field, traffic assignment

@Booklet{EasyChair:3110,
  author = {Tao Xiong and Yong Wang and Senlie Zheng},
  title = {Orthogonal Traffic Assignment in Online Overlapping A/B Tests},
  howpublished = {EasyChair Preprint no. 3110},

  year = {EasyChair, 2020}}