In this position paper, we formulate performance regression testing as an automated experimentation problem and focus on the problem of controlling the experiment so as to provide more computation time to experiments that are more likely to detect performance changes. Conversely, this requires detecting and stopping experiments early if they are unlikely to detect any performance changes. To this end, we present a method that uses results from previous performance testing experiments to predict the outcome of new experiments in early stages of their execution.