Evolution driven multivariate testing on the web

Supervisors: Denis Lalanne

Student: Jan Aurel Kühni

Project status: Finished

Year: 2015

Today, most web interaction is designed based on the assumption that developers and interaction designers are able to come up with the best and most appropriate solution for a given interaction problem. In general, developers work on a few possible implementations, internal user testing then decides which version is published on the web. The instantiated solution remains static and is only adjusted when a completely new version is created. The main problem with such an approach lies in the nature of web interaction. Having to create an optimal solution that works for an ever changing and strongly heterogeneous user group is very challenging. Fluctuation in user behavior over time as well as the large number of possible visual configurations make the task of finding an optimal solution impossible.

A first paradigm change can already be observed on the web today. More and more companies start to shift their focus from the developers to the user by implementing some aspects of A/B or multivariate testing. Adhering to this approach, developers come up with a series of possible solutions for a given problem and measure performance based on some success criteria in order to find the most appropriate implementation for that specific goal. Since this process is quite tedious and can only be performed by a web developer, a variety of testing frameworks have emerged in the past few years. While the A/B testing approach is very promising and can deliver great results if done correctly, it does not fully adhere to the described paradigm shift. In the standard A/B or multivariate testing approach, developers are still creating the versions that will be tested, forcing their view of how the interaction is best designed upon the users.

The aim of this master thesis lies in the exploration and extension of various A/B testing aspects as well as the development of a testing framework that will facilitate and demonstrate the acquired concepts. In order improve and push the A/B approach further, we will utilize genetic algorithms to find the optimal configuration of a visual web element through gradually changing a series of variables over time. Optimality in this context means a solution that performs best in terms of a defined success criteria. The automated testing framework will allow developers to specify certain parameters of their user interface that are able to evolve over time within a certain bound. In order to further improve this approach, the framework will adaptively change the probability for a certain mutation to be sown to a user based on its pervious scores.

Document: report.pdf