Many apps today employ the tactic of performing A/B tests with their features. This is a very valuable approach. For those unfamiliar with the approach, the basic notion is that you provide 2 or more variations of the same feature. You then assign each variation to your user population in a random manner. Next, you monitor your application stats and see which version of the feature does the best. The version that does the best can then be released to your entire user base.
Common Issues with A/B Testing
While there are great advantages to this approach, there are also pitfalls that need to be avoided. The main pitfall is that simply too many experiments can cause problems with each.
Duplicate Feature Code Paths
In many cases, when you add an experiment, the developer implementing that experiment will literally have to code two different versions of the A/B test. They usually don’t have to code it all from scratch, but generally speaking it requires a lot more code. More code equals more opportunities for issues to exist.
Testing Complexity
With each new A/B test you add a new set of variations that need to be tested. The more game features that the test interacts with, the greater the complexity of the testing.
Let's assume that we have 5 different A/B test that all influence a particular feature in a different way. Each test has 2 different variations. That means that there are now 10 different varieties of this feature now available to your users. Adding those new varieties complicated the testing matrix by a factor of 10.
Rules to Avoid A/B Testing Nightmares
The most basic rule to help keep over experimentation from occurring your app is to simply remove experiments when they are finished. This is a task that is often overlooked and skipped over. the reason is simple. It doesn’t have to be done and often requires coordination and effort. The product managers have to ensure that the right data was retrieved from the experiment. The developers need to remove the experiment in a fashion that will not ruin the game play for users. In addition the user migration from one experiment group to another has to be considered and dealt with.
A good rule to implement in your product is for every experiment that is added, a minimum of 1 old experiment must be removed. This helps to auto prune old experiments as it were as time goes by. It can be daunting to see dozens of experiments in your product and think, “it is going to take weeks to remove all these”. Well - if you look at it from that approach it might never get done. Instead go at it piece by piece, removing one or two experiments every time a new experiment is added.
An alternate version of this rule is to schedule the removal of one or two experiments per week. Write them down in a list and assign them out to various dev. This usually helps to keep their plate from being overloaded with the task while also ensuring that it does actually get done.
One final rule to implement in your group is the rule about experiments overlapping each other. Sometimes this cannot be avoided — but whenever possible, do not overlap multiple experiments in the same feature and / or same pieces of code. This is just asking for issues from an engineering / testing standpoint. In addition, overlapping experiments can taint the data collected from the experiment providing product managers with a false outcome in the data.
