AB-tests
On NetOnNet
Summary
As a compliment to usability testing and design guidelines, NetOnNet conducts AB-tests to measure how different metrics are affected by a desired design change.
What did the process look like?
When the CRO-specialist had a hypothesis, I started my work by sketching on ideas in order to start a conversation with developers as soon as possible. Usually I also read e-commerce design recommendations to be able to make informed decisions even when I had no other qualitative or quantitative insights.
After a few iterations, consisting of feedback from the AB-testing team and my design colleagues, I came forward with 1-3 variants that was going to be tested.
What did I learn?
1) To write hypotheses
2) To design based on a hypothis
3) Understanding of terms such as goal metrics/KPI:s and statistical significance
4) The developers’ work process and limitations when working with an AB-testing tool
5) To make suggestions for future AB-tests
6) When the UX-team was the source of a hypothesis, come up with and argument for a test plan that might span over several tests
How many AB-tests was I involved in?
I actually haven’t counted, but I would guess somewhere between 15-20. :)
What do I think of AB-testing?
It’s great as a tool to get qualitative data on metrics that are important to the business, and a great way to understand approximately “what” happens when a certain design change is made. However, it’s definitely not the right tool to use when figuring out “why” something works or doesn’t work.
It’s important to know why you’re spending resources on a whole AB-testing process, instead of any other, a lot cheaper, method.
Sometimes AB-testing has helped NetOnNet immensely, though, and kept us from making changes that everyone involved actually thought was good for the business.
What did I do?
As part of a team consisting of one CRO-specialist, a product owner, and a few developers, I was responsible for making design suggestions based on the hypothesis that was going to be tested. As each AB-test is assigned to one UX-designer, I did most of the design work on my own, except from when I took design feedback from my colleagues.
