Developing an A/B Testing Program
SUMMARY
When I started at Persona there was lots of talk about A/B testing the website, but there was very little action. No one understood how, what, or why to test, but they knew they should. Armed with a background in tech, mathematics, and UX, I developed an A/B testing process, together with the SVP of growth, that created actionable data for the business, improved the overall customer experience, and doubled conversion rates.
ROLE
Product Design, Product Strategy,
Research, Prioritization, Growth, A/B Testing & Experimentation
January 2020 - Present
Background
Data is everywhere. It is as much a product as the service or physical goods companies sell. Like many companies, Persona had no shortage of data, however, the data they had did little to inform marketers about their customers. Demographics and general statistics did not provide insights into the why of customer behavior.
Persona was also victim to the HiPPO conundrum, where the highest paid executive made decisions regarding the website based on “gut instincts”. This led to fluctuating conversion rates and changes made on best guesses, rather than definitive data.
The company needed a better way to understand customer behavior.
PROBLEM STATEMENT
How might we create an unbiased way to implement changes on our digital platforms such that we can objectively measure impact on customer behavior to learn what content, interactions, prices, and experiences resonate with our target audience.
Think
RESEARCH
The first thing we did to establish an A/B testing program was to garner executive support. To do this we researched testing tools and provided leadership with case studies from other similar companies where A/B testing was implemented successfully.
We also involved leadership in the A/B testing process, asking questions such as:
-
What are our key business objectives?
-
What are the goals of the website?
-
What are our key performance indicators?
-
What are our target metrics?
IDENTIFYING TRENDS
In order to build a successful A/B testing program, we needed to determine what items to test that would give us the biggest ROI. We looked at user surveys, customer care tickets, and reports from user interviews to identify key concepts to test. We analyzed analytics on pages with the most potential, looking at bounce rates, exit rates, and funnel drop off rates. Exit surveys were also done to determine why users did or did not complete a conversion goal. This helped us understand the motives behind the actions.
From the image above you can see that only 14% of traffic proceed from step 2 to step 3. It was clear we had identified a drop-off step in the transactional funnel, from which we asked:
What information were customers looking for that wasn't provided?
Was there anything stopping them from taking action on the page?
What were they expecting to see on the page?
Where are the visitors coming from?
Are they not motivated enough to proceed?
PRIORITIZATION
To get the biggest ROI, we decided to test pages in the core conversion funnel. In other words:
-
Landing pages
-
Vitamin Advisor Questionnaire
-
Recommendations page
-
Checkout pages
We built a backlog of hypotheses we wanted to test. Many of these exploited common cognitive biases and usability heuristics.
We also constructed a framework to prioritize testing efforts. Those with the highest score are prioritized first.
Make
The team and I have conducted well over 150 tests in the last two years. We have doubled conversion rates and provided an uplift of 200+ basis points. Below are screen shots of some of the tests we have conducted. We continue to build, measure, and learn.
Check
Before rolling out any variation site wide we require 95% statistical significance or higher to be sure that it will positively impact the business. We have developed a cycle that we adhere to of measure, analyze, test, optimize. It is an iterative process and one that has proven results.