When testing any hypothesis in marketing, you need to measure the effect of your actions. This is where control groups come in. Put simply it is an isolated group of customers who don’t receive the action, providing a baseline to compare against.
You may recognise this style of experimental design from medicine. When testing a new drug, you want to know the effect it is having on the patient. This is achieved by applying either no treatment or a neutral treatment such as the well-known placebo, to a randomised group of patients.
This allows you to test your experimental results against a baseline and say with certainty if you have made a statistically significant improvement.
An example
How this applies to marketing is best illustrated with an example. A marketing manager is looking to test the effectiveness of two offers currently made to customers.
The experimental designs proposed splits the user base in two. Group 1, undergoing experiment A and Group 2, undergoing experiment B.
The results from experiment A shows that user uptake is at 15% and experiment B shows an uptake of 20%.
While this appears to show that experiment B has had a greater effect, we can’t see if there is an improvement in user engagement as neither of these treatments has a baseline to contrast against.
If however these experiments use control groups, we see a very different trend emerging. The baseline for experiment A was at an uptake 5% and for experiment B was an uptake of 15%. Giving us an absolute difference of 10% for experiment A and 5% for experiment B. Showing that experiment A had the larger overall effect.
A note on control group integrity
When you run experiments with control groups there are two major considerations that have been mentioned already but are worth repeating.
The members must be randomly selected to prevent bias in the selection process.
Also, they must be protected from other interactions for the duration of the experiment. If you ignore either you jeopardise the integrity of your experiment.
Counterfactuals: knowing the road untraveled.
Continuing with the medicine parallel. If a patient is ill you have 2 choices, to treat them or not. If we do treat them, we can’t know what would have happened if we didn’t. And if we don’t treat them, we can’t know what would have happened if we did.
This is the dilemma of a counterfactual. As much as we would love to. Once we have decided, we can’t observe what the other choice would lead to.
This is where control groups come to the rescue. They allow us to build a model which can accurately estimate what would have happened as a result of the decision we didn’t make. Meaning we can predict the best course of action to take with a new patient.
In summary
Control groups are one of the foundational building blocks on which good experimental design is built. Using them well whilst occasionally tricky is well worth the effort as you will see improvements in accuracy and returns.
For if you fail to account for the status quo you won’t know if what you are doing is having the desired effect or even any effect at all.
Have control groups work for you
At IBEX we are aware of the need and usefulness of control groups in our modelling process. Our machine learning engines utilise them to provide accurate advice and plot the best course of action, across our suite of products.
Grow your business
Use Ibex to drive customer engagement and deliver the right messages at the right time
