Beating A/B testing with 20 lines of code
Finding the best interface choice and enticing banner ad relies on A/B testing by users. But machine learning can replace the focus groups with 20 lines of code.
That's the idea presented by Steve Hanov using the example of which slot machine in a casino you suspect will pay the best. For websites, that may become a Buy Now!! button color. Rather than have thousands of potential customers test it, Hanov suggests using code and the epsilon-greedy method, the simplest strategy to the most popular choice, derived from the casino example.
Using an example from one of his own sites, Hanov reports a green button outperformed orange and white buttons. Why don't people use this method? Because most don't understand machine learning algorithms, and mainstream tools don't support it. And if you need a refresher on A/B testing, try, "The Ultimate Guide to A/B Testing" relying on alternate pages based on user choices.
I like it
beating a random strategy (or the one that relies on random for a portion of its execution) is always a tough proposition. This is a really nice both pragmatic and practical approach - love it.
Tim on stevehanov.ca
Definitely going to use this. Perhaps this method should be called A/Z testing or A/n testing.
solecoder on stevehanov.ca
Decision optimization is the next step past A/B testing.
Kevin on stevehanov.ca
I think this comes down to the debate between the marketer and the designer. And it will always be a great debate.
Jen Walter on smashingmagazine.com
I've seen minor tweaks to a form raise conversions by 20%. The person running a particular test may not even know that a particular change was significant, or even that it happened.
btilly on news.ycombinator.com
One hypothetical case it could end up serving the worse design more often is if you had times of the day where your users behave very differently due to time zones, i.e. Europe vs North America.
medell on news.ycombinator.com
I disagree with the notion that you should never let artistic vision trump the results of a test.
Andrew Ingram on smashingmagazine.com
However, to claim "The strategy that has been shown to win out time after time in practical problems is the epsilon-greedy method" is false.
noelwelsh on news.ycombinator.com
Will this help resolve any arguments about building your website?