Beating A/B testing with 20 lines of code

By , ITworld |  Software, A/B testing, epsilon-greedy

taste test

Some writers of "CollegeHumor's Guide to College" participate in a Beer Tasting Test.

flickr/Zach Klein

Finding the best interface choice and enticing banner ad relies on A/B testing by users. But machine learning can replace the focus groups with 20 lines of code.

That's the idea presented by Steve Hanov using the example of which slot machine in a casino you suspect will pay the best. For websites, that may become a Buy Now!! button color. Rather than have thousands of potential customers test it, Hanov suggests using code and the epsilon-greedy method, the simplest strategy to the most popular choice, derived from the casino example.

Using an example from one of his own sites, Hanov reports a green button outperformed orange and white buttons. Why don't people use this method? Because most don't understand machine learning algorithms, and mainstream tools don't support it. And if you need a refresher on A/B testing, try, "The Ultimate Guide to A/B Testing" relying on alternate pages based on user choices.

I like it

beating a random strategy (or the one that relies on random for a portion of its execution) is always a tough proposition. This is a really nice both pragmatic and practical approach - love it.
Tim on stevehanov.ca

I actually noticed that Google was doing this with my Adwords account a couple of weeks ago.
steve8918 on news.ycombinator.com

Definitely going to use this. Perhaps this method should be called A/Z testing or A/n testing.
solecoder on stevehanov.ca

Next steps

Decision optimization is the next step past A/B testing.
Kevin on stevehanov.ca

I think this comes down to the debate between the marketer and the designer. And it will always be a great debate.
Jen Walter on smashingmagazine.com

I've seen minor tweaks to a form raise conversions by 20%. The person running a particular test may not even know that a particular change was significant, or even that it happened.
btilly on news.ycombinator.com

Other options

One hypothetical case it could end up serving the worse design more often is if you had times of the day where your users behave very differently due to time zones, i.e. Europe vs North America.
medell on news.ycombinator.com

I disagree with the notion that you should never let artistic vision trump the results of a test.
Andrew Ingram on smashingmagazine.com


However, to claim "The strategy that has been shown to win out time after time in practical problems is the epsilon-greedy method" is false.
noelwelsh on news.ycombinator.com

Will this help resolve any arguments about building your website?

For the latest IT news, analysis and how-tos, follow ITworld on Twitter, Facebook, and Google+.

Now read this:
Developer declares 'I am done with the Freemium Business Model'
Khan Academy offers JavaScript as their first computer language
Study says Facebook profile can predict job performance

  Sign me up for ITworld's FREE daily newsletter!
Email: 
 

Join us:
Facebook

Twitter

Pinterest

Tumblr

LinkedIn

Google+

Answers - Powered by ITworld

Join us:
Facebook

Twitter

Pinterest

Tumblr

LinkedIn

Google+

Ask a Question
randomness