Trending Now

Multivariate testing in action

I’ve written before about multivariate testing, the best way of optimizing your Web site to drive conversions. I was lucky today to speak with the president of ReallyGreatRate.com, Matt Schaub, who told me his story of how he’s improved his Web marketing through multivariate testing. If you’re not using this technique, read on to find out how it works.


ReallyGreatRate.com is an online financial lead generation company that connects lenders to people looking for student loans, debt consolidation loans, and home mortgages. The company does all of its marketing online, passing loan applicants to up to four lenders (getting compensated for each valid lead passed). You can imagine how valuable it is to increase the number of leads passed, which requires convincing more people to complete the online application form. ReallyGreatRate.com Submission Form
But how do you do that? Everyone has an opinion of what should be changed. Multivariate testing boils those opinions down to testable sizes. Matt started with the submission form you see above. He worked with Optimost, a leading multivariate testing vendor, to identify 13 separate areas on the page that could be modified, and came up with seven or eight different treatments for each area.
For example, the masthead at the top of the page was one such area. Matt worked with Optimost to come up with several variations, such as changing the location of the image to the left side, changing the words in the tag line, and many others.
Optimost ran several rounds of testing to cycle through each variant, identifying the ones that seemed to outperform the rest—the ones that correlated with more loan applicants pressing that “Submit” button. Within four months, a winning design was chosen. ReallyGreatRate.com Submission Form
It doesn’t look all that different, does it? It’s a bit less cluttered on the left side, removing some things that proved more distracting than helpful, and the most noticed text in the middle of the screen has been reworded to emphasize starting the process. But it looks very similar to what Matt already had.
Here’s what wasn’t so similar: Form completion went up 116%! A new form that didn’t look all that much different than the old one had dramatically different results. Now, no expert could look at these two forms and know for sure which would test better. And, remember, thousands of different variants were tested, which no expert could do either.
So now Matt is happy he’s got the best one, right? Wrong. “I’m starting a new round of testing this week,” he told me. “I’m going to constantly change my form. You have to! What works now may not work next month.”
When I tell people to “Do it wrong quickly,” this is what I am driving at. Experiment. Measure. Keep what is working and change all of it, working or not. Matt advised all Web marketers, “Never get complacent.”

Mike Moran

Mike Moran is a Converseon, an AI powered consumer intelligence technology and consulting firm. He is also a senior strategist for SoloSegment, a marketing automation software solutions and services firm. Mike also served as a member of the Board of Directors of SEMPO. Mike spent 30 years at IBM, rising to Distinguished Engineer, an executive-level technical position. Mike held various roles in his IBM career, including eight years at IBM’s customer-facing website, ibm.com, most recently as the Manager of ibm.com Web Experience, where he led 65 information architects, web designers, webmasters, programmers, and technical architects around the world. Mike's newest book is Outside-In Marketing with world-renowned author James Mathewson. He is co-author of the best-selling Search Engine Marketing, Inc. (with fellow search marketing expert Bill Hunt), now in its Third Edition. Mike is also the author of the acclaimed internet marketing book, Do It Wrong Quickly: How the Web Changes the Old Marketing Rules, named one of best business books of 2007 by the Miami Herald. Mike founded and writes for Biznology® and writes regularly for other blogs. In addition to Mike’s broad technical background, he holds an Advanced Certificate in Market Management Practice from the Royal UK Charter Institute of Marketing and is a Visiting Lecturer at the University of Virginia’s Darden School of Business. He also teaches at Rutgers Business School. He was a Senior Fellow at the Society for New Communications Research and is now a Senior Fellow of The Conference Board. A Certified Speaking Professional, Mike regularly makes speaking appearances. Mike’s previous appearances include keynote speaking appearances worldwide

Join the Discussion

Your email address will not be published. Required fields are marked *

Back to top