Quicken Loan's Regis Hadiaris on multivariate testing

I’ve written extensively in the past about multivariate testing, but I haven’t had many opportunities to interview a veteran of this important Internet marketing technique. For those of you who don’t know, multivariate testing is an advanced form of traditional A/B testing, where you randomly show two versions of content (two different Web pages, two different images or copy blocks on the same Web page, etc.) and see which version produces more conversions. Multivariate testing tools, such as Google Website Optimizer, take A/B testing to the next level by allowing thousands of different versions to be tested at the same time. I was able to corner Regis Hadiaris, whose title is “Leader of Leaders and Pursuer of WOW!” at top online lender Quicken Loans, who let us in on some of the secrets of this important testing technique.

Me: Tell me about your role at Quicken Loans. How long have you been doing that?
RH: I lead a team of project leaders responsible for improving conversion on our 15 websites and five blogs. I’ve been conducting A/B and multivariate tests at Quicken Loans for the past five years.
Me: How did you get that job? Tell me more about your background and how you think it prepared you for what you are doing now.
RH: I’ve been an Internet marketer since 1996, spending the last five years with Quicken Loans. I’ve personally lead and completed hundreds of high-profile Internet marketing programs and projects over the past 12 years. If you factor in the teams of project managers I have lead, that number of projects is easily in the thousands.
Prior to Quicken Loans, I worked for two different Internet marketing consulting firms managing websites for GM, the State of Michigan, and several startups.
I have a degree in finance (hence the interest in numbers and statistics used in testing) and a natural curiosity to continuously improve stuff.
Me: How do you measure success at Quicken Loans? Do you have a Web metrics facility that keeps score for you? What has your experience been?
RH: We are highly metrics focused at Quicken Loans. While I cannot share the specific metrics we track, I can tell you that we have analyzed every part of the customer experience and track our success at each step. Every team member plays a part in this.
Me: It seems like there were lots of things you could have done to improve a financial services site. What first attracted you to the idea of multivariate testing?
RH: The humility to believe that I don’t know everything!
Seriously, in the five years I’ve been running tests, I’m only about as correct in guessing the results as a major league baseball player is in hitting the ball. That’s right – I’ve been doing this for 5 years, and I can only “guess” the outcome of a test about 33% of the time! And a note to your readers: Once you open your mind to the idea that you don’t know what is “right” all the time, you’ll be amazed at what you learn!
Me: How did you choose your vendor? Were there features you just had to have? Or were other reasons more important to your choice.
RH: Any time we choose a company to partner with, the focus is on how they will adjust to our needs. We have worked with several companies over the years, and that is the biggest priority for us in those relationships.
I also believe that being able to easily create tests on your own, and easily bring test data into other reporting tools should be key priorities for any testing program.
Me: Did you have to work through any executive objections or culture changes to get multivariate testing accepted or to get quick decisions made based on test results?
RH: We have a saying at Quicken Loans: “it’s not about who’s right, it’s about what’s right.” Basically, that gives anyone in the company the right to challenge any idea. Testing often provides the “what” (i.e. objective results) so we don’t have to have big discussions about opinions.
Me: What results did you see with your first multivariate test? What’s your favorite multivariate test story at Quicken Loans?
RH: In our first test, we saw a triple-digit increase in a key metric. It was then that we knew there was something to this “testing stuff.” However, in that test we also quickly learned how “environmental variables” (things like time of year, economic conditions, etc) and statistical significance can all impact a test.
My favorite story was the time we were able to demonstrate a direct relationship between what we tested on the Web site, and its impact on the customer experience later, offline.
Me: What was the biggest surprise you’ve found from one of your tests?
RH: We’ve had several surprises and literally countless opportunities to learn over the years. A few highlights:

Knowing when to apply A/B testing vs. multivariate testing. For example, when you are looking for big gains, its best to try dramatically different ideas/pages in an A/B format, and then follow-up with a multivariate test on the winning page.
How too narrowly defining what you optimize for can actually hurt your business. For example, if you highly optimize for an on-page action (let’s say using an online calculator), you may inadvertently remove other things from the page that drove business for you.
How different “environmental variables” can yield different results. Different economic times, different months of the year, and different traffic sources can all impact your test.

Me: Is there anything about multivariate testing that could be better? Or any mistakes you had to correct as you first started testing this way? It would help my readers to hear about lessons you learned that they won’t have to. Do you have any advice for companies considering making this move?
RH: There are always ways multi-variable testing could be better: easier tools, better integration with WordPress and e-commerce platforms, and clearer reporting all come to mind.
Also, the folks at Marketing Experiments have done a good job of putting a formalized testing process in place (definitely check out their online testing class). We’ve used a modified version of that process for some time, with success.
Here are my top five recommendations when conducting a test:

Test your “top of the funnel” pages first. These are usually landing pages visitors see after clicking on your paid advertising. You’ll get a bigger “bang for your buck” by starting at these high volume pages instead of a page further “downstream” in your process.
Consider all metrics when planning your test. Are there other “on-page” or “downstream” metrics you need to optimize for? Will running the test inadvertently hurt other key metrics? Will the act of testing itself reduce your website conversion? These are the major questions to ask yourself and monitor while the test is running.
Run all of your tests to statistical significance, but be willing to “prune” poor performers out of the ideas you are testing at lower confidence levels. For example, I prefer to run tests to 95% confidence (i.e. 95 times out of 100 the result I’m seeing will be true). However, there are times when I make decisions off of data with 80% or 90% confidence. It all depends on the business need and how the other ideas in the test are performing.
Actively manage your stakeholder’s expectations about how long a test will take. Plan enough time to find and fix tracking issues. Predict, and constantly update predictions about, how long a test will take to reach your desired statistical significance level. It’s important to “arm” executives with the choice, for example: “If we end the test tomorrow, we can be 85% confident in its results. However, if we wait 10 days, we can be 95% confident. What would you like to do?”
Integrate your testing tools with your website analytics tools. It’s critical to have additional insights at your fingertips when analyzing test data. Let’s say you find a winning idea in a test. You’re next question will be “why did that work?” Having your testing and web analytics data combined dramatically helps you answer that question.

Me: Thanks so much, Regis, for sharing your experiences with my readers.