Visual Clarity Analysis
We analyzed 28 travel booking websites with EyeQuant's Visual Clarity algorithm, which assigns a 0-100 score to web designs. Lower scores indicate cluttered designs, and higher scores indicated cleaner, clearer interfaces. The algorithm was trained with real user research data from hundreds of websites, and is 90% as accurate as a 200-participant study. Booking.com scores a 32, lagging far behind the industry average of 71, proving that Booking's homepage is objectively much more cluttered than most other travel sites.
Visual Attention and Distractors
To evaluate visual distractors, we used EyeQuant's visual attention model, which predicts which content will initially be most/least eye-catching, and is trained with data from eye-tracking studies. Results are 85% as accurate as a real eye-tracking study. It turns out, the personalized ads for specific destinations are more eye-catching than the search widget itself. In this case, "Poznan" is the most visible word on the page. This destination has been chosen based on previous booking history, but this doesn't necessarily align with the user's current needs.
The Local Maximum Problem
Booking has so much traffic that they're able to isolate even the smallest variable to measure its impact. So that's what they do – they run small, isolated tests on every imaginable piece of the website. This makes sense because by isolating individual variables (an image choice, a font color), it's possible to measure the impact of each individual element of the website.
However, there's a problem with relying so heavily on small, isolated iterations: the scope of your tests will be very narrow, and if you run a lot of tests like Booking does, you'll eventually reach a point where it's difficult to generate uplifts through small tweaks. This happens as you approach a Local Maximum in conversion rates.
According to Alex Birkett from ConversionXL, a local maximum in the context of conversion optimization is: "When you hit the peak of your current design. It can't get much better – even if you make a thousand small tweaks, you can only improve so much. The site is as effective as it ever will be on its current structural foundation."
Evolution Over Time
In order to move from a local maximum to a global one, your CRO program needs to include not just iterative tests, but also a series of bigger, bolder tests as well. Paul Rouke from PRWD suggests that to reach their full potential, A/B testing programs need to leverage both iterative testing (small tweaks) and innovative testing (bolder changes).
This might explain why Booking.com, Amazon and other sites that perform a lot of A/B tests look the way they do. It's not because they've reached the "optimal design", it's because they've focused primarily on getting the most out of their current design.
A Radical Redesign Concept
Just for fun, our team decided to try our hand at re-designing the Booking.com homepage, using EyeQuant to guide us through the process and help us decide what's likely to work, and what isn't.
Here are some of the changes:
- Instead of having several distracting images on the page, this design sticks to one main image that blends into the background. This image can be personalized based on a user's search history or popular locations.
- The color scheme within the search widget allows for a high level of luminance contrast, which ensures that the most important content gets noticed first. This also improves accessibility.
- The current version of Booking.com has plenty of extra links, offers, and options which are only relevant to a specific sub-set of visitors. The design above has a "menu" section in the header where some of these options can be presented.
The EyeQuant results for this design show a laser-focus on the most important content: Users immediately see what the site is about, why they should use it, and where to go next.
Key Improvements
The biggest improvement comes in the reduction of clutter on the page. This design scores an 84/100 on our visual clarity index, triple the score of Booking's actual design. It also scores better than Priceline's homepage, and isn't too far behind SkyScanner. According to EyeQuant, this design would also be more exciting than the existing Booking homepage (80 vs. 64).
That's not to say this design is the right one – again, it's only a quick mockup to serve as an example of the type of approach Booking could take if they were to embrace innovative testing in addition to the iterative testing they're already doing.
Lessons Learned
The main take-away here is that Iterative vs. Innovative testing shouldn't be an either-or decision. We mentioned earlier in this post that radical re-designs have a major drawback: it's very hard to unpack the changes you've made and figure out which of them actually drove an uplift (or drop) in conversion rates. But that's something you figure out using iterative tests!
Indeed, one CRO approach that could work very well is to run iterative tests until you hit a plateau in conversion rates, then run a series of bolder tests (such as radical redesigns). Afterwards, you can go back to another series of iterative tests to measure the impact of specific components of the new design. It's the best of both worlds.