Helen Groom looks at the intense competition over internet search rankings and asks how the search engines are trying to stop companies fixing

How does the average consumer look for a motor insurer? Increasingly, by typing “car” and “insurance” into an internet search engine such as Google.

And the site which comes out top of the list is likely to be the first one the consumer visits, meaning Google and its competitors have enormous power over price comparison sites, which live and die by their internet traffic.

Earlier this month, price comparison site GoCompare.com, formerly the site that topped the search list for car insurance, was downgraded to the sixth page of results and consequently suffered an 87% drop in the proportion of search traffic it received.

There have been other examples of price comparison sites suddenly plunging down the rankings. In the brave new world of search advertising, few people understand what determines which company comes top of the results list, or why they might fall.

Can canny online marketing play a role, or are insurers, brokers and online aggregators at the mercy of the search engines? And if so, how much impact does a search result downgrade have on business?

The most popular search engines in the UK – Google, Microsoft Live Search, Yahoo! and Ask – all offer a variety of marketing proposals, the most common being paid for search and natural search.

Paid for search is fairly straightforward – you can pay for an advert against certain search terms, which on Google will appear at the top of the page or on the right hand side of the page. Natural search, however, is where things can get interesting, and the new (and increasingly important) art of search engine optimisation comes in.

So what can companies do to increase their natural search results? Good practice involves designing your site to be search engine friendly – allowing ‘spiders’ to easily assess the information in the site, having relevant links to respected sites, and having lots of content which is frequently updated.

But sometimes all those things are not enough, and companies start looking for other ways to boost their results. And that can take them into a grey area. The search engines themselves police what can and can’t be done, though there is no official regulation.

According to Internet Advertising Bureau (IAB) programme manager Jack Wallington, the most common sins committed by companies wishing to improve their results are spam blogging, using RSS feeds to boost the amount of new content on their site (often without permission), paying for links, and tagging irrelevant keywords, including the names of competitors.

“Search engines are about relevancy and best practice. They will rate sites on how relevant they are, and on all the links leading back to the site,” says Wallington. “If they find the relevancy isn’t as high as it should be, they can assess all the links and place the site in the results where they think it would have come without the paid for or irrelevant links.”

Google clearly warns in its guidelines that paying for excessive irrelevant links can have serious repercussions for natural search results: “This is in violation of Google’s webmaster guidelines and can negatively impact your site’s ranking in search results,” it says.

Yahoo! likewise warns those using “methods to artificially inflate search engine ranking”, and it “reserves the right, at its sole discretion, to take any and all action it deems appropriate to ensure the quality of its index”.

How long such a penalty lasts can vary, says Wallington, but it would normally take one to two months to climb back up the rankings to somewhere near where it had been. All of which can be pretty bad for business.

Punitive action

GoCompare.com is recently thought to have suffered from such punitive action – though Google refuses to confirm to individual companies whether or why it has taken action.

The results were dramatic: according to online monitoring company Hitwise, the number one position saw GoCompare.com take 17.49% of all search traffic for the term in the week ending 26 January 2008. Following the penalisation, in the week ending 9 February 2008, the aggregator received only 2.31% of search traffic for “car insurance” from Google, a fall of 87%.

“It is so competitive out there and there are a lot of smaller players who are prepared to do what they can to get up the ranking. It makes it harder for us to stay at the top.

Sarah Baldwin, Confused.com

However, the situation is not as bleak as those figures might initially suggest. Moneysupermarket.com had a similar run-in with Google in the summer of 2007, where its natural search result was downgraded for about a week.

Moneysupermarket.com head of insurance Richard Mason says the experience had been frustrating as Google would not acknowledge that there was a problem and it simply had to hope its pleas for forgiveness got through to the right people.

“Fortunately because we advertise a lot on TV and we have over 200 portal partners, we get a lot of our business from other partners, so for us it didn’t have that significant an impact,” Mason says.

Detrimental to business

But others say such a downgrade has the potential to be extremely damaging for business. “We get as much as 70% of our business through search engines,” says Comparethemarket.com marketing director Mark Vile. “It would be really detrimental to our business if we were no longer appearing.”

The loss of natural search placing can, of course, be offset by spending more on paid search and other online advertising, but doing so would mean increasing the advertising and marketing spend. Comparethemarket.com currently has a roughly 50/50 split between traffic coming from paid for and natural search, indicating the size of the problem any dramatic drop in natural search placing could cause.

Indeed GoCompare.com’s webtraffic appears to have been fairly stable over the past month, according to statistics from web information company Alexa, suggesting that sites may be more resistant to the actions of search engines than many would have thought. ‘

Confused.com marketing manager Sarah Baldwin says the relationship between search engine and site is more even than many people may think.

“We have done search engine optimisation since day one. They [search engines] don’t have us over a barrel as we have strong organic links for natural search. So it’s a more even relationship,” she says.

Mason says: “They are trying to deliver what customers are looking for. If someone puts car insurance into Google it will try to order the sites as to what it thinks the people are looking for. We have to play by its rules, but that doesn’t mean that we are frustrated by it: we know what we need to do.”

Increasingly resistant

In addition, the big price comparison sites, or aggregator brands, are increasingly resistant to the attempts of search engines to police their own results. As GoCompare.com, Moneysupermarket.com, Comparethemarket.com and others spend more on marketing their own brands, consumers can increasingly find them without the aid of search engines.

Paid for results are also excluded from any action which might be taken over manipulation of natural search results. The algorithms used for paid and natural search are totally different.

While GoCompare.com might be on page six of Google for natural search, they’re right up there on page two for paid search.

So why risk the displeasure of the search engines by trying to improve your rankings? “It’s so competitive out there and there are a lot of smaller players who are prepared to do what they can to get up the ranking. It makes it harder for us to stay at the top,” says Baldwin.

Mason agrees: “If you put car insurance into Google you get over 100 million results. It’s the most competitive part of the web. It is the absolute zenith of competition.”

Online advertising boom

Internet advertising spend appears to be on an unstoppable rise throughout the UK, but is the insurance industry following the trend?
The Internet Advertising Bureau (IAB) Adspend Study, conducted in partnership with PricewaterhouseCoopers, shows that in the first half of 2007 online adspend reached £1,334m, a rise of 41.3 per cent year on year.
During this period it accounted for 14.7 per cent of the total advertising market, and is fast closing in on classified press advertising. And the IAB predicts that adspend online could be as high as £2.75bn, once the full year figures for 2007 are taken into account.
Within these figures the UK financial industry, including insurance, is the third largest spender in terms of online, accounting for 11.7 per cent of online advertising and marketing spend in the first half of 2007.
This high percentage indicates that insurers and brokers are following the trend and putting an increasing share of their advertising and marketing spend online.
Online is not a new area for insurers to spend their budgets, however. Analysis by online measurement
company Hitwise shows that in the 2005/2006 financial year, the AAs motor insurance division spent 14 per cent of its advertising and marketing budget online, making it second only to television in terms of spend.
Carole Nash managing director David Newman says: Online is becoming more important to us because
consumers are going online and are prepared to transact online.
The motorcycle broker launched its web channel just under a year ago, and is supporting its online presence with search engine optimisation, paid for search and banner advertising. From a standing start a year ago, online has become a significant source of business, he adds.
Newman says the move into online transaction and supported marketing was prompted by a change in consumer behaviour. He predicts most insurance categories will see growth from the online sector. They need to respond to where their consumers are, and pay more attention to promoting their brands and products online.
Integrating digital marketing into the overall strategy is now one of the basics of media planning. While advertising on offline media, such as television and radio has traditionally been seen as the best way to build a public profile, Hitwise research says integrated offline and online campaigns yield the biggest gains in brand awareness and web traffic.
Comparethemarket.com marketing director Mark Vile says everyone
needs to have digital as part of their advertising strategy. “Quite often you see a disconnect between online and offline activity. What you need to do is make sure that on and offline strategies are integrated. Digital is just something that should be part of your strategy.
The aggregator started with a somewhat unusual marketing mix, with 80%-90% of its marketing and advertising budget going on digital, as it sought to carve out a place in the comparison
marketplace. Our strategy has moved towards making a brand that will build up position in the market and build position for our insurance partners, Vile says.
And that means shifting to a roughly equal split between offline and online spend over the next few years, something he thinks others in the insurance marketplace will tend towards.
Consumers online are growing in numbers. The UK has the most active online population in Europe, with the highest average number of daily visitors (21.8 million), the highest usage days per month (21 per user), and the highest average time spent per month per user (34.4 hours), according to E-Consultancy.
The amount of money being spent online has likewise rocketed. Figures from the IAB show that £30.2bn was spent by UK consumers online during 2006, with this figure predicted to rise to almost £52bn by 2011, indicating the size of the digital pie up for grabs.
Once people are spending time online, and are willing to transact online, the marketing policies of those companies wishing to attract part of that spend have to look online too.
Newman of Carloe Nash says: Financial services marketing is becoming much more sophisticated online, and where people started with banner ads which were good for brand recognition, now they are looking at search engines and affiliate sites. It has certainly changed from where it used to be. Online was an afterthought, but now it’s a starting point. In the past 10 years the level of expertise that you need to show in this area has increased exponentially.
Vile would support that. What we are seeing now is a much broader market in demographic terms of those buying insurance online, he says.
With talk of a credit crunch, and rumours of a recession on the horizon, Vile believes more people will be looking to save money by researching their insurance options online.
And where the customers go, the advertising and marketing money will surely follow.