Search engine rankings are a vital component of an Internet presence: More than three quarters of hits on Internet home pages are directed there via a search engine; and half of consumers research online prior to making purchases. But many of the world's largest businesses are failing to take advantage of this growing channel for acquiring customers.

Analyst group IDC says that £4 billion was spent online in the UK in 2004, making up 7% of all retail sales. But analysis of user behaviour suggests that many search engine users never scroll down the screen to look at lower ranking results. This leaves a narrow window of just half a dozen results. Anything outside the top 30 results is essentially invisible - though contrary to popular myth, 'eye-tracking' has found the first two rankings are both seen equally.

Worryingly, there appears to be little understanding of the importance of search engine ranking by leading businesses. Research by Screen Pages, a provider of managed ecommerce services, found that less than a third of 18 UK high street retailers' websites it surveyed had broad coverage on Google, Yahoo or MSN; only half showed evidence of a basic search engine optimisation (SEO) strategy; and less than a quarter had "high-street grade" page ranks - Google's measure of site importance. Another study, by search engine marketing company Oneupweb, found that only one in eight of the Fortune 100's websites were adequately optimised for web searches.

How is it that the world's largest companies are failing to make their content easy to find on the Internet almost a decade after most of them launched their websites?

Many companies use paid search listings to gain online visibility, buying keywords so that they are listed alongside results when shoppers use terms likely to be associated with the retailer's products. This can drive substantial traffic to a site but it is increasingly expensive. In the US, the most popular online gambling keywords cost up to $50 per user click, while UK insurance companies pay £2 to £5.

An unexpected side effect of this reliance on keyword advertising is the phenomenon known as 'click fraud': companies clicking repeatedly on their rivals' advertisements to drain the funds allocated to that campaign. By designing web pages that are sympathetic to search engines' methods of data gathering, companies improve their ranking. But as most online shoppers trust - and so click - on search results above the linked advertisements, experts suggest the best results come from using both approaches - paid-for and optimisation.

Gaining a high placing in search results can take months. It hinges on three factors: ease of indexing, relevance and link popularity (see box, 'Best practice checklist') - none of which can be perfected overnight. The value of search engine optimisation can be demonstrated by those locked out. Password-protected sites cannot be accessed by search engine 'spiders' any more than they can by unauthenticated people, meaning vast resources of valuable content are locked out of the index. The Wall Street Journal, the American financial newspaper, has been a particular victim of this, failing to appear under searches for which any subscriber would believe it to be an authority.

Yahoo's new "deep web search" promises to solve this by indexing the subscription-only content without giving away the valuable material - but only for 'big name' sites. However, protected sites will still fail to draw the credit from links. This technique is often used by search engines to rank sites, but the restricted sites are 'passively boycotted' by bloggers - a valuable source of getting that credit.

This makes alternative forms of optimisation all the more important. But a focus on improving ranking is far from a free option, warns Greg Boser of SEO consultancy Web Guerrilla : "There is no such thing as a free listing. You end up paying for it somehow." Search engines regularly change their indexing techniques and the way they rank sites to improve their results and filter out abuses. For those aggressively seeking top listings, it is a constant battle.

Innovations in the way web searches are conducted are set to make that task even harder. While today, Google's dominance makes optimising sites for its ranking algorithms an obvious approach, there are a growing number of contenders that use different techniques. Blinkx, a UK-based search engine, operates from the desktop, reading the whole content of an email or document and offering contextually relevant results, removing keywords altogether. French enterprise search specialist Exalead claims to merge Yahoo's directory approach and Google's indexing technique to provide results that are updated in real time. Yahoo Mindset, currently in testing, gives the user a horizontal slider to filter between commercial and research-based results.

A Google that knows you?

Personalisation is a key trend as information gathering enables "a Google that knows you", to use its CEO Eric Schmidt's now-infamous phrase. Google's My Search History and Yahoo's My Web are online versions of a browser's bookmarks or history, recording every site the user has visited. Combined with data from email services, these could enable search engines to use personal information and search results preferences to deliver more relevant results to each individual. "The personalisation of results over the next few years would change organic listings as we know it," explains Barry Lloyd.

But Danny Sullivan, editor of specialist news site Search Engine Watch, says that the move to more specialised searching presents opportunities as well as challenges. "You've got to learn about these verticals, you've got to make sure you're well placed in there. The nice thing is that since relatively nobody's paying attention to it, it's like rolling back to 1997 for the optimisation opportunities."

Ultimately, the aim of any search engine is to deliver the best answers to their users' questions. It is not in Google's interest any more than it is in Topshop's that the results from a search for "fashion clothing" will rank a news story about outfits being made for Japanese chickens higher than a leading high street brand. Eventually, search may evolve to meet the big brands. But in the meantime, brands need to catch up with search.

Don't be evil: The dark side of search engine optimisation

As a significant conduit for driving traffic to web sites, search engine Google is uniquely placed to influence the popularity of any given site. The company tries to take an ethical and even-handed approach to how it uses the power bestowed on it by the army of faithful users that follow its search results.

One of Google's informal corporate mottos is: "Don't be evil". It likes to remind its staff of their privileged position, but the phrase also serves as a warning for those web sites that might try to abuse Google's technology to boost their ratings. Some marketing firms' 'unethical' practices make search engine optimisation (SEO) a minefield for the uninitiated. Hidden text and links top Google's SEO black list. These are used to improve page rank by including content that visitors would not expect to see, with text coloured the same as the background to render it invisible. In June 2005, a hidden link to a personal finance site was found on pages in the Financial Times' online version, FT.com - a site Google would regard as a valuable referrer.

Similarly Wordpress, a popular personal web publishing package, was discovered to be renting out 200,000 pages of domain space with no real factual content or purpose. It appears that marketers hoped generous linkage and repetition of their keywords would lure in clicks to lucrative Google Adsense listings. The Wordpress homepage was banned from Google listings for 48 hours.

This punishment was relatively brief because if a site is too big, Google "can't keep them banned," says Danny Sullivan, editor of specialist news website Search Engine Watch. "If people search for BA they want to find British Airways. But you have to ask yourself, are you big enough?"

And it is not just hidden text that Google has to counteract. Spammers frequently use tactics such as 'cloaking', where sites show one highly optimised version of a web page to search engine crawlers and another to real people visiting it, by identifying the crawlers' IP addresses.

However, Google's position on cloaking is not always clear-cut. Some sites use it with impunity when moving to a new URL or to show crawlers firewalled content that visitors have to pay to see. "As long as the process is benevolent to the site's users, I would not call it cloaking," says Magnus Sandberg, a Google developer in Zurich.

Perhaps because of all this, Google is believed to have a "rater hub" of human editors. It certainly changes its methods frequently to ensure fresh content and prevent manipulation. The resulting fluctuations in sites' rankings are enough to drive some to the dark side of SEO.