How to avoid search engine blacklisting

By By Marcus Austin, 18 May 2006 at 16:19

There are two truisms when it comes to the Internet: if you open a connection to the Internet within a few minutes someone will try and hack into it, and if you build a website, it won't be long before someone from a SEO (search engine optimisation) company will ring you and ask you if you want to be number one on Google.

SEO companies have sprung up all over the Internet in the last three years, all claimin the same thing: to get your website to the top of the search engine listings. The problem is that many are unscrupulous about how they go about this, and search engines are beginning to wise up.

In February of 2006 BMW Germany's site was de-listed from Google's index, and for a period of 24 hours anyone searching Google for BMW.DE would not have been able to find it. BMW were let off quite lightly: normally a de-listed site can take weeks if not months to get back on the database.

Many companies don't realise quite how big an effect Google and the other search engines have on site traffic. A good natural listing (a site that appears in the main listings rather than in the paid-for listings) on the combined search engines can account for up to 80 per cent of some sites' traffic, and Google accounts for 80 per cent of all search engine traffic. Take away Google and you could, in theory, cut traffic to a site by up to 64 per cent.

De-listing can be painful

The key lesson to learn from the BMW situation is that de-listing can happen to anyone, no matter how big or small. As we've seen it can be paiful too, for your bottom line as well as your corporate image, and you should do everything possible to stop it from happening. This is, however, not as straightforward as it seems.

According to Google employee and blogger, Matt Cutts, the German BMW site had been removed for violating the guideline: "Don't deceive your users or present different content to search engines than you display to users."

Google and the other engines, however, neither reveal how their search algorithms work - as they don't want SEO companies to know how to unfairly influence the results - nor do they publish what is and isn't acceptable practice for SEO. Its outlines and those of the other search engines are hazy.

The key point is to be open in everything you do. A search engine catalogues websites with applications know as bots. These applications spider the Web and your site looking at all the information contained within it and following any links on the pages. They are "dumb" applications. They can't pull down drop-down menus, they can't read text in gif or jpg files, they can't type names into search boxes, they can cope with Flash but they'd rather not, and they're particularly frightened of getting stuck in loops. Above all they hate been cheated.

Cheap chicanery

SEO companies have traditionally used various practices that are designed to fool the dumb bot into thinking a page is more useful than it actually is. One method is called "cloaking", or presenting hundreds of keywords to the bot but hiding them from users by making them the same colour as the background or putting them at the foot of the page in very small text. Another is by producing so-called "doorway pages" that are full of keywords and phrases, but are never seen by the browser as the page automatically redirects to another page, often with completely different content on it.