 |
|
By: Sumantra Roy
Some people believe that they should create different pages for different search engines, each page optimized for one keyword and for one search engine. Now, while I don't recommend that people create different pages for different search engines, if you do decide to create such pages, there is one issue that you need to be aware of.
These pages, although optimized for different search engines, often turn out to be pretty similar to each other. The search engines now have the ability to detect when a site has created such similar looking pages and are penalizing or even banning such sites. In order to prevent your site from being penalized for spamming, you need to prevent the search engine spiders from indexing pages which are not meant for it, i.e. you need to prevent AltaVista from indexing pages meant for Google and vice-versa. The best way to do that is to use a robots.txt file.
You should create a robots.txt file using a text editor like Windows Notepad. Don't use your word processor to create such a file.
Here is the basic syntax of the robots.txt file:
User-Agent: [Spider Name]
Disallow: [File Name]
For instance, to tell AltaVista's spider, Scooter, not to spider the file named myfile1.html residing in the root directory of the server, you would write
User-Agent: Scooter
Disallow: /myfile1.html
To tell Google's spider, called Googlebot, not to spider the files myfile2.html and myfile3.html, you would write
User-Agent: Googlebot
Disallow: /myfile2.html
Disallow: /myfile3.html
You can, of course, put multiple User-Agent statements in the same robots.txt file. Hence, to tell AltaVista not to spider the file named myfile1.html, and to tell Google not to spider the files myfile2.html and myfile3.html, you would write
User-Agent: Scooter
Disallow: /myfile1.html
User-Agent: Googlebot
Disallow: /myfile2.html
Disallow: /myfile3.html
If you want to prevent all robots from spidering the file named myfile4.html, you can use the * wildcard character in the User-Agent line, i.e. you would write
User-Agent: *
Disallow: /myfile4.html
However, you cannot use the wildcard character in the Disallow line.
Once you have created the robots.txt file, you should upload it to the root directory of your domain. Uploading it to any sub-directory won't work - the robots.txt file needs to be in the root directory.
I won't discuss the syntax and structure of the robots.txt file any further - you can get the complete specifications from here.
Now we come to how the robots.txt file can be used to prevent your site from being penalized for spamming in case you are creating different pages for different search engines. What you need to do is to prevent each search engine from spidering pages which are not meant for it.
For simplicity, let's assume that you are targeting only two keywords: "tourism in Australia" and "travel to Australia". Also, let's assume that you are targeting only three of the major search engines: AltaVista, HotBot and Google.
Now, suppose you have followed the following convention for naming the files: Each page is named by separating the individual words of the keyword for which the page is being optimized by hyphens. To this is added the first two letters of the name of the search engine for which the page is being optimized.
Hence, the files for AltaVista are
tourism-in-australia-al.html
travel-to-australia-al.html
The files for HotBot are
tourism-in-australia-ho.html
travel-to-australia-ho.html
The files for Google are
tourism-in-australia-go.html
travel-to-australia-go.html
As I noted earlier, AltaVista's spider is called Scooter and Google's spider is called Googlebot.
A list of spiders for the major search engines can be found here.
Now, we know that HotBot uses Inktomi and from this list, we find that Inktomi's spider is called Slurp.
Using this knowledge, here's what the robots.txt file should contain:
User-Agent: Scooter
Disallow: /tourism-in-australia-ho.html
Disallow: /travel-to-australia-ho.html
Disallow: /tourism-in-australia-go.html
Disallow: /travel-to-australia-go.html
User-Agent: Slurp
Disallow: /tourism-in-australia-al.html
Disallow: /travel-to-australia-al.html
Disallow: /tourism-in-australia-go.html
Disallow: /travel-to-australia-go.html
User-Agent: Googlebot
Disallow: /tourism-in-australia-al.html
Disallow: /travel-to-australia-al.html
Disallow: /tourism-in-australia-ho.html
Disallow: /travel-to-australia-ho.html
When you put the above lines in the robots.txt file, you instruct each search engine not to spider the files meant for the other search engines.
When you have finished creating the robots.txt file, double-check to ensure that you have not made any errors anywhere in it. A small error can have disastrous consequences - a search engine may spider files which are not meant for it, in which case it can penalize your site for spamming, or, it may not spider any files at all, in which case you won't get top rankings in that search engine.
An useful tool to check the syntax of your robots.txt file can be found here. While it will help you correct syntactical errors in the robots.txt file, it won't help you correct any logical errors, for which you will still need to go through the robots.txt thoroughly, as mentioned above.
About the Author
Article by Sumantra Roy. Sumantra is one of the most respected and recognized search engine positioning specialists on the Internet. For more articles on search engine placement, subscribe to his 1st Search Ranking Newsletter by going to: http://the-easy-way.com/newsletter.html
|
 |
|
Australia Honeymoon Vacationstreasure Down Under
Australia honeymoon vacations are more than just being the pleasure of being down under. Australia is virtually a paradise on earth. Many of the people hail the continent as the capital for romance. Be they exotic beaches, fascinating wild life, rich culture, you will find all of them in Australia.
Where Can I Go On A Kitesurfing Vacation
If you've been kitesurfing for a while, you might find yourself getting tired of kitesurfing in the same place over and over. After all, different locations will have different conditions - which could mean a nice challenge, or at least a break from the ordinary. Therefore, if you've been kitesurfin
How Safe Do You Feel Going To Hospital To Be Cured
There has been a lot of concern in Australia over the performance of an overseas trained Doctor that had been made Director of Surgery in a Regional City.Currently there is a Commission of Inquiry into the allegations that have been made against this Doctor and in the interim report that has been re
Australia : A Continent In One Country
If you haven’t been to Australia or are planning to visit it a second time, then you must have all the information about the places to visit so that you can fully experience the wonderful land that is Australia.Australia is the sixth-largest country in the world and is divided into 6 states and sev
The 9 Ways Of Emigrating To Australia
Ever wondered how to go about emigrating to Australia and go about getting an Australian visa? Wondering what the various options were and how to go about the different schemes without too much red tape or expense? There are lots of ways to emigrate to australia from the UK. This article spells it o
Do My Ugg Australia Boots Put Me Ahead Of Fashion Or Behind
Ugg Australia boots are not just for winter anymore. These boots are showing up everywhere. Women are wearing these big boots with short skirts. Children and grown-ups alike are wearing their brightly colored boots regardless of the season. Celebrities are wearing these boots during filming breaks.
Computer Ewaste Recycling Now Fueled By Lack Of Australian Legislation
Motivated by the lack of Australian government legislation and limited options available concerning the disposal of computer e-waste, IT Recyclers has initiated a much needed computer recycling service aimed primarily at Sydney business who wish to play an essential part in reducing computer landfil
Creating A Robotstxt File
Some people believe that they should create different pages for different search engines, each page optimized for one keyword and for one search engine. Now, while I don't recommend that people create different pages for different search engines, if you do decide to create such pages, there is one i
Australian Wine Is More Than Yellow Tail
The [yellow tail] range of wines have taken the world by storm. And so they should. They are excellent Australian wines which are consistently good. They have clearly won the battle for everyday wines at their particular price range. But they are a made from classical French grape varieties, Chardon
Mandurah Western Australia | A Holiday Destination For All
Mandurah is located approximately 72kms south of Perth - Western Australia. Mandurah is a water wonderland visited by day trippers and holiday makers alike all year round. The towns population is approximately 50,000 and continuing to grow. In fact Mandurah is the fastest growing regional town in We
Albany Western Australia A Holiday Destination For All
Albany Western Australia is a city of approximately 30,000 people in the south-west corner of Western Australia, approximately 409km from Perth . The city is nestled between three large hills, Mount Clarence, Mount Melville and Mount Adelaide, facing the beautiful King George III Sound.The main indu
Mandurah Western Australia A Holiday Destination For All
Mandurah is located approximately 72kms south of Perth - Western Australia. Mandurah is a water wonderland visited by day trippers and holiday makers alike all year round. The towns population is approximately 50,000 and continuing to grow. In fact Mandurah is the fastest growing regional town in We
Big Aussie Wines
Australia’s wine industry has boomed in the past ten years. Employers have had to triple their staff numbers to cope with the demand for Aussie wine. Considering the lower average national population of Australia, compared to say the United States or even South Africa, 30,000 (2001) workers is prett
Ugly Man From Australia Shows You How To Increase Yours Sales 1200
*Do you have an incredible product but no one buys?*Do you know other people that are selling the same products earning a lot more then you?I can GUARANTEE if you would only take a few moments to listen to what I say, you can turn your dismal sales around and become a SUCCESS!How can I say this? Be
NEW ZEALANDAUSTRALIA TRAVEL SPECIALIST SPEAKS
NEW ZEALAND-AUSTRALIA TRAVEL SPECIALIST SPEAKSBY ALECE SCHREIBERHave you ever approached your local travel agent with a request for information on an out of the ordinary destination? Have you seen his or her eyes glaze over? You may be in need of a niche travel specialist, one who puts all of her en
|
|
|
