[ad_1]
Your website should be working for you, working to help itself climb the web rankings and helping drive traffic to your business. While the Top Position site offers a wealth of information on the various ways to make your site work for you there is another important role your site should fulfill. In addition to clever self-promotion and various optimization functions your site can help the search engines navigate your site and understand what it is you want to show the world. This point revolves around one of the most underused aspects of search engine optimization work, the Robots.TXT file.
Web site owners use the Robots.TXT file to give instructions about their site to web robots with a view to presenting the most suitable information to encourage the robots to boost the site up the rankings. It might be an idea for us to first explore what a web robot is. Web robots, or spiders as you may hear them referred to as, are software programs used to index websites. Search engines use these robots to assess your website by navigating around it, following links and looking for new pages. Basically this process is connected with the ultimate goal of making a judgment on where to rank your site. This is as much as you need to know to appreciate the importance of these robots for your SEO, and the importance of presenting them with the best possible view of your site.
Now, as we have discussed a Robots.TXT file is used to give the robots instructions about your site, but what sorts of things should this include and why do we need to action it? Well, this file is placed on your server to tell the various search engine robots not to index certain sections or pages of your site, and surprisingly such exclusion is beneficial to your SEO, and your site as a whole. If for example your site includes product pages for a range of shoes, it is likely that some of the product pages will be the same, ie if one particular make of shoe comes in a variety of colors. As the product pages for each different color of this shoe will be the same your site is in danger of being penalized, in terms of SEO and web rankings, for including duplicate pages. The search engines disapprove of duplicate pages so your robots.TXT file can instruct the robots to disregard these pages. This may also come in handy when your site is undergoing construction as you can instruct the robots not to index pages that are half finished, or in the middle of a rework, which could also hinder your site's progress.
All search engines, or at least all the important ones, now look for robots.TXT file as soon their robots arrive on your site, and so the very fact that search engines are looking for them is reason enough to put one on your site . If you look at your site statistics you are sure to see many entries in the 'files not found' section where search engine robots looked for, and failed to find, a Robots.TXT file on your site.
Creating a basic Robots.TXT file is not difficult and can be done using notepad. It needs to be saved to the root directory of your site, that is the directory where your home page or index page is. Each entry has just two lines and they should run that:
User-Agent: [Spider or Robot Name]
Disallow: [Directory or File Name]
This line can be repeated for each directory or file you want to exclude, or for each robot you want to exclude.
There are many other reasons you may want to instruct certain robots to ignore certain parts of your site and these relate to the varying nature of search engines and their views on search engine optimization as an activity. The different search engines work in different ways, perform different functions, and offer differenting values to different SEO activities. You may for example want to exclude some robots from indexing your site altogether, for example from those search engines you do not want to appear in, or those what main aim is collecting email addresses.
The reason you are exploring different methods of SEO is because you want your site to be visible on the search engines, but within this you have to accept that the engines are going to want reasons as to why they should give you higher rankings. The Robots.TXT file system allows sites to present what they want to the search engine assessment software and so it is of the utmost importance that you consider fully the implications of allowing robots to index each page of your site.
[ad_2]