Since the mid 90s it has been agreed amongst all tech savvy internet people, that a robots.txt file on a server could specify access policies for robots. Creating a robots.txt file is very easy.
In order to get started open a text editor (mac) or notepad (windows) to create a unformatted text file with the file extension .txt.
What should be included in a robots.txt file?
The file should provide answers to two questions robots want answers for:
- Do these rules apply to me?
- What on this server am I not allowed to access?
Keep in mind, that the robots.txt file has to be placed in the root of your website (which is the same location where you put your index.html or index.php file)
In your text editor/notepad, the first thing you’re going to write down, is to whom your policy applies. You’ll be using the User-agent command for this.
The value you’ll enter here is the name of the robot. If you use a * your policy will apply to all robots.
Example allowing all:
Example allowing one robot, in this example Google:
Using the disallow command you specify which part of your server or website a robot is not allowed to access. This is a great way to shield off any non-public part of your website or server. For instance, your webmail, CMS location or any pages that require user logins.
So again, keep in mind the location of this file (in the root of your web directory). Suppose your CMS is located on www.yourdomainname.com/admin , then you probably want your Disallow-line to look like this:
If you want to disallow access to everything, this is the way to do it:
You can add as many disallow lines as you want.
If, for whatever reason, you’d like to include comments, you can do so by starting your line with #
# this is a comment for me and not read by robots
Putting the robots.txt together
Your file should be looking something like this:
Sample disallowing everyone and everything:
Sample disallowing everyone, but Google
#disallow all, but Google. Google may access everything
#Google will now ignore your first command and use the policy you’ve set for them
Sample disallowing certain directories and files for all
#allowing all access, but blocking a few directories and files
Sample disallowing a particular directory, but allowing one file in that directory