How to create a robots.txt file
Since the mid 90s it has been agreed amongst all tech savvy internet people, that a robots.txt file on a server could specify access policies for robots. Creating a robots.txt file is very easy. In order to get started open a text editor (mac) or notepad (windows) to create a …