How to create a robots.txt file

Since the mid 90s it has been agreed amongst all tech savvy internet people, that a robots.txt file on a server could specify access policies for robots. Creating a robots.txt file is very easy. In order to get started open… Read on