If you ever heard of a robot.txt file then you always woundered if it really helps search engine ranking. the robot.txt file does not help search engine ranking but that doesn’t mean you don’t need it.
  A robots.txt file can also be set up to instruct specifically which robots
have permission to crawl your files. For [...]