Stop putting sensitive URLs in your robots.txt file!

Webmasters, the robots.txt does not provide any security. It merely provides hints to search engines about what they should include in their search indexes, and search engines don’t even have to listen to them. Stop putting sensitive URLs in your robots.txt file.

Even better, keep the URLS in there but tighten the access controls on them, so nobody can actually access them. The robots.txt file is one of the first things an attacker will examine for useful intelligence.

Here is a small sample from a cursory glance of some of the sites I frequent:

Disallow: /cron.php
Disallow: /install.php
Disallow: /setup.php
Disallow: /update.php
Disallow: /admin/
Disallow: /admin/?pass=abc123