robots.txt does not provide any security. It merely provides hints to search engines about what they should include in their search indexes, and search engines don’t even have to listen to them. Stop putting sensitive URLs in your
Even better, keep the URLS in there but tighten the access controls on them, so nobody can actually access them. The
robots.txt file is one of the first things an attacker will examine for useful intelligence.
Here is a small sample from a cursory glance of some of the sites I frequent:
Disallow: /cron.php Disallow: /install.php Disallow: /setup.php Disallow: /update.php Disallow: /admin/ Disallow: /admin/?pass=abc123