Dynamic robots.txt implementation with PHP

09 April 2013

The problem

Recently I discovered what on some multidomain sites I made, I had the same robots.txt file for all the sites, which has multiple domains. That created some confusions for search engines, even for external one, based on mnoGoSearch technology.


So, dynamic robots.txt should be done, which changes robots.txt content conditionally and dynamically.


First of all You should make a substituition of url /robots.txt with dynamic script, which will generate the output, for example : robots.php

With Apache You could make simple .htaccess rule to acheive that.

 RewriteRule  ^robots\.txt$ /robots.php [NC,L]

And after we could make whatever we want, to generate robots.txt of our dream in the best way with the help of PHP.

 // read static robots.txt template
 $robotsTxt = @file_get_contents('robots.txt');
 # make some operations with default template
 echo $robotsTxt;


You could make conditions based on user-agents to avoid the indexation by spiders You don’t need.


  • http://forums.oscommerce.com/topic/380271-dynamic-robotstxt-to-stop-worthless-traffic/
  • http://blog.bannasties.com/2012/07/how-to-implement-your-dynamic-robots-txt/

© 2018 Fedir RYKHTIK