# File created/last updated 20090323T18:45+10.5Z (Karl) # Apparently, Google, MSN and Yahoo! support regex. # Most of the others seem to support the regex too (seems that from observation) # Not convinced Yahoo is actually following the rules properly, but its much # less spammy then before this went in. # As for MSN... forget it. # Delay between crawls in seconds for MSN only User-Agent: msnbot Crawl-delay: 3600 # All robots User-Agent: * # Do not perform these actions: Disallow: /*?action=login Disallow: /*?action=edit Disallow: /*?action=editform Disallow: /*?action=print