# For more on robots.txt and its purose, see # http://www.searchengineworld.com/robots/robots_tutorial.htm # # 2009-01-31: Getting insane amounts of web crawler traffic, don't want to pay # for that, so disallow more stuff User-agent: * Disallow: /webstat/ Disallow: /awstats/ Disallow: /vic/ Disallow: /pub/restricted/ Disallow: /gallery/main.php Disallow: /gallery/*