User-Agent: MJ12bot Crawl-Delay: 10 Disallow: /bat/login Disallow: /bat/guest Disallow: /bat/userprefs Disallow: /bat/member Disallow: /bat/poard/post_answer Disallow: /bat/content/search User-Agent: ICCrawler Disallow: / # fail # fix your crawler, it's crawling an awful lot of non existant links User-Agent: Pixray-Seeker Disallow: / User-Agent: * Disallow: /bat/login Disallow: /bat/guest Disallow: /bat/userprefs Disallow: /bat/member Disallow: /bat/poard/post_answer Disallow: /bat/content/search Crawl-Delay: 10