# Robots.txt file from this website # All robots will spider the domain User-agent: * Disallow: / User-agent: Googlebot Disallow: User-agent: Baiduspider Disallow: User-agent: MSNBot Disallow: User-agent: Slurp Disallow: