User-agent: Googlebot Disallow: User-agent: MSNBot Disallow: User-agent: Slurp Disallow: User-agent: Teoma Disallow: User-agent: Twiceler Disallow: User-agent: Gigabot Disallow: User-agent: Scrubby Disallow: User-agent: Robozilla Disallow: User-agent: Yahoo-slurp Disallow: User-agent: 80legs Disallow:/public_html/ Disallow:/*.html$ User-agent: * Disallow: /cgi-bin/ Disallow: /JoomlaTest/ Disallow: /images/ Disallow: /cgi-bin/*.pl$ Disallow: /*.PDF$ Disallow: /*.jpeg$ Disallow: /*.exe$ Disallow: /RBScontact.html Crawl-delay: 10 # Custom robot.txt, allows major search engines, trailing / will disallow obiedient engines. This is set to dissallow certain directories and specific file types. Do not use * in disallow line by itself, use trailing / to stop or dissallow. Mainly you can only tell them NO, not to allow.