# http://www.robotstxt.org/wc/exclusion.html#robotstxt User-agent: * Disallow: /cgi-bin/ # http://validator.w3.org/docs/checklink.html#bot User-Agent: W3C-checklink Disallow: