User-agent: gigabot Disallow: / # no exclusion at all User-agent: * Crawl-delay: 5 Disallow: User-agent: Mediapartners-Google Disallow: