# /robots.txt file for gsa-crawler (This is a comment line. )
User-agent: mmu-gsa-crawler
# Disallow: /*.cgi (The gsa-crawler will not be allowed to crawl any CGI files.)
# Disallow: /*.pl (The gsa-crawler will not be allowed to crawl any Perl scripts.)
# Allow: /$ (The gsa-crawler is allowed to crawl everything else.)
# Disallow: /private_stuff/ (blocks access to the directory and sub directories of /private_stuff and their contents)
Disallow: / #(This prevents the gsa-crawler from crawling anything on the site.)