database of robots.txt-derived permissions

http://search.cpan.org/dist/WWW-RobotRules/

This module parses _/robots.txt_ files as specified in "A Standard for
Robot Exclusion", at Webmasters
can use the _/robots.txt_ file to forbid conforming robots from accessing
parts of their web site.

Refresh
Refresh
Source Files (show merged sources derived from linked package)
Filename Size Changed Actions
Comments for perl-WWW-RobotRules 0