database of robots.txt-derived permissions

This module parses _/robots.txt_ files as specified in "A Standard for
Robot Exclusion", at Webmasters
can use the _/robots.txt_ file to forbid conforming robots from accessing
parts of their web site.

Source Files (show merged sources derived from linked package)
Filename Size Changed
Comments for perl-WWW-RobotRules 0
openSUSE Build Service is sponsored by