database of robots.txt-derived permissions

Edit Package perl-WWW-RobotRules
http://search.cpan.org/dist/WWW-RobotRules/

This module parses _/robots.txt_ files as specified in "A Standard for
Robot Exclusion", at Webmasters
can use the _/robots.txt_ file to forbid conforming robots from accessing
parts of their web site.

Refresh
Refresh
Source Files (show merged sources derived from linked package)
Filename Size Changed
Latest Revision
Yuchen Lin's avatar Yuchen Lin (maxlin_factory) committed (revision 3)
Comments 0
openSUSE Build Service is sponsored by