database of robots.txt-derived permissions
http://search.cpan.org/dist/WWW-RobotRules/This module parses _/robots.txt_ files as specified in "A Standard for
Robot Exclusion", at Webmasters
can use the _/robots.txt_ file to forbid conforming robots from accessing
parts of their web site.
- Sources inherited from project openSUSE:Leap:15.1:Rings:1-MinimalX
- Links to openSUSE:Leap:15.1 / perl-WWW-RobotRules
- Download package
Refresh
Refresh
Source Files (show merged sources derived from linked package)
Filename | Size | Changed |
---|---|---|
_link | 0000000109 109 Bytes | over 2 years |
Comments for perl-WWW-RobotRules 0