robot_rules 0.9.3
A tool to determine if the robots.txt would prevent a given user agent from making a request to a given URI.
Gemfile:
=
installieren:
=
Runtime Abhängigkeiten (1):
addressable
>= 2.1.2
Besitzer:
Autoren:
-
James Edward Gray II, Jeremy Friesen, Edgar Gonzalez
SHA 256-Prüfsumme:
=