polipus 0.3.5
An easy to use distributed web-crawler framework based on Redis
Gemfile:
=
instalar:
=
Runtime Dependencies (9):
aws-s3
>= 0.6.3, ~> 0.6
bson_ext
>= 1.9.2, ~> 1.9
hiredis
>= 0.4.5, ~> 0.4
http-cookie
>= 1.0.1, ~> 1.0
mongo
>= 1.9.2, ~> 1.9.0
nokogiri
>= 1.6.0, ~> 1.6
redis
>= 3.0.4, ~> 3.0
redis-bloomfilter
>= 0.0.1, ~> 0.0
redis-queue
>= 0.0.3, ~> 0.0