Ebot crawler
Ebot
http://www.redaelli.org/matteo-blog/projects/ebot/
Erlang Bot ( Ebot ) is an opensource web crawler written on top of Erlang , a NOSQL database ( Apache CouchDB or Riak ), RabbitMQ , Webmachine ( Mochiweb ), RRDTOOL , .. Using a NOSQL instead of a Relational Database, Ebot can grow easily and cheaply… Ebot is a solid and highly scalable, distribuited and customizable web crawler .
The Ebot crawler project is hosted at http://github.com/matteoredaelli/ebot
ebot web crawler
Thanks to Ebot crawler I’ve been improving my knowledge about Erlang , the AMQP protocol (RabbitMQ) and NOSQL databases ( Apache CouchDB and Riak ) with the distribuited map/reduce queries
Below there is an example of a url document generated by the ebot crawler (with apache couchdb backend)
Below you find a sample image of Statistics generated by ebot web crawler using RRDTOOL