/stilio

🦎 A DHT crawler with a minimalistic frontend, written in Python

Primary LanguagePythonApache License 2.0Apache-2.0

stilio

https://travis-ci.org/seik/stilio.svg?branch=master https://readthedocs.org/projects/stilio/badge/?version=latest https://img.shields.io/github/license/seik/stilio

Goals

  • Completely written in python.
  • Fast. But never trade code readability for speed.
  • Easy set up.

Quickstart

A docker-compose file is included to allow a fast and easy set up.

docker-compose up

Note

For a production enviroment it's recommended to run the frontend and database separated from the crawler since the last one generates a lot of traffic and can make a poor experience for the users. You can read a guide about how to do this here.

FAQ

The crawler speed is not so good.

Try to run this commands if you are on linux:

iptables -I OUTPUT -t raw -p udp --sport PORT_NUMBER -j NOTRACK
iptables -I PREROUTING -t raw -p udp --dport PORT_NUMBER -j NOTRACK