/acrawler-prometheus

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

acrawler-prometheus

The handler working with aCrawler and Prometheus.

Export statistics for:

  • Concurrent requests
  • Task (Requests, Items) counts
  • Queue status

Installation

$ pip install acrawler_prometheus

Usage

Add Handler:

class MyCrawler(Crawler):
    middleware_config = {
        "acrawler_prometheus.PromExporter": 100,
    }

    config = {
        "PROMETHEUS_INTERVAL": 5
    }

Avaliable Config:

PROMETHEUS_ADDR = "localhost"
PROMETHEUS_PORT = 8000
PROMETHEUS_INTERVAL = 1  # exporting interval, in second