README for Inveneo Crawler The Inveneo Crawler reads a set of host IPs from OpenNMS import XML files, and then visits those hosts to do some housekeeping, such as pulling the latest configuration and rebooting the host if it has been running a long time. It generates a report in a form suitable for import into a spreadsheet. Written by jwiggins@inveneo.org 2011-2012 ===== Files (top down hierarchy) ===== nightly.sh - a wrapper script for cron to handle logging/reporting crawler.sh - a wrapper script ensuring root, limiting runtime host_visitor.py - top level module; gets host list and does housekeeping host_walker.py - module that pulls together a host list from XML files h3c_control.py - class for controlling H3C switches mikrotik_control.py - class for controlling Mikrotik routers ubiquiti_control.py - class for controlling Ubiquiti radios host_control.py - base class with funtionality common to all hosts crawler_util.py - utility data and functions used by several modules crawler_conf.py - per-site configuration README - this file ===== Discussion ===== Normally you will run this from /etc/crontab using a line like this: 00 02 * * * root /opt/inveneo/sbin/nightly.sh /opt/inveneo/sbin/crawler.sh 05 00 nightly.sh - This shell script handles three things: making sure only one instance of the desired program is running, putting its output into a logfile (and rotating log files), and sending an email report of the run. crawler.sh - This shell script puts a time limit on the running of the program, killing it if it goes past its end time (which is passed in via two arguments). host_visitor.py - This Python script makes a list of hosts to visit, picking up where it left off on its last run. It tries to ping the host, and then to log in and get the uptime. After that it tries to download the config file(s) of the host. And sometimes it may then try to reboot the host. host_walker.py - This Python script parses OpenNMS provisioning XML files, considered to be the "master list" of what host nodes are out there on the network, and presents the list. h3c_control.py, mikrotik_control.py, ubiquiti_control.py - These Python scripts are subclasses of host_control.py, extending its functions for specific devices (namely, H3C switches, Mikrotik routers, and Ubiquiti radios). host_control.py - A Python base class for presenting a generic interface to a network node: you can query the uptime, version, and configuration, as well as reboot the device (but this is an abstract base class: you need to use one of the specific subclasses above to work with a given device). crawler_util.py, crawler_conf.py - These are common utilities and site-specific configuration data. README - A helpful text file, no?