Painless screen scraping for network engineers - Control your network with Python
Essentially, this is a simple wrapper around pexpect. Standard usage:
import time
from clijockey.traits import TraitTable, CUnicode, CUnicodeRegex
from clijockey.traits import CMacAddressCisco, CIPv4AddressStr
from clijockey.util import RotatingTOMLLog
from clijockey.util import Account
from clijockey.lib import CLIMachine
# TextFSM template...
TEMPLATE = """Value INTF (\S+)\nValue IPADDR (\S+)\nValue STATUS (up|down|administratively down)\nValue PROTO (up|down)\n\nStart\n ^${INTF}\s+${IPADDR}\s+\w+\s+\w+\s+${STATUS}\s+${PROTO} -> Record"""
# Interfaces() is used to map TextFSM template values to the TOML Log
class Interfaces(TraitTable):
intf = CUnicode() # See the traitlets documentation for CUnicode usage
addr = CUnicode()
status = CUnicode()
# Usage of CUnicodeRegex illustrated below... it errors if no match
proto = CUnicodeRegex(r'down|up')
_map = ('intf', 'addr', 'status', 'proto') # TextFSM field order
# Create a log named rviews_intfs, which automatically rotates at midnight
# category must be unique per logging file
log = RotatingTOMLLog('rviews_intfs', category='route-views')
## Define a tuple of username and password pairs here...
## The first is an expected failure to illustrate how it works
accts = (Account('itsa-mistake', ''), Account('rviews', 'secret2'),)
## You can optionally disable auto-enable mode if you like...
conn = CLIMachine('route-views.routeviews.org', accts,
auto_priv_mode=False, log_screen=True, debug=False, command_timeout=5)
conn.execute('term len 0', wait=0.5) # Wait 0.5 seconds after the cmd
conn.execute('show version', regex='test>') # regex is another prompt string
conn.execute('show users', timeout=60) # 'show users' outputs slowly...
## Get the result of the 'show users' command...
user_output = conn.response
## Automatically parse with TextFSM and log to a TOML log
for ii in range(0, 5):
intf_list = conn.execute('show ip int brief', template=TEMPLATE)
for vals in intf_list:
## NOTE: info must be a dictionary to be parsed by .write_table_list()
info = dict(Interfaces(vals)) # <---- This is the info table
## Write a timestamped list of tables named 'rview_intf' to the log
log.write_table_list(table='rview_intf', info=info, timestamp=True)
time.sleep(1)
conn.logout()Install with pip (-U to auto-upgrade to the latest version)
pip install -U clijockey
Short answer:
Because libraries like this should "just work" regardless of what you're screen scraping.
Longer answer:
I have been writing network screen scraping scripts for fun and profit over the last two decades; in the process, I have accumulated some opinions about how things should be done.
As of this writing, there are several similar Python command / response libraries... some even have a battery of vendor-specific plugins. The obvious question is why I think another library is required. Am I merely guilty of the not invented here syndrome?
I hope not.
1. The popular Python libraries with vendor-specific CLI drivers are pointlessly finicky and sometimes don't even work for all permutations from that vendor. All credit to the tireless souls who write and maintain them, but I'm tired of hacking around quirks in libraries; I just want to get things done.
2. Many of the existing libraries drive SSH sessions slowly because they use paramiko (pure-python SSH)
3. Unit tests should stand alone without needing a real network to test them on. This isn't easy when it comes to testing screen scraping, but Samuel Abel's exscript tests are a good example of one way you can do this well. I leveraged his ideas in clijockey
- Maximum flexiblity from a single CLI driver... no vendor-specific plugins.
- Get the most common authentication prompt sequences right
- Try a list of credentials until one works.
- Don't assume the credentials always grant enable privs mode
- Speed
- Optional parsing with TextFSM (gtextfsm to be exact)
- Verbose error messages and debugs.
- Support both telnet and ssh
- Per-session TOML logging (partially implemented)
- Python3 support (not implemented yet)
clijockey only supports *nix (OpenSSH is required); no Windows support.
Right now, I recommend Python 2.x; Python3 support is forthcoming, but a lower priority
I am extremely grateful to my employer (Samsung Data Services) for allowing me to develop parts of this at work.