The Meraki Settings Archive & Differ project extracts Meraki device settings from the Meraki Dashboard API, stores as JSON files in a local git repo and creates web pages showing differences betweeen scans in colorized form.
The business driver for developing this was for Cisco customers that may have sophisticated IT Service Management requirements and also use cloud-managed Meraki equipment. The solution archives the settings into a local git repo allowing for archive and difference analysis. This solution is the foundation for possible follow-on functionality, such as settings scans for compliance. As the settings are stored in feature-specific JSON files, it is easy to write JSON Path Queries to check for compliance to your corporate standards or any other compliance framework (eg. HIPAA, PCI/DSS, Basel II/III, FIPS, etc). Note, the code in the current project does not provide those rules or functionality. It is expected that Cisco Customer Experience (CX) will be releasing a service that performs the compliance specific function.
-
Technology stack: Python, REST APIs, Meraki Dashboard API, HTML/CSS
Figure 1 - Meraki Settings Archive & Differ architecture/framework -
Status: v1 (Released to Automation Exchange 07/01/2021)
-
Screenshots:
Figure 2 - Main Meraki Scan Summary web page
Figure 3 - Individual Scan Results web report
Figure 4 - Feature/Setting-specific Diff report
Meraki equipment is cloud-managed and access via the Meraki web portal. It doesn't have a traditional configuration file in the sense of IOS-XE, IOS-XR, NX-OS, etc. Some IT environments that embrace ITSM, ITIL and TOGAF methodologies may require archive of device settings for disaster recovery, difference analysis and compliance scanning. This project queries the latest Meraki OpenAPI specification and works from a CSV of known API end-points, extracting all readable (GET) end-points and storing them as JSON file objects in a local git repository. The organization, network and device hierarchies are obtained and stored in a file directory structure per organization identifier (orgid). Scan dates, times and git commit hashs are collected and reported in a web report for reference.
Difference scans can be initiated against latest and previous scan (git HEAD and HEAD~1) or any combination of scanned instances. Difference reports are logged to a summary web page for easy reference with clickable links to the specific difference scan results. Individual different reports show all feature category changes, eg. orgAdministrators, and can be individually clicked for review, showing differences in a colorized side-by-side report.
Two options are provided: self-directed installation with requirements OR Docker. Pick whichever suites your preference, security requirements and deployment capabilities. When running as a self-directed installation with requirements, it is suggested to run this project in its own Linux Virtual Machine. Development was done on CentOS Stream 8.3, but releases at or above the 7 train or other Linux variants should be fine. Additionally, Apache and Python environments should be installed - CentOS may have provided one in your installation. Our guidance is to keep the system-supplied version and install the latest Python and use virtual environments (venv) to maintain separation of environments from the core operating system. Our development was done on Python v3.9.1, but anything above v3.7 should be sufficient.
Suggested docs for Centos 8.3:
- https://linuxconfig.org/installing-apache-on-linux-redhat-8
- https://linuxconfig.org/how-to-install-mod-ssl-on-redhat-8
[root@my_vm ~]# dnf install httpd
[root@my_vm ~]# systemctl enable httpd
[root@my_vm ~]# systemctl start httpd
[root@my_vm ~]# firewall-cmd --zone=public --permanent --add-service=http
[root@my_vm ~]# firewall-cmd --reload
Pay attention to the web publishing directory, typically /var/www/html. This directory should be specified in the project-system environment settings file, GMSIGconfig.py with the web_publishing_dir key. For modularity with possible new DevNet projects the value defaults to:
web_publishing_dir = '/var/www/html/DevNetDashboards/MerakiGit'
Suggested docs for Centos 8.3:
[me@my_vm ~]$ mkdir Python
[me@my_vm ~]$ cd Python
[me@my_vm Python]$ python3.9 -m venv MerakiGit
[me@my_vm Python]$ cd MerakiGit
[me@my_vm MerakiGit]$ source bin/activate
Unzip the repo files into this MerakiGit folder or use git to clone the repo locally
(MerakiGit) [me@my_vm MerakiGit]$ git clone git://github.com/CiscoDevNet/[REPONAME].git
Install package dependencies with pip using supplied requirements.txt
(MerakiGit) [me@my_vm MerakiGit]$ pip install -r ./requirements.txt
Ensure you have docker and docker-compose installed in your environment. In the project's docker directory is the docker-compose.yaml file which defines the environment's service requirements. Essentially, Apache and Python images are created. Remember your first time running 'docker-compose up' will require some time to download images from standard repositories. For the docker image we will mount a msaad-vol directory which maintains the dynamic content for the Apache we server and the git repo. Ensure you monitor the host system's file storage. Development was done with about 20G disk space and easily managed a Meraki environment of over 3,000 networks and devices. Make sure the GMSIGconfig.py file has the web_publishing_dir variable set to '/apache-data/DevNetDashboards/MerakiGit' as the default is '/var/www/html/DevNetDashboards/MerakiGit', which is suited for local installs. Follow the remaining instructions for setting Meraki Dashboard API key and setting up cron job inside the Python container. Access the Python container from the host server running docker, with 'docker exec -it msaad_python_1 /bin/sh'
A Meraki Dashboard API key must be enabled and supplied for the Python scripts to access and extract the Meraki settings. If you don't have an API key, generate one by following Meraki's documentation at: https://documentation.meraki.com/General_Administration/Other_Topics/Cisco_Meraki_Dashboard_API
The API key should be stored in the executing user's environment profile for optimal security. Assuming a Bash shell, perform this by:
[me@my_vm MerakiGit]$ cd ~
[me@my_vm ~]$ echo 'export MERAKI_DASHBOARD_API_KEY=' > .bashrc
Restart your shell and observe the correct settings:
[me@my_vm MerakiGit]$ echo $MERAKI_DASHBOARD_API_KEY]
There are two main Python scripts for the project, GetMerakiSettingsIntoGit.py and CreateMerakiGitDiffWebreport.py The GetMerakiSettingsIntoGit.py extracts the Meraki settings and stores them in the local git repository [the repo is created, if needed]. The CreateMerakiGitDiffWebreport.py file is run afterwards, as desired, to generate the difference report web pages. Both scripts are executed at the command line.
(MerakiGit) [me@my_vm MerakiGit]$ python GetMerakiSettingsIntoGit.py
usage: GetMerakiSettingsIntoGit.py [-h] {listorgs,estimatescan,getsettings} ...
List available Meraki OrgIds or Execute data collection of settings into git repo.
positional arguments:
{listorgs,estimatescan,getsettings}
Help for subcommand
listorgs Show Meraki Organizations currently available to the API_KEY - helpful in determining which orgs to collect into git repo
estimatescan Estimate length of time to scan Meraki settings for Organization Id identified
getsettings Get Meraki settings for Organization Ids (orgids) identified - archive into git repo
optional arguments:
-h, --help show this help message and exit
(MerakiGit) [me@my_vm MerakiGit]$ python GetMerakiSettingsIntoGit.py listorgs
This API_KEY has access to the following Meraki Orgs:
OrgId: 123456 - Org name
OrgId: 234567 - Another org name
(MerakiGit) [me@my_vm MerakiGit]$ python GetMerakiSettingsIntoGit.py getsettings 123456
Repo description: Meraki settings git repo for *OrgName* with orgid 123456
Repo active branch is master
...(script output of scan)
(MerakiGit) [me@my_vm MerakiGit]$ python CreateMerakiGitDiffWebreport.py
usage: CreateMerakiGitDiffWebreport.py [-h] {listorgs,listcommits,getdiff} ...
Create Meraki git diff web reports or lists available commits based on OrgId.
positional arguments:
{listorgs,listcommits,getdiff}
Help for subcommand
listorgs Show Meraki Organizations currently in git repo - helpful in determining which orgs and commits to diff
listcommits Show commit entries - helpful in determining which commits to diff
getdiff Scan for diffs and report against Meraki Organization Id (orgid) archived in git repo
optional arguments:
-h, --help show this help message and exit
(MerakiGit) [me@my_vm MerakiGit]$ python CreateMerakiGitDiffWebreport.py listorgs
This API_KEY has access to the following Meraki Orgs:
OrgId: 123456 - Org name
OrgId: 234567 - Another org name
(MerakiGit) [me@my_vm MerakiGit]$ python CreateMerakiGitDiffWebreport.py listcommits
usage: CreateMerakiGitDiffWebreport.py listcommits [-h] orgid
CreateMerakiGitDiffWebreport.py listcommits: error: the following arguments are required: orgid
(MerakiGit) [me@my_vm MerakiGit]$ python CreateMerakiGitDiffWebreport.py listcommits 123456
Git commits for Meraki org id 123456 - Org Name are:
[-- Commit Hash -----------------------] 'Branch commit message...'
cbd9b2ddb6520d6cdae4a7f0f7c3ecdea9cc0841 'Commit from Meraki scan finished on Friday, May 28, 2021 at 13:44:22 '
c2e5d99d65340302124ab1e08ab769d5b5d0becb 'Commit from Meraki scan finished on Thursday, May 20, 2021 at 13:41:36 '
ce913a84f0f33721372dc852722028880360be8d 'Commit from Meraki scan finished on Sunday, May 16, 2021 at 17:11:38 '
4741ae5a0471dac914c72ab13f6789ed9d2be72b 'Commit from scan finished at 2021-05-13--15-00'
fca57f6e7c42b81ce7824e40aa47264aad996da8 'Commit from scan finished at 2021-05-10--09-02'
a6a10ce0386af12bddf776c30323ce55323844c7 'Commit from scan finished at 2021-05-09--21-15'
0030f3de5c306d2c133c1f21d4f26f1751c84202 'Commit from scan finished at 2021-05-09--21-07'
b6d472f32e82ce6008c4979a2bcdfa433270afd5 Initial commit
(MerakiGit) [me@my_vm MerakiGit]$ python CreateMerakiGitDiffWebreport.py getdiff
usage: CreateMerakiGitDiffWebreport.py getdiff [-h] orgid [FirstCommit] [SecondCommit]
CreateMerakiGitDiffWebreport.py getdiff: error: the following arguments are required: orgid
(MerakiGit) [me@my_vm MerakiGit]$ python CreateMerakiGitDiffWebreport.py getdiff 123456 HEAD HEAD~2
Starting at: Friday, May 14, 2021 at 13:54:38
M devices/Q2AA-AAAA-AAAA - MX250/device_ApplianceDhcpSubnets.json
M networks/L_123456789012345678 - Name/network_FirmwareUpgrades.json
M networks/L_123456789012345678 - Name/network_FloorPlans.json
M networks/L_987654321098765432 - Another Name/network_FirmwareUpgrades.json
D networks/L_987654321098765432 - Another Name/network_Devices.json
...(extra output omitted)...
Note, it is possible and desirable to put these into a user's cron environment to perform periodic scans without manual interventions.
Meraki Always On Sandbox - non-reservable, always on Meraki Enterprise Sandbox - reservable Meraki Small Business Sandbox - reservable
How to test the solution functionality - the scripts should generate entries in the meraki_base_path setting value of GMSIGconfig.py. Assuming default recommendations in the GMSIGconfig.py file are used, the directory path will be:
/opt/
└── MerakiGit/
└── orgid/
└── <OrgId>/
├── scaninfo
└── settingsThe scaninfo directory contains historical scan log information. The settings directory contains the Meraki settings data stored as JSON files. The top-level is organizational-level settings, then devices and networks directories contain their subordinate settings.
None known at this time.
For additional help, look to the DevNet Developer Support team. If major enhancements are requested, the Cisco Customer Experience (CX) team can be engaged for transactional assistance.
If you have questions, concerns, bug reports, etc., please create an issue against this repository.
How to contribute is found in the CONTRIBUTING file.
- Meraki REST API documentation https://developer.cisco.com/meraki/api/
- DevNet Meraki Developer Hub https://developer.cisco.com/meraki/
- Python Asyncio https://docs.python.org/3/library/asyncio.html
- Meraki 'backup_configs' by Maria Papazoglou (mpapazog) https://github.com/meraki/automation-scripts/tree/master/backup_configs
- diff2html by Rodrigo Fernandes (rtfpessoa) https://github.com/rtfpessoa/diff2html
This Cisco DevNet Automation Exchange Premium project is provided under the 'Cisco Sample Code License, Version 1.1'. The terms can be viewed at https://developer.cisco.com/site/license/cisco-sample-code-license/
Run the GetMerakiSettingsIntoGit script a few times before attempting to generate a diff. Observe how long the GetMerakiSettingsIntoGit script runs, then schedule the CreateMerakiGitDiffWebreport with some buffer time between executions in cron.