This main repository for Data.gov's stack deployment onto AWS Infrastructure. The repository is broken into the following roles all created/provisioned using Ansible:
Included in this Repository:
- Software
- Data.gov (Wordpress)
- Catalog.data.gov (CKAN 2.3)
- Inventory.data.gov (CKAN 2.5)
- Labs.data.gov/CRM (Open311 CRM)
- Labs.data.gov/Dashboard (Project Open Data Dashboard)
- Security
- Baseline OS Hardening
- GSA IT Security Agents
- Fluentd (Logging)
- New Relic (Infrastructure Monitoring)
- New Relic (Application Performance Monitoring)
- Trendmicro (OSSEC-HIDS)
See our Roadmap.
Moved to datagov-infrastructure-live
- Ansible > 1.10
- SSH access (via keypair) to remote instances
- ansible-secret.txt:
export ANSIBLE_VAULT_PASSWORD_FILE=~/ansible-secret.txt
- run all provisioning/app deployment commands from repo's
ansible
folder - to update
ansible/roles/vendor
roles run there:ansible-galaxy install -r requirements.yml
Update/deploy all data.gov assets.
$ ansible-playbook site.yml
Note: the above playbook is incomplete. There are a few playbooks that must be
run with specific parameters. For that we include them in site.sh
:
$ ./site.sh {{ inventory }}
If the playbooks failed to apply to a few hosts, you can address the failures
and then retry with the --limit
parameter and the retry file.
$ ansible-playbook site.yml --limit @site.retry
Reboot any hosts requiring reboot after an update. Note: this takes a while since we only reboot one host at a time.
$ ansible-playbook actions/reboot.yml
Force reboot hosts even if no reboot is required. Use this if you just need to reboot hosts for any reason.
$ ansible-playbook actions/reboot.yml -e force_reboot=true
Install the trendmicro agent.
$ ansible-playbook trendmicro.yml
Upgrade OS packages as a one-off command on all hosts.
$ ansible -m apt -a 'update_cache=yes upgrade=dist' all
Reload the apache2 service for catalog.
$ ansible-playbook -m service -a 'name=apache2 state=reload' catalog-web-v1
Run a one-off shell command.
$ ansible -m shell -a "/usr/bin/killall dhclient && dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0" all
Tail the logs using dsh
.
$ dsh -g catalog-web-v1 -M -c tail -f /var/log/apache2/ckan.custom.log
cd ansible
ansible-playbook --help
See example(s) below
provision vm & deploy app: ansible-playbook datagov-web.yml --tags="provision" --limit wordpress-web
deploy app: ansible-playbook datagov-web.yml --tags="deploy" --limit wordpress-web
deploy rollback: ansible-playbook datagov-web.yml --tags="deploy-rollback" --limit wordpress-web
-
You can override branch to be deployed via
-e project_git_version=develop
e.g.
ansible-playbook datagov-web.yml --tags=deploy --limit wordpress-web -e project_git_version=develop
provision vm & deploy app: ansible-playbook dashboard-web.yml --tags="provision" --limit dashboard-web
deploy app: ansible-playbook dashboard-web.yml --tags="deploy"
deploy rollback: ansible-playbook dashboard-web.yml --tags="deploy-rollback"
provision vm & deploy app: ansible-playbook crm-web.yml --tags="provision" --limit crm-web
deploy app: ansible-playbook crm-web.yml --tags="deploy"
deploy rollback: ansible-playbook crm-web.yml --tags="deploy-rollback"
provision vm - web: ansible-playbook catalog.yml --tags="frontend,ami-fix,bsp" --skip-tags="solr,db,cron" --limit catalog-web
provision vm - harvester: ansible-playbook catalog.yml --tags="harvester,ami-fix,bsp" --skip-tags="apache,solr,db,saml2" --limit catalog-harvester
provision vm - solr: ansible-playbook catalog.yml --tags="solr,ami-fix,bsp" --limit solr
provision vm && deploy app: ansible-playbook inventory.yml --skip-tags="solr,db,deploy-rollback" --limit inventory-web
provision vm - solr: ansible-playbook inventory.yml --tags="solr,ami-fix,bsp" --limit solr
provision vm && deploy app: ansible-playbook jekyll.yml --limit jekyll-web
provision vm && deploy app: ansible-playbook elasticsearch.yml
provision vm && deploy app: ansible-playbook kibana.yml
provision vm && deploy app: ansible-playbook efk_nginx.yml
The CIS hardening benchmark sets a 027
umask, which means by default files are
not world-readble. This is often a source of problems, where a service cannot
read a configuration file.
This section describes how the Ansible inventories are organized and variables defined.
catalog-web
Web hosts for the catalog app.
catalog-harvester
Worker hosts for the catalog app.
jumpbox
Jumpbox host where Ansible playbooks are executed from.
solr
Solr hosts.
inventory-web
Web hosts for the inventory app.
crm-web
Web hosts for the CRM app.
dashboard-web
Web hosts for the Dashboard app.
wordpress-web
Web hosts for the datagov/wordpress app.
jekyll-web
Web hosts for the static/jekyll app.
elasticsearch
Elasticsearch hosts in mgmt vpc only.
kibana
Kibana hosts in mgmt vpc only.
efk_nginx
EFK hosts in mgmt vpc only.
v1
v1 hosts. These hosts run Ubuntu Trusty. Each group name will have a -v1
suffix e.g.
catalog-web-v1
.
This helps us transition between stacks.
v2
v2 hosts. These hosts run Ubuntu Bionic. Each group name will have a -v2 suffix e.g.
catalog-web-v2
. This helps us transition between stacks.
web
Meta group containing any hosts with a web server (e.g. apache2 or nginx).
Install the dependencies (from a python virtualenv).
$ make setup
Run the playbooks locally.
$ make test
You can set the concurrency parameter with make's -j
parameter.
$ make -j4 test
This runs all the suites, both molecule and kitchen tests. See below for more on how to work with individual suites. Both suites rely on docker for running tests within containers.
Lint your work.
$ make lint
Molecule is the preferred test suite for testing roles. Playbooks can be tested by including them in the molecule playbook.
Molecule is modular, so you must cd
to the directory of the role you are
testing.
$ cd roles/software/ckan/native-login
$ molecule test
During development, you'll want to run only the converge playbook to avoid creating/destroying the container every time.
$ molecule converge
If you have multiple scenarios, you can specify them individually.
$ molecule test -s <scenario>
We use Kitchen for testing playbooks, although we are moving suites to molecule.
Run a single suite.
$ cd ansible
$ bundle exec kitchen test catalog
Log into the instance to debug.
$ cd ansible
$ bundle exec kitchen login catalog
Re-run the playbook from a particular step.
$ ANSIBLE_EXTRA_FLAGS='--start-at-task="software/ckan/apache : make sure postgresql packages are installed"' bundle exec kitchen converge catalog
Refer to kitchen commands for more information.