This repo provides starter code and documentation for new Councilmatic instances.
The councilmatic family is a set of web apps for keeping tabs on city representatives and their legislative activity.
Councilmatic started as a Code for America project by Mjumbe Poe, who designed the earliest version of a Councilmatic site – Councilmatic Philadelphia. DataMade then implemented Councilmatic in New York City (in partnership with the Participatory Politics Foundation), Chicago (in partnership with the Sunlight foundation), and Los Angeles (in partnership with the Los Angeles County Metropolitan Transportation Authority).
To simplify redeployment of Councilmatic instances, DataMade identified two necessities: (1) Open Civic Data, a data standard for describing people, organizations, events, and bills, and (2) the abstraciton of core functionality into a separate app. And so, DataMade built django-councilmatic – a django app with base functionality, common across all cities, which implements the OCD data standard.
With this template, you can create a Councilmatic site for your own city, built using the django-councilmatic app. Read on!
You need data about your city in the Open Civic Data API.
How you get your data into an instance of the OCD API is up to you. What does DataMade do? We use scrapers, which run nightly to update the API by scraping data from Legistar-backed sites operated by the cities for which we built Councilmatic. Your city may be running a Legistar-backed site, and if so, you can checkout python-legistar-scraper and the pupa framework to get a head start on scraping.
If you need examples of how to customize your scraper, look at scrapers-us-municipal as well as DataMade, which hosts several municipal-level scrapers. You can find information about what cities and other governmental bodies are already covered on the OCD DataMade site.
NOTE: This guide focuses on setting up your app for development. It does not discuss in detail the process of finding, scraping, and importing city data or deploying your site.
- Python 3.4
- PostgreSQL 9.4 +
git clone https://github.com/datamade/councilmatic-starter-template.git yourcity_councilmatic
We recommend using virtualenv and virtualenvwrapper for working in a virtualized development environment. Read how to set up virtualenv.
Once you have virtualenvwrapper set up, do the following in your bash profile:
mkvirtualenv councilmatic
cd yourcity_councilmatic
pip install -r requirements.txtAfterwards, whenever you want to use this virtual environment, run:
workon yourcity_councilmaticInside the git repository that you cloned above, you should see a folder called city. Rename this folder to something that makes sense for your project, e.g., "chicago."
mv city chicagoNow, update the main settings file - councilmatic/settings.py. First, in INSTALLED_APPS, add the name of the folder that you just renamed:
INSTALLED_APPS = (
...
~'city',~
'chicago',
)Then, change the TIME_ZONE. You can use this list to find the correct formatting.
TIME_ZONE = 'America/Chicago'Look for councilmatic/settings_jursidiction.py. This settings file tells your Councilmatic instance how to populate different parts of the UI and get fresh data from the OCD API. The following table explains why and how to adjust these values.
| Name | Description | Example value | Optional? |
|---|---|---|---|
| OCD_JURISDICTION_ID |
For scrapers hosted on the Datamade OCD API, you can find the jurisdiction id here. |
Look for councilmatic/settings_deployment.py.example. Make a copy of it so that you can customize it for your city:
cp councilmatic/settings_deployment.py.example councilmatic/settings_deployment.py
This file is important! It's where you keep the parts of your Councilmatic that should not end up in version control (e.g., passwords, database connection strings, etc., which will vary based on where the app is running and which you probably won't want other people to know). Most of these variables are generic Django settings, which you can look up in the Django docs; but some of these variables require customization. This table shows only the variables that you need to customize.
| Name | Description | Optional? |
|---|---|---|
| HAYSTACK_CONNECTIONS | Dictionary of connections used by Haystack. More on that here. | No |
| FLUSH_KEY | Used by the `flush` view in `django-councilmatic` to clear Django's cache. | No |
| DISQUS_SHORTNAME | Will cause Disqus comment threads to be attached to individual bill pages. | Yes |
| ANALYTICS_TRACKING_CODE | Google Analytics property ID | Yes |
| HEADSHOT_PATH | Absolute path to where the data loader will save images of members of the legislative body that are often available through Legistar. | Yes |
| EXTRA_APPS | List of additional django apps that you want to use with your custom councilmatic | Yes |
Before we can run the website, we need to create a database.
createdb yourcity_councilmaticThen, run migrations. (Be sure that you are "working on" the correct virtual environment.)
python manage.py migrate --no-initial-dataCreate an admin user. Set a username and password when prompted.
python manage.py createsuperuserGet an image (suggested 310x310), and transform it in device-specific favicon images using this site. Then, move the images into <city_app>/static/images/icons.
The django-councilmatic app comes with a import_data management command, which populates bills, people, committees, and events, loaded from the OCD API. You can explore the nitty-gritty of this code here. Note: Earlier releases of django-councilmatic (< 0.7) use loaddata, instead of import_data.
Running import_data will take a while, depending on volume (e.g., NYC may require around half an hour).
python manage.py import_dataBy default, the import_data command carefully looks at the OCD API; it is a smart management command. If you already have bills loaded, it will not look at everything on the API - it will look at the most recently updated bill in your database, see when that bill was last updated on the OCD API, and then look through everything on the API that was updated after that point. If you'd like to load things that are older than what you currently have loaded, you can run the import_data management command with a --delete option, which removes everything from your database before loading.
The import_data command has some more nuance than the description above, for the different types of data it loads. If you have any questions, open up an issue and pester us to write better documentation.
Now, you are ready to run your unique version of Councilmatic!
python manage.py runserverNavigate to http://localhost:8000/.
On a Councilmatic site, users can search bills according to given query parameters. To power our searches, we use Solr, an open source tool, written in Java.
Requirements: Open JDK or Java
On Ubuntu:
$ sudo apt-get update
$ sudo apt-get install openjdk-7-jre-headlessOn OS X:
-
Download latest Java from http://java.com/en/download/mac_download.jsp?locale=en
-
Follow normal install procedure
-
Change system Java to use the version you just installed:
sudo mv /usr/bin/java /usr/bin/java16 sudo ln -s /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java /usr/bin/java
Download & setup Solr
wget http://mirror.sdunix.com/apache/lucene/solr/4.10.4/solr-4.10.4.tgz
tar -xvf solr-4.10.4.tgz
sudo cp -R solr-4.10.4/example /opt/solr
# Copy schema.xml for this app to solr directory
sudo cp solr_scripts/schema.xml /opt/solr/solr/collection1/conf/schema.xmlRun Solr
# Next, start the java application that runs solr
# Do this in another terminal window & keep it running
# If you see error output, somethings wrong
cd /opt/solr
sudo java -jar start.jarIndex the database
# back in the nyc-councilmatic directory:
python manage.py rebuild_indexOPTIONAL: Install and configure Jetty for Solr
Just running Solr as described above is probably OK in a development setting. To deploy Solr in production, you'll want to use something like Jetty. Here's how you'd do that on Ubuntu:
sudo apt-get install jetty
# Backup stock init.d script
sudo mv /etc/init.d/jetty ~/jetty.orig
# Get init.d script suggested by Solr docs
sudo cp solr_scripts/jetty.sh /etc/init.d/jetty
sudo chown root.root /etc/init.d/jetty
sudo chmod 755 /etc/init.d/jetty
# Add Solr specific configs to /etc/default/jetty
sudo cp solr_scripts/jetty.conf /etc/default/jetty
# Change ownership of the Solr directory so Jetty can get at it
sudo chown -R jetty.jetty /opt/solr
# Start up Solr
sudo service jetty start
# Solr should now be running on port 8983Regenerate Solr schema
While developing, if you need to make changes to the fields that are getting indexed or how they are getting indexed, you'll need to regenerate the schema.xml file that Solr uses to make it's magic. Here's how that works:
python manage.py build_solr_schema > solr_scripts/schema.xml
cp solr_scripts/schema.xml /opt/solr/solr/collection1/conf/schema.xml
In order for Solr to use the new schema file, you'll need to restart it.
Using Solr for more than one Councilmatic on the same server
If you intend to run more than one instance of Councilmatic on the same server, you'll need to take a look at this README to make sure you're configuring things properly.
- Forest Gregg, DataMade - Open Civic Data (OCD) and Legistar scraping
- Cathy Deng, DataMade - data models, front end
- Derek Eder, DataMade - front end
- Eric van Zanten, DataMade - search and dev ops
- Regina Compton, DataMade - developer
If something is not behaving intuitively, it is a bug, and should be reported. Report it here.
We welcome your ideas and feedback! Here's how to make a contribution:
- Fork the project.
- Make your feature addition or bug fix.
- Commit - please be descriptive, but concise.
- Send a pull request. Bonus points for well-titled topic branches!
Copyright (c) 2015 Participatory Politics Foundation and DataMade. Released under the MIT License.