#ODI Summit 2013 - London fire stations project This is the repository for the Open Data Institute "London fire stations project" presented during the ODI summit on 29 October 2013. It includes two branches:
- the master branch includes the data preprocessing scripts and the API server
- the gh-pages branch is the website that you can visit at http://london-fire.labs.theodi.org/, including the client-side code for the interactive map.
##Setup instructions
###Dependencies All software distributed in this repository is written in either R or Node.js.
All R scripts have been tested vs version 3.0.2 of the interpreter on MacOS 10.8.5. The comments in the source code for each of the functions explain how to use them. The function signatures often include working default values, e.g. the relative position of the input files they have to be applied to.
For instructions on how to run R see here. After installing R, you will have to install packages rjson, sp and data.table.
All Node.js scripts have been tested vs version 0.10.21 of the interpreter on MacOS 10.8.5. Before execution, all dependencies must be installed by running npm install in the same folder of the scripts. By running the scripts without parameters, e.g. node stations.preprocess.js, any required instructions are provided on screen.
For instructions on how to run Node.js see here. Node.js's package manager npm can be used to install any dependencies.
###Preprocessing Before the server can be run, all pre-processing activities must be completed according to the data workdflow diagram at this url (PDF).
All source, raw data but for the Telefónica Dynamic Insights files are in this repository at data/raw. For convenience, data/preprocessed instead has copies of all the files expected as the output of the preprocessing stage.
Data must be processed in the same order as indicated by the arrows in the diagrams. The arrows are labelled with the name of the function to be run, e.g. incidents.preprocess. readAndClean corresponds to the function of the same name in the incidents.preprocess.R file in the preprocessing folder.
The entire preprocessing stage can take several hours to complete due to the volume of the data being processed. The output of the stage is the list of files below:
- To be used by the web client
- A set of [borough name].json "GeoJSON" files, each of which defines the boroughs' boundaries on the map and stores their default scoring, that is when all stations are open
- stations.csv, with all stations' addresses and coordinates, for displaying on the map
- boroughs_by_first_responders.json, with the list of stations that supported each of the boroughs' incidents, grouped by borough
- To be used by the server
- An incidents.csv file with all pre-processed incident records
- A census.csv file with the Office for National Statistics' key census information for the boroughs
###Server The server is a Node.js script, requiring a MongoDB database to read the incidents data from. Before execution, all dependencies must be installed by running npm install in the same folder where api_server.js and package.json are.
Assuming that we have console access to the database server, we can use the mongoimport utility to load the data in the database, as follows:
mongoimport -d databaseName -c incidentsData --type csv --file incidents.csv --headerline
mongoimport -d databaseName -c censusData --type csv --file census.csv --headerline
To optimise the speed of execution, it is necessary to create a few dedicated indexes in the database. Start a MongoDB console session by running:
mongo databaseName
and then execute the statements listed below:
db.incidentsData.createIndex({ "borough": 1});
db.incidentsData.createIndex({ "firstPumpStation": 1});
db.incidentsData.createIndex({ "borough": 1, "firstPumpStation": 1});
db.censusData.createIndex({ borough: 1});
Next copy config.env.example to config.env and update this to point to your database and set the port you want to run the server on.
Then, we can start the server by doing:
node api_server.js
You will see that the server starts caching a series of calculations that are the most common, to speedup execution. The server will be available for use at completion of this process.
2013/11/01 19:20:44 - The server is listening on port 8080.
2013/11/01 19:20:44 - Caching getBoroughResponseTime(borough)...
2013/11/01 19:20:44 - Calculating for the first time getBoroughResponseTime for Barking and Dagenham with closed stations:
2013/11/01 19:20:44 - Calculating for the first time getBoroughResponseTimes for Barking and Dagenham with closed stations:
(...)
2013/11/01 19:22:11 - Caching completed.
You can then test that the server is up and running by calling any of the APIs from a web browser, for example http://serverName:port/getAllBoroughsScores .