payanam

A browser-based user-friendly tool for progressively mapping public transport routes when there wasn't geo-data available to begin with, and converting it to static GTFS standard.

Payanam (pronounced as "Paa-ya-nam") means "pathway" in Telugu.

Created by a joint collaboration between World Resources Institute (WRI) and Telangana Open Data Portal, Telangana, India.
Lead programmer: Nikhil VJ from Pune, India

Initially deployed on https://server.nikhilvj.co.in/payanam/

Getting Started : Running

Download this repository and unzip / clone to a folder on your computer.

A. Deploying the tool

Currently the tool needs to be run using Python 3 programming language - executable binaries are not yet available.
Set up a Python 3.6 Virtual environment. See this guide or lookup on the net.
Install the required packages: pip install -r requirements.txt (if not running in virtual env, use: pip3 install --user -r requirements.txt )
Run python launch.py (or python3 launch.py). This should deploy the tool on your web browser on a URL like localhost:5050. In case your default web browse is IE (Internet Explorer - default browser of Windows), please change to another web browser like Chrome or Firefox, as certain front end features may not work so well on IE.
If you're running this for the first time, you may see a popup asking to enter your API key. See the "access.csv" section below to know more. You can press "Cancel" to go with the default "Guest" user, and can change it from the bottom of the screen later.
Head to the Data Entry page and create your first route. After you have saved it, see the 'routes/' folder in the payanam folder and you'll see a new sub-folder created and a .json in it. That's your route.

B. Allied scripts

Once at least one route is saved from Data Entry page, run python reports_creation.py. This will generate the CSVs and shapefiles needed for the pages Overview, Stops Reconciliation, Map.
For making the above pages reflect the latest data, run reports_creation.py repeatedly at regular intervals. You can set this up as a cronjob on a linux laptop / server.
Similary, gtfs_creation.py reads the CSVs generated by reports script and generates GTFS feed in gtfs/ subfolder : both the .txt files and a gtfs.zip. Run that only after you have run reports_creation.py at least once. This too can be scheduled as a cron job or so, so that you have the latest GTFS output ready once you've set up your routes.
Please run both the scripts once manually to get an estimate of the time needed for them to run and thus the proper interval between runs. Typically, 5 to 10 mins interval is good for reports_creation and 1 to 2 hrs interval is good for gtfs_creation.
Here are sample crontab entries that can be set up on a unbuntu system by crontab -e command:

# Payanam: Reports generation, run every 10 mins 
*/10 * * * * /usr/bin/python3 /home/user1/payanam/reports_creation.py

# Payanam: GTFS generation, run every 2 hours
0 */2 * * * /usr/bin/python3 /home/user1/payanam/gtfs_creation.py

Cron tips:
- Ensure that the user account has write access to the payanam folder
- Use absolute paths in all things cron.
- You can redirect the stdout and stderr outputs to other log files or to /dev/null (aka the black hole); these files are logging everything to reports/logs/log.txt anyways.
- This website will help you figure out your cron: https://crontab.guru

Getting Started : Setup and Customization of options

Payanam comes packed with a plethora of assistance features to aid the user in mapping the routes. There are several options and settings laid out to configure the same. Upon changing any of the below, please re-launch the tool and empty you browser cache or Ctrl+F5 at every page to ensure that the latest settings are being loaded.

A. Local parameters: `js/common.js` :

Change STARTLOCATION, STARTZOOM values to reflect your region.
Change MBaccessToken, mapillaryClientID values to put in access tokens created at 3rd party services.
MBaccessToken : Mapbox Access Token. click here to create your free account and see this to get your access token (only public scopes needed).
mapillaryClientID : Mapillary Client ID. This is for the feature where you can fetch all street-view photos taken around 1km of your map's center. Click here to create your free account and see this to get your own Client ID.
Note: You can keep using the values already there, but this project's maintainer might dispose of them at any point (particularly if too many users are hogging the monthly free quota), so you're on your own. The mapbox access token is a little more critical for the various map backgrounds, so shoot for getting your own for that at least.
After any change here, Ctrl+F5 (or force-refresh) all the pages when you deploy the tool, so that the latest changes are reflected and browser doesn't use cached files.

B. Data writing access: `config/access.csv` :

Open this in Excel or another spreadsheet editor.
Here you can set up API keys (or passwords) for different levels of users.
First, edit the ADMIN row and assign a secret password and your email id.
For your team members, create rows with different access values, and assign each user a unique email id and key.
Do not share the 'key' values with everybody, and please keep them unique (one key = one user).
DATAENTRY level will allow to create routes in the Data Entry page.
MAPPER level will allow to map created routes in the Route Mapper page. This is the most basic level with any exective powers.
REVIEW level will let the user "lock" a route once its mapping and timings info is fully set.
ADMIN level allows for bulk actions in the Admin and Stops Reconciliation pages.
Please keep the "guest" line as-is to enable default visitors to see the site, to load up the individual routes, etc. Change it if you don't want the tool (if deployed on an intranet or the web) to work properly for regular visitors.

C. Configuration file: `config/config.json` :

databanksList has the list of databanks that can be used to assist in stop locations finding. They will appear under "Databank table" heading on Route Mapper page. The first one in the sequence will be the databank used (in addition to mapped stops) in Automatic Suggestions and Bulk Mapping.
timeDefaults contains the default trip times that will be assumed as trip timings for a route in GTFS calculation if its timings aren't saved. Also, these are the values that will load up on pressing "load Defaults" button(s) in Timings page.
sanityDistance is max distance in km from neighboring stops used in the sanity-check on auto-mapped routes (See Admin page). Auto-map picks that are too far away from the neighboring stops get dropped. (provided sanity check is enabled) fuzzyTopN: If the automatic suggestions function is working in fuzzy mode, if the number of matches exceeds this number then only this many top matches will be taken. This is to ensure cases like 50 stops getting suggested because the matching algorithm was too broad.

D. Databanks

In many cases, there IS a list of possible stop locations available; only it's not been matched with the routes yet.
User can store this 'databank' in the databank/ folder, configure it in config/config.json and then Payanam tool will show it on the map as yellow dots that one can click when mapping a route.
Multiple databank files can be stored in the databank/ folder.
There is a dummy databank 'randomPoints.csv' set up by default. See the structure and adapt your databank(s) accordingly (this follows the GTFS stops.txt structure).
You can use 'rank' column in your databank to prioritise which stops should be favoured in the matching algorithm.

E. Depots file: `databank/depots.csv` :

This is a simple lat-long listing of prominent depots / stations / terminuses that user may want to see marked on the map for reference while mapping routes. They show up as pink dots on the map pages. Edit the CSV to update to your location.

Use case scenario that this tool targets

This tool is for folks who need to digitize their public transport data to GTFS, but do not have a cleaned, mapped central database of stops to begin with.
Rather, the tool allows the user build up their data organically, starting from a route-centric perspective which traditionally transit agencies operate from.
At the Data Entry page, you can create a route by simply entering in a text list of stop names, which is how data is more traditionally managed. The mapping comes later.
After entering in the data, user can use the Route Mapper page to map each route.
Timings page enables one to enter timings per route. This tool works with trip times and with frequencies.

Assistive features

If we're mapping route by route, that can quickly lead to a lot of duplication of efforts plus messy redundant data with same stop names getting mapped just next to each other by different routes. Also, it does not make sense to keep manually mapping the same stop name again and again under different routes. To mitigate these issues, several assistive have been incorporated.

Data-banks of possible stop locations
Depots / Major terminuses being marked on map
Auto-suggestions
Bulk auto-mapping
Stops Reconciliation
Jumping stops detection and fixing
Dashboard to show progress of mapping
Convex Hull area calculation in routes overview to indicate which routes may need more intervention after auto-mapping
Integration with 3rd party street-level photography services like Mapillary and pic4carto

Difference from static-GTFS-Manager

For editing and managing GTFS feeds, we've published another tool, check it out: https://github.com/WRI-Cities/static-GTFS-manager
The two tools are complimentary with some overlap. The crucial difference is in the way stops are managed.
The static GTFS data standard has a strict starting requirement: a centralized, uniquely coded, mapped database of stops. The static-GTFS-Manager tool adheres to this.
But there are scenarios where that is the last thing a transit agency will have in hand. Rather, the starting data would be routes, and the only stops data available is in form of simple text names under each route - which is non-unique and non-mapped.
Therefore, the Payanam tool was created specifically to address this kind of situation.

What this tool does NOT do

Because Payanam is engineered towards one very specific use case of traditional public transport systems, it is therefore not able to cover many other scenarios that are covered under the broad GTFS standard. Some of these might get built in as features over time, so we'll update about it here. Listing those cases here, of what is NOT covered by Payanam tool

Different time schedules for different days of the week (this is first on the list of incorporating, at least for making a weekday-weekend distinction if not per-day-of-week)
Different trip patterns under the same route + direction
Fares and transfers
Grouping of stops under common stations/terminals
Different schedules for specific dates (calendar_dates.txt)
Per-stop timings. Instead, each trip's start timing is taken and duration is taken or computed. So in stop_times.txt, first and last stops in each trip will have timings (no waiting time at stop) and intermediate stops' arrival and departure times are left blank.

We advise users to use this tool only if their situation matches the described use case, and for the further tweaks mentioned, take it to static-GTFS-Manager or other GTFS management tools or services. This community site: Awesome Transit gives a good listing of all things GTFS.

WRI-Cities/payanam