Add Origin and Destination Census Tracts
Closed this issue · 10 comments
Add census tract geocoding to trip processing and include origin/destination census tracts in output.
Tasks:
- created geocoded census tract geojson covering city of austin
- udate processing script
- create database fields
- add fields to public dataset
data processing point in poly is updated and ready for deployment. going to first process the backlog of trips by running locally. estimating ~4 hours of runtime. the prod point-in-poly will still be running and updating new trip data with hexbins (although those are no longer exposed to a public enpoint). once the backlog processing is successful will deploy the new branch, which will stop hexbin processing and those fields can then be dropped from the db.
the socrata publisher is going to be picking up these records because their modified dates will be timestamped from the census tract geocoding. socrata api does not handle large loads well so we may end up choking it. will see.
derp. despite my checkboxes above i forgot to add the census tract fields to the socrata dataset. i just made the changes to a new draft of the dataset, but the editing interface is not letting me publish this new version (the "update" button is inactive). It seems that it can take a while for it to be possible to publish dataset changes because the draft version first processes these updates. I don't really know and will open a Socrata issue if it's not available for publication in the AM.
Once the new fields are added to the socrata dataset I will kick off a total refresh of the data from postgres. It's going to take days...
I'm still unable to update the Socrata dataset. Just submitted an issue w/ their support.
Socrata resolved the issue and the Socrata dataset has now been updated with census_geoid_start and census_geoid_end.
I'm not sure how useful the data is with a Census Tract granularity.
Could the Military Grid Reference System (MGRS) be used?
For areas with low traffic density, a 1km precision reference could be stored. For areas with high traffic density, a 100m precision reference could be stored.
This would protect anonymity in the 'burbs, while still allowing decently detailed analysis and visualization in the most important corridors.
Thanks @avickers. For now we've settled on census tracts for the individual trip data, and intend to produce an aggregated dataset at a higher spatial resolution. Do you have a specific use case or methods in mind? That will help us define what the aggregated data looks like.
@avickers we're using Zenhub so you may not see this parent issue:
https://github.com/cityofaustin/atd-dockless-processing/issues/6
all data processed and db views updated. kicking off socrata csv upload. it's a lot easier than trying to chunk/script it and handle the many failures
Socrata import is not working. The columns are empty after I import the data. I opened another helpdesk ticket.
Not especially. I did have one idea in mind, but it would be scope creep, so I won't bring it up until after you've finished the basic implementation.
One thing I can say from having done some traffic stuff for the Smart Cities initiative is that it would be cool if:
- Austin released its updated Traffic Analysis Zones (TAZ) on Socrata; and,
- You released this data using TAZ in addition to Census Tracts.
Traffic studies and scholarly works are more likely to utilize TAZs than Census Tracts. The CoA defines its TAZs in conjunction with TXDoT, but the only extant copy that I could find is from 2000 and made available by the US Census Bureau.