sketch-city/harvey-api

Roadmap 2017-09-07

Opened this issue · 2 comments

ironcoconut commented

API Team's Roadmap for the next 2 weeks.

Goals

First, we have built something that works and provides real value to people on the ground. Because so many groups are using the api, it is important that we test thoroughly to ensure we do not break v1 features. That being said, goals are:

Version 2

  1. Simplify the import process for adding spreadsheets from various organization.
  2. Deploy a v2 version of the html and api endpoints that programatically builds tables, forms, navbar and json endpoints for each sheet for each organization.
  3. Update schema to use a single, unified location table for all data. Data from the spreadsheet import will be stored in a JSONB field.
  4. Build a migration that can convert the current tables to v2. (optional, based on the needs of the various orgs)

These will allow us to rapidly on board organizations, train them in how to use the software and enable to use their own methodologies for collecting, managing and displaying data. @ironcoconut is the process of achieving these 4 goals. Having a single endpoint will facilitate further normalization of the data model and enable us to build an organization/role based permissioning system if necessary.

UI Improvements

  1. Update data tables in real time (#38)
  2. Show status of row on tables (#38)
  3. Allow admins to update columns from the tables
  4. Allow admins to approve updates from the tables
  5. Build a single location interface, an interface with a simple navbar that shows only the location name and the user's status (logged in/out). This will allow us to simplify the workflow for less tech savy volunteers.

Data Normalization and Version 3

A downside of the current approach for v1 and v2 is that all data is served from multiple endpoints b/c the data lives in multiple tables in the db. Once all data is migrated to v2 on the locations table, we can build out tables for each type (shelters, rescues, needs) and then create a form that allows volunteers to manually migrate the data from the legacy JSONB field to the proper table. Once the data is normalized and migrated, we can serve all data in a given project (Irma, Harvey, City of Houston) from a single endpoint. This normalization should be optional and should not interfere with v1 or v2. I am unsure if v3 will require an update to the HTML pages.

Issues, Thoughts, etc...

A problem I see is that once we move an organization from v1 to v2 or from v2 to v3, the old data sources, specifically the endpoints, will no longer be up to date. The data will still be there, but because the volunteers are using data entry for the new version, the old version's data will not get updated. This means that some orgs may choose not to upgrade. It means that we need to clearly communicate the costs and risks of the migration.

If we can coordinate with the maps people, perhaps we can build a single map to rule them all for each version. This way the orgs can simply migrate to using the new map once their data has been moved to the new system.

Migrating or normalizing data during the storm is not an option.

rmoscowitz commented

Really great stuff in here!

  • Can you explain how normalizing the data and making it available on one endpoint would work? If I try to GET from api.futurestorm.org, do I get one big response with everything in it? If I'm building a new app, I might not want all of the data, so child endpoints like /needs or maybe selectors or more filters would be helpful.
  • "programmatically builds tables, forms, navbar and json endpoints for each sheet for each organization" - Can you explain this more too? Does this mean an organization could decide they want to deploy the api with just Needs, nothing else? That would be AMAZING! Who configures this and how?
  • Any plans to support POST/PUT in v2?
  • Allowing editing/approving from the main table would be a nice ux improvement. It's definitely a familiar action if a user has spreadsheet experience.

Edit - one more thing! This is probably still nice-to-have, but a workflow that allows transitioning a shelter to a need (i.e. a donation-only site) would be pretty sweet. If those models are being revamped, planning for this would be awesome.

ironcoconut commented

Can you explain how normalizing the data and making it available on one endpoint would work?

There would be filters so that you would only get the data you need. The specifics are yet to be determined. We would want to be able to filter by type (shelter, rescue, etc), by managing organization, by status and probably more.

"programmatically builds tables, forms, navbar and json endpoints for each sheet for each organization" - Can you explain this more too?

Your org would tell use which google sheets to import. After we import them, you could visit the following links for html pages:

  • your_org/table_name: a table with all the data
  • your_org/table_name/123: a specific entry on the table

For the api, you could get the same data but in JSON form for maps and other apps.

Any plans to support POST/PUT in v2?

Not at the moment. We believe it is easier to include a link to the show and edit pages for a record on the map. Then the user can click the link and update the info on the api page. This way maps only worry about displaying data and links to the pages to edit it. It saves us from having to build authentication on the json endpoints, which could be a real pain.