NYCPlanning/db-template-repo

POC: PLUTO+

Closed this issue · 2 comments

damonmcc commented

goals

  • create new repo to produce data product
  • output version for OSE and Lynn review and feedback

tasks to get output

  • Load in source data: mapPLUTO and 7 zoning files
    Build
  • Decide approach for assigning zoning information - update output table as we go or create intermediary table (i.e. _pluto) or?
  • Do we need to update zoning_create_priority.sql? If so, how to we maintain this moving forward?
  • Adapt 10 zoning scripts to new data product. These can be split out into individual tasks.
  • Determine what other fields depend on zoning district fields. Currently known: FAR
  • Adapt scripts to update values dependent on Zoning fields, such as FAR fields
  • Assign updated versioning information with new versioning schema (i.e. 22v3.1)
    QAQC
  • Determine if we want PLUTO intermediary versions to flow into QAQC app or not
  • If so, build process to incorporate new dataset. The app should have reports for all the things we we need to check.
  • If not, develop script(s) to confirm nothing but values in expected fields have changed and all else (number of records, etc) has stayed constant compared to source PLUTO.
  • Brainstorm how to QAQC Zoning Tax Lot Database (ZTLDB) and PLUTO+ in one go since both data products will be created by the GIS team monthly and both have most of the same source data.
    Export
  • Output same series of files available for PLUTO, which included additional resources, being mindful of Esri formats and schema. This probably gets broken out into more discrete tasks.

notes

  • prioritizing making build process work locally, CI run is a stretch goal
damonmcc commented

questions and notes

  • Should this output new PLUTO files or only MapPLUTO files?

    • aka if a user didn't want MapPLUTO 22v3.1, just PLUTO 22v3.1, would any fields be different from 22v3
    • both sets of outputs live in edm-publishing/db-pluto/main/latest/output/ ...
    • ** MapPLUTO is a subset of PLUTO records**, idea is to update MapPLUTO and apply changes to PLUTO

  • last major release of PLUTO was 22v3. should our first minor release be 22v3.1 or should we do 23v1 and then 22v1.1?

    • perhaps 22v3.1 is better so we can troubleshoot and compare to a major release that went through QAQC we know to be valid
    • we don't have to publish 22v3.1
    • if 23v1 is expected ASAP then we should maybe wait

  • where is CAMA data coming from?

    • there's ssh_cmd get Prod_FromDOF/CAMA.zip cama.zip and it seems to use envars related to "ginger"
    • the zip file isn't there
damonmcc commented

closing since this project was completed in PLUTO repo and this will become a template repo