/geospatial

Primary LanguageJavaScript

-Caution-

if you are running your own db...
  • the insert script that is generated is huge...
    • the following errors indicate the script size is a problem:
      • [ERROR] error executing sql script { [Error: Packets out of order. Got: 2 Expected: 3] code: 'PROTOCOL_PACKETS_OUT_OF_ORDER', fatal: true }
      • [ERROR] error executing sql script { [Error: Connection lost: The server closed the connection.] fatal: true, code: 'PROTOCOL_CONNECTION_LOST' }
  • HACK/bandaid/fix: change max_allowed_packet to something like 50MB in the [mysqld] section of your my.cnf file - There is a tech debt item listed below to fix this

-To Setup App Deps-

npm run setup

-To Setup Data For App-

npm run load:db
npm run load:pre:solr
npm run load:solr

-To Run App-

npm start

-Available Commands, Descriptions & Usages-

npm run help
  • shows help doc
npm run setup
  • runs npm install and sudo npm install -g gulp
  • this is buggy --- it's better just to manually npm install then sudo npm install -g gulp
npm run load:db
  • drops an existing table called GeoPoints in package.json config.db
  • creates a table called GeoPoints in package.json config.db
  • gets csv from here
    • writes request result package.json > [config.inputDataFilePath].csv
    • writes [inputDataFilePath].sql in package.json config
      • rows from package.json [config.inputDataFilePath].csv file are converted and written into a sql script
    • runs package.json [config.inputDataFilePath].sql script against package.json config.db
npm run load:csv
  • fetches all records from existing GeoPoints table in package.json config.db section
  • writes all data rows for pickups and dropoffs to:
    • [inputDataFilePath]_pickups_keyed.csv in package.json config
    • [inputDataFilePath]_dropoffs_keyed.csv in package.json config
npm run load:pre:solr
  • fetches all records from existing GeoPoints table in package.json config.db section.
  • creates solr docs for each record
    • creates grid-based cluster keys/values
      • creates key/value by truncating lat/lon values from between 0 to 6 decimal places
    • calculates centroids for grid-based cluster levels
    • writes solr docs in csv formate to [centroidedSolrDocsCSVPath] in package.json config section.
npm run load:solr
  • curls [centroidedSolrDocsCSVPath] file contents (package.json config section) to solr, creating documents
npm start
  • starts app
npm install
  • installs app deps

-Tech Debt & Things To Improve-

  • User should only be able to draw one polygon at a time
    • existing polygon should disappear when starting a new polygon
  • Clustering needs fixed/improved
    • Implement visual cue to denote whether cluster is server-side or client-side cluster?
    • Make rules around clustering smarter
      • server/client clustering could be used exclusively when it makes the sense to use either/or
  • Improve documentation
    • Install jsdoc and make sure output is correct
  • Minify/uglify and cache html, js, etc
  • turn on server compression
  • npm run load:db sql insert script should be broken into reasonably-sized chunks of scripts
  • Automatically draw polygon when route param reflects a user-drawn polygon.
  • Add a groupings tool to the toolbar, eg: pickups and dropoffs
  • change db user permissions
    • less privileged user should be used
  • change initial points-loading call to a post-init function
    • it's currently done as a matter of set up, which means, depending on conditions, it could get fired more than necessary
  • Everything should be URL driven
    • url should contain current geometry and zoom
  • keep track of points loaded into the map
    • only load diff points NOT in the map for each new user action
    • break up clusters as the user zooms in, and retain the loaded and more precise points as the user zooms back out
    • look at performance implications of retaining all points of broken up clusters
      • consider re-clustering as a means of mitigating performance concerns.
      • consider caching or storing user temporal cluster data