patw/ElasticSensorDump

Re-write elastic rest API handling

patw opened this issue · 1 comments

patw commented

Currently the application works by collecting sensor data into a hash structure in memory, then dumping that hash to JSON periodically into an array. The array is consumed one element at a time in the ESDataLogger which results in 1 thread and 1 submit per item. This is fire and forget, we don't look at the response code from ES. On a fast ES host and fast Wifi this isn't much of a problem but most people do not have either one.

Proposal:

  • Switch to Bulk insert of all documents in the array
  • Pay attention to the elastic return codes so we can better track errors
  • Throttle application based on performance we are seeing from ES (maybe?)
patw commented

Didn't need to switch to bulk. Just proper resource handling seems to have fixed it.