Re-write elastic rest API handling
patw opened this issue · 1 comments
patw commented
Currently the application works by collecting sensor data into a hash structure in memory, then dumping that hash to JSON periodically into an array. The array is consumed one element at a time in the ESDataLogger which results in 1 thread and 1 submit per item. This is fire and forget, we don't look at the response code from ES. On a fast ES host and fast Wifi this isn't much of a problem but most people do not have either one.
Proposal:
- Switch to Bulk insert of all documents in the array
- Pay attention to the elastic return codes so we can better track errors
- Throttle application based on performance we are seeing from ES (maybe?)
patw commented
Didn't need to switch to bulk. Just proper resource handling seems to have fixed it.