Ability to move pcap & indexes to "cold storage"
dcode opened this issue · 3 comments
I've had some users of RockNSM express the need for longtime archival of pcap data, potentially from multiple sensors. I'm curious if it would be sufficient to simply move the matching data and index files to another filesystem, and move back and query stenographer after a restart to pull that pcap.
So, there could be a main thread directory, for instance. On beefier sensors I put this on NVMe storage. From there, I can move to regular SSD or perhaps a SAN, which is more affordable for long-term storage. If this could be managed with something like filerotate or similar as noted above, then easy, peasy. Otherwise, it could be useful to allow an option to move the file (with some magic so we don't tie up the thread) rather than unlink the file.
pcap? Why not do it in Docket?
Sorry if this is a dumb question - I'm not sure what the exact use case is.
@JeffKwasha - When using Docket to export a large amount of PCAP, it ends up filling up the RAM and swap space quickly since it apparently loads the pcap into memory... I tried using Docket just that way, last 24 hours, and I think around the 60 second mark had my swap filled up and then looks like the task scheduler killed Java - was running Elasticsearch on the same system. I sat down with @dcode and he said - don't do that and use stenoread, but that uses enormous resources while running.
Yep. Your use-case is outside of Docket's intended capabilities (at least last I checked).
You might find curl options (stream to file) that would work for you (and you might add them to stenocurl).
If using curl (or stenocurl) is too resource intensive, I would investigate stenographer for a non-web-api solution.