SecurityCenter 5 - sc.analysis method - design issue - analysis/download
Closed this issue · 2 comments
To the point:
Using sc.analysis to down load large amounts (30 gigs) of json data requires you to use the "page" feature.
What is not documented by Tenable, but which I was told by Tenable premier (paid) support is:
- The first call must be to /analysis REST API
- Any subsequent 2 thorough nth calls must be done to /analysis/download
SecurityCenter5 has no capability to issue the second call.
What is not documented by Tenable is the quirky way in which the filter parameter works is:
lastSeen=n:m
where n, m are positive integer values and n must be less than m and they represent the number of days prior as a range. What is not well known or documented by Tenable is that these numbers represent the number of 24 hour periods from the wall clock time at which you submit your API call, not a calendar day, starting at midnight.
So, if you want to "pull" an entire 7 day period, you must specify 0:8 and throw out days 0 and 8. Yes, it is true. This make no sense and a very unworkable API.
So repeatably calling the /analysis with page 2 through n will cause you to miss some vulnerabilities in the json output. Let's say it takes 3 minutes to pull a page. If you call /analysis 500 time, you have 500 * 3 minute gaps that might cause you to loose/skip data.
There is no documentation on the web, that I can find anywhere, about the requirement to first call /analysis and then analysis/download n times. No doc, no examples, nothing.
Further update from Tenable Elite Support:
The analysis/download endpoint is only for CSV data download, not json data.
Hence, this is not an appropriate solution. There is nothing missing from SecurityCenter analysis for downloading json data.
However, the comments about the funky lastSeen=n:m still applies according to Tenable Elite support.
If you have any scan that completes in between calls to analysis endpoint, you will skip it and not download it.
First, sorry for the delay. Between travel and whatnot, I haven't had the time to get back to you on this one.
As for the Analysis call, /analysis is for JSON, /analysis/download is for the CSV report of the same data. I'd highly recommend using /analysis for programmatic purposes and then work your way through the pages (take a look at how the analysis endpoint is constructed in pySC). pySC isn't really receiving any more improvements however, as all current work is being focused on the pyTenable library that's more encompassing. If I can ask you to file an issue on the pytenable library covering the lastSeen filter oddities, I'll add it to the list of things that need to be done.
As for how the data is downloaded, pySC does make multiple calls behind the scenes and will stitch the data together into a singular list be default. If you want to overload this behavior, pass a page function to the call and it will feed the data to that function for every call thats made. This behavior predates my undertanding on iterators in python, and pyTenable is using an iterator instead to make this much easier to work with.