TrafficFlowListReport TimeFilter
ericgcoker opened this issue · 5 comments
I am using Python to create a TimeFilter object. I am using datetime objects in PANDAs and looping through them to create the TimeFilter object. Example and Output appear like this.
for index,row in df.iterrows():
print TimeFilter.parse_range(row['startdt'] + " to " + row['enddt'])
<steelscript.netprofiler.core.filters.TimeFilter(start=2017-05-25 12:35:00, end=2017-05-25 14:06:00> <steelscript.netprofiler.core.filters.TimeFilter(start=2017-05-25 22:49:00, end=2017-05-26 00:34:00> <steelscript.netprofiler.core.filters.TimeFilter(start=2017-05-25 22:19:00, end=2017-05-25 23:15:00>
However, I am also looping through for my IP from the same "df" dataframe; and not getting the correct date range. The only time I get the correct date range is with using something like "last 10 m".
Is this output I've presented the expected TimeFilter object output for the TrafficFlowListReport?
Is it possible that when I insert a time string into 'TimeFilter', it gets converted into UTC time, and I actually receive the results in my local time plus difference between local and UTC?
It's quite possible that timezone is affecting you here. There are some helper methods in steelscript.common.timeutils which may be of use, see ensure_timezone and force_to_utc. Feed one of those functions your datetime value and you will get a datetime-aware object in return.
Another question, if using TrafficFlowListReport, are
report.run(columns, timefilter=timefilter,limit=30000,trafficexpr=trafficexpr)
the only options, and is 10,000 the maximum allowable records? Because I can't receive more than that? I'm assuming I can't adjust the 'resolution' like some of the other 'realms'?
I'm essentially just looking for Client Host IP/Application Group Aggregation of Bytes
B/c the API is limited to 10,000, in some cases a DateTime range are needed. Anything can be converted to a string in Python, as it appears "parse_range" only accepts string input? But then the string is not TimeZone Aware? Further, it should be noted the Python's timeutils assumes CET, and the OUTPUT for all THESE API calls is epoch. So, unless you were inherently familiar with Python's Timeutils, you'd not know why the strings you were inputting into "parse_range" are actually being interpreted as CET, yet the results are in epoch (UTC). Is there another function other than "parse_range" for defining time endpoints?
It's been a while - but just to document some things before closing the issue:
The limit argument in the code snippet above should definitely allow greater than 10k results in the output. If that's still limited to 10k in what comes back there may be some other issues going on. There isn't really an upper bound to how many rows can be returned via a reporting call, but once you get above 100k or more performance can be impacted so that should be kept in mind.
For the time filter issues, the parse_range method will return naive timestamps (no timezone attached), and these will ultimately get interpreted to the local timezone when they are packaged to get sent in the report query.
Usually this isn't an issue - but if the NetProfiler you are querying is in a different time zone than where you are creating the query you can get some odd side effects such as querying for a time that happens in the future from the perspective of the NetProfiler.