Average temp and humidity calculations resulting in saw-tooth rendering in D3
Opened this issue · 15 comments
I have not been able to pin down the cause of this, but the averages are being plotted as a saw-tooth on the website.
I have noticed the errors that are likely causing this behavior, but I am having difficulty pinpointing where in the callbacks the error is created. The error is being detected at line 670 of D3.js with an invalid path (see attached image)
Any help or guidance would be appreciated.
Here's another capture that demonstrates the issue in the data stream:
Error: Invalid value for attribute d="M0,NaNC0,NaN,0,NaN,0,NaNS0,NaN,0,NaNS61.44723652342241,NaN,61.44723652342241,NaNS61.44723652342241,NaN,61.44723652342241,NaNS92.16959834959353,NaN,92.16959834959353,NaNS92.16959834959353,NaN,92.16959834959353,NaNS92.16959834959353,NaN,92.16959834959353,NaNS92.16959834959353,NaN,92.16959834959353,NaNS122.8894473046845,NaN,122.8894473046845,NaNS153.61683487301593,NaN,153.61683487301593,NaNS153.61683487301593,NaN,153.61683487301593,NaNS184.3341709570267,NaN,184.3341709570267,NaNS215.05653278319784,NaN,215.05653278319784,NaNS215.05653278319784,NaN,215.05653278319784,NaNS245.778894609369,NaN,245.778894609369,NaNS276.5062821777004,NaN,276.5062821777004,NaNS276.5062821777004,NaN,276.5062821777004,NaNS276.5062821777004,NaN,276.5062821777004,NaNS307.24120835927243,NaN,307.24120835927243,NaNS337.94598008788233,NaN,337.94598008788233,NaNS337.94598008788233,NaN,337.94598008788233,NaNS337.94598008788233,NaN,337.94598008788233,NaNS337.94598008788233,NaN,337.94598008788233,NaNS368.6708547851336,NaN,368.6708547851336,NaNS368.6708547851336,NaN,368.6708547851336,NaNS399.3907037402246,NaN,399.3907037402246,NaNS399.3907037402246,NaN,399.3907037402246,NaNS399.3907037402246,NaN,399.3907037402246,NaNS430.11055269531556,NaN,430.11055269531556,NaNS460.8354273925668,NaN,460.8354273925668,NaNS491.557789218738,NaN,491.557789218738,NaNS491.557789218738,NaN,491.557789218738,NaNS522.2751253027487,NaN,522.2751253027487,NaNS522.2751253027487,NaN,522.2751253027487,NaNS553.0100514843207,NaN,553.0100514843207,NaNS553.0100514843207,NaN,553.0100514843207,NaNS583.7223618261711,NaN,583.7223618261711,NaNS614.442210781262,NaN,614.442210781262,NaNS614.442210781262,NaN,614.442210781262,NaNS645.1620597363531,NaN,645.1620597363531,NaNS675.8894473046845,NaN,675.8894473046845,NaNS706.6143220019358,NaN,706.6143220019358,NaNS737.3291452148663,NaN,737.3291452148663,NaNS737.3291452148663,NaN,737.3291452148663,NaNS768.0615585253582,NaN,768.0615585253582,NaNS768.0615585253582,NaN,768.0615585253582,NaNS798.7738688672086,NaN,798.7738688672086,NaN"
It appears that when the vertices are forced to redraw, the d variable on the SVG object are becoming corrupted, possibly by the scale function, and have difficulty recovering. I'll explore more for a cause and fix.
@dankline I'm also facing same issue. The scaleMargin is NaN or Infinity so the value of path "d" has NaN
/cc: @olivierbloch
Thanks @punitganshani. I tried setting a Min and Max value based on the first value for the node (min = .75 x value and max = 1.5 x value) but it didn't give me the result expected. With all the callbacks in the code, it is very tedious to trace the variables. I'll dive back into it in a few days, but for now, I'm buried in tax preparation. Keep my posted if you make progress.
@punitganshani I noticed you aren't displaying averages. Does the chart resize when the scale is exceeded? I wasn't able to get the averages to not render. When I turned off the aggregator analytics job, the scales never rendered resulting in a plot that was never rendered. I'm curious how you accomplished this. I found the averages annoying at best and would rather not have them render. I think they were added to sample just for the stream analytics demonstration.
@dankline yes the scale changes. The scaleMargin was becoming Infinity so the domain had invalid range values. self._y0 evaluation was giving NaN which was not rendering the values correctly. Yet to try the averages but this resolved the NaN issue.
Thanks. Is it possible to get a copy of what you have done?
From: Punit Ganshani [mailto:notifications@github.com]
Sent: Thursday, March 03, 2016 9:01 AM
To: Azure/connectthedots connectthedots@noreply.github.com
Cc: Daniel C. Kline dan@klines.org
Subject: Re: [connectthedots] Average temp and humidity calculations resulting in saw-tooth rendering in D3 (#233)
@dankline https://github.com/DanKline yes the scale changes. The scaleMargin was becoming Infinity so the domain had invalid range values. self._y0 evaluation was giving NaN which was not rendering the values correctly. Yet to try the averages but this resolved the NaN issue.
—
Reply to this email directly or view it on GitHub #233 (comment) . https://github.com/notifications/beacon/AC8ECjBr-T6sPXTvCGh0dhYT7AXp2FhPks5ppveWgaJpZM4HWAnE.gif
Yes, the commit is merged into this repo (see #238) so you can try with the latest code. Average Stream Analytics works with this as well.
Thanks ... I'll try it.
Congratulations @punitganshani. Nice job on the scale redraw issue. I even held my finger on the sensor to cause the temp and humidity to climb rapidly, and the scaling and redraw worked perfectly. I'm still seeing an issue with the averages, but curiously, not on the leading edge of the rendering, but rather on the redraw of previous data. If time allows, I'll set up a stream capture and compare it against the D3 render data for the values. I suspect that it may have to do with a mismatch between timing of the node data and the streaming analytic data sampling. I'll let you know if I figure it out. Thanks again. That was an awesome piece of work.
Here's a capture of the average rendering issue.
Interestingly, if I leave the application rendering for a long enough time, the averages disappear from the renderings. I'll have to capture the streams to see if we have another NaN issue with the Stream Analytics data.
I have stumbled, somewhat intuitively on the solution to the saw tooth averages in my renderings and would appreciate if anyone can verify my assumptions before I make recommendation for changes to the project. In the configuration for the BI service I added some handling for late arrivals and it appears to have stabilized the renderings. Can someone try this in their code and tell me if the result are consistent.
