Dashboard errors
faister opened this issue · 10 comments
Type of issue
- Bug
- New feature
- Improvement
Description
When I start ingesting telemetry from about 400 devices sending around 162K D2C messages over a 12-hour period, the dashboard stopped rendering. The error messages across all the frames were the following:
Error: Oops, we were not able to find the service.
Screenshot
As attached
Steps to reproduce
- Deploy remote monitoring solution.
- Create physical devices to send telemetry to IoT Hub. I followed one of the tutorials from https://docs.microsoft.com/en-au/azure/iot-accelerators/iot-accelerators-connecting-devices-node
- Issues started popping up around after 3 hours of operation.
Expected behavior
Dashboard renders correctly.
Current behavior
Error: Oops, we were not able to find the service.
When I checked the telemetry container, issues started popping up around 3 hours after devices started sending telemetry. It started with StorageClient:QueryDocuments returning 0, and then errors. See logs below:
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:29:47Z][DEBUG][Alarms:List] Created Alarm Query, {"s
ql":"SELECT TOP 1000 * FROM c WHERE (c["doc.schema"] = "alarm" AND c["device.id"] IN ("750262","750263","
750334","781146","896190","896439","899160","7C7AA6","7C01C2","7CF4A1","7C7A36","7CF466","7C6D2A
","7C6D35","7CF45B","7C7BD5","7C6C74","7C6DB8","7C6C75","7CF451","7C6B0C","7C6D33","7C6D9B","7
C6DEC","06A146","7C77F9","7CF4B2","7C1473","7C6B3C","7C6B1B","7C6DE6","7CF483","7C4487","7C7801
","7C6D2B","7C6D36","7C530A","7CF49A","7CF456","A874BF","7CF47C","7C1465","7C8026","7C6D97","7C
6D34","7C752B","C88006","7C7701","7C8101","7C37C5","7C351B","7C6C27","7C6C5A","76CEF2","7CF447"
,"7C7AA1","7C6D37","7C6C57","7C6CA1","7C6277","7C6178","7C7527","7C81E0","7C2F54","7CF452","7C7
A38","7C7747","7C1474","7C6289","7CF441","7C617C","7C7BD7","7C4515","7CF45F","7C6C32","7CF4AA",
"7C6D90","7C6A8C","7C6D9D","7C7C98","7C4E6B","7CF4B6","7C146E","7C6C24","7C6DE3","7C7AA7","7C1C
55","7C6DB2","7C75B8","7C089A","7C01C0","7C1C5D","7C6C2B","7C6DD5","7C6C9F","780A36","7C1C5C",
"A61C53","7C6DB9","7C6DDD","7C531D","7C7A43","7C6DED","7C6C9C","7C530E","7C6C97","A47244","7CF4E
4","7C6DD6","7CF860","7C1469","7C81D1","7C6D94","7C6C29","7C6D99","C8187E","7C6DB6","7C6D27","
7C6D75","7C4990","7C6D39","7C5963","7C617E","7C7A45","7C6C94","7C6C9B","7C6D24","7CF480","7C6725
","7C81D2","7C6D25","7CF4BE","7CF4A6","7C6DBB","A782DD","7CF7BD","7C390B","7CF44C","7C6D20","7
C15AF","7C6B18","7C6B09","7C6DB3","7C7A4B","7C6D78","7C7802","7CF4B0","7C6DDB","76CEF9","7C01C1
","7C7A44","7C6A8D","7C6DB0","7C6D26","7CF3CE","7C6B3D","7C5307","7C6B2F","7C7C77","C81DF5","7C
6276","C8178C","7C6DB1","C82258","78115C","7C5B45","7C6D96","7CF47B","8963E5","7CF46C","7C7A41"
,"7C5310","7CF4B4","7C495B","780B33","7C146D","89645B","7C6D23","7C4838","C816DC","7C39FB","7CF
4B7","86EB2E","C82347","7CF4B8","7C6DE5","7C6D8E","7CF43B","7CF460","7C7AB6","7C81DE","7CF43A",
"7C6B12","7C5357","7811E5","89618C","7C7C74","7C77FE") AND c["device.msg.received"] >= 1529695787647 A
ND c["device.msg.received"] <= 1529699387647) ORDER BY c["device.msg.received"] ASC"}
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:29:47Z][INFO][StorageClient:QueryDocuments] Query re
sults count:, {"Count":0}
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:29:47Z][INFO][StorageClient:QueryDocuments] Query re
sults count:, {"Count":0}
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:29:56Z][DEBUG][AuthMiddleware:Invoke] Skipping auth for service to service request
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:29:56Z][DEBUG][HttpClient:SendAsync] Sending request, {"httpMethod":{"Method":"GET"},"Uri":"http://storageadapter:9022/v1/collections/rules/values","Options":{"EnsureSuccess":false,"AllowInsecureSSLServer":false,"Timeout":10000}}
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:30:03Z][ERROR][AuthMiddleware:Invoke] Authorization header not found
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:30:03Z][ERROR][AuthMiddleware:Invoke] Authorization header prefix not found
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:30:03Z][WARN][AuthMiddleware:Invoke] Authentication required
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:30:03Z][ERROR][AuthMiddleware:Invoke] Authorization header not found
[WebService.cdb863fa-9406-48bc-ac94-6b3c5f3d31a1][2018-06-22 20:30:03Z][ERROR][AuthMiddleware:Invoke] Authorization header prefix not found
Known workarounds
None
Possible solution
What's creating the authorization and authentication errors?
Context and Environment
- Operating System: n/a
- GitHub branch: Current
- .NET Runtime: n/a
@zhang-hua Can you take a look at this? You can investigate this as part of 1808.
@zhang-hua Can you please close the issue if it has been fixed.
@ppathan Check if latest PR contains this fix.
will investigate 9/21
This requires sending a post instead of a get when the get query is more than 2048 characters. This will require 3 PRs:
Java & .NET versions of adding POST requests for any GET requests that require a list of devices
Web UI change to use a POST request when the query is too long
@asdonald To link back to the WebUI issue here.
Is there are any estimates when those issue will be fixed?
@AlexShkabura end of November
@AlexShkabura these changes are all in master. They should be in the release coming at the end of the month
@molly-moen Thanks. Will be waiting for those release.