Dashboards with a large number of widgets no longer return all data
mikewalkerblackpatch opened this issue · 4 comments
Since the change introduced by #173 some of our dashboards, which contain a huge number of widgets don't return all the data.
The GET request now being made (included below) does not contain all the data id's. There are 104 in the request shown below but we have 168 on this particular dashboard.
127.0.0.1 - - [02/Mar/2021:13:11:22 +0000] "GET /events?ids=applivery%2Capplivery.sit%2Capplivery.uat%2Capplivery.preprod%2Capplivery.live%2Cclient-api%2Cclient-api.sit%2Cclient-api.uat%2Cclient-api.preprod%2Cclient-api.live%2Ccollection-streams-processor%2Ccollection-streams-processor.sit%2Ccollection-streams-processor.uat%2Ccollection-streams-processor.preprod%2Ccollection-streams-processor.live%2Ce2e-parcel-events-connector%2Ce2e-parcel-events-connector.sit%2Ce2e-parcel-events-connector.uat%2Ce2e-parcel-events-connector.preprod%2Ce2e-parcel-events-connector.live%2Ce2e-tour-reporting-sink-connector%2Ce2e-tour-reporting-sink-connector.sit%2Ce2e-tour-reporting-sink-connector.uat%2Ce2e-tour-reporting-sink-connector.preprod%2Ce2e-tour-reporting-sink-connector.live%2Cemail-sender%2Cemail-sender.sit%2Cemail-sender.uat%2Cemail-sender.preprod%2Cemail-sender.live%2Chht-auth-service%2Chht-auth-service.sit%2Chht-auth-service.uat%2Chht-auth-service.preprod%2Chht-auth-service.live%2Chht-config-api%2Chht-config-api.sit%2Chht-config-api.uat%2Chht-config-api.preprod%2Chht-config-api.live%2Chht-event-feed%2Chht-event-feed.sit%2Chht-event-feed.uat%2Chht-event-feed.preprod%2Chht-event-feed.live%2Chht-parcel-api%2Chht-parcel-api.sit%2Chht-parcel-api.uat%2Chht-parcel-api.preprod%2Chht-parcel-api.live%2Chht-routing-api%2Chht-routing-api.sit%2Chht-routing-api.uat%2Chht-routing-api.preprod%2Chht-routing-api.live%2Chht-tour-api%2Chht-tour-api.sit%2Chht-tour-api.uat%2Chht-tour-api.preprod%2Chht-tour-api.live%2Ckafka-delayed-topics%2Ckafka-delayed-topics.sit%2Ckafka-delayed-topics.uat%2Ckafka-delayed-topics.preprod%2Ckafka-delayed-topics.live%2Cmars-et-services%2Cmars-et-services.sit%2Cmars-et-services.uat%2Cmars-et-services.preprod%2Cmars-et-services.live%2Cmars-opunits-connector%2Cmars-opunits-connector.sit%2Cmars-opunits-connector.uat%2Cmars-opunits-connector.preprod%2Cmars-opunits-connector.live%2Cmars-parcel-feed%2Cmars-parcel-feed.sit%2Cmars-parcel-feed.uat%2Cmars-parcel-feed.preprod%2Cmars-parcel-feed.live%2Cmars-parcel-feed-jdbc-connectors%2Cmars-parcel-feed-jdbc-connectors.sit%2Cmars-parcel-feed-jdbc-connectors.uat%2Cmars-parcel-feed-jdbc-connectors.preprod%2Cmars-parcel-feed-jdbc-connectors.live%2Cmars-parcel-target-hist-connector%2Cmars-parcel-target-hist-connector.sit%2Cmars-parcel-target-hist-connector.uat%2Cmars-parcel-target-hist-connector.preprod%2Cmars-parcel-target-hist-connector.live%2Cparcel-date-processor%2Cparcel-date-processor.sit%2Cparcel-date-processor.uat%2Cparcel-date-processor.preprod%2Cparcel-date-processor.live%2Croute-analysis-connector%2Croute-analysis-connector.sit%2Croute-analysis-connector.uat%2Croute-analysis-connector.preprod%2Croute-analysis-connector.live%2Croute-analysis-tour-data-connector%2Croute-analysis-tour-data-connector.sit%2Croute-analysis-tour-data-connector.uat%2Croute-analysis-tour-data-connector.preprod HTTP/1.1" 200 - 90.1307
The fact that you were still receiving data for part of widget didn't give me rest, so I've generated a dashboard with 160 widgets:
<% content_for :title do %>My super sweet dashboard<% end %>
<script type='text/javascript'>
$(function() {
Dashing.widget_base_dimensions = [50, 50]
Dashing.numRows = 10
Dashing.numColumns = 16
});
</script>
<div class="gridster">
<ul>
<% 10.times do |a| %>
<% 16.times do |b| %>
<li data-row="<%= a + 1 %>" data-col="<%= b + 1 %>" data-sizex="1" data-sizey="1">
<div data-id="test-<%= a %>-<%= b %>" data-view="Text" data-moreinfo="<%= a %>×<%= b %>"></div>
</li>
<% end %>
<% end %>
</ul>
</div>Looking at console I realised that EventSource is created too early and after failing to find anything in documentation I dug through batman.js code and found apparently the only event that should work without changing any other logic.
A bit more explanation in #182
@mikewalkerblackpatch releasing 1.3.3 now, if you could test your dashboard again with that fix. @toy 's PR linked this issue, but I am not sure if that fix will resolve the issue with large number of widgets that have also long names. Even if you manage to shorten the name of the widget ID's, we should still add the new flag to have the /events/ endpoint returning all the events without ids=abc,def etc. Thanks!
Thank you for your patience and help with this issue @mikewalkerblackpatch . All props go to @toy for his work on the past two releases 🥇