netdata/netdata-cloud

[Bug]: Windows node has gone stale and no longer showing in the parent Netdata instance.

Closed this issue · 9 comments

mcdent commented

Bug description

A windows node, running the prometheus endpoint is now showing as 'stale' in the netdata cloud (NC) this was added as a vnode. It used to show but for some reason now it does now.
It also does not show at the end of the list in the Linux node which is scraping its data. I can connect

Expected behavior

I should actually be seeing 2 windows nodes at the bottom of the screenshot on the right side. As you can see in the screenshot neither are showing.

Steps to reproduce

Try to view the windows node in the list under the linux host that is configured locally to run netdata.
This is the same output when viewing via the cloud login or going direct to the local IP of netdata.
I can also curl to the prometheus endpoint running on the windows host.

Screenshots

here there should be 2 windows nodes showing windows and vnode config files metrics access ok vnode shows as stale curl to the windows endpoint

Error Logs

No response

Desktop

OS: MAcOS
Browser Firefox
Browser Version 123.0 64 bit

Additional context

No response

hugovalente-pm commented

@mcdent do you have the same issue if you open your Linux machine local agent dashboard?
Have you also tried to run the go.d.plugin in debug mode from the Linux machine? https://learn.netdata.cloud/docs/collecting-metrics/windows-systems/windows#debug-mode

mcdent commented

Thanks for the tip @hugovalente-pm. I recreated the windows.conf and vnode.conf files, with fresh UIDs and now there seems to be no errors in the debug mode.
My remaining problem is the node which is marked as 'stale' is not collecting/sending any data, despite me being able to curl to the prometheus metric endpoint from the linux host. It shows as stale still in the linux host dashboard.

Is the fact it is marked as stay preventing this?

One other thing, no hardware spec of the windows machine is showing, I assume this is a limitation of prometheus?

Thanks
Mike

hugovalente-pm commented

My remaining problem is the node which is marked as 'stale' is not collecting/sending any data, despite me being able to curl to the prometheus metric endpoint from the linux host. It shows as stale still in the linux host dashboard.

can you try just having just that node configured on your Linux machine and then run the debug mode? also try to access your Linux machine local agent dashboard?

Is the fact it is marked as stay preventing this?

Stale means your Linux node has past data for that node but it doesn't "find" any current data.
a gut feeling, could there be a clock sync issue on that machine?

One other thing, no hardware spec of the windows machine is showing, I assume this is a limitation of prometheus?

yes and no, atm we only get those specs for nodes where the agent is locally running there. we know we need to find a way to get this for remote Windows machines but it something currently on our backlog

mcdent commented

My remaining problem is the node which is marked as 'stale' is not collecting/sending any data, despite me being able to curl to the prometheus metric endpoint from the linux host. It shows as stale still in the linux host dashboard.

can you try just having just that node configured on your Linux machine and then run the debug mode? also try to access your Linux machine local agent dashboard?

Ok, doing this has now brought the errant asdf123 windows machine back. I have not tried adding the config file again for the windows box wiresx1, I will do that next?
Notice although the asdf123 is now reporting, it still shows as stale?

Screenshot 2024-03-14 at 19 13 39

Is the fact it is marked as stay preventing this?

Stale means your Linux node has past data for that node but it doesn't "find" any current data. a gut feeling, could there be a clock sync issue on that machine?

I checked the time on both windows boxes and they are both set to automatic and the time seems to align with that from the linux box collecting the stats.

One other thing, no hardware spec of the windows machine is showing, I assume this is a limitation of prometheus?

yes and no, atm we only get those specs for nodes where the agent is locally running there. we know we need to find a way to get this for remote Windows machines but it something currently on our backlog

IS this why I am missing memory stats for one thing?
Screenshot 2024-03-14 at 19 14 13

I also note that the average CPU utilisation does not seem to align with that of the Windows 11 pc?
Task manager on windows shows a fairly steady 20% cpu (whilst running some programs) however looking back over the same period on Netdata shows it as much lower?
Screenshot 2024-03-14 at 16 44 28
IMG_4504
Am I expecting too much from the windows clients? I don't really want to run the full agent on these boxes if possible.

Thanks
Mike

mcdent commented

Hmm, now I set the configs back I have ended up with 2 stale asdf123 windows nodes?
Screenshot 2024-03-14 at 19 34 00

mcdent commented

Ok, lets update this here, think I have 3 days left on my trial and have been unsuccessful to add more than 1 windows node at a time.
I tried adding a completely new windows 10 pro machine, installed the prometheus endpoint via the instructions, I can curl to the endpoint just fine. Added the new windows node to the windows.conf and vnode.conf and restarted net data.
Now I see the last working windows node as stale and my new one shows up? What am I doing wrong?

Screenshot 2024-03-17 at 16 57 59
Screenshot 2024-03-17 at 16 58 56

hugovalente-pm commented
hugovalente-pm commented

seems there was already feedback on the possible issue, shared on Discord here

Remove the extra jobs, keep just one on top
That's an array in yaml

@mcdent if this works please update the result here

mcdent commented

Ok, the issue here was that I had a separate job entry for each host. They should all have been under the one job section. e.g.
Screenshot 2024-03-18 at 19 18 46

My bad all sorted.

  • 👍1