get backfill stats from sdiag

[jlec]

Hi

It would be great if the backfill stats from sdiag would be exported as well.

Best
Justin

[mtds]

Hi.

According to the man page, sdiag exports 14 different stats related to the backfill algorithm.

Three of them are already exported:

• (Backfill) Last cycle: Time in microseconds of last backfilling cycle.
• (Backfill) Mean cycle: Mean of backfilling scheduling cycles in microseconds since last reset.
• (Backfill) Depth mean: Mean of processed jobs during backfilling scheduling cycles since last reset.

Is there something in particular that you are thinking about? I am not so sure exporting all the backfilling stats indiscriminately will be useful.

Three of them caught my eye and could proably be added to the exporter in the near future:

• Total backfilled jobs (since last slurm start)
• Total backfilled jobs (since last stats cycle start)
• Total backfilled heterogeneous job components

Regards,
Matteo

[jlec]

Hi @mtds

Yes the three mentioned ones are definitely of interest. http://giovannitorres.me/graphing-sdiag-with-graphite.html has some interesting ideas.

Maybe you could at some flags to filter.

Best
Justin

[mtds]

There are some interesting ideas on that post, in particular the use of Pyslurm to query Slurm.

I am not sure it will be faster or more efficient: as you have seen from the source code, we are basically launching the Slurm utilities from the exporter itself and then grepping the output. The rest is 'just' Golang syntactic sugar to feed the data into Prometheus using the official APIs.

Anyway, Python interface aside, I guess it's possible to extend the exporter to include these addtional Backfill metrics about the scheduled jobs with its algorithm. Initially we were interested only in the performance of Slurm itself, since we did switch from time to time from the backfill to the fifo and viceversa whenever our Slurm scheduler is overloaded (we are still running a very old version here, so we are subjected to bugs), so it would not change so much our statistics to differentiate between the number of scheduled jobs depending on the algorithm used.

I will try to work on this feature this week but I cannot be sure at the moment if it will be ready soon or in the coming weeks.

Regards,
Matteo

[jlec]

I really like your work @mtds . We are currently exploring the feasibility to deploy it at scale.

[mtds]

Thanks @jlec but I must say that I share the merits with my colleague (@vpenso) which actually started the development and brought the first version into being. I added later the dashboard, adjusted the documentation and fixed a couple of bugs along the way.

Regarding deploying it at scale, I can provide you with some numbers: a single instance of this exporter is able to collect easily Slurm stats from 755 nodes (bare metal ones), on a cluster which sees roundabout 10K jobs running per week.

The (so to say) weak point is mainly related to Slurm itself than the exporter: querying frequently the master to extract the stats means adding a little bit more 'stress' to the Slurm scheduler and the reason why it's adviseable to restrict the 'scrape interval' to no less than 30 seconds (new versions
of Slurm may be able to sustain the load better but we are quite behind its release schedule).

Regards,
Matteo

[mtds]

This issue can be closed since the backfill stats were added back in July.