Strategy for Meter & ActionScheduler when schedules cannot be served at requested intervals
slogen opened this issue · 1 comments
I am missing a strategy for what to do when ActionScheduler cannot meet the requested interval between invocations.
I am using the (very nice) Metrics.NET library to monitor thousands of Meters. Unfortunately that means that the Meters default Tick() refresh-rate at MeterMetric.TickInterval = 5s cannot be serviced.
As a result, the ThreadPool spins up more Threads, which actually makes the problem worse, by causing a cascade of Thread creations. Eventually upto ThreadPool.MaxThreads and consumes all OS resources changing between threads that update Meters :(
The below code demonstrates the issue (on my machine, milage may vary as it depends on slowness of Meter.Tick()-updates).
[Fact]
public void ActionScheduler_ShouldNotCascadeThreadCreationWhenManyMetersAreInstantiated()
{
var count = 50000;
var meters = Enumerable.Range(0, count)
.Select(i =>
{
var meter = Metric.Meter(string.Format("M{0}", i), Unit.Calls, TimeUnit.Seconds);
meter.Mark();
return meter;
})
.ToList();
// The mere *existence* of many meters makes the ActionScheduler schedule a large
// amount of tasks that cannot execute at the desired (5s) intervals on some machines
// 1. leading to the TaskPool extending the number of threads,
// 2. leading to even more scheduling time spent, and even worse execution (goto 1)
Func<int> threadCount = () => System.Diagnostics.Process.GetCurrentProcess().Threads.Count;
var threadCountInitial = threadCount();
Thread.Sleep(TimeSpan.FromSeconds(20));
var threadCountAfterSomeTime = threadCount();
Assert.True(threadCountAfterSomeTime <= threadCountInitial + 1);
}
I am using a workaround that extends the ActionScheduler with a TaskScheduler to run the task on, defaulting to TaskPool.Default (with exactly the old behaviour). But I can inject a thread-limited TaskScheduler via a static default to use for the Scheduler. This means that I can limit the amount of effort dedicated to Meter updates and live with the slow update-times that gives.
Since nobody else has this issue, and I stopped using Metrics.net, I am closing this issue as not-planned