SGudbrandsson/csharp-threadpool-starvation-example

This is an example of a threadpool starvation, awith a way to do healthchecks even when we are starved

languages	products	page_type	name	urlFragment	description
csharp	dotnet	sample	Diagnostic scenarios sample debug target	diagnostic-scenarios	A .NET Core sample with methods that trigger undesirable behaviors to diagnose.

To get threadpool starvation

Build and run the application locally

dotnet build
dotnet run

To monitor the threadpool starvation, use dotnet-counters in a separate terminal and monitor the ThreadPool Queue Length increase

dotnet-counters monitor -n DiagnosticScenarios

Then in a different terminal, run bombardier with 250 threads for 2 minutes

bombardier -c 250 -d 120s http://localhost:5000/api/diagscenario/tasksleepwait

To see if you can get health checks despite the threadpool starving, you should go to the health check endpoint

# Broken liveness check
curl -v http://localhost:5000/health/live
# Working liveness check .. hopefully
curl -v http://localhost:7000/live

Diagnostic scenarios sample debug target

The sample debug target is a simple webapi application. The sample triggers undesirable behaviors for the .NET Core diagnostics tutorials to diagnose.

Download the source

To get the code locally on your machine, click on '<> Code' in the top left corner of this page. This will take you to the root of the repo. Once at the root, clone the samples repo onto your local machine and navigate to samples/core/diagnostics/DiagnosticScenarios/.

Build and run the target

After downloading the source, you can easily run the webapi using:

dotnet build
dotnet run

Target methods

The target triggers undesirable behaviors when hitting specific URLs.

Deadlock

http://localhost:5000/api/diagscenario/deadlock

This method will cause the target to hang and accumulate many threads.

High CPU usage

http://localhost:5000/api/diagscenario/highcpu/{milliseconds}

The method will cause to target to heavily use the CPU for a duration specified by {milliseconds}.

Memory leak

http://localhost:5000/api/diagscenario/memleak/{kb}

This method will cause the target to leak memory (amount specified by {kb}).

Memory usage spike

http://localhost:5000/api/diagscenario/memspike/{seconds}

This method will cause intermittent memory spikes over the specified number of seconds. Memory will go from base line to spike and back to baseline several times.