Enqueue doesn't interrupt event loop run (race condition)

[ysbaddaden]

I was fixing MT codegen in Crystal (see #14748) and noticed it had a pitfall (starving threads because of blocked fiber) that could be solved by using execution contexts.

I thus went to try it out, and I quickly identified an issue in the execution contexts: the process eventually comes to a halt 🤨

The DEFAULT thread (ST) whose main thread is sending to a channel is waiting on the event loop, while the CODEGEN threads wait to read on an empty channel (with no pending sender). A debug sessions showed that the main fiber was properly enqueued in the global queue of the DEFAULT thread, but the thread is still waiting on the event loop.

That MUSN'T happen. There's a race condition that fails to interrupt or fails to prevent a thread from blocking on the event loop.

[ysbaddaden]

Now that I know, it's obvious that there are a bunch of races between stop spinning & start blocking:

Scenario A:

1. thread A: checks global queue (empty)
2. thread B: enqueues Fiber to global queue
3. thread B: checks for spinning threads, finds one => skips wakeup
4. thread A: stops spinning
5. thread A: starts blocking
6. thread A: waits on the EventLoop (oops)

Scenario B:

1. thread A: checks global queue (empty)
2. thread B: enqueues Fiber to global queue
3. thread A: stops spinning
4. thread B: checks for spinning threads: none
5. thread B: checks for blocked threads: none => skips wakeup
6. thread A: starts blocking
7. thread A: waits on the EventLoop (oops)

It's so obvious that it's embarrassing 😩