Reconsider fixed gpu spike buffer size

[schmitts]

Running on GPU I've encountered RuntimeError: gpu/threshold_watcher: gpu spike buffer overflow, which is triggered by:

        // TODO: allocates enough space for 10 spikes per watch.
        // A more robust approach might be needed to avoid overflows.
        stack_(10*size(), ctx.gpu)
        
        ...
        
        if (stack_.overflow()) {
            throw arbor_internal_error("gpu/threshold_watcher: gpu spike buffer overflow");
        }

in arbor/backends/gpu/threshold_watcher.hpp.

It would be nice if a magic number for the buffer would not be necessary.

[brenthuisman]

Can I ask how you ran into this?

[schmitts]

Running my Graupner-Brunel example (@boeschf is polishing it) on GPU (RTX 3090).

[thorstenhater]

Good to bring this up. There's some places[1] where we could use a per-epoch/-step scratch pad, this might be one of those. A simple bump-allocator will do the trick.

[1] One instance I readily remember is the reduction step, where we allocate a single double result.