Polygonal overlay renderer

[JujuAdams]

Requires triangulation algo

[xotmatrix]

Requires triangulation algo

Not necessarily. If you can draw it to a dedicated surface and then composite it, you can do it very cheaply. Draw it as a triangle fan using an XOR-like blending mode. As each triangle is drawn, any out-of-bounds pixels that might be formed from concave polygons erase themselves like magic once the polygon is complete. This is akin to the "crossing number" point-in-polyon test, where any pixel that is hit an even number of times is outside the polygon and does not appear in the resulting rasterization. It does mean that self-intersecting polygons can appear to have holes in them, which may or may not be a desired feature.

I use the technique here:
https://xotmatrix.github.io/demos/gm-niccc/index.html

I simplified this code a bit but I think it is still valid:

gpu_set_blendmode_ext(bm_inv_dest_alpha, bm_zero);
draw_set_color(palette[polyColor]);
surface_set_target(blit);
draw_clear_alpha(c_black, 0.0);
            
draw_primitive_begin(pr_trianglefan);
            
for (var i=0; i<polyVerts; i++) {
    var vertIndex = buffer_read_byte(scene);
    draw_vertex(vertX[vertIndex], vertY[vertIndex]);
}
            
draw_primitive_end();

surface_reset_target();
gpu_set_blendmode(bm_normal);
            
draw_surface_ext(blit, 0, 0, scaleB, scaleB, 0, c_white, 1.0);

[xotmatrix]

I just realized I had a much simpler demonstration that lets you draw polygons so you can see how it behaves.
https://xotmatrix.github.io/demos/concave-poly-trick/index.html