Seperate frankenphp(caddy) from ingress for local development

[usu]

After merging #5123, I'd propose that we separate the path-forwarding functionality (ingress) from frankenphp.

Combining this has a few disadvantages:

• The need to maintain and test 2 different Caddyfiles (1 for local dev and 1 for production)
• The need for some extra code in Varnish vcl to detect different cases between dev and prod (see for example https://github.com/ecamp/ecamp3/blob/devel/api/docker/varnish/vcl/default.vcl#L24)

The 2nd point comes from the fact that the processing order is different between dev and prod:

• on prod: ingress --> varnish --> api
• od dev: varnish --> caddy (as ingress) --> api

Adding a separate ingress on local dev would make the two more similar and hence less error prone.

As ingress we could use:

• Caddy: similar config as we already have today
• Nginx: if we want to stay as similar as possible to the deployment

[manuelmeister]

Core Meeting Decision

We vote for NGINX. We want the setup to be similar to the deployment.