Help for configuring Mercure behind nginx as proxy-server

Question

Help for configuring Mercure behind nginx as proxy-server

Astro-Otter-Space opened this issue 9 months ago · 6 comments

Astro-Otter-Space commented 9 months ago

Hello,

I'm deploying Mercure hub on my server (VPS hosted by OVH) where an API REST (Symfony+FOSRestBundle) and a front app (VueJS 3, vue-cli) are installed yet, served with Nginx. API and FO have their own domains (https://api.exemple.space/ and https://www.exemple.space/). I've created a domain for Mercure (https://mercure.exemple.com/). Mercure (version 0.15) is installed with binary, not with docker.

I added an nginx host configuration for working as reverse-proxy (https://mercure.rocks/docs/hub/nginx):

# should i keep these lines commented or not ?
#server {
#  listen      80 http2;
#  server_name mercure.exemple.space;
#  return 301 https://mercure.exemple.space;
#}

server {
  listen      443 ssl http2;
  listen [::]:443 ssl http2;
  server_name mercure.exemple.space;

  ssl_certificate /etc/letsencrypt/live/mercure.exemple.space/fullchain.pem; 
  ssl_certificate_key /etc/letsencrypt/live/mercure.exemple.space/privkey.pem;
  
  location / {
    proxy_pass http://127.0.0.1:3000; # <-- is correct ?
    proxy_read_timeout 24h;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 300s;

    #proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  # Configuration des logs
  access_log  /var/log/nginx/mercure_access.log;
  error_log   /var/log/nginx/mercure_error.log;
}

i set public and private keys and set them as env variables in /etc/profile.d/mercure.sh

export MERCURE_PUBLISHER_JWT_KEY=$(cat /home/me/mercure/publisher.key.pub)
export MERCURE_PUBLISHER_JWT_ALG=RS256
export MERCURE_SUBSCRIBER_JWT_KEY=$(cat /home/me/mercure/subscriber.key.pub)
export MERCURE_PUBLISHER_JWT_ALG=RS256
export SERVER_NAME=localhost:3000 # <--  mercure.exemple.space or mercure.exemple.space:3000, which one is correct ?

I run with command MERCURE_PUBLISHER_JWT_KEY=$MERCURE_PUBLISHER_JWT_KEY MERCURE_SUBSCRIBER_JWT_KEY=$MERCURE_SUBSCRIBER_JWT_KEY /usr/bin/mercure run --config /home/stephane/mercure/Caddyfile

My Caddyfile :

{
	{$DEBUG:debug}

	{$CADDY_GLOBAL_OPTIONS}
	order mercure after encode
	
	# Ports
	http_port 3001 # Should i keep this line ?
	https_port 3000 # Should i keep this line ?
	{$GLOBAL_OPTIONS}
}

{$CADDY_EXTRA_CONFIG}

{$SERVER_NAME:localhost} {
        # tls line is commented because Mercure cant read keys, change rights/owner ?
	# tls /etc/letsencrypt/live/mercure.exemple.space/fullchain.pem /etc/letsencrypt/live/mercure.exemple.space/privkey.pem

	log {
		format filter {
			wrap console
			fields {
				uri query {
					replace authorization REDACTED
				}
			}
		}
	}

	encode zstd gzip

	mercure {
		# Transport to use (default to Bolt)
		transport_url {$MERCURE_TRANSPORT_URL:bolt://mercure.db}
		# Publisher JWT key
		publisher_jwt {env.MERCURE_PUBLISHER_JWT_KEY} {env.MERCURE_PUBLISHER_JWT_ALG}
		# Subscriber JWT key
		subscriber_jwt {env.MERCURE_SUBSCRIBER_JWT_KEY} {env.MERCURE_SUBSCRIBER_JWT_ALG}
		# Extra directives
		# CORS
		cors_origins https://www.exemple.space https://exemple.local:8080 https://localhost:8080 <-- maybe some error here ?
		publish_origins *
		anonymous
		subscriptions
		{$MERCURE_EXTRA_DIRECTIVES}
	}

	{$CADDY_SERVER_EXTRA_DIRECTIVES}

	respond /healthz 200
	respond "Not Found" 404
}

In log i have :

2024/01/03 11:07:15.053	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "mercure.exemple.space", "issuer": "acme.zerossl.com-v2-DV90", "error": "[mercure.astro-otter.space] creating new order: attempt 1: https://acme.zerossl.com/v2/DV90/newOrder: performing request: Post \"https://acme.zerossl.com/v2/DV90/newOrder\": context deadline exceeded (Client.Timeout exceeded while awaiting headers) (ca=https://acme.zerossl.com/v2/DV90)"}
2024/01/03 11:07:15.053	DEBUG	events	event	{"name": "cert_failed", "id": "533a237d-c07a-4cd4-a12d-dfe931a9c5aa", "origin": "tls", "data": {"error":{},"identifier":"mercure.exemple.space","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}

2024/01/03 11:07:15.053	ERROR	tls.obtain	will retry	{"error": "[mercure.exemple.space] Obtain: [mercure.exemple.space] creating new order: attempt 1: https://acme.zerossl.com/v2/DV90/newOrder: performing request: Post \"https://acme.zerossl.com/v2/DV90/newOrder\": context deadline exceeded (Client.Timeout exceeded while awaiting headers) (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 1, "retrying_in": 60, "elapsed": 127.939511407, "max_duration": 2592000}

If i run curl from my local env :

$ curl -X GET https://mercure.exemple.space/.well-known/mercure
Client sent an HTTP request to an HTTPS server.
$ curl -X GET https://mercure.exemple.space:3000/.well-known/mercure
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

In my JS app (from local or prod env), i have CORS error with this code :

  const hubUrl = new URL('https://mercure.exemple.space');
  const domain = 'https://api.exemple.space'
  const topic = 'notifications/all';
  hubUrl.searchParams.append('topic', `${domain}/${topic}`);
  const eventSource = new EventSource(hubUrl.toString(), {withCredentials: true});
  eventSource.onmessage = (event) => {
    console.log(event.data);
  }

I need help for wrtting good configuration for Nginx and Caddyfile. I'll fix CORS errors later.
Thank you for help :)

Answer 1 · 2024-01-03T16:38:07.000Z

Try to disable TLS on Mercure. For instance, set SERVER_NAME to http://localhost (notice the http:// prefix).

Answer 2 · 2024-01-04T08:15:38.000Z

ok I'll do that. I changed SERVER_NAME value in /etc/environment

$ echo $SERVER_NAME 
http://localhost

and changed Caddyfile like this :

{
	{$DEBUG:debug}

	{$CADDY_GLOBAL_OPTIONS}
	order mercure after encode
	
	# Ports
	http_port 3000
	auto_https off
	{$GLOBAL_OPTIONS}
}

{$CADDY_EXTRA_CONFIG}

{$SERVER_NAME:localhost} {
	# tls /etc/letsencrypt/live/mercure.exemple.space/fullchain.pem /etc/letsencrypt/live/mercure.exemple.space/privkey.pem

line tls ... commented and disabled like i saw here or here
Is it enough ?

I have no more error in console (a good point ^^).

$ MERCURE_PUBLISHER_JWT_KEY=$MERCURE_PUBLISHER_JWT_KEY \
MERCURE_SUBSCRIBER_JWT_KEY=$MERCURE_SUBSCRIBER_JWT_KEY \
SERVER_NAME=$SERVER_NAME \
DEBUG=debug \
/usr/bin/mercure run --config Caddyfile
2024/01/04 08:32:28.347	INFO	using provided configuration	{"config_file": "Caddyfile", "config_adapter": ""}
2024/01/04 08:32:28.350	WARN	Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies	{"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2024/01/04 08:32:28.351	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/01/04 08:32:28.352	WARN	http.auto_https	automatic HTTPS is completely disabled for server	{"server_name": "srv0"}
2024/01/04 08:32:28.352	DEBUG	http.auto_https	adjusted config	{"tls": {"automation":{"policies":[{}]}}, "http": {"http_port":3000,"servers":{"srv0":{"listen":[":3000"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"set":{"Content-Type":["text/html; charset=utf-8"]}}}],"match":[{"path":["/"]}]},{"handle":[{"encodings":{"gzip":{},"zstd":{}},"handler":"encode","prefer":["zstd","gzip"]},{"anonymous":true,"cors_origins":["https://news.exemple.space","https://exemple.local:8080","https://localhost:8080"],"handler":"mercure","publish_origins":["*"],"publisher_jwt":{"alg":"{env.MERCURE_PUBLISHER_JWT_ALG}","key":"{env.MERCURE_PUBLISHER_JWT_KEY}"},"subscriber_jwt":{"alg":"{env.MERCURE_SUBSCRIBER_JWT_ALG}","key":"{env.MERCURE_SUBSCRIBER_JWT_KEY}"},"subscriptions":true,"transport_url":"bolt://mercure.db"}]},{"handle":[{"handler":"static_response","status_code":200}],"match":[{"path":["/healthz"]}]},{"handle":[{"body":"\u003c!DOCTYPE html\u003e\n\t\u003chtml lang=en\u003e\n\t\u003cmeta charset=\"utf-8\"\u003e\n\t\u003cmeta name=\"robots\" content=\"noindex\"\u003e\n\t\u003ctitle\u003eWelcome to Mercure\u003c/title\u003e\n\t\u003ch1\u003eWelcome to Mercure\u003c/h1\u003e\n\t\u003cp\u003eThe URL of your hub is \u003ccode\u003e/.well-known/mercure\u003c/code\u003e.\n\tRead the documentation on \u003ca href=\"https://mercure.rocks\"\u003eMercure.rocks, real-time apps made easy\u003c/a\u003e.","handler":"static_response"}],"match":[{"path":["/"]}]},{"handle":[{"body":"Not Found","handler":"static_response","status_code":404}]}]}],"terminal":true}],"automatic_https":{"disable":true},"logs":{"logger_names":{"localhost":"log0"}}}}}}
2024/01/04 08:32:28.354	DEBUG	http	starting server loop	{"address": "[::]:3000", "tls": false, "http3": false}
2024/01/04 08:32:28.355	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/01/04 08:32:28.355	INFO	autosaved config (load with --resume flag)	{"file": "/home/stephane/.config/caddy/autosave.json"}
2024/01/04 08:32:28.355	INFO	serving initial configuration
2024/01/04 08:32:28.357	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc0004c0000"}
2024/01/04 08:32:28.359	WARN	tls	storage cleaning happened too recently; skipping for now	{"storage": "FileStorage:/home/stephane/.local/share/caddy", "instance": "359fed6e-f64a-4f98-a8cb-e2b7bb24d40c", "try_again": "2024/01/05 08:32:28.359", "try_again_in": 86399.999999583}
2024/01/04 08:32:28.359	INFO	tls	finished cleaning storage units

And i saw my request in log :

2024/01/04 08:37:37.796	INFO	http.log.access	handled request	{"request": {"remote_ip": "127.0.0.1", "remote_port": "36484", "client_ip": "127.0.0.1", "proto": "HTTP/1.1", "method": "GET", "host": "127.0.0.1:3000", "uri": "/.well-known/mercure?topic=https%3A%2F%2Fapi.exemple.space%2Fnotifications%2Fdso", "headers": {"Sec-Fetch-Site": ["cross-site"], "Sec-Fetch-Mode": ["cors"], "Referer": ["https://localhost:8080/"], "Accept-Encoding": ["gzip, deflate, br"], "X-Forwarded-For": ["185.101.209.57"], "Cache-Control": ["no-cache"], "Sec-Ch-Ua-Mobile": ["?0"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"], "Accept-Language": ["fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7"], "Sec-Ch-Ua-Platform": ["\"Linux\""], "Sec-Fetch-Dest": ["empty"], "X-Forwarded-Host": ["mercure.exemple.space"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua": ["\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Google Chrome\";v=\"120\""], "Accept": ["text/event-stream"], "Origin": ["https://localhost:8080"]}}, "bytes_read": 0, "user_id": "", "duration": 0.000004338, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"]}}

If i curl from local :

curl -k -I -X GET https://mercure.exemple.space/.well-known/mercure
HTTP/2 200 
server: nginx
date: Thu, 04 Jan 2024 08:33:39 GMT
content-length: 0

JS side, i got 200 too but CORS error

Access to resource at 'https://mercure.exemple.space/.well-known/mercure?topic=https%3A%2F%2Fapi.astro-otter.space%2Fnotifications%2Fall' from origin 'https://localhost:8080' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Credentials' header in the response is '' which must be 'true' when the request's credentials mode is 'include'.
mercure:1 
        
        
       GET https://mercure.exemple.space/.well-known/mercure?topic=https%3A%2F%2Fapi.exemple.space%2Fnotifications%2Fall net::ERR_FAILED 200 (OK)
home:1 EventSource's response has a MIME type ("text/plain") that is not "text/event-stream". Aborting the connection.

It seems it's on a good way ^^

Answer 3 · 2024-01-05T09:17:29.000Z

I added in my nginx vhost these lines :

  location / {
    if ($http_origin ~ '^https?://(localhost(:[0-9]+)?|[^/]*\.exemple\.space)') {
        set $cors "true";
    }    

    if ($cors = "true") {
      add_header 'Access-Control-Allow-Origin' '$http_origin' always;    
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
      add_header 'Access-Control-Allow-Methods' 'GET,POST,OPTIONS';
      add_header "Content-Type" "text/event-stream";
    }	    

    proxy_pass http://127.0.0.1:3000;
    proxy_read_timeout 24h;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 300s;

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

And no more CORS errors.

Answer 4 · 2024-01-05T20:18:18.000Z

This shouldn't be necessary but thanks for the workaround! I'll take a look.

Answer 5 · 2024-01-08T09:17:47.000Z

Without Access-Control-Allow-Origin, Access-Control-Allow-Credentials and Content-Type, i have CORS error JS-side.

With this nginx configuration, what should be values of cors_origins and publish_origins in my Caddyfile ?
You told me to set http://localhost for env variable SERVER_NAME. Should i set proxy_pass http://localhost:3000; instead of proxy_pass http://127.0.0.1:3000; ?

UPDATE 09/01/2024 :
i've changed proxy_pass http://127.0.0.1:3000; into proxy_pass http://localhost:3000; in nginx.
Now when i do curl http://localhost:3000 (on server) and curl https://mercure.astro-otter.space in local, i have the response from Caddy same as defined in Caddyfile,good point (youpi)

If i'm publishing from my Symfony controller or POST curl request:

i got exception HTTP/1.1 401 Unauthorized returned for "http://localhost:3000/.well-known/mercure"." with symfony
in log: Topic selectors not matched, not provided or authorization error {"remote_addr": "127.0.0.1:59610", "error": "unable to parse JWT: signature is invalid"}

Answer 6 · 2024-01-22T08:38:21.000Z

Finally i found a way to make it working correctly. I resume :
First, i'm working with binary, not docker image.
Env file /etc/environment :

MERCURE_PUBLISHER_JWT_KEY="mySecretKey"
MERCURE_SUBSCRIBER_JWT_KEY="mySecretKey"
# Mercure URL
SERVER_NAME=http://localhost
MERCURE_PUBLIC_URL=https://mercure.exemple.space/.well-known/mercure

My nginx vhost :

server {
  listen      443 ssl http2;
  listen [::]:443 ssl http2;
  server_name mercure.exemple.space;

  ssl_certificate /etc/letsencrypt/live/mercure.exemple.space/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/mercure.exemple.space/privkey.pem; # managed by Certbot
  
  location / {
    if ($http_origin ~ '^https?://(localhost(:[0-9]+)?|127.0.0.1(:[0-9]+)?|[^/]*\.exemple\.space)') {
        set $cors "true";
    }    

    if ($cors = "true") {
      add_header 'Access-Control-Allow-Origin' '$http_origin' always;    
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Allow-Headers' 'Authorization,Accept,Origin,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
      add_header 'Access-Control-Allow-Methods' 'GET,POST,OPTIONS';
      add_header 'Content-Type' 'text/event-stream';
    }	    

    proxy_pass http://localhost:3000;
    proxy_read_timeout 24h;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 300s;

    #proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  # Configuration des logs
  access_log  /var/log/nginx/mercure_access.log;
  error_log   /var/log/nginx/mercure_error.log;

I create a systemd file for running mercure as service /etc/systemd/system/mercure.service :

[Unit]
Description=Mercure.Rocks service
After=network.target
StartLimitBurst=5
StartLimitIntervalSec=33

[Service]
Type=simple
WorkingDirectory=/tmp
EnvironmentFile=-/etc/environment
ExecStart=/usr/bin/bash -c "MERCURE_PUBLISHER_JWT_KEY=$MERCURE_PUBLISHER_JWT_KEY MERCURE_SUBSCRIBER_JWT_KEY=$MERCURE_SUBSCRIBER_JWT_KEY SERVER_NAME=$SERVER_NAME /usr/bin/mercure run --config /var/www/mercure/Caddyfile"
StandardOutput=file:/var/log/nginx/mercure.log
StandardError=file:/var/log/nginx/mercure.log
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

In /var/www/mercure, here's my caddyfile :

{
	{$DEBUG:debug}

	{$CADDY_GLOBAL_OPTIONS}
	order mercure after encode
	
	# Ports
	http_port 3000
	auto_https off
	{$GLOBAL_OPTIONS}
}

{$CADDY_EXTRA_CONFIG}

{$SERVER_NAME:localhost} {
	log {
		output file /var/log/caddy/mercure.log {
			roll true
			roll_size_mb 10
		        roll_keep 5
		}
		format filter {
			wrap console
			fields {
				uri query {
					replace authorization REDACTED
				}
			}
		}
		level INFO
	}

	encode zstd gzip

	mercure {
		# Transport to use (default to Bolt)
		transport_url {$MERCURE_TRANSPORT_URL:bolt://mercure.db}
		# Publisher JWT key
		publisher_jwt {env.MERCURE_PUBLISHER_JWT_KEY} {env.MERCURE_PUBLISHER_JWT_ALG}
		# Subscriber JWT key
		subscriber_jwt {env.MERCURE_SUBSCRIBER_JWT_KEY} {env.MERCURE_SUBSCRIBER_JWT_ALG}
		# Extra directives
		# CORS
		cors_origins /** add here URL **/
		publish_origins *
		anonymous
		subscriptions
		{$MERCURE_EXTRA_DIRECTIVES}
	}

	{$CADDY_SERVER_EXTRA_DIRECTIVES}

	header / Content-Type "text/html; charset=utf-8"
	respond / `<!DOCTYPE html>
	<html lang=en>
	<meta charset="utf-8">
	<meta name="robots" content="noindex">
	<title>Welcome to Mercure</title>
	<h1>Welcome to Mercure</h1>
	<p>The URL of your hub is <code>/.well-known/mercure</code>.
	Read the documentation on <a href="https://mercure.rocks">Mercure.rocks, real-time apps made easy</a>.`

	respond /healthz 200

Code-side

My backend is a symfony (upgrade to API Platform ne day ^^).
/path/to/backend/c/.env

MERCURE_URL=http://localhost:3000/.well-known/mercure
MERCURE_PUBLIC_URL=https://mercure.exemple.space/.well-known/mercure # use variable from /etc/environment ?
MERCURE_JWT_SECRET="PutYoutJWTHere"

/path/to/backend/config/packages/mercure.yaml

mercure:
  hubs:
    default:
      url: '%env(MERCURE_URL)%'
      public_url: '%env(MERCURE_PUBLIC_URL)%'
      jwt: 
        value: '%env(MERCURE_JWT_SECRET)%'
        publish: '*'

Front-side is a VueJS application, using EventSource, nothing exotic.

I think i need some adjustments but with all these it's working :).