Regex path matching isn't working.

Question

Regex path matching isn't working.

KieronWiltshire opened this issue 2 years ago · 10 comments

KieronWiltshire commented 2 years ago

Preflight checklist

I could not find a solution in the existing issues, docs, nor discussions.
I agree to follow this project's Code of Conduct.
I have read and am following this repository's Contribution Guidelines.
This issue affects my Ory Network project.
I have joined the Ory Community Slack.
I am signed up to the Ory Security Patch Newsletter.

Describe the bug

I have two of these matching rules:

This is for kratos

"url": "<(http|https):\\/\\/([\\w-]+(\\.[\\w-]+)*|[\\d.]+)(:\\d+)?\\/\\.ory\\/kratos\\/public(\\/.*|(\\?.*)?)?$>"

This is for my custom service

"url": "<(http|https):\\/\\/([\\w-]+(\\.[\\w-]+)*|[\\d.]+)(:\\d+)?\\/api\\/account(\\/.*|(\\?.*)?)?$>"

Both are configured identically, I've even tried swapping them around and one thing always remains constant. The .ory regex is the only one that works. If i switch /api/account to /.api/account that appears to work. So I thought, okay it must be an issue with my regex, but no when I swap the match paths to literal strings like http://localhost:4455/api/account and go to this URL in the browser, I get the exact same error

{"error":{"code":404,"status":"Not Found","message":"Requested url does not match any rules"}}

Reproducing the bug

Honestly no idea, but I'm using the latest ory oathkeeper docker image and my config files are as follows:

docker-compose.yml

version: "3.9"
services:
  timescaledb:
    image: timescale/timescaledb:latest-pg14
    container_name: postgres
    restart: always
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    volumes:
      - ./.docker-compose/data/timescaledb:/var/lib/postgresql/data
    networks:
      - dev
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres" ]
      interval: 5s
      timeout: 5s
      retries: 5

  redis:
    image: redis:alpine
    container_name: redis
    restart: always
    ports:
      - 6379:6379
    volumes:
      - ./.docker-compose/data/redis:/data
    networks:
      - dev

  pgadmin:
    image: dpage/pgadmin4:latest
    container_name: pgadmin
    restart: always
    depends_on:
      - timescaledb
    ports:
      - 8080:80
    environment:
      - PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
      - PGADMIN_CONFIG_SERVER_MODE=False
      - PGADMIN_DEFAULT_EMAIL=admin@admin.org
      - PGADMIN_DEFAULT_PASSWORD=admin
      - PGADMIN_LISTEN_PORT=80
    volumes:
      - ./.docker-compose/data/pgadmin:/var/lib/pgadmin
    networks:
      - dev

  redis-commander:
    image: rediscommander/redis-commander:latest
    container_name: redis-commander
    restart: always
    depends_on:
      - redis
    ports:
      - 8081:8081
    environment:
      - REDIS_HOSTS=local:redis:6379:1
    networks:
      - dev

  mailslurper:
    image: oryd/mailslurper:latest-smtps
    container_name: mailslurper
    ports:
      - "4436:4436"
      - "4437:4437"
    networks:
      - dev

  hydra-migrate:
    image: oryd/hydra:latest
    container_name: hydra-migrate
    restart: on-failure
    depends_on:
      - timescaledb
    command:
      migrate -c /etc/config/hydra/hydra.yml sql -e --yes
    environment:
      DSN: postgres://postgres:password@timescaledb:5432/ory-hydra?sslmode=disable&max_conns=20&max_idle_conns=4
    volumes:
      -
        type: bind
        source: ./.docker-compose/configs/ory/hydra
        target: /etc/config/hydra
    networks:
      - dev

  kratos-migrate:
    image: oryd/kratos:latest
    container_name: kratos-migrate
    restart: on-failure
    depends_on:
      - timescaledb
    command:
      migrate -c /etc/config/kratos/kratos.yml sql -e --yes
    environment:
      DSN: postgres://postgres:password@timescaledb:5432/ory-kratos?sslmode=disable&max_conns=20&max_idle_conns=4
      LOG_LEVEL: debug
    volumes:
      -
        type: bind
        source: ./.docker-compose/configs/ory/kratos
        target: /etc/config/kratos
    networks:
      - dev

  hydra:
    image: oryd/hydra:latest
    container_name: hydra
    restart: unless-stopped
    depends_on:
      - hydra-migrate
    command:
      serve -c /etc/config/hydra/hydra.yml all --dev
    environment:
      DSN: postgres://postgres:password@timescaledb:5432/ory-hydra?sslmode=disable&max_conns=20&max_idle_conns=4
      OIDC_SUBJECT_IDENTIFIERS_SUPPORTED_TYPES: public
      LOG_LEVEL: trace
    volumes:
      -
        type: bind
        source: ./.docker-compose/configs/ory/hydra
        target: /etc/config/hydra
    ports:
      - 4444:4444
      - 4445:4445
      - 5555:5555
    networks:
      - dev

  kratos:
    image: oryd/kratos:latest
    container_name: kratos
    restart: unless-stopped
    depends_on:
      - kratos-migrate
    command: serve -c /etc/config/kratos/kratos.yml --dev
    environment:
      DSN: postgres://postgres:password@timescaledb:5432/ory-kratos?sslmode=disable&max_conns=20&max_idle_conns=4
      LOG_LEVEL: trace
    volumes:
      -
        type: bind
        source: ./.docker-compose/configs/ory/kratos
        target: /etc/config/kratos
    ports:
      - 4433:4433
      - 4434:4434
    networks:
      - dev

  oathkeeper:
    image: oryd/oathkeeper:latest
    container_name: oathkeeper
    restart: unless-stopped
    depends_on:
      - timescaledb
    command: serve --config=/etc/config/oathkeeper/oathkeeper.yml
    environment:
      - LOG_LEVEL=debug
      - LOG_LEAK_SENSITIVE_VALUES="true"
      - TRACING_PROVIDER=jaeger
      - TRACING_SERVICE_NAME=Oathkeeper
      - TRACING_PROVIDERS_JAEGER_SAMPLING_SERVER_URL=http://jaeger:5778/sampling
      - TRACING_PROVIDERS_JAEGER_LOCAL_AGENT_ADDRESS=jaeger:6831
      - TRACING_PROVIDERS_JAEGER_SAMPLING_TYPE=const
      - TRACING_PROVIDERS_JAEGER_SAMPLING_VALUE=1
    volumes:
      - type: bind
        source: ./.docker-compose/configs/ory/oathkeeper
        target: /etc/config/oathkeeper
    ports:
      - 4455:4455
      - 4456:4456
    networks:
      - dev

  jaeger:
    image: jaegertracing/all-in-one:1.22
    container_name: jaeger
    ports:
      - 16686:16686
#     These are ports for collecting, sampling, agents, ...
#      - "5775:5775/udp"
#      - "6831:6831/udp"
#      - "6832:6832/udp"
#      - "5778:5778"
#      - "14268:14268"
#      - "9411:9411"
    networks:
      - dev

  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
    command: [ "--config=/etc/otel-collector-config.yaml" ]
    volumes:
      - ./.docker-compose/configs/otel-collector/otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"
      - "4318:4318"
      - "1234:1234"
    networks:
      - dev

networks:
  dev:
    driver: bridge

Here is my .docker-compose/configs directory with my config files inside
configs.zip

Relevant log output

I'm not sure how to access the logs, there isn't enough documentation around this or in fact a majority of things.

Relevant configuration

No response

Version

latest

On which operating system are you observing this issue?

Windows

In which environment are you deploying?

Docker Compose

Additional Context

Please help, I'm in the slack, I follow the issues on github, but the lack of support and relevant documentation for this and other ory projects make it really difficult for anyone to adopt. Which is really annoying because I like the idea behind the way this works.

Answer 1 · 2023-04-16T21:22:00.000Z

Here is additional info:

When trying to use full regex like the one provided below, the response is 404 "does not match any rules."

[
    {
      "id": "account:protected",
      "upstream": {
        "preserve_host": true,
        "url": "http://host.docker.internal:80",
        "strip_path": "/api/account"
      },
      "match": {
        "url": "<(http|https):\/\/([w-]+(.[w-]+)*|[d.]+)(:d+)?\/api\/account(/.*|\\?.*)?>",
        "methods": ["GET", "POST" ,"PUT", "DELETE", "PATCH"]
      },
      "authenticators": [{
        "handler": "cookie_session"
      }, {
        "handler": "oauth2_introspection"
      }, {
        "handler": "anonymous"
      }],
      "authorizer": {
        "handler": "allow"
      },
      "mutators": [{
        "handler": "id_token"
      }],
      "errors": [{
        "handler": "json"
      }]
    }
  ]

However, changing the regex rule to something like http://localhost:4455<\/api\/account(/.*|\\?.*)?> sort of works... well Oathkeeper says it works, but I still get the same response. Here are the logs:

2023-04-16 22:19:17 time=2023-04-16T21:19:17Z level=info msg=started handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 accept-encoding:gzip, deflate, br accept-language:en-GB,en-US;q=0.9,en;q=0.8 cache-control:max-age=0 connection:keep-alive sec-ch-ua:"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99" sec-ch-ua-mobile:?0 sec-ch-ua-platform:"Windows" sec-fetch-dest:document sec-fetch-mode:navigate sec-fetch-site:none sec-fetch-user:?1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36] host:localhost:4455 method:GET path:/api/account query:<nil> remote:172.26.0.1:36986 scheme:http]
2023-04-16 22:19:18 time=2023-04-16T21:19:18Z level=info msg=started handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 accept-encoding:gzip, deflate, br accept-language:en-GB,en-US;q=0.9,en;q=0.8 authorization:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". cache-control:max-age=0 connection:close content-length:0 sec-ch-ua:"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99" sec-ch-ua-mobile:?0 sec-ch-ua-platform:"Windows" sec-fetch-dest:document sec-fetch-mode:navigate sec-fetch-site:none sec-fetch-user:?1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 x-forwarded-for:172.26.0.1] host:localhost:4455 method:GET path:/ query:<nil> remote:172.26.0.1:36990 scheme:http]
2023-04-16 22:19:18 time=2023-04-16T21:19:18Z level=warning msg=Access request denied audience=application error=map[debug: message:Requested url does not match any rules reason: status:Not Found status_code:404] granted=false http_host=localhost:4455 http_method=GET http_url=http://localhost:4455/ http_user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 service_name=ORY Oathkeeper service_version=v0.40.2
2023-04-16 22:19:18 time=2023-04-16T21:19:18Z level=info msg=An error occurred while handling a request audience=application error=map[debug: message:Requested url does not match any rules reason: status:Not Found status_code:404] http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 accept-encoding:gzip, deflate, br accept-language:en-GB,en-US;q=0.9,en;q=0.8 authorization:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". cache-control:max-age=0 content-length:0 sec-ch-ua:"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99" sec-ch-ua-mobile:?0 sec-ch-ua-platform:"Windows" sec-fetch-dest:document sec-fetch-mode:navigate sec-fetch-site:none sec-fetch-user:?1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 x-forwarded-for:172.26.0.1, 172.26.0.1] host:localhost:4455 method:GET path:/ query:<nil> remote:172.26.0.1:36990 scheme:http] http_response=map[status_code:404] service_name=ORY Oathkeeper service_version=v0.40.2
2023-04-16 22:19:18 time=2023-04-16T21:19:18Z level=info msg=completed handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 accept-encoding:gzip, deflate, br accept-language:en-GB,en-US;q=0.9,en;q=0.8 authorization:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". cache-control:max-age=0 connection:close content-length:0 sec-ch-ua:"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99" sec-ch-ua-mobile:?0 sec-ch-ua-platform:"Windows" sec-fetch-dest:document sec-fetch-mode:navigate sec-fetch-site:none sec-fetch-user:?1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 x-forwarded-for:172.26.0.1] host:localhost:4455 method:GET path:/ query:<nil> remote:172.26.0.1:36990 scheme:http] http_response=map[headers:map[content-type:application/json] size:95 status:404 text_status:Not Found took:992.413µs]
2023-04-16 22:19:18 time=2023-04-16T21:19:18Z level=info msg=Access request granted audience=application granted=true http_host=localhost:4455 http_method=GET http_url=http://host.docker.internal:80 http_user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 service_name=ORY Oathkeeper service_version=v0.40.2 subject=guest
2023-04-16 22:19:18 time=2023-04-16T21:19:18Z level=info msg=completed handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 accept-encoding:gzip, deflate, br accept-language:en-GB,en-US;q=0.9,en;q=0.8 cache-control:max-age=0 connection:keep-alive sec-ch-ua:"Chromium";v="112", "Google Chrome";v="112", "Not:A-Brand";v="99" sec-ch-ua-mobile:?0 sec-ch-ua-platform:"Windows" sec-fetch-dest:document sec-fetch-mode:navigate sec-fetch-site:none sec-fetch-user:?1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36] host:localhost:4455 method:GET path:/api/account query:<nil> remote:172.26.0.1:36986 scheme:http] http_response=map[headers:map[content-length:95 content-type:application/json date:Sun, 16 Apr 2023 21:19:18 GMT] size:95 status:404 text_status:Not Found took:339.500784ms]

Here is the response I get for both instances:

{"error":{"code":404,"status":"Not Found","message":"Requested url does not match any rules"}}

Answer 2 · 2023-04-17T15:24:15.000Z

I'm not sure I get the point of your regex, to be honest. Have you tried debugging with https://regex101.com/?

Answer 3 · 2023-04-18T01:57:11.000Z

Well the regex is to match http or https protocol scheme, with any url or IP on any port with the specified path. Yes I’ve tested the regex rules and they both work except for in Ory Oathkeeper. I always get a 404.

Answer 4 · 2023-04-18T08:06:40.000Z

Are you choosing Golang in the list of languages? Golang's regex is limited

Answer 5 · 2023-04-18T08:37:23.000Z

This is the regex you're trying, right?

(http|https)://([w-]+(.[w-]+)*|[d.]+)(:d+)?/api/account(/.*|\?.*)?

And you want it to match this path?

https://localhost:12345/api/accounts

Check out the interactive debugger on this page to see why it doesn't match: https://regex101.com/r/KPw8ca/1

Answer 6 · 2023-04-18T08:39:32.000Z

Specifically, I don't think [w-]+ does what you think it does.

Answer 7 · 2023-04-18T22:41:08.000Z

@alnr @aeneasr So I see you guys are saying that golang's regex is limited, but the rule for /.ory/public/kratos works fine?

Answer 8 · 2023-04-19T08:03:27.000Z

That is because the regex doesn't match the pattern, not because Oathkeeper is doing something weird. If the regular expression is correct (yours isn't https://regex101.com/r/KPw8ca/1) then Oathkeeper will also match. TLDR You need to write the correct regex ;)

Example of a correct regex: https://regex101.com/r/rbUdxO/1

Closing, because this isn't an Oathkeeper technical issue, but rather an incorrectly written regular expression.

Answer 9 · 2023-04-20T00:52:12.000Z

Sorry, that regex won't work for ip addresses too, not that it matters that much, but would it be a good idea to create matching rules for protocol + hostname which are separate to sort of routes? It might not be, I'm just curious, because for me to create a regex that would match protocol, ip address/hostname + route would be pretty big in size and won't really be eloquent in terms of config readability.

Answer 10 · 2023-04-20T08:14:15.000Z

In the end it's just a matter of writing the correct regular expression. There are for sure some good examples out there on Google what a regex for domain|ip:port looks like for Go's regexp implementation