[Bug]: Sync on self-hosted server returns 413

Question

[Bug]: Sync on self-hosted server returns 413

Mnikley opened this issue a month ago · 12 comments

What did you expect to happen?

Sync working without returning an error

What happened?

I switched from the official api.atuin.sh server to my own, following the documentation. I run a traefik server and atuin with a postgres db in docker-compose. I increased the max_history_length to 16384, the rest is set to the defaults.

Registering and logging in works, and I can see changes in the database, but when trying to sync i get:

❯ atuin sync
Error: There was an error with the atuin sync service: Status 413.
If the problem persists, contact the host

Location:
    /home/runner/work/atuin/atuin/atuin-client/src/api_client.rs:171:9

Is there something else i can try to import my ~13k entries? My longest history entry is 1245 characters long:

❯ atuin history list | awk '{ print length, $0 }' | sort -n -s -r | cut -d" " -f2- | less -S | head -n 1 | wc -c
1245

Atuin doctor output

Atuin Doctor
Checking for diagnostics


Please include the output below with any bug reports or issues

atuin:
  version: 18.2.0
  sync:
    cloud: false
    records: false
    auto_sync: true
    last_sync: 2024-04-27 15:42:54.424677903 +00:00:00
shell:
  name: zsh
  default: unknown
  plugins:
  - atuin
system:
  os: Debian GNU/Linux
  arch: x86_64
  version: '12'
  disks:
  - name: /dev/mapper/glorypc--vg-root
    filesystem: ext4
  - name: /dev/nvme0n1p2
    filesystem: ext2
  - name: /dev/nvme0n1p1
    filesystem: vfat

Code of Conduct

I agree to follow this project's Code of Conduct

Answer 1 · 2024-04-27T17:45:02.000Z

Does increasing max_record_size fix it for you?

Answer 2 · 2024-04-27T19:12:46.000Z

unfortunately no, i multiplied the default value by 64:

max_record_size = 68719476736

also, i tested increaseing the page_size from 1100 to:

page_size = 2500

Answer 3 · 2024-04-28T09:09:24.000Z

on mac the same problem

Atuin Doctor
Checking for diagnostics


Please include the output below with any bug reports or issues

atuin:
  version: 18.2.0
  sync:
    cloud: false
    records: false
    auto_sync: true
    last_sync: 2024-04-28 5:03:49.001966 +00:00:00
shell:
  name: zsh
  default: zsh
  plugins:
  - atuin
system:
  os: Darwin
  arch: arm64
  version: 14.4.1
  disks:
  - name: Macintosh HD
    filesystem: apfs
  - name: Macintosh HD
    filesystem: apfs
  - name: Macintosh HD
    filesystem: smbfs

and
ATUIN_LOG=debug atuin sync -f

…
[2024-04-28T08:59:16Z DEBUG h2::codec::framed_read] received frame=Headers { stream_id: StreamId(97), flags: (0x4: END_HEADERS) }
[2024-04-28T08:59:16Z DEBUG h2::codec::framed_read] received frame=Data { stream_id: StreamId(97), flags: (0x1: END_STREAM) }
Error: There was an error with the atuin sync service: Status 413.
If the problem persists, contact the host

Location:
    /private/tmp/atuin-20240415-5422-dpc6s2/atuin-18.2.0/atuin-client/src/api_client.rs:171:9

full log

server in k8s

server.toml

host = "0.0.0.0"
port = 8888
open_registration = false
db_uri="#DBURI#"

Answer 4 · 2024-04-28T11:03:52.000Z

syncing from the same machine running the atuin server, with a smaller history (<1k entries) works, so I thought the problem might be caused by either the traefik configuration or the reverse proxy im running on the machine that is hosting traefik.. so i added the following labels to the reverse proxy:

      - "traefik.http.routers.proxy-atuin-rtr.middlewares=buffering-atuin"
      - "traefik.http.middlewares.buffering-atuin.buffering.memRequestBodyBytes=20971520" # increase from 1MB (default) to 20MB limit
      - "traefik.http.middlewares.buffering-atuin.buffering.retryExpression=IsNetworkError()" # retry on networking errors

and increased the client_max_body_size of my nginx proxy:

client_max_body_size 20M;

.. unfortunately both without success. I also tried playing around with proxy_buffering and proxy_request_buffering. Previously to those changes I got this error log:

proxy-atuin  | 2024/04/28 10:41:35 [error] 30#30: *1 client intended to send too large body: 2543883 bytes, client: 99.99.99.254, server: atuin.domain.sh, request: "POST /history HTTP/1.1", host: "atuin.domain.sh"

that is now gone after the changes, now i simply get:

proxy-atuin  | 99.99.99.254 - - [28/Apr/2024:11:00:11 +0000] "GET /sync/status HTTP/1.1" 200 79 "-" "atuin/18.2.0"
proxy-atuin  | 99.99.99.254 - - [28/Apr/2024:11:00:11 +0000] "GET /sync/count HTTP/1.1" 200 11 "-" "atuin/18.2.0"
proxy-atuin  | 99.99.99.254 - - [28/Apr/2024:11:00:13 +0000] "POST /history HTTP/1.1" 413 56 "-" "atuin/18.2.0"

and on the client with atuin:

❯ atuin sync
Error: There was an error with the atuin sync service: Status 413.
If the problem persists, contact the host

Location:
    /home/runner/work/atuin/atuin/atuin-client/src/api_client.rs:171:9

Answer 5 · 2024-04-29T08:00:19.000Z

I'm going to close this for the time being as we cannot offer hosting support in our issue tracker - this is a configuration issue and not a bug

max_record_size = 0
max_history_length = 0

Will disable length checking for your server and resolve the issue. Otherwise, you can use the prometheus metrics we export to tune the value to your liking (which is what I've done for Atuin cloud).

I'd suggest testing without any reverse proxies. Get Atuin sync working, and then mess around with other software sat in the middle.

If you continue to have issues, please open another issue - ensure that you are only connecting Atuin client to Atuin server, and only following our documentation.

Answer 6 · 2024-04-29T08:34:28.000Z

max_record_size = 0
max_history_length = 0

didn't helped
im think it is problem with atuin server
can you offer any debug flags for atuin server

when connecting without traefik

…
[2024-04-29T08:26:27Z DEBUG hyper::proto::h1::conn] incoming body is content-length (56 bytes)
[2024-04-29T08:26:27Z DEBUG hyper::proto::h1::conn] incoming body completed
[2024-04-29T08:26:27Z DEBUG hyper::client::pool] pooling idle connection for ("http", 127.0.0.1:8888)
Error: There was an error with the atuin sync service: Status 413.
If the problem persists, contact the host

Location:
    /private/tmp/atuin-20240415-5422-dpc6s2/atuin-18.2.0/atuin-client/src/api_client.rs:171:9

atuin-sync-local.log

Answer 7 · 2024-04-29T08:46:48.000Z

didn't helped

Where did you put this config?

can you offer any debug flags for atuin server

RUST_LOG=atuin_server=debug

Answer 8 · 2024-04-29T09:07:57.000Z

Where did you put this config?

in server configuration server.toml

and log from server looks now like that

| atuin 2024-04-29T09:04:20.259710Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:20.498594Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items 
│ atuin 2024-04-29T09:04:20.721178Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items 
│ atuin 2024-04-29T09:04:20.933890Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items 
│ atuin 2024-04-29T09:04:21.165340Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:21.408839Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:21.731991Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:22.229897Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:22.439476Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:22.682373Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:22.905213Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:23.060108Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:23.311512Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:23.536330Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:23.735201Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:23.942056Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:24.172916Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:24.428093Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:24.628646Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:24.834622Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:25.076950Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:25.327205Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items
│ atuin 2024-04-29T09:04:25.514084Z DEBUG add{user.id=1}: atuin_server::handlers::history: request to add 1100 history items

same 413 in client

Answer 9 · 2024-04-30T02:25:02.000Z

Setting client_max_body_size 512M; in the nginx.conf for my site fixed this issue for me. I set it in the server block as well as the location / block that is doing the proxy_pass for good measure. You should probably check that you don't have anything ridiculously long in your history that would go over that limit too (see #1961 (comment)).

Answer 10 · 2024-04-30T07:39:05.000Z

@thallada i get this error with proxy and without
and longest command is 634 so i think it is not so long
via atuin history list | awk '{ print length, $0 }' | sort -n -s -r | less -S

Answer 11 · 2024-05-01T10:26:39.000Z

What worked for me is decreasing the page_size and removing the limits as @ellie mentioned. The reverse proxy settings or traefik middlewares did not matter at all - skipping the proxy led to the same results, so I removed any client_max_body_size restrictions in my nginx.conf. The important parameters of the server.toml file are:

max_record_size = 0
max_history_length = 0
page_size = 500

I suppose the page_size could be set even lower to make sure longer entries are not leading to 413 errors.

Answer 12 · 2024-05-01T10:31:25.000Z

Regardless - you will run into far fewer 413 and other sync related issues if you use sync v2

Put this into your client config, for all clients

[sync]
records = true

The only relevant setting on the server for this is max_record_size, neither page_size nor history_length do anything there.

If you are

Using sync v2
Not using a reverse proxy
Have disabled record size limits, as detailed above
STILL running into 413 issues

Then please open another issue, with your specific setup and what is going wrong.