Running addon in docker results in an error

Question

Running addon in docker results in an error

tiborrr opened this issue a year ago · 7 comments

I am trying to run my addon in docker, but am getting the following error:

Wed, June 26th, 2024, 8:56: Running external command: docker run --rm --name rita-addon-0.35.0-34CELZ --mount type=bind,source=/tmp/docspell-addons/addon-3233020791394861928,target=/mnt/work --mount type=bind,source=/tmp/docspell-addons/addon-output-11812519887303900956,target=/mnt/output --mount type=bind,source=/tmp/docspell-addon-cache/CwHRw9KBUb5-bCosvo9UC54-ZLTrGqjGduk-DhNdCLqQFcn,target=/mnt/cache --network none --env ADDON_DIR=addons/rita-addon-0.35.0 --env ITEM_DATA_JSON=item/item-data.json --env DSC_DOCSPELL_URL=http://localhost:7880 --env ITEM_DIR=item --env TMP_DIR=temp --env TMPDIR=temp --env CACHE_DIR=/mnt/cache --env ITEM_ARGS_JSON=item/given-data.json --env ITEM_ORIGINAL_JSON=item/source-files.json --env DSC_SESSION=redacted --env ITEM_PDF_JSON=item/pdf-files.json --env OUTPUT_DIR=/mnt/output --env ITEM_ORIGINAL_DIR=item/originals --env ITEM_PDF_DIR=item/pdfs -w /mnt/redacted/docspell-rita-addon:0.35.0 arguments/user-input
Wed, June 26th, 2024, 8:56: Waiting for command to terminate…
Wed, June 26th, 2024, 8:56: >> [rita-addon-0.35.0 (err)] docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /tmp/docspell-addons/addon-3233020791394861928.
Wed, June 26th, 2024, 8:56: >> [rita-addon-0.35.0 (err)] See 'docker run --help'.
Wed, June 26th, 2024, 8:56: >> [rita-addon-0.35.0 (err)]

It seems that a local directory /tmp/docspell-addons/addon-3233020791394861928 that it's trying to bind to has not been created before the container is being run.

Answer 1 · 2024-06-27T07:19:30.000Z

@eikek you solved this error in #2666 . But did not give an answer on how you fixed it. I am currently running the joex:nightly image so I'd expect not to see this error.

Answer 2 · 2024-07-03T06:28:33.000Z

Hi @tiborrr I'm not sure what you mean. #2666 was fixed with #2660 and when reading the comments in #2666 I thought you tested it as well? My tests with your plugin were successful at the time.

This error here seems new? Can you give me access to this addon so I could reproduce it?

Answer 3 · 2024-07-05T08:58:43.000Z

I am using the same example as you used, but for me it gives different results. I did not have the ability to test it. I just figured what should happen given your input. But now that I am testing it, I am getting different results.

I tried running the raw command in the docker container and get the following results:

/opt # docker run --rm --name docspell-addon-example-1.0.0-7WGrEN --mount type=bind,source=/tmp/docspell-addons/addon-13846520185996341526,target=/mnt/work --mount type=bind,source=/tmp/docspell-addons/addon-output-7096050851327249133,target=/mnt/output --mount type=bind,source=/tmp/docspell-addon-cache/CwHRw9KBUb5-bCosvo9UC54-ZLTrGqjGduk-DhNdCLqQFcn,target=/mnt/cache --
network none --env ADDON_DIR=addons/docspell-addon-example-1.0.0 --env ITEM_DATA_JSON=item/item-data.json --env DSC_DOCSPELL_URL=http://localhost:7880 --env ITEM_DIR=item --env TMP_DIR=temp --env TMPDIR=temp --env CACHE_DIR=/mnt/cache --env ITEM_ARGS_JSON=item/given-data.json --env ITEM_ORIGINAL_JSON=item/source-files.json --env DSC_SESSION=1720166031019-MS9ldHMvNnl5dEdY
dGRwcXctZTVGZzdNNWhKRXMtZ0J2dE1xUkd6bW8teGdIRUhLSlZhUWUvYWRtaW4=-false-900-$2a$10$LggARkurvnb7h8/IR6Z/H.-TM5rl8vlhwBrwtzUsCcZ2SesbTU= --env ITEM_PDF_JSON=item/pdf-files.json --env OUTPUT_DIR=/mnt/output --env ITEM_ORIGINAL_DIR=item/originals --env ITEM_PDF_DIR=item/pdfs -w /mnt/work tiborrr/docspell-addon-example:1.0.0 arguments/user-input
docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /tmp/docspell-addons/addon-13846520185996341526.
See 'docker run --help'.

/opt # mkdir /tmp/docspell-addons/addon-13846520185996341526

/opt # docker run --rm --name docspell-addon-example-1.0.0-7WGrEN --mount type=bind,source=/tmp/docspell-addons/addon-13846520185996341526,target=/mnt/work --mount type=bind,source=/tmp/docspell-addons/addon-output-7096050851327249133,target=/mnt/output --mount type=bind,source=/tmp/docspell-addon-cache/CwHRw9KBUb5-bCosvo9UC54-ZLTrGqjGduk-DhNdCLqQFcn,target=/mnt/cache --
network none --env ADDON_DIR=addons/docspell-addon-example-1.0.0 --env ITEM_DATA_JSON=item/item-data.json --env DSC_DOCSPELL_URL=http://localhost:7880 --env ITEM_DIR=item --env TMP_DIR=temp --env TMPDIR=temp --env CACHE_DIR=/mnt/cache --env ITEM_ARGS_JSON=item/given-data.json --env ITEM_ORIGINAL_JSON=item/source-files.json --env DSC_SESSION=1720166031019-MS9ldHMvNnl5dEdY
dGRwcXctZTVGZzdNNWhKRXMtZ0J2dE1xUkd6bW8teGdIRUhLSlZhUWUvYWRtaW4=-false-900-$2a$10$LggARkurvnb7h8/IR6Z/H.-TM5rl8vlhwBrwtzUsCcZ2SesbTU= --env ITEM_PDF_JSON=item/pdf-files.json --env OUTPUT_DIR=/mnt/output --env ITEM_ORIGINAL_DIR=item/originals --env ITEM_PDF_DIR=item/pdfs -w /mnt/work tiborrr/docspell-addon-example:1.0.0 arguments/user-input
docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /tmp/docspell-addons/addon-13846520185996341526.
See 'docker run --help'.

/opt # ls -ld /tmp/docspell-addons/*
drwxr-xr-x    2 root     root          4096 Jul  5 10:54 /tmp/docspell-addons/addon-13846520185996341526

/opt # docker run --rm --name docspell-addon-example-1.0.0-7WGrEN --network none --env ADDON_DIR=addons/docspell-addon-example-1.0.0 --env ITEM_DATA_JSON=item/item-data.json --env DSC_DOCSPELL_URL=http://localhost:7880 --env ITEM_DIR=item --env TMP_DIR=temp --env TMPDIR=temp --env CACHE_DIR=/mnt/cache --env ITEM_ARGS_JSON=item/given-data.json --env ITEM_ORIGINAL_JSON=ite
m/source-files.json --env DSC_SESSION=1720166031019-MS9ldHMvNnl5dEdYdGRwcXctZTVGZzdNNWhKRXMtZ0J2dE1xUkd6bW8teGdIRUhLSlZhUWUvYWRtaW4=-false-900-$2a$10$LggARkurvnb7h8/IR6Z/H.-TM5rl8vlhwBrwtzUsCcZ2SesbTU= --env ITEM_PDF_JSON=item/pdf-files.json --env OUTPUT_DIR=/mnt/output --env ITEM_ORIGINAL_DIR=item/originals --env ITEM_PDF_DIR=item/pdfs -w /mnt/work tiborrr/docspell-addo
n-example:1.0.0 arguments/user-input
Hello arguments/user-input

Running (just pulled the latest versions to be sure)
docspell/joex:nightly
docspell/restserver:latest

It seems that the binds do not work in your container, or that docker does not like the way those binds are being used because without the binds, my container works just fine.

I have tried mapping ./tmp:/tmp instead of /tmp:/tmp to see if it was a permissions issue --> same result
I have tried disabling the volume mount /tmp:/tmp so /tmp is just a folder in joex --> same result

Locally the command that you run in the container works. @eikek do you have any ideas? I am lost at this moment.

Answer 4 · 2024-07-08T07:36:14.000Z

Hi @tiborrr hm, that really looks strange. So if I understand correctly, you even created the directory with mkdir and re-running the command still said it wouldn't exist?

The addon you are trying to run, is it still https://github.com/tiborrr/docspell-addon-example ? I'll test it again here and report back.

Answer 5 · 2024-07-08T12:37:30.000Z

Yes that is exactly what you see in those commands. Very curious if you can recreate it.

Answer 6 · 2024-07-08T19:56:47.000Z

I tried with your addon version 1.0.0 and it worked for me. This is the output:

Mon, July 8th, 2024, 19:51: About to run 1 addon(s) in /tmp/docspell-addons/addon-18077140888600799738
Mon, July 8th, 2024, 19:51: Extract 1 addons to /tmp/docspell-addons/addon-18077140888600799738/addons
Mon, July 8th, 2024, 19:51: Extracting AddonArchive(docspell-file:///1/addon/214E3SB5zrB4h3urwvdF9KfHH3bVmnZpT2RwNCtXjS99,docspell-addon-example,1.0.0)
Mon, July 8th, 2024, 19:51: Executing addon docspell-addon-example-1.0.0
Mon, July 8th, 2024, 19:51: Storing user input into file
Mon, July 8th, 2024, 19:51: Running external command: docker run --rm --name docspell-addon-example-1.0.0-7QZQyp --mount type=bind,source=/tmp/docspell-addons/addon-18077140888600799738,target=/mnt/work --mount type=bind,source=/tmp/docspell-addons/addon-output-10439851884482481856,target=/mnt/output --mount type=bind,source=/tmp/docspell-addon-cache/J5QnTyfi7o6-vDWGyp32pVz-s9Te5iVxcCC-94cVVoem1mX,target=/mnt/cache --network none --env ADDON_DIR=addons/docspell-addon-example-1.0.0 --env ITEM_DATA_JSON=item/item-data.json --env ITEM_DIR=item --env TMP_DIR=temp --env TMPDIR=temp --env CACHE_DIR=/mnt/cache --env ITEM_ARGS_JSON=item/given-data.json --env ITEM_ORIGINAL_JSON=item/source-files.json --env ITEM_PDF_JSON=item/pdf-files.json --env OUTPUT_DIR=/mnt/output --env ITEM_ORIGINAL_DIR=item/originals --env ITEM_PDF_DIR=item/pdfs -w /mnt/work tiborrr/docspell-addon-example:1.0.0 arguments/user-input
Mon, July 8th, 2024, 19:51: Waiting for command to terminate…
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] Unable to find image 'tiborrr/docspell-addon-example:1.0.0' locally
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 1.0.0: Pulling from tiborrr/docspell-addon-example
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 09f376ebb190: Already exists
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 276709cbedc1: Already exists
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 2e133733af76: Already exists
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] ded8879d9a79: Already exists
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 3cf9507408dc: Already exists
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 50a938d3527f: Pulling fs layer
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] c32789aa1105: Pulling fs layer
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] b13636b2e773: Pulling fs layer
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] dffddee3c109: Pulling fs layer
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 0515451d000d: Pulling fs layer
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] dffddee3c109: Waiting
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 0515451d000d: Waiting
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 50a938d3527f: Verifying Checksum
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 50a938d3527f: Download complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 50a938d3527f: Pull complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] b13636b2e773: Verifying Checksum
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] b13636b2e773: Download complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] c32789aa1105: Download complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] c32789aa1105: Pull complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] b13636b2e773: Pull complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] dffddee3c109: Verifying Checksum
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 0515451d000d: Download complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] dffddee3c109: Pull complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] 0515451d000d: Pull complete
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] Digest: sha256:a750904e5ffd38079a17fdae5076144c6ed6563e20b57f74dea4d73612f209eb
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)] Status: Downloaded newer image for tiborrr/docspell-addon-example:1.0.0
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (out)] Hello arguments/user-input
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (err)]
Mon, July 8th, 2024, 19:51: >> [docspell-addon-example-1.0.0 (out)]
Mon, July 8th, 2024, 19:51: Closing process: `docker run --rm --name docspell-addon-example-1.0.0-7QZQyp --mount type=bind,source=/tmp/docspell-addons/addon-18077140888600799738,target=/mnt/work --mount type=bind,source=/tmp/docspell-addons/addon-output-10439851884482481856,target=/mnt/output --mount type=bind,source=/tmp/docspell-addon-cache/J5QnTyfi7o6-vDWGyp32pVz-s9Te5iVxcCC-94cVVoem1mX,target=/mnt/cache --network none --env ADDON_DIR=addons/docspell-addon-example-1.0.0 --env ITEM_DATA_JSON=item/item-data.json --env ITEM_DIR=item --env TMP_DIR=temp --env TMPDIR=temp --env CACHE_DIR=/mnt/cache --env ITEM_ARGS_JSON=item/given-data.json --env ITEM_ORIGINAL_JSON=item/source-files.json --env ITEM_PDF_JSON=item/pdf-files.json --env OUTPUT_DIR=/mnt/output --env ITEM_ORIGINAL_DIR=item/originals --env ITEM_PDF_DIR=item/pdfs -w /mnt/work tiborrr/docspell-addon-example:1.0.0 arguments/user-input`
Mon, July 8th, 2024, 19:51: Addon docspell-addon-example-1.0.0 executed successfully!
Mon, July 8th, 2024, 19:51: Addon stdout: Hello arguments/user-input
Mon, July 8th, 2024, 19:51: Addon result: AddonExecutionResult(List(DecodingError(expected json value got 'Hello ...' (line 1, column 1))),true)
Mon, July 8th, 2024, 19:51: Job execution successful

The last line only shows that it could not json-decode the addons output, but it ran successfully. I have no idea what's going on on your side. My docker version is

❯ docker --version
Docker version 24.0.9, build v24.0.9

Maybe there is some difference?

Edit: Also worth noting, that I'm not running docspell itself from docker.

Answer 7 · 2024-08-15T10:37:27.000Z

I have found the issue. This was caused by a missing line in the volumes.

volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp:/tmp # this line was missing