How to use FILESYSTEM as IMAGE_UPLOAD_TYPE with docker
Closed this issue · 15 comments
How can I create persistent storage for the codimd-container to allow filesystem as CMD_IMAGE_UPLOAD_TYPE? For example setting up a volume. And is there any downside with this? What needs to be considered when to update?
I added a volumes line to my docker-compose.yml - the only downside seems to be that sometimes the read and execute permissions for the group get lost (I haven't yet figured out when this happens). I only need that because I let my frontend HTTP proxy deliver static images as well, so for a basic setup this shouldn't bother you.
Here's most of my docker-compose.yml, I removed things that are not related to image upload. Please let me know if this is in any way helpful!
version: '3'
services:
database:
image: postgres:9.6-alpine
environment:
- POSTGRES_USER=hackmd
- POSTGRES_PASSWORD=hackmdpass
- POSTGRES_DB=hackmd
volumes:
- ./database:/var/lib/postgresql/data
restart: always
app:
image: hackmdio/hackmd:1.2.1
environment:
- CMD_DB_URL=postgres://hackmd:hackmdpass@database:5432/hackmd
- CMD_SESSION_SECRET=garbled stuff in here.
- CMD_IMAGE_UPLOAD_TYPE=filesystem
volumes:
- ./hackmd-uploads:/codimd/public/uploads
ports:
- "9003:3000"
restart: always
depends_on:
- databaseThanks a lot. This was really helpful. So far working as expected.
I noticed the following. In your docker-compose.yml you are using a relative path for your database volume ./database:/var/lib/postgresql/data. So the data is stored in the /codimd-container path. In my it's database:/var/lib/postgresql/data and I assume the volume is in var/lib/docker/volumes. Is there any advantage to have it in the codimd-container path except from having it centralized for easier backup?
Do I need to think about something special when having the upload folder like you in /codimd-container for example during update?
Regarding the relative path: I think I was just getting started with docker, and this way I felt I had better control over where data ended up. It may also have been something I read in some tutorial. So: if the regular way works for you: no need to change it :-)
Same for uploads, of course! They can be anywhere on your host system.
You shouldn't need to take any special care about the uploads. I like to test an upload after every upgrade, as there has been a mixup with permissions on my setup before (I hinted at this in my previous post, but I haven't looked into it so far).
Thanks for your really fast support. Issue resolved / question answered.
@ccoenen Hi, I updated today to 1.3.0 and now may have the issue you described with the permissions. Folder permissions are set to 755 for the upload folder and owner www-data / www-data but I get the following error:
2019-03-05T22:34:09.839Z error: uncaughtException: Invalid URL: /uploads/
app_1 | TypeError [ERR_INVALID_URL]: Invalid URL: /uploads/
app_1 | at onParseError (internal/url.js:219:17)
app_1 | at parse (internal/url.js:228:3)
app_1 | at new URL (internal/url.js:311:5)
app_1 | at new URL (internal/url.js:309:14)
app_1 | at Object.exports.uploadImage (/codimd/lib/web/imageRouter/filesystem.js:19:19)
app_1 | at /codimd/lib/web/imageRouter/index.js:31:22
app_1 | at IncomingForm.<anonymous> (/codimd/node_modules/formidable/lib/incoming_form.js:107:9)
app_1 | at emitNone (events.js:106:13)
app_1 | at IncomingForm.emit (events.js:208:7)
app_1 | at IncomingForm._maybeEnd (/codimd/node_modules/formidable/lib/incoming_form.js:557:8)
app_1 | at /codimd/node_modules/formidable/lib/incoming_form.js:238:12
app_1 | at WriteStream.<anonymous> (/codimd/node_modules/formidable/lib/file.js:79:5)
app_1 | at Object.onceWrapper (events.js:313:30)
app_1 | at emitNone (events.js:111:20)
app_1 | at WriteStream.emit (events.js:208:7)
app_1 | at finishMaybe (_stream_writable.js:613:14)
app_1 | 2019-03-05T22:34:09.840Z error: An uncaught exception has occured.
app_1 | 2019-03-05T22:34:09.840Z error: Invalid URL: /uploads/
app_1 | 2019-03-05T22:34:09.841Z error: Process will exit now.
See below my docker-compose.yml file:
app:
image: hackmdio/hackmd:1.3.0
environment:
- CMD_DB_URL=postgres://hackmd:hackmdpass@database:5432/hackmd
- CMD_ALLOW_EMAIL_REGISTER=false
- CMD_ALLOW_FREEURL=false
- CMD_DEFAULT_PERMISSION=private
- CMD_ALLOW_PDF_EXPORT=true
- CMD_ALLOW_ANONYMOUS=false
- CMD_PROTOCOL_USESSL=false
- CMD_HSTS_ENABLE=false
- CMD_IMAGE_UPLOAD_TYPE=filesystem
volumes:
- ./codimd-uploads:/codimd/public/uploads
ports:
- "127.0.0.1:3000:3000"
networks:
backend:
restart: always
depends_on:
- database
After docker-compose up and trying to upload a file the ownership of the upload folder is 10000.
What do I need to do?
The container runs with an own users that uses the UID 10000. When the container detects that you use the filesystem upload type, it'll run chown and chmod on the volume to make sure it's writeable by the container:
This is intended behavior to make the usage of the container as easy as possible while keeping your system safe.
@SISheogorath Thanks for clarifying. But after the system has set the owner of the folder right upload is still not possible and the error message is the same. In addition container is not reachable and I need to reboot. What to do? Same configuration worked before I updated and pictures in existing notes are shown.
Thanks for identifying this part of the script! This explains my issue very well. This happened to me once every full moon, so I usually just cmodded my uploads dir and didn't look into it further.
I added #25 as a proposal on how this could be changed.
For now, @MelBourbon could you try to chmod it after codi instance is running? If this fixes your issue, we have the same problem :-).
@ccoenen I set the permissions of my upload folder to 770 (u+rwx,g+rwx) after codi instance is running and the ownership is 10000/www-data. But if I want to upload a file the same error message is occurring. Afterward permission on the folder are set to 700 again. I wonder why it worked before.
@MelBourbon do you see a warning on startup, saying that serverURL is not set correctly? I think this is your problem.
Also you shouldn't set the permission to 0770 but 0750 at best (there is no need for the www-data group to write into the directory)
@SISheogorath I once tried to set CMD_DOMAIN the app section of docker-compose.yml but with that variable set user can not login since they will be redirected to https://my-domain.de:3000/login but my nginx server is listening on 443 and redirecting locally to port 3000 only for codimd. How can I resolve this?
@MelBourbon as the warning states:
Make sure
protocolUseSSLandurlAddPortorCMD_PROTOCOL_USESSLandCMD_URL_ADDPORTare configured properly.
Which is in your case:
CMD_PROTOCOL_USESSL=true and CMD_URL_ADDPORT=false
@SISheogorath Ok, you got me. First reading, then asking. It works now with all needed variables set as described in your and @ccoenen comments. Thanks for your support both!
Is there any way to do this without having all logins redirect to the CMD_DOMAIN? We use the same Codimd instance on multiple domains and dont want them all redirected to one, but we're fine with uploads only being hosted on one of the domains.
When I remove CMD_DOMAIN uploads crash the system and stop the container.
Right now, due to the usage of the URL module to make sure we generate valid URLs, it requires the CMD_DOMAIN.
But you can use an minio container instead (which I recommend in general) to serve uploaded files. That makes you independent from this bug/feature.