Not able to configure S3 on EC2 Instance
tanmay-predisai opened this issue · 7 comments
Hello - We are trying to configure S3 by setting the parameters in the .env file.
When we try and upload a file, we get a 400 bad request on https:///api/admin/projects/1/images with the response as "{"status":"error","error":"An error occurred with this request."}". We observe this in the browser network tab.
If i hit the same URL directly, https://pv.predis.ai/api/admin/projects/1/images, i get this kind of output -
{"results":[{"id":1,"created_at":"2024-07-24T06:20:39.000Z","updated_at":"2024-07-24T06:20:40.000Z","project_id":1,"uuid":"33537321-1a57-4bff-af17-bf8ae6ecb18a","name":"screenshot_2024_07_24_115023","original_name":"Screenshot 2024-07-24 115023","extension":".png","alt":null,"file_size":141192,"URL":"<STORAGE_BASE_URL>/33537321-1a57-4bff-af17-bf8ae6ecb18a.png"}],"limit":25}
Is there a way I can see the logs on what's going wrong?
We have set the following params in our .env file
STORAGE_DRIVER=s3
STORAGE_BASE_URL=https:///parcelvoy-assets/
STORAGE_S3_BUCKET=(our s3 bucket name)
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
STORAGE_S3_ENDPOINT=parcelvoy-assets
STORAGE_S3_FORCE_PATH_STYLE=true
Thank you for your help!
Assuming you are running this in Docker Compose you should be able to view the logs by running docker compose logs to see what is being output
Thank you for your reply. This is the error i see in the docker compose logs when I try to upload a new file -
error: {
api-1 | "input": "parcelvoy-assets",
api-1 | "code": "ERR_INVALID_URL",
}
Is there a problem in one of these configurations?
STORAGE_DRIVER=s3
STORAGE_BASE_URL=https://assets.predis.ai/parcelvoy-assets/
STORAGE_S3_BUCKET=user-uploaded-img
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
STORAGE_S3_ENDPOINT=parcelvoy-assets
STORAGE_S3_FORCE_PATH_STYLE=true
Or would it be possible to add some more description on how to get these values - esp - STORAGE_S3_BUCKET, STORAGE_S3_ENDPOINT,STORAGE_S3_FORCE_PATH_STYLE please.
Thank you for your help.
here is the log output :
[1723090496892] ERROR (9 on 6fb2dfceaa3a): error
api-1 | ctx: {
api-1 | "request": {
api-1 | "method": "POST",
api-1 | "url": "/api/admin/projects/1/images",
api-1 | "header": {
api-1 | "host": "127.0.0.1:3001",
api-1 | "connection": "close",
api-1 | "content-length": "1204962",
api-1 | "sec-ch-ua": ""Not)A;Brand";v="99", "Google Chrome";v="127", "Chromium";v="127"",
api-1 | "accept": "application/json, text/plain, /",
api-1 | "sec-ch-ua-platform": ""macOS"",
api-1 | "sec-ch-ua-mobile": "?0",
api-1 | "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
api-1 | "content-type": "multipart/form-data; boundary=----WebKitFormBoundaryvLZTTAKvkSulN4ZC",
api-1 | "origin": "https://pv.predis.ai",
api-1 | "sec-fetch-site": "same-origin",
api-1 | "sec-fetch-mode": "cors",
api-1 | "sec-fetch-dest": "empty",
api-1 | "referer": "https://pv.predis.ai/projects/1/campaigns/11/editor?locale=en",
api-1 | "accept-encoding": "gzip, deflate, br, zstd",
api-1 | "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
api-1 | "cookie": "_fbp=fb.1.1712838844905.406076715; crisp-client%2Fsession%2F94435076-ab60-4eec-8a60-50044deec72f=session_d8152aec-a117-4c0d-945d-cd5df2188775; _tt_enable_cookie=1; _ttp=vKlXB2mRMYkO8lcK2qgExgNLypN; _gcl_au=1.1.325747883.1720617126; _fbc=fb.1.1721824136118.IwZXh0bgNhZW0BMAABHXRtM8ELVnfFWFg7Zq8x5OH2TmT-RNIAxExzI4Et8fWOAbK4Y1cRZOKFdg_aem_reko4qsaQNXgBjJUVQZRsA; pred_ref=page-fam; pref_ref_location=use-cases/facebook-ad-maker/; pred_history=1721913454002|page-fam|/use-cases/facebook-ad-maker; cf_clearance=M_XQhgDMjCrsGOq6oo5W5bCGlvJw_ShVdpAHFm4Do34-1722860950-1.0.1.1-pp2yfCimyywKbnr0nXKPtBHFMaeOPkJuxk8pneNQA.6_TB37BEoPsijGG.WlEH7jl0Q4sNiZWOwqDSATethxFA; _ga=GA1.1.632535694.1712838844; _clck=76buwp%7C2%7Cfo4%7C0%7C1562; oauth={"access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpasI6MSwib3JnYW5pemF0aW9uX2lkIjoxLCJyb2xlIjoib3duZXIiLCJleHAiOjE3MjMxMTQ3NDksImlhdCI6MTcyMzAyODM0OX0.uzgz-0bsB_TFb7kzY-YqDYgHOeirHIN7xotg_ko3yaE","expires_at":"2024-08-08T10:59:09.916Z"}; oauth.sig=zJh0uQ-TIQNo94nas1xYkuVMMzFc; _ga_PQJ6V7Z34C=GS1.1.1723090210.429.0.1723090210.60.0.165529106"
api-1 | }
api-1 | },
api-1 | "response": {
api-1 | "status": 404,
api-1 | "message": "Not Found",
api-1 | "header": {
api-1 | "vary": "Origin",
api-1 | "access-control-allow-origin": ""
api-1 | }
api-1 | },
api-1 | "app": {
api-1 | "subdomainOffset": 2,
api-1 | "proxy": true,
api-1 | "env": "production"
api-1 | },
api-1 | "originalUrl": "/api/admin/projects/1/images",
api-1 | "req": "",
api-1 | "res": "",
api-1 | "socket": ""
api-1 | }
api-1 | error: {
api-1 | "input": "parcelvoy-assets",
api-1 | "code": "ERR_INVALID_URL",
api-1 | "headers": {
api-1 | "Access-Control-Allow-Origin": "",
api-1 | "vary": "Origin"
api-1 | }
api-1 | }
api-1 |
api-1 | TypeError [ERR_INVALID_URL]: Invalid URL
api-1 | at new NodeError (node:internal/errors:405:5)
api-1 | at new URL (node:internal/url:676:13)
api-1 | at parseUrl (/usr/src/app/node_modules/@smithy/url-parser/dist-cjs/index.js:29:21)
api-1 | at toEndpointV1 (/usr/src/app/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js:112:41)
api-1 | at Object.customEndpointProvider [as endpoint] (/usr/src/app/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js:232:65)
api-1 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
api-1 | at async Promise.all (index 1)
api-1 | at async _Upload.__uploadUsingPut (/usr/src/app/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:230:22)
api-1 | at async _Upload.__doConcurrentUpload (/usr/src/app/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:293:18)
api-1 | at async Promise.all (index 0)
api-1 | at async _Upload.__doMultipartUpload (/usr/src/app/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:379:5)
api-1 | at async _Upload.done (/usr/src/app/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:203:12)
api-1 | at async S3StorageProvider.upload (/usr/src/app/build/storage/S3StorageProvider.js:32:9)
api-1 | at async Storage.upload (/usr/src/app/build/storage/Storage.js:44:9)
api-1 | at async Storage.save (/usr/src/app/build/storage/Storage.js:32:9)
api-1 | at async uploadImage (/usr/src/app/build/storage/ImageService.js:11:20)
Use this variable like this:
STORAGE_S3_ENDPOINT=https://s3.us-east-1.amazonaws.com
More info on https://docs.aws.amazon.com/general/latest/gr/s3.html
@leobarcellos Thank you for your help. This works!
Another question- Would you know how can I specify the key (subfolder) name here? The structure we wanted to have here was /parcelvoy-assets/<files.jpg>. As of now, it's working for /<files.jpg>.
Not sure, I use it like you said (example.com/<filename.ext>)
Try removing the trailing slash: STORAGE_BASE_URL=https://assets.predis.ai/parcelvoy-assets
@tanmay-predisai the solution above is correct, however you can also just remove STORAGE_S3_FORCE_PATH_STYLE from your config or set it to false and it should work as well. That setting disables AWS defaults for endpoint generation for if you want to use another S3-compatible service for uploads such as Digital Ocean Spaces. You can find the original implementation here: #338. It is not possible currently to upload to a given folder, you are welcome to contribute changes to make this possible however!
@pushchris this works for us. I appreciate your help!
We plan to start using PV fully from next week. I will keep you posted on the progress :)