Ghost 2.x + Storage Adapter not using CDN URL?
necevil opened this issue · 2 comments
Hey guys,
So this one is weird, and a non-standard use case.
I am hoping I am missing something simple here as I have more than a few hours into this (after previously quickly running through and adding S3 storage adapter on earlier ghost versions).
I had previously been running a ghost 0.11.x install with S3 storage adapter baked in to the docker file (essentially just added as you normally would to the base Alpine image (https://hub.docker.com/_/ghost/).
I am in the process of rebuilding that setup on Ghost 2.x (2.7.1 to be specific).
I basically thought I had everything working again with 2.x — everything except for Ghost S3 Storage Adapter. I know that this one could be docker related but I figured I would post here in case someone else runs into the issue.
Problem 1
My first problem was that for whatever reason my docker based ghost Container always seems to be looking in the following directory:
/var/lib/ghost/versions/2.7.1/core/server/adapters/storage/s3
That's obviously different from the directory discussed in the docs (see: https://github.com/colinmeinke/ghost-storage-adapter-s3#installation) or in the blog mentioned in some of the other Ghost 2.x threads (https://it.ismy.fun/2018/08/26/ghost-custom-storage-module/) all of which indicate that the storage module should be copied into:
/var/lib/ghost/content/adapters/storage/s3
I couldn't get Ghost to look in that location no matter what I did — so instead I just copied my S3 storage adapter director into the alternate directory: /var/lib/ghost/versions/2.7.1/core/server/adapters/storage/s3
That worked. No more adapter not found — and I can see my images (which I bulk uploaded to S3 previously). The images are not available locally in any of the content directories.
Problem 2
The next issue is that I can't (for whatever reason) upload a new image to S3. When uploading a new image from the editor I see the image visually in the editor for a split second before the editor shows an error like:
In my console (keep in mind the container name is printed here) I get:
ghost-s3-os | ERROR [2018-12-18 20:46:16] "POST /ghost/api/v2/admin/uploads/" 403 2023ms
ghost-s3-os |
ghost-s3-os | NAME: InternalServerError
ghost-s3-os | CODE: AccessDenied
ghost-s3-os | MESSAGE: Access Denied
ghost-s3-os |
ghost-s3-os | level:normal
ghost-s3-os |
ghost-s3-os | empty
ghost-s3-os | empty
ghost-s3-os | ERROR DETAILS:
ghost-s3-os | empty
ghost-s3-os |
ghost-s3-os | InternalServerError: Access Denied
ghost-s3-os | at new GhostError (/var/lib/ghost/versions/2.7.1/core/server/lib/common/errors.js:10:26)
ghost-s3-os | at _private.prepareError (/var/lib/ghost/versions/2.7.1/core/server/web/shared/middlewares/error-handler.js:42:19)
ghost-s3-os | at Layer.handle_error (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/layer.js:71:5)
ghost-s3-os | at trim_prefix (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:315:13)
ghost-s3-os | at /var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:284:7
ghost-s3-os | at Function.process_params (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:335:12)
ghost-s3-os | at next (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:275:10)
ghost-s3-os | at Layer.handle_error (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/layer.js:67:12)
ghost-s3-os | at trim_prefix (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:315:13)
ghost-s3-os | at /var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:284:7
ghost-s3-os | at Function.process_params (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:335:12)
ghost-s3-os | at next (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:275:10)
ghost-s3-os | at /var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:635:15
ghost-s3-os | at next (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/index.js:260:14)
ghost-s3-os | at next (/var/lib/ghost/versions/2.7.1/node_modules/express/lib/router/route.js:127:14)
ghost-s3-os | at apiImpl.then.catch (/var/lib/ghost/versions/2.7.1/core/server/api/shared/http.js:59:17)
ghost-s3-os |
ghost-s3-os | AccessDenied: Access Denied
ghost-s3-os | at Request.extractError (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/services/s3.js:283:35)
ghost-s3-os | at Request.callListeners (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/sequential_executor.js:103:18)
ghost-s3-os | at Request.callListeners (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/sequential_executor.js:104:14)
ghost-s3-os | at Request.emit (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
ghost-s3-os | at Request.emit (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/request.js:584:14)
ghost-s3-os | at Request.transition (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/request.js:16:12)
ghost-s3-os | at AcceptorStateMachine.runTo (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/state_machine.js:14:12)
ghost-s3-os | at /var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/state_machine.js:26:10
ghost-s3-os | at Request.<anonymous> (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/request.js:22:9)
ghost-s3-os | at Request.<anonymous> (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/request.js:586:12)
ghost-s3-os | at Request.callListeners (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/sequential_executor.js:87:20)
ghost-s3-os | at Request.callListeners (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/sequential_executor.js:104:14)
ghost-s3-os | at Request.emit (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
ghost-s3-os | at Request.emit (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/request.js:584:14)
ghost-s3-os | at Request.transition (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/request.js:16:12)
ghost-s3-os | at AcceptorStateMachine.runTo (/var/lib/ghost/versions/2.7.1/node_modules/aws-sdk/lib/state_machine.js:14:12)
Basically that seems (to me) to indicate that AWS for whatever reason is not able to upload the images (possibly due to IAM permissions). That isn't the case since I can execute the cp command to manually utilize the same Key / Secret to do the upload from within the container, for example:
aws s3 cp file.txt s3://some-bucket/
Problem 3 (the weirdest one)
The last problem is that it appears that S3 is in fact working properly, my images are available, but when I inspect the site — the images are loading from the container (which DOES NOT have the images stored anywhere inside).
The reason I know there are no images in the container is because a bulk find from within the contianer shows nothing except theme images:
$ find / -name "*jpg"
/var/lib/ghost/versions/2.7.1/content/themes/casper/assets/screenshot-desktop.jpg
/var/lib/ghost/versions/2.7.1/content/themes/casper/assets/screenshot-mobile.jpg
/var/lib/ghost/versions/2.7.1/core/built/assets/img/themes/nurui-146ba228f415aa6dc073911c5eedd524.jpg
/var/lib/ghost/versions/2.7.1/core/built/assets/img/themes/farafra-800f5fbb100aaefbefe42e8f79fa3391.jpg
/var/lib/ghost/versions/2.7.1/core/built/assets/img/themes/valkyrie-3ce1c51ff5a8fb77494a97ee192f199a.jpg
/var/lib/ghost/versions/2.7.1/core/built/assets/img/themes/sente-3c2bd8202c626b11048c3bc6bddc250a.jpg
/var/lib/ghost/versions/2.7.1/core/built/assets/img/themes/massively-06edf00108429f7fb8e65f190fba34fe.jpg
/var/lib/ghost/versions/2.7.1/core/built/assets/img/themes/pacific-182fa717198992c893b4afd7b1dd1096.jpg
/home/node/.cache/yarn/v2/npm-browserify-zlib-0.1.4-bb35f8a519f600e0fa6b8485241c979d0141fb2d/test/fixtures/person.jpg
So basically the net result appears to be that the Ghost install is some how grabbing images from S3 — but then serving them as if they are local (ie. from the standard /content/whatever.jpg URL, rather than the CDN / cloudfront URL specified in my config). How that could be happening I have NO idea.
(Bonus) Problem 4
Last issue — probably related to the above — is that some images that are in S3 (manually uploaded) apparently are not accessible on S3 (which would indicate that yes S3 can be connected to). Specifically I get a single error (where this file DOES exist in S3):
ghost-s3-os | [2018-12-18 20:41:26] ERROR
ghost-s3-os |
ghost-s3-os | NAME: InternalServerError
ghost-s3-os | CODE: IMAGE_SIZE_STORAGE
ghost-s3-os | MESSAGE: /2018/11/Original-Skateboards-Logo-175.png is not stored in s3
ghost-s3-os |
ghost-s3-os | level:critical
ghost-s3-os |
ghost-s3-os | "/2018/11/Original-Skateboards-Logo-175.png"
ghost-s3-os | empty
ghost-s3-os | ERROR DETAILS:
ghost-s3-os | {"originalPath":"http://dev.originalskateboards.com/content/images/2018/11/Original-Skateboards-Logo-175.png","reqFilePath":"/2018/11/Original-Skateboards-Logo-175.png"}
ghost-s3-os |
ghost-s3-os | InternalServerError: /2018/11/Original-Skateboards-Logo-175.png is not stored in s3
ghost-s3-os | at new InternalServerError (/var/lib/ghost/versions/2.7.1/node_modules/ghost-ignition/lib/errors/index.js:71:23)
ghost-s3-os | at storage.getStorage.read.then.catch.catch (/var/lib/ghost/versions/2.7.1/core/server/lib/image/image-size.js:214:35)
ghost-s3-os | at tryCatcher (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/util.js:16:23)
ghost-s3-os | at Promise._settlePromiseFromHandler (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/promise.js:512:31)
ghost-s3-os | at Promise._settlePromise (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/promise.js:569:18)
ghost-s3-os | at Promise._settlePromise0 (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/promise.js:614:10)
ghost-s3-os | at Promise._settlePromises (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/promise.js:690:18)
ghost-s3-os | at _drainQueueStep (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/async.js:138:12)
ghost-s3-os | at _drainQueue (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/async.js:131:9)
ghost-s3-os | at Async._drainQueues (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/async.js:147:5)
ghost-s3-os | at Immediate.Async.drainQueues (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/async.js:17:14)
ghost-s3-os | at runCallback (timers.js:810:20)
ghost-s3-os | at tryOnImmediate (timers.js:768:5)
ghost-s3-os | at processImmediate [as _immediateCallback] (timers.js:745:5)
ghost-s3-os |
ghost-s3-os | Error: /2018/11/Original-Skateboards-Logo-175.png is not stored in s3
ghost-s3-os | at /var/lib/ghost/versions/2.7.1/core/server/adapters/storage/s3/index.js:174:16
ghost-s3-os | at Promise._execute (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/debuggability.js:313:9)
ghost-s3-os | at Promise._resolveFromExecutor (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/promise.js:483:18)
ghost-s3-os | at new Promise (/var/lib/ghost/versions/2.7.1/node_modules/bluebird/js/release/promise.js:79:10)
ghost-s3-os | at Store.read (/var/lib/ghost/versions/2.7.1/core/server/adapters/storage/s3/index.js:168:12)
ghost-s3-os | at getImageSizeFromStoragePath (/var/lib/ghost/versions/2.7.1/core/server/lib/image/image-size.js:188:10)
ghost-s3-os | at Object.getImageSizeFromUrl (/var/lib/ghost/versions/2.7.1/core/server/lib/image/image-size.js:88:16)
ghost-s3-os | at Object.getCachedImageSizeFromUrl (/var/lib/ghost/versions/2.7.1/core/server/lib/image/cached-image-size-from-url.js:21:26)
ghost-s3-os | at getImageDimensions (/var/lib/ghost/versions/2.7.1/core/server/data/meta/image-dimensions.js:17:24)
ghost-s3-os | at getMetaData (/var/lib/ghost/versions/2.7.1/core/server/data/meta/index.js:97:26)
ghost-s3-os | at Object.ghost_head (/var/lib/ghost/versions/2.7.1/core/server/helpers/ghost_head.js:124:12)
ghost-s3-os | at Object.returnAsync (/var/lib/ghost/versions/2.7.1/core/server/helpers/register.js:16:28)
ghost-s3-os | at Function.Waiter.resolve (/var/lib/ghost/versions/2.7.1/node_modules/express-hbs/lib/async.js:83:3)
ghost-s3-os | at Object.<anonymous> (/var/lib/ghost/versions/2.7.1/node_modules/express-hbs/lib/hbs.js:396:18)
ghost-s3-os | at Object.eval [as main] (eval at createFunctionContext (/var/lib/ghost/versions/2.7.1/node_modules/handlebars/dist/cjs/handlebars/compiler/javascript-compiler.js:254:23), <anonymous>:22:166)
ghost-s3-os | at main (/var/lib/ghost/versions/2.7.1/node_modules/handlebars/dist/cjs/handlebars/runtime.js:173:32)
Config File
For reference my config file looks like:
{
"server": {
"host": "0.0.0.0",
"port": 2368
},
"process": "local",
"paths": {
"contentPath": "content/"
},
"storage": {
"active": "s3",
"s3": {
"accessKeyId": "KEYHERE",
"secretAccessKey": "SECRETHERE",
"assetHost": "https://somecloudfronturl.cloudfront.net",
"region": "us-east-1",
"bucket": "some-bucket",
"forcePathStyle": true
}
},
}
I can properly access the host in my browser (and as mentioned) I can even see the images — which do not exist locally. The weirdness is that those images appear to be HOSTED locally some how — and I can't upload anything.
Here's hoping you guys might have some ideas!
If you made it this far — Thanks in advance.
OK so as you might imagine this ended up being a few different issues stacked together.
The first one was an issue with my Paths configuration. In order to make the /var/lib/ghost/content host my files I needed to change my configuration for my paths to an absolute path within the container:
"paths": {
"contentPath": "/var/lib/ghost/content"
},
The next piece that needed to be resolved was the fact that the default description in the documentation of copying ONLY the node_modules/ghost-storage-adapter-s3 package did not work for me inside of the container. This had to do with the fact that once I placed my adapter at /var/lib/ghost/content/adapters/storage/s3 the dependencies that get installed along with ghost-storage-adapter-s3 were not accessible to that module.
My solution in my Dockerfile was:
ADD node_modules /var/lib/ghost/content/adapters/storage/s3/node_modules
ADD node_modules/ghost-storage-adapter-s3 /var/lib/ghost/content/adapters/storage/s3
Basically I am adding ALL dependencies for ghost-storage-adapter-s3 into the container from my local machine in the node_modules folder that is relative to the storage module — and then adding the storage adapter it's self in that directory (which then looks in the correct place for the node_modules file.
Last but not least I had an error where for some reason at least one of the list permissions did not exist on my IAM user. This was causing image upload to WORK but caused some weird issues — and specifically was responsible for 'Access Denied' messaging when uploading from Ghost EVEN THOUGH I could upload from the command line within the container.
Alright.
So now ghost can find the module properly and we have working upload from the Editor.
Last Problem Left to Solve
The last issue that I need to address is the fact that my CDN / assetHost has an additional back slash in it.
When I load a page that contains an image that was uploaded to S3 it does not show up because the assethost being prepended has an additional backslash within the URL.
Wrong: https://whatevercdninstance.cloudfront.net//2018/12/Screen-Shot-2018-12-11-at-7.00.41-PM.png
Right: https://whatevercdninstance.cloudfront.net/2018/12/Screen-Shot-2018-12-11-at-7.00.41-PM.png
For reference my storage config block looks like:
"storage": {
"active": "s3",
"s3": {
"accessKeyId": "MyKeyHere",
"secretAccessKey": "MySecretHere",
"assetHost": "https://somelettersandnumbers.cloudfront.net/",
"region": "us-east-1",
"bucket": "ghost-assets",
"forcePathStyle": true
}
},
If I edit the URL within Chrome DevTools the image does appear so I know that the image is uploaded properly. I also did some searching of other closed issues within the current project but wasn't able to find anything related to an extra backslash being appended to the back of the CDN / Asset Host URL in front of the file name / directory.
I am going to do some more digging but figured I would update anyone who might ALSO happen to end up trying to build ghost-storage-adapter-s3 into a docker container.
Ok the last piece of this is now solved.
Basically for whatever reason I needed to ALSO add the pathPrefix set to "" in order to remove the additional slash. After that everything worked. My final configuration looks like:
"paths": {
"contentPath": "/var/lib/ghost/content"
},
"process": "local",
"storage": {
"active": "s3",
"s3": {
"pathPrefix": "",
"accessKeyId": "MyKey",
"secretAccessKey": "MySecret",
"assetHost": "https://lettersandnumbers.cloudfront.net",
"region": "us-east-1",
"bucket": "ghost-assets",
"forcePathStyle": true
}
},
I think that's about it. Hopefully it helps out someone working to create a custom Ghost 2.x docker container that also includes ghost-storage-adapter-s3.