some resized files are missing

Question

some resized files are missing

berkaytheunicorn opened this issue 7 years ago · 12 comments

Hey there, I've been trying to figure out the package and I notice whatever I do some files just wont't generate.

my config:

{
    "bucket": "xxx-test",
    "resizes": [
        {
            "size": 128,
            "suffix": "_sm",
            "quality": 90
        },
        {
            "size": 512,
            "suffix": "_md",
            "quality": 90
        },
        {
            "size": "512X512^",
            "gravity": "Center",
            "crop": "512X512",
            "suffix": "_md_sq",
            "quality": 90
        },
        {
            "size": 1024,
            "quality": 90,
            "suffix": "_lg"
        },
        {
            "size": 1440,
            "quality": 90,
            "suffix": "_xl"
        }
    ]
}

sometimes it created _sm sometimes up to _md but I couldn't make it fully work.
(I tested with various images 1024px(50kb) to 2560px(1mb))
Memory of Lambda 1280mb and timeout at 20s.

I guess I am missin something. (or could be a bug?)

Answer 1 · 2018-04-04T06:45:27.000Z

btw: logs and triggers settings are like this, and files alcs are set to private

Answer 2 · 2018-04-04T13:10:56.000Z

@berkaey Thank you for reporting.
It seems that your configuration satisfies to work as well.

And looking at the log, it looks like some error occurred, but I cannot see because it's interrupted.

{"errorMessage": Error processing demo/xxxx ..

Can I see the full log message about this line? it looks like TypeError, it might be an issue in a program. And I'm happy to get an image which error occurred if you can.

Answer 3 · 2018-04-04T14:45:52.000Z

here you go ;)

2018-04-04T06:28:16.657Z	ddf8df64-37d0-11e8-8e6e-b76df8d653e4	Parsing S3 event...
2018-04-04T06:28:16.658Z	ddf8df64-37d0-11e8-8e6e-b76df8d653e4	Downloading: demo/cf90285e-70a9-447d-b7ce-19e2a177e624.png
2018-04-04T06:28:16.757Z	ddf8df64-37d0-11e8-8e6e-b76df8d653e4
{
    "errorMessage": "Error processing demo/cf90285e-70a9-447d-b7ce-19e2a177e624.png: S3 getObject failed: NoSuchKey: The specified key does not exist."
}

END RequestId: ddf8df64-37d0-11e8-8e6e-b76df8d653e4
REPORT RequestId: ddf8df64-37d0-11e8-8e6e-b76df8d653e4	Duration: 100.84 ms	Billed Duration: 200 ms Memory Size: 1280 MB	Max Memory Used: 48 MB	
START RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465 Version: $LATEST
2018-04-04T06:28:38.474Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Parsing S3 event...
2018-04-04T06:28:38.475Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Downloading: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66.png
2018-04-04T06:28:38.534Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:28:38.974Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Reducing to: in-place
2018-04-04T06:28:38.994Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Uploading to: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_sm.png (799 bytes)
2018-04-04T06:28:39.075Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:28:39.579Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Reducing to: in-place
2018-04-04T06:28:39.620Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Uploading to: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_md.png (1550 bytes)
2018-04-04T06:28:39.708Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:28:39.710Z	68facee4-37d1-11e8-b9ff-4d43a4e05465
{
    "errorMessage": "Error processing demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66.png: TypeError: Cannot read property '1' of null"
}

END RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465
REPORT RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465	Duration: 1249.44 ms	Billed Duration: 1300 ms Memory Size: 1280 MB	Max Memory Used: 79 MB	
START RequestId: 69caa24f-37d1-11e8-9952-e7e3fa59258f Version: $LATEST
2018-04-04T06:28:39.845Z	69caa24f-37d1-11e8-9952-e7e3fa59258f	Parsing S3 event...
2018-04-04T06:28:39.845Z	69caa24f-37d1-11e8-9952-e7e3fa59258f	Downloading: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_md.png
2018-04-04T06:28:39.874Z	69caa24f-37d1-11e8-9952-e7e3fa59258f	Image already processed
END RequestId: 69caa24f-37d1-11e8-9952-e7e3fa59258f
REPORT RequestId: 69caa24f-37d1-11e8-9952-e7e3fa59258f	Duration: 30.44 ms	Billed Duration: 100 ms Memory Size: 1280 MB	Max Memory Used: 79 MB	
START RequestId: 49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1 Version: $LATEST
2018-04-04T06:28:56.879Z	49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1	Parsing S3 event...
2018-04-04T06:28:56.880Z	49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1	Downloading: demo/83d32285-b4c3-4b18-ba8d-a5d7fb747ec0.png
2018-04-04T06:28:56.975Z	49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1
{
    "errorMessage": "Error processing demo/83d32285-b4c3-4b18-ba8d-a5d7fb747ec0.png: S3 getObject failed: NoSuchKey: The specified key does not exist."
}

END RequestId: 49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1
REPORT RequestId: 49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1	Duration: 111.03 ms	Billed Duration: 200 ms Memory Size: 1280 MB	Max Memory Used: 79 MB	
START RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465 Version: $LATEST
2018-04-04T06:29:38.914Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Parsing S3 event...
2018-04-04T06:29:38.915Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Downloading: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66.png
2018-04-04T06:29:39.058Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:29:39.498Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Reducing to: in-place
2018-04-04T06:29:39.503Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Uploading to: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_sm.png (799 bytes)
2018-04-04T06:29:39.560Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:29:40.059Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Reducing to: in-place
2018-04-04T06:29:40.098Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Uploading to: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_md.png (1550 bytes)
2018-04-04T06:29:40.200Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:29:40.201Z	68facee4-37d1-11e8-b9ff-4d43a4e05465
{
    "errorMessage": "Error processing demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66.png: TypeError: Cannot read property '1' of null"
}

END RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465
REPORT RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465	Duration: 1288.17 ms	Billed Duration: 1300 ms Memory Size: 1280 MB	Max Memory Used: 79 MB	
START RequestId: 8dd885bd-37d1-11e8-9c60-e3f5b738ea44 Version: $LATEST
2018-04-04T06:29:40.298Z	8dd885bd-37d1-11e8-9c60-e3f5b738ea44	Parsing S3 event...
2018-04-04T06:29:40.299Z	8dd885bd-37d1-11e8-9c60-e3f5b738ea44	Downloading: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_md.png
2018-04-04T06:29:40.352Z	8dd885bd-37d1-11e8-9c60-e3f5b738ea44	Image already processed
END RequestId: 8dd885bd-37d1-11e8-9c60-e3f5b738ea44
REPORT RequestId: 8dd885bd-37d1-11e8-9c60-e3f5b738ea44	Duration: 54.72 ms	Billed Duration: 100 ms Memory Size: 1280 MB	Max Memory Used: 79 MB	
START RequestId: 23285c0e-37d1-11e8-87aa-49521f6c7411 Version: $LATEST
2018-04-04T06:29:59.536Z	23285c0e-37d1-11e8-87aa-49521f6c7411	Parsing S3 event...
2018-04-04T06:29:59.536Z	23285c0e-37d1-11e8-87aa-49521f6c7411	Downloading: demo/95aadd5c-5c1a-4f08-9280-5e915ba5ee5a.png
2018-04-04T06:29:59.623Z	23285c0e-37d1-11e8-87aa-49521f6c7411
{
    "errorMessage": "Error processing demo/95aadd5c-5c1a-4f08-9280-5e915ba5ee5a.png: S3 getObject failed: NoSuchKey: The specified key does not exist."
}

END RequestId: 23285c0e-37d1-11e8-87aa-49521f6c7411
REPORT RequestId: 23285c0e-37d1-11e8-87aa-49521f6c7411	Duration: 88.17 ms	Billed Duration: 100 ms Memory Size: 1280 MB	Max Memory Used: 79 MB	
START RequestId: 49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1 Version: $LATEST
2018-04-04T06:31:01.128Z	49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1	Parsing S3 event...
2018-04-04T06:31:01.129Z	49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1	Downloading: demo/83d32285-b4c3-4b18-ba8d-a5d7fb747ec0.png
2018-04-04T06:31:01.183Z	49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1
{
    "errorMessage": "Error processing demo/83d32285-b4c3-4b18-ba8d-a5d7fb747ec0.png: S3 getObject failed: NoSuchKey: The specified key does not exist."
}

END RequestId: 49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1
REPORT RequestId: 49ae6ad5-37d1-11e8-a2d2-c7a4ff1b1fe1	Duration: 55.10 ms	Billed Duration: 100 ms Memory Size: 1280 MB	Max Memory Used: 79 MB	
START RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465 Version: $LATEST
2018-04-04T06:31:39.359Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Parsing S3 event...
2018-04-04T06:31:39.372Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Downloading: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66.png
2018-04-04T06:31:39.440Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:31:39.892Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Reducing to: in-place
2018-04-04T06:31:39.898Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Uploading to: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_sm.png (799 bytes)
2018-04-04T06:31:39.948Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:31:40.458Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Reducing to: in-place
2018-04-04T06:31:40.498Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Uploading to: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_md.png (1550 bytes)
2018-04-04T06:31:40.550Z	68facee4-37d1-11e8-b9ff-4d43a4e05465	Resizing to: in-place
2018-04-04T06:31:40.551Z	68facee4-37d1-11e8-b9ff-4d43a4e05465
{
    "errorMessage": "Error processing demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66.png: TypeError: Cannot read property '1' of null"
}

END RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465
REPORT RequestId: 68facee4-37d1-11e8-b9ff-4d43a4e05465	Duration: 1198.72 ms	Billed Duration: 1200 ms Memory Size: 1280 MB	Max Memory Used: 81 MB	
START RequestId: d595ffc6-37d1-11e8-a91b-27569d8fba99 Version: $LATEST
2018-04-04T06:31:40.656Z	d595ffc6-37d1-11e8-a91b-27569d8fba99	Parsing S3 event...
2018-04-04T06:31:40.656Z	d595ffc6-37d1-11e8-a91b-27569d8fba99	Downloading: demo/d09b20c4-c522-42fe-a60f-58fb5ffaea66_md.png
2018-04-04T06:31:40.703Z	d595ffc6-37d1-11e8-a91b-27569d8fba99	Image already processed
END RequestId: d595ffc6-37d1-11e8-a91b-27569d8fba99
REPORT RequestId: d595ffc6-37d1-11e8-a91b-27569d8fba99	Duration: 47.91 ms	Billed Duration: 100 ms Memory Size: 1280 MB	Max Memory Used: 81 MB	```

Answer 4 · 2018-04-04T14:47:55.000Z

BTW, is the package somehow handles the recurring s3 triggering ( I'm reading and writing the same s3 bucket so new generated images can triggers the script again (not sure about this) ? )

Answer 5 · 2018-04-04T15:55:28.000Z

Thanks. And I got it.
Perhaps S3 put event had been triggered even if the uploaded file isn't ready to get.
it's S3 issue, but I'm not sure how S3 deal with store file and triggers the uploaded file.

And yes, your point is right. if this package works on the same bucket, the S3 put event will be triggered again.

To avoid it, I recommend that you should add prefix setting for triggering event files as following steps:

Set prefix as images/ to filter triggering script only under the images/* files
Update your configuration as a processed file upload to /, not includes images/
Upload file to images/ by your program
The script will be triggered only uploaded images/* file, and processed file won't be triggered again.

Totally, we shouldn't set a configuration of destination file as the same directory :)

Answer 6 · 2018-04-04T16:05:08.000Z

Thanks, I will do that and while checking for error I notice you are using regex on crop props. I'm not sure but in my config I set as config: "512X512" but your regex can handle only lowercase like 512x512. I will report If this solves the issue.

Answer 7 · 2018-04-04T16:06:23.000Z

Yes, I just noticed it 😄 I'm happy if you solved it!

Answer 8 · 2018-04-04T16:23:54.000Z

I confirm that using as 512x512 fixes the issue. but now last item on resizes won't be generated. new config

{
    "bucket": "xxx-test",
    "resizes": [
        {
            "size": 128,
            "suffix": "_sm",
            "quality": 90
        },
        {
            "size": 512,
            "suffix": "_md",
            "quality": 90
        },
        {
            "size": "512x512^",
            "gravity": "Center",
            "crop": "512x512",
            "suffix": "_md_sq",
            "quality": 90
        },
        {
            "size": 1024,
            "quality": 90,
            "suffix": "_lg"
        },
        {
            "size": 1440,
            "quality": 90,
            "suffix": "_xl"
        }
    ]
}

Answer 9 · 2018-04-04T16:25:16.000Z

test image 2560x1440 png 18KB

Answer 10 · 2018-04-04T16:26:56.000Z

is package won't generate file if resized dimensions is same with original ?

Answer 11 · 2018-04-04T16:49:48.000Z

No, on my local, test image resized properly.

{
    "size": 1440,
    "quality": 90,
    "suffix": "_xl"
}

2560x1440 image will resize to 1440x822.

Answer 12 · 2018-04-04T16:50:28.000Z

Ok, I'll continue to investigate to new issue, thanks.