S3 Multi-Region Access Point(MRAP) is not available unless with additional dependency

Question

S3 Multi-Region Access Point(MRAP) is not available unless with additional dependency

AllanZhengYP opened this issue 3 years ago · 22 comments

Description

Package @aws-sdk/client-s3 launched the support for S3 Multi-Region Access Point (MRAP) feature in v3.31.0. This feature relies on signer class from aws-crt package. This dependency requires native addons that takes up to 10.5 MB on disk. Because the dependency is only required by MRAP feature, we are omitting the dependency in @aws-sdk/client-s3.

To use this feature, you need to install this dependency explicitly in your application:

$ npm install @aws-sdk/signature-v4-crt
$ yarn add @aws-sdk/signature-v4-crt

Because aws-crt contains platform-specific binary code, the following use cases are not supported in MRAP:

browsers
React Native
bundle as Node.js application

The suggested use case for supporting MRAP in Node.js app is to deploy your application with the complete node_modules folder.

Describe alternatives you've considered

We are working on solutions to unblock the MRAP for all the supported platforms.

Related issues:

MicrosoftJones commented 10 months ago

Patch

Answer 1 · 2021-10-16T06:09:47.000Z

FYI: One might consider porting the signer class of the common runtime to webassembly. Web assembly is supported in all major browsers and in NodeJs 8+ See also: https://developer.mozilla.org/en-US/docs/WebAssembly/C_to_wasm

Kind regards,
Hans

Answer 2 · 2021-10-26T17:04:36.000Z

Thank you for the advice! @seriousme

We are currently weigh in the options including WASM, ASM.js, and pure JS with WebCrypto. Currently one of the obstacle is that WASM seems not supported in all React Native platforms, so we might need to fallback to use ASM.js in React Native.

On the other hand, we need to make sure the signer is cryptographically secure. If the implementation involves 3rd party crypto primitives, we need to audit them. For example, the signing algorithm in SigV4a requires constant time implementation.

Answer 3 · 2021-10-26T18:28:22.000Z

Good to see you have options :-)

You can mix and match, e.g. use the browsers webcrypto from WASM ;-)
e.g. https://ihsavru.medium.com/calling-javascript-code-from-c-c-using-webassembly-a9445c11bc6d

Kind regards,
Hans

Answer 4 · 2021-12-27T20:40:15.000Z

Adding aws-crt blocks me from updating the deno port of aws-sdk-js-v3 past v3.33.0.

👍 for WASM, which would also allow using this in deno.

Answer 5 · 2022-01-26T11:35:22.000Z

Just want to weigh in with the impact this issue currently has on our team. Hopefully it helps to motivate a fix. Our developers use a mix of macOS and Windows machines.

We depend on the @aws-sdk/client-s3 package. This causes the package manager to complain about the missing @aws-sdk/signature-v4-crt peer dependency.

The Windows users go ahead and add the dep to package.json, and everything appears to be resolved. Then a macOS user pulls the latest changes and suddenly package installation is completely broken.

This isn't a great experience. Given that @aws-sdk/signature-v4-crt has ~63k downloads per week, I can only imagine how many developers have run headlong into this so far.

Answer 6 · 2022-03-17T06:28:30.000Z

I just ran into this issue, seeing the same peer dependency warnings as @rossng. Using the peerDependenciesMeta field of package.json might be a good solution: https://docs.npmjs.com/cli/v8/configuring-npm/package-json#peerdependenciesmeta

Adding this to package.json in middleware-sdk-s3 should fix this:

"peerDependenciesMeta": {
   "@aws-sdk/signature-v4-crt": {
     "optional": true
   }
 }

Answer 7 · 2022-03-22T06:26:21.000Z

@jekh Thanks a lot for pointing this out! It's a miss on our side. I will address them a separate PR.

Answer 8 · 2022-04-01T19:08:06.000Z

I just tried this with 3.58.0 and I'm still seeing the error on macOS:

WARNING in ../node_modules/@aws-sdk/util-user-agent-node/dist-es/is-crt-available.js 3:96-114
Module not found: Error: Can't resolve 'aws-crt' in '/Users/craig/Documents/PMI/node_modules/@aws-sdk/util-user-agent-node/dist-es'
 @ ../node_modules/@aws-sdk/util-user-agent-node/dist-es/index.js 5:0-52 16:23-37
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/runtimeConfig.js 12:0-65 22:898-914
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/SSMClient.js 10:0-73 15:24-42
 @ ../node_modules/@aws-sdk/client-ssm/dist-es/index.js 2:0-28 2:0-28
...

...where I have the library referenced like so:

    "@aws-sdk/client-ssm": "^3.58.0"

and it's warning on this line:

import { SSM } from "@aws-sdk/client-ssm";

Answer 9 · 2022-08-24T12:05:56.000Z

This issue is affecting our team and our entire organization will run into the same issue soon as we are moving towards a DNS regional failover which requires SigV4A signing in the browser to make API calls to unkown AWS region (using @aws-sdk/signature-v4-crt).

Are there alternatives for the time being? Can we do SigV4A signing with existing parts of the SDK that work on browser?

Answer 10 · 2022-09-09T02:44:19.000Z

This is insane.
No update from Amazon? This has put an entire build on hold.

Minimal documentation for MRAP as it is.

Answer 11 · 2023-01-18T01:10:27.000Z

Still nothing for this?

Answer 12 · 2023-03-30T14:16:48.000Z

any updates on this?

Answer 13 · 2023-04-19T09:47:31.000Z

any updates on this?

We're trying to use apigateway in multiple regions with custom domains with iam authorization.
The request can be routed to different region based on latency, so regular sigV4 doesn't work.

Is there any other workaround to make apigateway works in multiple regions with the same domain name?

Answer 14 · 2023-04-21T12:18:13.000Z

@AllanZhengYP I'm curious does c++ code has any platform specific dependencies?
It seems like the latest version of react native has first class support c++ code with turbomodules. Does the team consider the option to create a separate rn library for that

Answer 15 · 2023-06-21T17:32:39.000Z

Would also like like updates on this. Getting error during build Module not found: Can't resolve '@aws-sdk/signature-v4-crt' on latest version 3.354.0

Answer 16 · 2023-08-17T09:36:53.000Z

I'm getting the following error in CloudWatch, when trying to do a putObjectCommand to an MRAP...

ERROR	Invoke Error 	
{
    "errorType": "TypeError",
    "errorMessage": "Cannot read properties of undefined (reading 'enable_logging')",
    "$metadata": {
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "stack": [
        "TypeError: Cannot read properties of undefined (reading 'enable_logging')",
        "    at new CrtSignerV4 (/var/task/[bundleName].js:41108:22)",
        "    at SignatureV4MultiRegion.getSigv4aSigner (/var/task/[bundleName].js:41056:33)",
        "    at SignatureV4MultiRegion.sign (/var/task/announcementHandlerBundle.js:41029:25)",
        "    at /var/task/[bundleName].js:9155:31",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "    at async /var/task/[bundleName].js:6896:46",
        "    at async /var/task/[bundleName].js:54832:20",
        "    at async /var/task/[bundleName].js:5636:26"
    ]
}

Using the latest S3 and cert lib., in my package.json ...

"@aws-sdk/client-s3": "^3.391.0",
"@aws-sdk/signature-v4-crt": "^3.391.0",

S3Client setup as follows (in side a Ts Lambda function, which works if I just use a standard s3 bucket):

const s3Client = new S3Client({ useGlobalEndpoint: true });

I've tried with and without useGlobalEndpoint; as well as with and without setting the region: 'eu-west-2'; the region the lambda is running in - all the same issue.

NB. Bucket name is set as the MRAP:
arn:aws:s3::123456789:accesspoint/[alias].mrap

Any help appreciated.

Answer 17 · 2023-08-18T08:27:23.000Z

Just read the doesn’t work with bundled nodejs applications; with advice to copy in the node_modules.
That’s not going to be easy

Answer 18 · 2023-09-18T18:26:15.000Z

The AWS SDK for JavaScript team is planning to change the usage of the optional CRT dependency in a future update.

The change is described in this announcement: #5229.

The change is aimed at reducing errors encountered by bundlers and other static analysis tools. The @aws-sdk/signature-v4-crt package remains optional and is only required when using S3 MRAP or EventBridge.

Answer 19 · 2023-10-14T04:59:02.000Z

Description

Package @aws-sdk/client-s3 launched the support for S3 Multi-Region Access Point (MRAP) feature in v3.31.0. This feature relies on signer class from aws-crt package. This dependency requires native addons that takes up to 10.5 MB on disk. Because the dependency is only required by MRAP feature, we are omitting the dependency in @aws-sdk/client-s3.

To use this feature, you need to install this dependency explicitly in your application:
$ npm install @aws-sdk/signature-v4-crt
$ yarn add @aws-sdk/signature-v4-crt
Because aws-crt contains platform-specific binary code, the following use cases are not supported in MRAP:

browsers

React Native

bundle as Node.js application

The suggested use case for supporting MRAP in Node.js app is to deploy your application with the complete node_modules folder.

Describe alternatives you've considered

We are working on solutions to unblock the MRAP for all the supported platforms.

Related issues:

@aws-sdk/client-s3 makes node_modules too large #2747

@aws-sdk/client-s3 (@aws-sdk/signature-v4-crt) 3.29.0 introduces aws-crt and deprecated dependencies #2750

AWS CRT binary not present in any of the following locations #2753

Missing dependency @aws-sdk/signature-v4-crt after updating to client-s3 3.33.0 #2806

Missing Peer Dependency | @aws-sdk/client-s3 missing @aws-sdk/signature-v4-crt #2960

Answer 20 · 2024-02-09T15:04:39.000Z

So I have spent the last weeks setting up buckets, replication rules, an Multi Region Access Point and I have gotten STS to sign an access token that I have now passed to my client. And here I am, just now finding out that oh btw, multi region access points cannot be queried from the browser. Que? I'm fuming right now.

Isn't that like a very common use case to query s3 from the browser?
At least put this information in some MRAP docs 😢 Very sad to have spent so much time on this.

Please tell me that there is a way to do GetObject from a browser via an mrap.

Answer 21 · 2024-02-23T17:22:03.000Z

anywhere I can find a sample code or documentation how to use @aws-sdk/signature-v4-crt to sign an http request sent to aws service. I'm using aws opensearch client, but it has a gap in covering all apis supported by opensearch (elasticSearch), compared to elasticSearchClient. So I can either switch to elasticSearch client, or keep using openSearchClient, but with a polyfill of sending a few signed http request directly to opensearch service. Either way, I need to work on this signing request. I used to use v2 to sign elasticSearch client request, but I believe it's deprecated now. Any suggestions?