cloudydeno/deno-aws_api

S3 api is missing both `getSignedUrl` and `createPresignedPost`

justinmchase opened this issue · 15 comments

justinmchase commented

Here's the link to the javascript api for createPresignedPost.

Its possible that these functions in javascript skd have been manually added these since they're not actually api endpoints. Is there some utility class in here where I could effectively sign urls still?

For context, in case you're not aware, these two singing apis will create a url with an encrypted token in it which you can then hand off to someone else, including a browser, and it can then be used to fetch or upload a file directly from the browser. This is how you'd manage access to private buckets and also its a pretty slick way to handle file uploads without having to go through your api server at all.

justinmchase commented

It looks like the AWSSignerV4 might cover it?

const signer = new AWSSignerV4();
const body = new TextEncoder().encode("Hello World!")
const request = new Request("https://test-bucket.s3.amazonaws.com/test", {
  method: "PUT",
  headers: { "content-length": body.length.toString() },
  body,
});
return await signer.sign("s3", request);
justinmchase commented

It looks like its not quite going to work though its close...

Down in the sign function it has:

const payloadHash = sha256(body ?? new Uint8Array()).hex();
if (service === 's3') {
  headers.set("x-amz-content-sha256", payloadHash);
}

This assumes that a body is provided. In these cases you need to be able to not include the body as part of the signature, because you can't know what the body is in this case. You're giving them a blank check basically to upload whatever they want. I will have the contentType and the contentLength but not the actual content.

Here is the generated code in the node aws-sdk:

function createPresignedPost(params, callback) {
    if (typeof params === 'function' && callback === undefined) {
      callback = params;
      params = null;
    }

    params = AWS.util.copy(params || {});
    var boundParams = this.config.params || {};
    var bucket = params.Bucket || boundParams.Bucket,
      self = this,
      config = this.config,
      endpoint = AWS.util.copy(this.endpoint);
    if (!config.s3BucketEndpoint) {
      endpoint.pathname = '/' + bucket;
    }

    function finalizePost() {
      return {
        url: AWS.util.urlFormat(endpoint),
        fields: self.preparePostFields(
          config.credentials,
          config.region,
          bucket,
          params.Fields,
          params.Conditions,
          params.Expires
        )
      };
    }

    if (callback) {
      config.getCredentials(function (err) {
        if (err) {
          callback(err);
        } else {
          try {
            callback(null, finalizePost());
          } catch (err) {
            callback(err);
          }
        }
      });
    } else {
      return finalizePost();
    }
  }

function preparePostFields(
  credentials,
  region,
  bucket,
  fields,
  conditions,
  expiresInSeconds
) {
  var now = this.getSkewCorrectedDate();
  if (!credentials || !region || !bucket) {
    throw new Error('Unable to create a POST object policy without a bucket,'
      + ' region, and credentials');
  }
  fields = AWS.util.copy(fields || {});
  conditions = (conditions || []).slice(0);
  expiresInSeconds = expiresInSeconds || 3600;

  var signingDate = AWS.util.date.iso8601(now).replace(/[:\-]|\.\d{3}/g, '');
  var shortDate = signingDate.substr(0, 8);
  var scope = v4Credentials.createScope(shortDate, region, 's3');
  var credential = credentials.accessKeyId + '/' + scope;

  fields['bucket'] = bucket;
  fields['X-Amz-Algorithm'] = 'AWS4-HMAC-SHA256';
  fields['X-Amz-Credential'] = credential;
  fields['X-Amz-Date'] = signingDate;
  if (credentials.sessionToken) {
    fields['X-Amz-Security-Token'] = credentials.sessionToken;
  }
  for (var field in fields) {
    if (fields.hasOwnProperty(field)) {
      var condition = {};
      condition[field] = fields[field];
      conditions.push(condition);
    }
  }

  fields.Policy = this.preparePostPolicy(
    new Date(now.valueOf() + expiresInSeconds * 1000),
    conditions
  );
  fields['X-Amz-Signature'] = AWS.util.crypto.hmac(
    v4Credentials.getSigningKey(credentials, shortDate, region, 's3', true),
    fields.Policy,
    'hex'
  );

  return fields;
}

function preparePostPolicy(expiration, conditions) {
  return AWS.util.base64.encode(JSON.stringify({
    expiration: AWS.util.date.iso8601(expiration),
    conditions: conditions
  }));
}
danopia commented

Thanks for the report. As you noted, presigned URLs aren't actually an API call and thus weren't in the scope of this API client codegen effort. I can see the usefulness though and it would make sense to expose the necessary aspects + include an example of making a presigned URL for S3.

justinmchase commented

Do you have any idea of a work around? I'm blocked so hard on this and I cannot figure it out. Presigned URL's are a core feature and I can't seem to unwind their horrible code into a simple function. I'll have to abandon Deno just so I can use the amazon sdk.

All 3 of the Deno projects for the amazon SDK have this same bug where they're generating code off of the json definitions and lack the presigned url apis, its a real bummer.

danopia commented

You can use the real full-fat SDK to presign URLs today, as long as you're comfortable with the flags the main port needs (--unstable --allow-read --allow-env)

import { getSignedUrl } from "https://deno.land/x/aws_sdk@v3.22.0-1/s3-request-presigner/mod.ts";
import { S3Client } from "https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/S3Client.ts";
import { GetObjectCommand } from "https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/commands/GetObjectCommand.ts";
// set the credentials
const client = new S3Client({
  region: "ap-south-1",
  credentials: {
    accessKeyId: 'AKIAANDSOON',
    secretAccessKey: 'thisismysecret',
  },
});
// build the command to presign
const command = new GetObjectCommand({
  Bucket: 'my-bucket',
  Key: 'my/key/is/here',
});
const url = await getSignedUrl(client, command, { expiresIn: 3600 });
  • ❤️1
justinmchase commented

Awesome, I was just trying this out too so its good to see.

Though now that I have the URL I cannot seem to figure out how to use it via curl.

I had this working via the v2 api last time I went to do this but it seems like its different now and its not clear why it doesn't work :(

justinmchase commented 
echo "testing 123" > hello.txt
URL=$(deno run --unstable -A main.ts)
curl "$URL" -T hello.txt

I'm using minio and so I had to add an endpoint and forcePathStyle

import { getSignedUrl } from "https://deno.land/x/aws_sdk@v3.22.0-1/s3-request-presigner/mod.ts";
import { S3Client } from "https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/S3Client.ts";
import { PutObjectCommand } from "https://deno.land/x/aws_sdk@v3.22.0-1/client-s3/commands/PutObjectCommand.ts";
// set the credentials
const client = new S3Client({
  region: "us-east-1",
  endpoint: "http://localhost:9000",
  forcePathStyle: true,
  credentials: {
    accessKeyId: 'AjAOk2gNRU',
    secretAccessKey: 'Wk1HVyV8WP2Nh3O9QfLvTW9dOwR0ysqthZrP2Smf',
  },
});
// build the command to presign
const command = new PutObjectCommand({
  Bucket: 'uploads',
  Key: 'test123',
});
const url = await getSignedUrl(client, command, { expiresIn: 3600 });
console.log(url)

All of the other apis work so the creds are right its just somehow this getSignedUrl is doing something its not expecting that or I have to add some headers that I don't know about, you didn't have to do that in the v2 api...

danopia commented

Good to hear you got somewhere with the official SDK. I would still consider this in-scope to add in this repository somewhere, but I'll let this stay closed unless someone wants to revive the feature request.

dansalias commented

In case it's useful as a reference - I've published a Deno module specifically for creating S3 presigned urls: https://deno.land/x/aws_s3_presign@1.2.1.

Here you can see how S3 presigned URLs relate to signatures: https://github.com/dansalias/aws_s3_presign/blob/trunk/mod.ts#L102-L111.

  • 👍2
danopia commented

Thanks, that looks like a pretty clean and tidy module for anyone who wants to specifically presign S3 URLs!

Given that this codebase is pretty married to a signingFetcher interface which conflates both tasks, I don't think I'll be able to offer any code nearly as concise in /x/aws_api. (Having signingFetcher fetch without signing was pretty workable but signing without a fetch doesn't fit into the types without some refactor. If I eventually work presigning into /x/aws_api it would handle some extra goodies like path-style routing & other AWS partitions, so there'd still be some benefit I suppose.)

Until that happens I'd recommend your /x/aws_s3_presign to anyone else with this usecase. Brief example of using both libraries together:

import {
  DefaultCredentialsProvider,
  getDefaultRegion,
} from "https://deno.land/x/aws_api@v0.5.0/client/credentials.ts";
import {
  getSignedUrl,
} from "https://deno.land/x/aws_s3_presign@1.2.1/mod.ts";

async function presignGetObject(bucket: string, key: string) {
  const credentials = await DefaultCredentialsProvider.getCredentials();
  return getSignedUrl({
    accessKeyId: credentials.awsAccessKeyId,
    secretAccessKey: credentials.awsSecretKey,
    sessionToken: credentials.sessionToken,
    region: credentials.region ?? getDefaultRegion(),

    bucketName: bucket,
    objectPath: `/${key}`,
  });
}

console.log(await presignGetObject('my-bucket', 'my-key'));

This way the credential loading is consistent with the rest of the application.

  • 👍2
dansalias commented

Perfect, thanks for the example. I'm sure it'll prove useful for others. And great work on the Deno ports so far!

  • ❤️1
yogesnsamy commented

@dansalias Currently there's an error in using your module. Could you please have a look at this PR: dansalias/aws_s3_presign#4

danopia commented

🚀 There's now a basic presigner in v0.8.1. It's similar to /x/aws_s3_presign except it uses /x/aws_api's credential fetching and request signing. This presigner is thus async (returns a Promise).

Two different ways of using:

  1. AWSSignerV4 offers presigning given a full URL and this is pretty straightforward but you have to construct the signer with credentials yourself.
import { DefaultCredentialsProvider } from "https://deno.land/x/aws_api@v0.8.1/client/credentials.ts";
import { AWSSignerV4 } from "https://deno.land/x/aws_api@v0.8.1/client/signing.ts";

const credentials = await DefaultCredentialsProvider.getCredentials();
const signer = new AWSSignerV4('us-east-2', credentials);

const url = await signer.presign('s3', {
  method: 'GET',
  url: 'https://my-bucket.s3.amazonaws.com/my-key',
});
  1. New module /extras/s3-presign.ts adds S3-specific presigning logic and constructs credentials and endpoints automatically.
import { getPresignedUrl } from "https://deno.land/x/aws_api@v0.8.1/extras/s3-presign.ts";

const url = await getPresignedUrl({
  region: 'us-east-2',
  bucket: 'my-bucket',
  path: '/my-key',
});
  • 👍1
  • 🎉1
  • ❤️1
dansalias commented

@danopia super useful, thanks for the update!

randallb commented

One other note: I don't think it's possible to sign headers, etc. I'm looking to add metadata to the request, and using the integrated signer, it doesn't look like it supports this. Would this be something folks would be open to me contributing?