compwright/aws-elasticsearch-connector

Support for IAM Roles?

mramato opened this issue · 15 comments

mramato commented

First thanks for trying to pick up https://github.com/TheDeveloper/http-aws-es and update it to the latest ES npm module.

We use IAM Roles to control permissions on our servers and therefore never need to provide an access key or secret because all of the AWS libraries/calls pick up the role and it "just works".

This was supported in http-aws-es but does not appear to be supported here. Is it just a matter of allowing credentials to be undefined and passing everything through without them?

Thanks!

mramato commented

I think I figured it out. The AWS credentials, though automatic when using aws-sdk, were not initialized at the time aws-elasticsearch-connector was being instantiated. If I wrap the call in a AWS.config.getCredentials myself, then the credentials are initialized and everything appears to work. Perhaps there's a better way to do it? Or an update to the readme for this use case is all that's needed?

Thanks again.

  • 👍1
compwright commented

@mramato that does sound like a workable solution. However, v8.1.1 of this library should handle refreshed credentials automatically. Feel free to take it for a spin, and re-open this if you still have the issue.

mramato commented

Unfortunately, this didn't fix the issue for me. It think the problem is that if you try to use ES before any unrelated AWS calls, there is no config object loaded at all. I think it was an incremental improvement to always use the latest credentials, but without making a explicit getCredentials request, there's no guarantee that it they exist. Here's the callstack.

TypeError: Cannot read property 'accessKeyId' of null
    at AmazonConnection.get credentials [as credentials] (/var/app/node_modules/aws-elasticsearch-connector/src/index.js:13:47)
    at AmazonConnection.buildRequestObject (/var/app/node_modules/aws-elasticsearch-connector/src/index.js:37:32)
    at AmazonConnection.request (/var/app/node_modules/@elastic/elasticsearch/lib/Connection.js:58:32)
    at makeRequest (/var/app/node_modules/@elastic/elasticsearch/lib/Transport.js:179:30)
    at Transport.request (/var/app/node_modules/@elastic/elasticsearch/lib/Transport.js:301:15)
    at Promise (/var/app/node_modules/@elastic/elasticsearch/lib/Transport.js:75:14)
    at new Promise (<anonymous>)
    at Transport.request (/var/app/node_modules/@elastic/elasticsearch/lib/Transport.js:74:14)
    at msearch (/var/app/node_modules/@elastic/elasticsearch/api/api/msearch.js:114:12)
    at Client._lazyLoad [as msearch] (/var/app/node_modules/@elastic/elasticsearch/api/index.js:510:12)
compwright commented

@mramato can you give me a little more context so that I reproduce this issue correctly? I'd like to see how you are initializing your ES client, and this library. Are you passing in awsConfig?

Also:

  • What version of the ES client are you using?
  • What version of the aws-sdk are you using?
mramato commented

In the working version, I'm doing:

this._esClient = new Promise((resolve, reject) => {
    AWS.config.getCredentials(function (err) {
        if (err) {
            reject(err);
            return;
        }
        resolve(new elasticsearch.Client({
            node: 'xxxxx',
            Connection: AwsConnection,
            awsConfig : { credentials: AWS.config.credentials }
        }));
    });
});

But as you noted, these credentials may expire. I'm also pretty sure if I leave off awsConfig, the above works as well because your code just grabs AWS.config by default anyway.

What I would like to do is the below, but that leads to the call stack I pasted above because AWS.config is undefined without my initial getCredentials call.

    this._esClient = new elasticsearch.Client({
        node: 'xxxxx',
        Connection: AwsConnection
    });
  • What version of the ES client are you using?

@elastic/elasticsearch@6.8.2

  • What version of the aws-sdk are you using?

aws-sdk@2.519.0

compwright commented

@mramato I think your problem is this line:

{
  awsConfig : { credentials: AWS.config.credentials }
}

By passing in an awsConfig object with explicit credentials, you are preventing AmazonConnection.credentials from seeing the updated credentials which exist on AWS.config.

Just delete that line and let it read from AWS.config, and you should be good to go. It may also work to pass AWS.config in explicitly like this:

{
  awsConfig: AWS.config
}
compwright commented

Also I should note that waiting for AWS.config.getCredentials() will still be necessary if you are getting your credentials from an async instead of a static source, such as reading from an IAM role. See #4 (comment) for an explanation of why this is the case.

I'm glad you were able to get it working! Thanks for all the feedback.

mramato commented

@compwright Unfortunately this still isn't a solution for me. Everything works up front but eventually the credentials go stale. Here's what I have now:

    this._esClient = new Promise((resolve, reject) => {
        // AWS ElasticSearch requires we sign our connections,
        // which we do with the help of a Utility class and
        // connections retrieved from the config.
        AWS.config.getCredentials(function (err) {
            if (err) {
                reject(err);
                return;
            }

            resolve(new elasticsearch.Client({
                node: undefined,
                Connection: AwsConnection
            }));
        });
    });

Which works when the app first starts up, but stops working once the credentials expire.

It looks like to use this library I need to call getCredentials before every elasticSearch client call, not just up front. Which is annoying, but may be the only solution if there's no way for a Connection to perform asynchronous operations.

mramato commented

I dug more into this. The fundamental issue (as you pointed out up thread) is that there doesn't appear to be a way to implement buildRequestObject asynchronously. However, the request method is asynchronous and provides a callback, so you can perform the getCredentials in request and then use them in buildRequestObject.

Below is a re-written implementation showing what I mean. It should work in all cases and there's no need to ever supply credentials manually because it just gets them from the library as-needed. They should never be stale because it retrieves them with every request.

class AmazonConnection extends Connection {
    request(params, callback) {
        AWS.config.getCredentials(function (err) {
            if (err) {
                callback(err, null);
                return;
            }
            const credentials = AWS.config.credentials;
            params.credentials = {
                accessKeyId: credentials.accessKeyId,
                secretAccessKey: credentials.secretAccessKey,
                sessionToken: credentials.sessionToken
            };
        });
        return super.request(params, callback);
    }

    buildRequestObject(params) {
        const req = super.buildRequestObject(params);

        if (!req.headers) {
            req.headers = {};
        }

        // Fix the Host header, since HttpConnector.makeReqParams() appends
        // the port number which will cause signature verification to fail
        req.headers.Host = req.hostname;

        if (params.body) {
            req.headers['Content-Length'] = Buffer.byteLength(params.body, 'utf8');
            req.body = params.body;
        } else {
            req.headers['Content-Length'] = 0;
        }

        return aws4.sign(req, params.credentials);
    }
}
mramato commented

Actually, I found some issues with the above code (worked by accident first time I ran it). Looking into whether the approach is still possible or not. Sorry for the noise.

compwright commented

No worries! Looks like you're on the right track. I'd love to get a PR from you to fix this for good if you can find a way.

mramato commented

Thanks, unfortunately it looks like we are hitting a limitation of the library. The request method is not async in the way I thought it it was. It requires you to return the request object synchronously and it only calls the callback when the entire request is finished. So everything before "make the request" is synchronous. You would have to do something hacky like intercept the req object at the node level or something.

As a short term workaround, I started to request credentials before every request. Hopefully that will fix the issue. I plan on asking the ES team if there is something we are missing or if they are open to a PR to add async request building to the library.

compwright commented

Sounds good. I thought about possibly trying to wrap the stream inside another stream which would first grab the latest credentials before executing the request stream, but I'm not sure I want to go down that road or that it is worth it.

So I understand your use case, why do your credentials expire so often? Are you running server-side or in browser? Are using static credentials an option?

compwright commented

@mramato this is a new library that attempts to solve the renewing credentials problem. This is a different approach I haven't thought of. Interested to know if it works for you or not: https://github.com/mergermarket/acuris-aws-es-connection

compwright commented

Resolved by #2