pnxtech/hydra

Running Hydra on AWS Elasticache

waglik opened this issue · 23 comments

I am trying to run Hydra app with AWS Elasticache but unfortunately I got

ReplyError: ERR unknown command 'config'

error. It is caused by the fact that config command is restricted on AWS ElastiCache. Is there a way to start hydra without calling config command?

cjus commented

@waglik where do you see that error? Not sure I follow. Please provide more information on how you're running your application that uses Hydra and perhaps an example of your config file.

@cjus I've created simple hydra application and then wrapped it into Docker image. When I run it locally on Redis running on local host it is fine. But when I run Docker image on AWS ESC it fails when starting. Reason is that AWS ElastiCache (this is my Redis instance) does not allow CONFIG command.

here is my config file

{ "environment": "development", "hydra": { "serviceName": "facebook-service", "serviceIP": "", "servicePort": 0, "serviceType": "facebook", "serviceDescription": "Fetch facebook comments", "plugins": { "logger": { "logRequests": false } }, "redis": { "url": "<url to AWS ElastiCache>", "port": 6379, "db": 15 } } }

Error happens when npm start is called. My guess is that it is happening here :

config.init('./config/config.json')

cjus commented

@waglik one thing you could try is backing the config into the service - not ideal - but good for seeing whether this is the only issue.

Just create a JS object for your config and pass it into the hydra.init() call. Are you using hydra or hydraExpress? If hydraExpress then just pass the JavaScript object representing your config to hydraExpress.init.
https://github.com/cjus/hello-service/blob/master/hello-service.js#L19

cjus commented

@waglik also, are you also using AWS Elastic Container Service? ECS?

@cjus I am using just hydra (not hydraExpress). I've added couple of loggers so I am pretty confident that this fails in the init method. If I remove config.init() it works fine.

Yes I am using ECS. I also checked connection with telnet so this is not an issue.

Stack tracke :


events.js:160
      throw er; // Unhandled 'error' event
      ^
ReplyError: ERR unknown command 'config'
    at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:193:12)
    at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:303:14)

npm info lifecycle facebook-service@0.0.2~start: Failed to exec start script
npm ERR! Linux 4.4.51-40.58.amzn1.x86_64
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
npm ERR! node v6.3.1
npm ERR! npm  v3.10.3
npm ERR! code ELIFECYCLE

cjus commented

@waglik Are you saying that if you don't use config.init() to load a config file then you don't have an issue?

Does your redis url looking something like this? test-dev-redis1.p42rev.0001.usw1.cache.amazonaws.com and do you have a password on it?

You can try to format your config this way:

{
  redis: {
    url: 'redis://test-dev-redis1.p42rev.0001.usw1.cache.amazonaws.com:6379/15'
  }
}

Let me know if that makes a difference.

@cjus I've added some more logging. Code looks like that :

console.log('here');
config.init('./config/config.json')
  .then(() => {
    console.log('here too 2');
    config.version = version;
    config.hydra.serviceVersion = version;
    /**
    * Initialize hydra
    */
    return hydra.init(config);
  })
  .then(() => {console.log('xxx');hydra.registerService();console.log('here 3')})
  .then(serviceInfo => {
    console.log('here 4');
   ....

Output is :

here
here too 2

events.js:160
      throw er; // Unhandled 'error' event
      ^
ReplyError: ERR unknown command 'config'
    at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:193:12)
    at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:303:14)

I'm using xyz-cluster.rmz72x.0001.euc1.cache.amazonaws.com cluster link and in config.json :

"redis": {
      "url": "redis://xyz-cluster.rmz72x.0001.euc1.cache.amazonaws.com:6379/15"
    }
  }

I'm not sure why it is trying to execute a CONFIG command. I couldn't find any calls to it in the node-redis module source, and we're not calling it directly from Hydra or Hydra Express. I can't see how it could have anything to do with loading config.json. @waglik, your code and config look OK to me. We're also using AWS Elasticache and haven't run into this issue.

As a test to help narrow this down, could you try connecting to redis directly after config.init? E.g. something like:

const redis = require('redis');
const config = require('fwsp-config');
config.init('./config/config.json')
  .then(() => {
    let client = redis.createClient(config.hydra.redis);
    client.time(redis.print);
  });

That should print out two values, e.g.

$ node testredis
Reply: 1493046174,987530
cjus commented

@waglik @emadum I think the issue might be that hydra.init requires a config JSON and not a config object. It's necessary to call hydra.init this way:

hydra.init(config.getObject())

Also see: https://github.com/flywheelsports/fwsp-config/blob/master/index.js#L18

@cjus just tried and gives same result. Beside same docker image works locally against Redis running on my localhost. So I am confident that issue is with AWS Redis and not with the code itself. From what I found AWS Redis does not allow calling CONFIG command. Is hydra calling that command inside init() method?

@waglik did you see my comment above?

@emadum no, sorry.

I've made changes you suggested and I got :

here
/usr/local/bin/node[21]: ../src/pipe_wrap.cc:268:static void node::PipeWrap::Connect(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `args[1]->IsString()' failed.
 1: node::Abort() [node]
 2: node::Assert(char const* const (*) [4]) [node]
 3: node::PipeWrap::Connect(v8::FunctionCallbackInfo<v8::Value> const&) [node]
 4: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [node]
 5: 0x9d5b0b [node]
 6: 0x9d60b1 [node]
 7: 0x23900f10961b
Aborted


code :


console.log('here');

config.init('./config/config.json')
  .then(() => {
    let client = redis.createClient(config.hydra.redis);
    client.time(redis.print);
  });

@waglik what version of node are you using?

cjus commented

@waglik Hydra requires Node 6.2.1 or greater.

@emadum @cjus Docker file says "FROM node:6.3" , it's the one generated by Hydra.

cjus commented

@waglik Would you be interested in doing a skype call? I'm at cjustini34
If so I'd like to find a time when we can step through the issue you're experiencing. I'm in New York.

@cjus I really appreciate that. I've sent you skype invite, let me know when is a good time. I am in EET time zone most of the time.

cjus commented

@waglik accepted. I should be able to do a call during your evenings.

cjus commented

@waglik I'm still interested in a skype call if you have time ;-)

@cjus I am sorry I missed the notifications. I will ping you today. thanks!

cjus commented

An update on this ticket. @waglik and I had a skype call to discuss and I'm in the process of setting up an AWS ECS cluster for testing this issue.

Hi I'm looking into using hydra, it will be running on AWS - I just noticed this issue and wanted to check in and see if this is still an ongoing problem? :)
Excited to use Hydra, it looks great!

cjus commented

@j3ddesign I never did get to test this. If you try it let us know what you find and we'll offer support. Recommend you sign up for our slack channel where others are on-hand to weigh in.