FINRAOS/herd

EC2 User Data Limit Exceeded

stevejhall opened this issue · 7 comments

stevejhall commented

When following the demo install shown here: https://github.com/FINRAOS/herd/wiki/demo-install

It appears that the user data exceeds an AWS limit. This is the error I got:
The following resource(s) failed to create: [herdApplicationServer]. . Rollback requested by user.
CREATE_FAILED | AWS::EC2::Instance | herdApplicationServer | User data is limited to 16384 bytes (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 3a...)

A bit of Googling seems to indicate that AWS has an upper limit of 16K

By way of troubleshooting I hacked out a big chunk of the UserData, just to see if I could get past this error. That test was successful. However, I do not fully understand the impacts of doing so.

This is what I deleted
"psql -f herd.postgres.0.01.0-to-0.02.0.upgrade.sql\n",
"psql -f herd.postgres.0.02.0-to-0.03.0.upgrade.sql\n",
"psql -f herd.postgres.0.03.0-to-0.04.0.upgrade.sql\n",
"psql -f herd.postgres.0.04.0-to-0.05.0.upgrade.sql\n",
"psql -f herd.postgres.0.05.0-to-0.06.0.upgrade.sql\n",
"psql -f herd.postgres.0.06.0-to-0.07.0.upgrade.sql\n",
"psql -f herd.postgres.0.07.0-to-0.08.0.upgrade.sql\n",
"psql -f herd.postgres.0.08.0-to-0.09.0.upgrade.sql\n",
"psql -f herd.postgres.0.09.0-to-0.10.0.upgrade.sql\n",
"psql -f herd.postgres.0.10.0-to-0.11.0.upgrade.sql\n",
"psql -f herd.postgres.0.11.0-to-0.12.0.upgrade.sql\n",
"psql -f herd.postgres.0.12.0-to-0.13.0.upgrade.sql\n",
"psql -f herd.postgres.0.13.0-to-0.14.0.upgrade.sql\n",
"psql -f herd.postgres.0.14.0-to-0.15.0.upgrade.sql\n",
"psql -f herd.postgres.0.15.0-to-0.16.0.upgrade.sql\n",
"psql -f herd.postgres.0.16.0-to-0.17.0.upgrade.sql\n",
"psql -f herd.postgres.0.17.0-to-0.18.0.upgrade.sql\n",
"psql -f herd.postgres.0.18.0-to-0.19.0.upgrade.sql\n",
"psql -f herd.postgres.0.19.0-to-0.20.0.upgrade.sql\n",
"psql -f herd.postgres.0.20.0-to-0.21.0.upgrade.sql\n",
"psql -f herd.postgres.0.21.0-to-0.22.0.upgrade.sql\n",
"psql -f herd.postgres.0.22.0-to-0.23.0.upgrade.sql\n",
"psql -f herd.postgres.0.23.0-to-0.24.0.upgrade.sql\n",
"psql -f herd.postgres.0.24.0-to-0.25.0.upgrade.sql\n",
"psql -f herd.postgres.0.25.0-to-0.26.0.upgrade.sql\n",
"psql -f herd.postgres.0.26.0-to-0.27.0.upgrade.sql\n",
"psql -f herd.postgres.0.27.0-to-0.28.0.upgrade.sql\n",
"psql -f herd.postgres.0.28.0-to-0.29.0.upgrade.sql\n",
"psql -f herd.postgres.0.29.0-to-0.30.0.upgrade.sql\n",
"psql -f herd.postgres.0.30.0-to-0.31.0.upgrade.sql\n",
"psql -f herd.postgres.0.31.0-to-0.32.0.upgrade.sql\n",
"psql -f herd.postgres.0.32.0-to-0.33.0.upgrade.sql\n",
"psql -f herd.postgres.0.33.0-to-0.34.0.upgrade.sql\n",
"psql -f herd.postgres.0.34.0-to-0.35.0.upgrade.sql\n",
"psql -f herd.postgres.0.35.0-to-0.36.0.upgrade.sql\n",
"psql -f herd.postgres.0.36.0-to-0.37.0.upgrade.sql\n",
"psql -f herd.postgres.0.37.0-to-0.38.0.upgrade.sql\n",
"psql -f herd.postgres.0.38.0-to-0.39.0.upgrade.sql\n",
"psql -f herd.postgres.0.39.0-to-0.40.0.upgrade.sql\n",
"psql -f herd.postgres.0.40.0-to-0.41.0.upgrade.sql\n",
"psql -f herd.postgres.0.41.0-to-0.42.0.upgrade.sql\n",
"psql -f herd.postgres.0.42.0-to-0.43.0.upgrade.sql\n",
"psql -f herd.postgres.0.43.0-to-0.44.0.upgrade.sql\n",
"psql -f herd.postgres.0.44.0-to-0.45.0.upgrade.sql\n",
"psql -f herd.postgres.0.45.0-to-0.46.0.upgrade.sql\n",
"psql -f herd.postgres.0.46.0-to-0.47.0.upgrade.sql\n",
"psql -f herd.postgres.0.47.0-to-0.48.0.upgrade.sql\n",
"psql -f herd.postgres.0.48.0-to-0.49.0.upgrade.sql\n",
"psql -f herd.postgres.0.49.0-to-0.50.0.upgrade.sql\n",
"psql -f herd.postgres.0.50.0-to-0.51.0.upgrade.sql\n",
"psql -f herd.postgres.0.51.0-to-0.52.0.upgrade.sql\n",
"psql -f herd.postgres.0.52.0-to-0.53.0.upgrade.sql\n",
"psql -f herd.postgres.0.53.0-to-0.54.0.upgrade.sql\n",
"psql -f herd.postgres.0.54.0-to-0.55.0.upgrade.sql\n",
"psql -f herd.postgres.0.55.0-to-0.56.0.upgrade.sql\n",
"psql -f herd.postgres.0.56.0-to-0.57.0.upgrade.sql\n",
"psql -f herd.postgres.0.57.0-to-0.58.0.upgrade.sql\n",
"psql -f herd.postgres.0.58.0-to-0.59.0.upgrade.sql\n",
"psql -f herd.postgres.0.59.0-to-0.60.0.upgrade.sql\n",
"psql -f herd.postgres.0.60.0-to-0.61.0.upgrade.sql\n",
"psql -f herd.postgres.0.61.0-to-0.62.0.upgrade.sql\n",
"psql -f herd.postgres.0.62.0-to-0.63.0.upgrade.sql\n",
"psql -f herd.postgres.0.63.0-to-0.64.0.upgrade.sql\n",
"psql -f herd.postgres.0.64.0-to-0.65.0.upgrade.sql\n",
"psql -f herd.postgres.0.65.0-to-0.66.0.upgrade.sql\n",
"psql -f herd.postgres.0.66.0-to-0.67.0.upgrade.sql\n",
"psql -f herd.postgres.0.67.0-to-0.68.0.upgrade.sql\n",
"psql -f herd.postgres.0.68.0-to-0.69.0.upgrade.sql\n",
"psql -f herd.postgres.0.69.0-to-0.70.0.upgrade.sql\n",
"psql -f herd.postgres.0.70.0-to-0.71.0.upgrade.sql\n",
"psql -f herd.postgres.0.71.0-to-0.72.0.upgrade.sql\n",
"psql -f herd.postgres.0.72.0-to-0.73.0.upgrade.sql\n",
"psql -f herd.postgres.0.73.0-to-0.74.0.upgrade.sql\n",
"psql -f herd.postgres.0.74.0-to-0.75.0.upgrade.sql\n",
"psql -f herd.postgres.0.75.0-to-0.76.0.upgrade.sql\n",
"psql -f herd.postgres.0.76.0-to-0.77.0.upgrade.sql\n",
"psql -f herd.postgres.0.77.0-to-0.78.0.upgrade.sql\n",
"psql -f herd.postgres.0.78.0-to-0.79.0.upgrade.sql\n",
"psql -f herd.postgres.0.79.0-to-0.80.0.upgrade.sql\n",
"psql -f herd.postgres.0.80.0-to-0.81.0.upgrade.sql\n",

Ultimately, the CloudFormation script still fails:
CREATE_FAILED | AWS::CloudFormation::WaitCondition | herdServerWaitCondition | WaitCondition timed out. Received 0 conditions when expecting 1
  | 14:52:36 UTC-0700 | CREATE_IN_PROGRESS | AWS::CloudFormation::WaitCondition | herdServerWaitCondition

How can I help restructure this go get around this error. Happy to contribute to the project, just need some idea how I could do so.

nateiam commented
mrperson2015 commented

As a update, I ran through the demo install steps using herd-scripts-cloud-formation-0.88.0.jar. This issue is still very much open.

I am more interested in the data catalog piece than anything else. Our storage platform is not S3. This will catalog an event store in another system.

ssaha01 commented

Nate,
Following your suggestion, I am trying to install herd-mdl through the provided installMDL.yml file. The installation is failing with the following error:

2019-07-01 12:11:23 UTC-0400 ice-poc-herd-mdl ROLLBACK_IN_PROGRESS The following resource(s) failed to create: [MdlStack]. . Rollback requested by user.
2019-07-01 12:11:23 UTC-0400 MdlStack CREATE_FAILED Embedded stack arn:aws:cloudformation:us-east-1:681636940503:stack/ice-poc-herd-mdl-MdlStack-16AJ6J361EUU6/ca1a8850-9c1a-11e9-8652-0eb49284c068 was not successfully created: The following resource(s) failed to create: [PrerequisitesPrimaryStack].

I tried to get the PowerUserAccess policy, but the link provided to get the sample policy is giving 404. Can you please help?

nateiam commented

Hi @ssaha01 -

Glad you are looking at Herd! The best course of action here is to use the Herd-MDL install as discussed in the comment above. We also recommend setting 'DeployComponents' to 'Herd' -- unless you are interested in the integrated Hive Metastore and Presto cluster.

@rongwang is an engineer from our team and she can dig into the current resource creation issue.

rongwang0930 commented

@ssaha01 For better assist you, we'd like you retry the creation and send us more detail information.
1.can you retry your stack creation with Rollback on failure set to No (Rollback on failure setting is under advanced section when you create cft from console)?
2. find the actual failed nested stack inside PrerequisitesPrimaryStack, and send us a screenshot of the Events history for the failed nested stack

dlutz2 commented

I found that you can get below the EC2 user data limit by removing the parts of herd.localdb.template that loads the demo data: all the curl POSTs between lines 633-729.
That allows the CloudFormation script to run but herd-app doesn't start. Tomcat logs show a permission denied exception when trying to write "velocity.log". Caused by: java.io.FileNotFoundException: velocity.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_242]
at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_242]
at java.io.FileOutputStream.(FileOutputStream.java:213) ~[?:1.8.0_242]
at java.io.FileOutputStream.(FileOutputStream.java:133) ~[?:1.8.0_242]
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.FileAppender.(FileAppender.java:110) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.RollingFileAppender.(RollingFileAppender.java:79) ~[log4j-1.2.17.jar:?]
at org.apache.velocity.runtime.log.Log4JLogChute.initAppender(Log4JLogChute.java:118) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.Log4JLogChute.init(Log4JLogChute.java:85) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.LogManager.createLogChute(LogManager.java:157) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.LogManager.updateLog(LogManager.java:269) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeInstance.initializeLog(RuntimeInstance.java:871) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeInstance.init(RuntimeInstance.java:262) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeSingleton.init(RuntimeSingleton.java:112) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.app.Velocity.init(Velocity.java:74) ~[velocity-1.7.jar:1.7]
at org.finra.herd.service.helper.VelocityHelper.(VelocityHelper.java:41) ~[herd-service-0.116.0.jar:?]

Tomcat is apparently trying to write velocity.log in /root. Running Tomcat as root (very bad idea) works. Add to the cft script:
"/bin/sed -i 's/TOMCAT_USER=\"tomcat\"/TOMCAT_USER=\"root\"/g' /usr/share/tomcat8/conf/tomcat8.conf\n",

Also as far as I can tell, there is not a default user/password created for herd-ui, so it is not possible to login.

I am trying to switch to herd-mdl but our restricted environment doesn't allow creation of the required IAM entities, so having to rewrite the CloudFormation scripts to use existing entities. (which we did for the herd scripts but herd-mdl is more complex)

TakumiHaruta commented

I tried to use version 0.136.0 and could pass through User data is limited to 16384 bytes error by replacing the db upgrade script with for loop.

          "psql -f herd.postgres.0.1.0.create.sql\n",
          "psql -f herd.postgres.0.1.0.refdata.sql\n",
          "psql -f herd.postgres.0.1.0.cnfgn.sql\n",
          "for i in {1..136}\n",
          "do\n",
          "  in=`expr $i + 1`\n",
          "  v_old=`printf %03d $i`\n",
          "  v_new=`printf %03d $in`\n",
          "  psql -f herd.postgres.0.${v_old}.0-to-0.${v_new}.0.upgrade.sql\n",
          "done\n",
          "psql -f activiti.postgres.create.engine.sql\n",
          "psql -f activiti.postgres.create.history.sql\n",
          "psql -f activiti.postgres.create.identity.sql\n",
          "psql -f quartz_tables_postgres.sql\n",
          "psql -f elasticsearch.configuration.values.sql\n",

But the UserData script still failed in the end. I got 404 during wget http://localhost:8080/herd-app/rest/buildInfo.
Sorry for the Japanese output, but it means you got 404.

/usr/bin/wget http://localhost:8080/herd-app/rest/buildInfo
--2021-01-19 01:48:49--  http://localhost:8080/herd-app/rest/buildInfo
localhost (localhost) をDNSに問いあわせています... 127.0.0.1
localhost (localhost)|127.0.0.1|:8080 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 404 
2021-01-19 01:48:49 エラー 404: (説明なし)。

I think herd.localdb.template doesn't work anymore. I just want to know what herd is like and don't want to use herd-mdi because it requires multiple resources, so I gave up on it.