YARN memory default vritual memory settings limits
Hokan-Ashir opened this issue · 0 comments
Hokan-Ashir commented
If you would like to write custom YARN application, i.e. via Spring Boot, using this manual (https://spring.io/guides/gs/yarn-basic/) you will face couple of errors:
- One issue about guide.
Make sure you have corresponding versions of "spring-yarn-boot", "spring-boot-starter-parent" and "spring-boot-maven-plugin" in pom-files. Simpliest solution go to maven central and use latest versions.
Otherwise you may face "cannot instantiate 'yarnClient' bean" error, which tell you almost nothing why it failed - Another about this docker image.
Make sure you set limitless virtual memory for YARN.
For example, you may set in yarn-site.xml this properties:
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
<description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4</value>
<description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>
Cause you use it in Docker, you have to change it somehow after hadoop cluster initiation, or before, mapping volumes with "-v" option
Another solution is to use fork of this docker image, which has this settings by default, i.e. - https://github.com/sfedyakov/hadoop-271-cluster