Permission denied
vgmartinez opened this issue · 2 comments
Hi guys,
I installing genie in Elasticbeanstalk and use EMR cluster, but i have a problem when execute a spark-submit command.
Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=tomcat, access=WRITE, inode="/user/tomcat/.sparkStaging/application_1493729336953_0001":hdfs:hadoop:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:320) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1728) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1712) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1695) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3896) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
Thx
Hi @vgmartinez,
This looks like because your spark client process is being kicked off as the "tomcat" user it doesn't have permission in HDFS to write data into the directory you're trying to write to. If you're going to have genie run all jobs as user "tomcat" you'll need to grant it write access to your HDFS. Other option is to enable genie to run jobs as user (this requires the tomcat user to have sudo access on the box so it can create the user if they don't exist). This assumes you have users in your hadoop system as well that would match.
Tom
Hi @tgianos,
I have solved it by creating the tomcat user in hdfs (EMR) using two steps in EMR one to create the /user/tomcat directory and another to give tomcat user write permission.
hadoop fs -mkdir /user/tomcat
hadoop fs -chown tomcat /user/tomcat
This works.
Thanks for support.
Victor