Fail to read dir at specified location

[jmercier-lbi]

Dear,

The command below fails with nfs while the native tool hdfs dfs works

 hdfs dfs -ls nfs://xxx.yy.zzz.aa:pppp/datalake/
Found 1 item
drwxrwxrwx   - root root       4096 2019-07-02 17:05 nfs://xxx.yy.zzz.aa:pppp/datalake/user

where xxx.yy.zzz.aa:pppp is an IP

$ hdfs-shell
 _   _  ___    ___    ___           ___    _   _  ___    _      _
( ) ( )(  _`\ (  _`\ (  _`\        (  _`\ ( ) ( )(  _`\ ( )    ( )
| |_| || | ) || (_(_)| (_(_)______ | (_(_)| |_| || (_(_)| |    | |
|  _  || | | )|  _)  `\__ \(______)`\__ \ |  _  ||  _)_ | |  _ | |  _
| | | || |_) || |    ( )_) |       ( )_) || | | || (_( )| |_( )| |_( )
(_) (_)(____/'(_)    `\____)       `\____)(_) (_)(____/'(____/'(____/'
                                                   by Vity (2017-2019)


Welcome to HDFS-shell CLI
foo@bar.org /home/foo$ ls nfs://xxx.yy.zzz.aa:pppp/datalake/
ls: No FileSystem for scheme: nfs

[Vity01]

Do you use same Hadoop client version + same configuration (HADOOP home dir) for both hdfs-shell (2.6 by default) and the hdfs dfs client?
I am not able to reproduce it.

[jmercier-lbi]

Thanks @Vity01 for your quick answer,

I made some test and now able to provides more information.

So If I build the tools with these 2 new lines after line 184

conf.addResource(new Path("file:///etc/hadoop/conf/core-site.xml")); 
conf.addResource(new Path("file:///etc/hadoop/conf/hdfs-site.xml"));

I have the real error:

Command failed java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.netapp.fs.nfs.NFSv3FileSystem not found
java.lang.ClassNotFoundException: Class org.apache.hadoop.netapp.fs.nfs.NFSv3FileSystem not found

Indeed we use a netApp technology to communicate with nfs through hdfs.
Thus we have a lib named: hadoop-hdfs-nfs-3.1.1.3.1.0.0-78.jar

So should I rebuild hdfs-shell in order to use this library ?
by inserting the below gradle code int build.gradle ?

configurations{
    //declare custom config if necessary, otherwise just use compile
    myLibs
}
dependencies {
    //add lib/*.jar files to myLibs
    myLibs fileTree(dir: 'lib', include: '*.jar')
    compile {
        //set compile configuration to extend from myLibs
        extendsFrom myLibs
    }
}
// task to copy libs to output/lib dir
task copyToLib(type: Copy) {
    into "$buildDir/output/lib"
    from configurations.myLibs
}

jar {
    //include contents of output dir
    from "$buildDir/output"
    manifest {
        //...
    }
}

//set build task to depend on copyToLib
build.dependsOn(copyToLib)

Thanks

Have a nice day

[Vity01]

Hmms, I would try to rebuild that with a hadoop client 3.1.1 which is obviously version you are using on your system.
In the build.gradle just replace this line:
compile "org.apache.hadoop:hadoop-client:2.6.0" with compile "org.apache.hadoop:hadoop-client:3.1.1"
Rebuild it with gradlew build.

Make sure your HADOOP_CONF_DIR directory is set on your system (via echo $HADOOP_CONF_DIR default value is /etc/hadoop/conf.
Then run hdfs-shell.

[Vity01]

Can I close the issue?

[jmercier-lbi]

I am working on this issue:
1/ Using HADOOP_CONF_DIR do not help , I continue to investigate...
2/ about nfs I solved it by compiling with jar provided by NetApp

diff --git a/build.gradle b/build.gradle
index c451291..c737a98 100644
--- a/build.gradle
+++ b/build.gradle
@@ -80,6 +80,8 @@ dependencies {
     compile "org.apache.commons:commons-lang3:3.3.2"
     compile "org.apache.hadoop:hadoop-client:2.6.0"

+   compile fileTree(include: ['*.jar'], dir: 'libs')
+
     testCompile 'junit:junit:4.12'
     testCompile("org.springframework.boot:spring-boot-starter-test:1.4.2.RELEASE")
 }

$ ls libs
hadoop-nfs-2.7.1.jar hadoop-nfs-connector-3.0.1.jar

3/ I am looking how to provides to hdfs-shell a Kerberos token

$ hdfs-shell
foo@domain $ ls
ls: SIMPLE authentification is not enabled. Available:[TOKEN, KERBEROS]

Thanks
Have a nice day