mmolimar/kafka-connect-fs

Unable to connect to remote host

andy-agap opened this issue · 3 comments

Whenever I supply the "fs.uri" for the connector, I am limited to the local file system with a file URI of "file:///...", otherwise it throws an illegal argument exception:

java.lang.IllegalArgumentException: Wrong FS: file://hostname/path expected: file:///
...

Is there any way to supply a URI that points to an external file system? (e.g., remote host)

Have looked at similar posts:
https://stackoverflow.com/questions/32078441/wrong-fs-expected-file-when-trying-to-read-file-from-hdfs-in-java

Do I need to supply additional configuration as above in the class files?

Hi!
Yes, the connector can connect to "any" kind of file systems (local or remote). For instance, HDFS, S3, GCP, FTP, etc. You only need to set the URI and the custom configurations for the file system you want to connect to.

In case you want to connect to HDFS, the URI would be something like hdfs://host:port/dir or using the corresponding prefix for your FS.

Thanks for the confirmation! Am I able to connect to an NFS mounted onto local using file:/// and LocalFileSystem?

Sure!