SFTP Connectivity with Private key

[tmanipra]

Hi,

I'm trying to access Source System using SFTP with Username & Passwordless Entry(Private key). But I'm getting AUTH Failure with below settings.

Even i tried, "fs.sftp.keyfile". but no luck.

Here is my Property file:

name=SourceConnector
connector.class=com.github.mmolimar.kafka.connect.fs.FsSourceConnector
tasks.max=1
policy.fs.fs.sftp.impl=org.apache.hadoop.fs.sftp.SFTPFileSystem
fs.uris=sftp://username:@hostname:22/home/user/output/
fs.sftp.keyfile=/home/user/.ssh/id_rsa
topic=sampletopic
policy.class=com.github.mmolimar.kafka.connect.fs.policy.CronPolicy
policy.recursive=true
file_reader.delimited.settings.data_type_mapping_error=false
file_reader.delimited.settings.allow_nulls=true
policy.regexp=^SOURCE_1.*.gz$
policy.batch_size=0
policy.cleanup=none
file_reader.class=com.github.mmolimar.kafka.connect.fs.file.reader.CsvFileReader
file_reader.batch_size=3000
policy.cron.expression=0/30 * * ? * * *
file_reader.delimited.compression.type=gzip

Please help me to connect with private key. Thanks

[tmanipra]

Error code:
ERROR FsSourceTask Cannot retrieve files to process from the FS: [[]]. There was an error executing the policy but the task tolerates this and continues: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:519)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:169)
at org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:151)
at org.apache.hadoop.fs.sftp.SFTPFileSystem.listStatus(SFTPFileSystem.java:623)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1941)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1983)
at org.apache.hadoop.fs.FileSystem$4.(FileSystem.java:2149)
at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2131)
at org.apache.hadoop.fs.FileSystem$5.(FileSystem.java:2255)
at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:2252)
at com.github.mmolimar.kafka.connect.fs.policy.AbstractPolicy$1.(AbstractPolicy.java:159)
at com.github.mmolimar.kafka.connect.fs.policy.AbstractPolicy.listFiles(AbstractPolicy.java:158)
at com.github.mmolimar.kafka.connect.fs.policy.CronPolicy.listFiles(CronPolicy.java:22)
at com.github.mmolimar.kafka.connect.fs.policy.AbstractPolicy.execute(AbstractPolicy.java:137)
at com.github.mmolimar.kafka.connect.fs.FsSourceTask.filesToProcess(FsSourceTask.java:125)
at com.github.mmolimar.kafka.connect.fs.FsSourceTask.poll(FsSourceTask.java:84)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:259)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:226)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
(com.github.mmolimar.kafka.connect.fs.FsSourceTask:129)

[tmanipra]

Hi, After a struggle for a week now, I'm able to resolve it with below config.

policy.fs.fs.sftp.keyfile=/home/user/.ssh/id_rsa