mmolimar/kafka-connect-fs

Question - File is not getting moved /delete after processing

iamrc90 opened this issue · 1 comments

I am trying to move the csv file to another folder after processing is done. I can see it being processed to kafka topic but the file is not moved to the given path. Also delete is not working.

Please find my config below

{
    "name": "csv-source-second-conn8",
    "connector.class": "com.github.mmolimar.kafka.connect.fs.FsSourceConnector",
    "tasks.max": "1",
    "fs.uris": "file:///data/unprocessed",
    "topic": "csv4",
    "policy.class": "com.github.mmolimar.kafka.connect.fs.policy.SimplePolicy",
    "policy.recursive": "true",
    "policy.regexp": ".*",
    "policy.batch_size": "0",
    "policy.cleanup": "delete",
    "file_reader.class": "com.github.mmolimar.kafka.connect.fs.file.reader.CsvFileReader",
    "file_reader.batch_size": "0",
    "file_reader.delimited.settings.header": "true",
    "file_reader.delimited.settings.line_separator_detection": "true",
    "file_reader.delimited.settings.delimiter_detection": "true"
}
{
    "name": "csv-source-second-conn7",
    "connector.class": "com.github.mmolimar.kafka.connect.fs.FsSourceConnector",
    "tasks.max": "1",
    "fs.uris": "file:///data/unprocessed",
    "topic": "csv4",
    "policy.class": "com.github.mmolimar.kafka.connect.fs.policy.SimplePolicy",
    "policy.recursive": "true",
    "policy.regexp": ".*",
    "policy.batch_size": "0",
    "policy.cleanup": "move",
    "policy.cleanup.move": "file:///data/processed",
    "file_reader.class": "com.github.mmolimar.kafka.connect.fs.file.reader.CsvFileReader",
    "file_reader.batch_size": "0",
    "file_reader.delimited.settings.header": "true",
    "file_reader.delimited.settings.line_separator_detection": "true",
    "file_reader.delimited.settings.delimiter_detection": "true"
}

Please suggest if anything needs to be modified.

@mmolimar - Could you please suggest ?