SETL-Framework/setl

FileConnector doesn't list path recursively when there are subdirectories inside the given path

qxzzxq opened this issue · 0 comments

Describe the bug
In such a folder structure, file connector's listFiles method will not return all the files. It only returns the path of file1.csv
image

To Reproduce

val spark: SparkSession = new SparkSessionBuilder().setEnv("local").build().get()
val connector: FileConnector = new FileConnector(Map[String, String](
  "path" -> "src/test/resources/test-list-files",
  "inferSchema" -> "true",
  "header" -> "true",
  "saveMode" -> "Overwrite",
  "storage" -> "CSV"
)) {
  override val storage: Storage = Storage.CSV
}
println("files:")
connector.listFiles().foreach(println)
println("pathes")
connector.listPaths().foreach(println)
println("pathes to load")
connector.listFilesToLoad(false).foreach(println)

Expected behavior
All the files should be listed recursively.