FileConnector doesn't list path recursively when there are subdirectories inside the given path
qxzzxq opened this issue · 0 comments
qxzzxq commented
Describe the bug
In such a folder structure, file connector's listFiles method will not return all the files. It only returns the path of file1.csv
To Reproduce
val spark: SparkSession = new SparkSessionBuilder().setEnv("local").build().get()
val connector: FileConnector = new FileConnector(Map[String, String](
"path" -> "src/test/resources/test-list-files",
"inferSchema" -> "true",
"header" -> "true",
"saveMode" -> "Overwrite",
"storage" -> "CSV"
)) {
override val storage: Storage = Storage.CSV
}
println("files:")
connector.listFiles().foreach(println)
println("pathes")
connector.listPaths().foreach(println)
println("pathes to load")
connector.listFilesToLoad(false).foreach(println)
Expected behavior
All the files should be listed recursively.