Read CSV files quickly

[Beliavsky]

Since R is used for analyzing large data sets, there are several libraries for reading CSV files quickly, one of them being csvread, written in C++. It would be nice to have a fast CSV file reader in Fortran, even if it's just an interface to one in C or C++.

[arjenmarkus]

Tangential remark: A problem and a rather fundamental one that occurs with CSV files is that the format leaves a lot of freedom. Just think of the use of comma's, semicolons etc. You would need some detection mechanism or options. I do not know that library, but it should not be too hard to get something reasonably fast - these degrees of freedom are the main bottleneck I'd say. Op di 5 nov 2024 om 14:46 schreef Beliavsky ***@***.***>:

…

Since R is used for analyzing large data sets, there are several libraries for reading CSV files quickly, one of them being csvread <https://cran.r-project.org/web/packages/csvread/index.html>, written in C++. It would be nice to have a fast CSV file reader in Fortran, even if it's just an interface to one in C or C++. — Reply to this email directly, view it on GitHub <#884>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR5IJ4JTBN5RT25X2ILZ7DDZRAVCNFSM6AAAAABRGSZ5CCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZTKNJRGMYDGNY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sakamoti]

I have never loaded large data, so I am not sure how fast it is for your use case, but as a package for reading CSV files in Fortran, there is csv-fortran. This package is also introduced on fortran-lang.org.

[jalvesz]

This could be developed using to_num_from_stream. The fastest apporach I have found to load large numeric ASCII files into memory is to load it fully into a single large string and then stream throught the string pointer.

[feenberg]

Can you say more about what you would expect from such a feature beyond what a list directed read can do? Would it be faster than the list directed read? Did you expect to avoid declaring the variables in the Fortran source? That would be super convenient, but probably out-of-scope for a compiled language like Fortran.