denisidoro/floki

Floki hangs on large JSON files

Opened this issue · 3 comments

emiln commented

Floki doesn't seem able to render large JSON files at all even though JSON.parse handles them just fine.

# It's a fairly big file, sure.
ls -lh
... 31M ... big-file.json

# But node has no trouble parsing it and returning its keys.
time node -p "var fs = require('fs'); Object.keys(JSON.parse(fs.readFileSync('big-file.json')))"
[ 'totalHits', 'hitsPerPage', 'startIndex', 'hits' ]

real	0m0.422s
user	0m0.416s
sys	0m0.046s

Running floki on this file just produces a blank terminal screen even if I wait for a couple of minutes.

Do you think there's anything that can be done to improve the performance on big files? I think floki would be particularly appealing exactly in these cases of big files I'm trying to explore.

denisidoro commented

Regarding the parsing, I think it should be trivial to fix: the problem is that first I try to parse the input as an EDN (Clojure's JSON-like structure) then I try to parse as a JSON. I can come up with something more intelligent.

However, browsing the structure will probably remain slow because it isn't optimized: it always tries to walk all the possible paths. I would need to refractor the code a little bit.

I'll probably take a look at it next week. Thanks for pointing it out.

emiln commented

Thanks for looking into it. What's your stance on contributions to the project? Are you interested in pull requests for issues like this?

denisidoro commented

Hi! Sorry, I haven't been able to look at it yet. Any pull request is welcome! 😀

In case you try to fix this yourself I can help you out browsing the code if there's any doubt