simeji/jid

Performance is extremely slow with a 7MB JSON file

Closed this issue · 3 comments

I have a 2016 MacBook Pro (pretty fast machine), but if I give jid as input a 7MB JSON file that is on my local SSD drive, performance is very slow; each character I type takes about 3 seconds to respond.

The JSON file consists of about 27,000 items. The file is structured like this:

{
  (item),
  (item),
  (item)
}

Each (item) looks like this:

"128708": {
   "id_1": 132499,
   "id_2": "1939",
   "lang": {
     "qc": 1
   },
   "items": [
     130556,
     126889,
     129103,
     131423,
     126300,
     131270,
     134335,
     134483,
     141317,
     136120
   ],
   "date": "7/7",
   "category": 133089,
   "cr_dt": "7/8/2001",
   "d_year": "2004",
   "some_id2": 132589,
   "b_year": "1/22",
   "l_en": {
     "qc": 3,
     "path": "jeffsmith",
     "name": "Jeff Smith"
   }
 }

I do:
jid < /tmp/some.json

With that as an input if I type "1" it takes about 3 seconds before jid reponds, then if I type "2" it takes another 3 seconds, and so on.

Is there a way this could be improved to be faster?

I've got same issues and ended creating my own tool: https://github.com/antonmedv/fx
Can handle 20MB files easily.

@dashfwd
Improved performance issue below release.
https://github.com/simeji/jid/releases/tag/v0.7.4

If you had requests any more, please reopen this issue or create another one.