JSON parsing not always completely loaded (and hence fails to parse)
Closed this issue · 4 comments
billkendrick commented
Playing with the MASTODON.BAS example on the Atari, I'm frequently getting "ERROR 136" (end of file) when trying to read queried data (command Q), which turns out is due to the JSON not successfully parsing (command P).
At first, I altered the BASIC client, in an attempt to have the FujiNet load 10 items from the example Mastadon timeline, via the URL https://oldbytes.space/api/v1/timelines/public?limit=10 -- it truncated around 8,192 bytes (8K).
However, going back to the original code in the ATR disk image (/Networking/mastodon (BASIC).atr on atari-apps.irata.online, as of 2022-06-01 -- Update now located at /Atari 8-bit/Internet/mastodon (BASIC).atr on apps.irata.online, as of 2022-10-13), I noticed I was getting the same issue most (but not all) of the time. Further experimentation showed I could load, for example, 3 items from the timeline at once, which resulted in a JSON payload of 10,023 bytes (just under 10K). 🤷
Without knowing more, it seems like sometimes this bit of code from fnjson.cpp escapes the while() loop prematurely:
_protocol->status(&ns);
while (ns.connected)
{
_protocol->read(ns.rxBytesWaiting);
_parseBuffer += *_protocol->receiveBuffer;
_protocol->receiveBuffer->clear();
_protocol->status(&ns);
vTaskDelay(10);
}
Debug_printf("S: %s\n",_parseBuffer.c_str());
An example of this that I observed (shared on Discord) was seen in this log output from the FujiNet:
[00:19:01] CF: 71 50 0c 00 cd
[00:19:01] sioNetwork::sio_process 0x50 'P': 0x0c, 0x00
[00:19:01] inq_dstats = 0
[00:19:01] ACK!
[00:19:01] fnHttpClient::GET
[00:19:01] 000150f4 _perform
[00:19:04] 000159a1 _perform status = 200, length = -1, chunked = 1
[00:19:04] NetworkProtocolFS::read_file(1388)
[00:19:04] NetworkProtocolHTTP::read_file_handle(0x3fff2698,1388)
[00:19:04] NetworkProtocolHTTP::read_file_handle_data()
[00:19:04] NetworkProtocol::read(1388)
[00:19:04] S: [{"id":"108400996681740128","created_at":"2022-06-01T07:18:32.000Z","in_reply_to_id":null,"in_reply_to_account_id":null,"sensitive":false,"spoiler_text":"","visibility":"public","language":"en","uri":"https://botsin.space/users/scream/statuses/108400994684932537","url":"https://botsin.space/@scream/108400994684932537","replies_count":0,"reblogs_count":0,"favourites_count":0,"edited_at":null,"local_only":null,"content":"\u003cp\u003eAAAAAAAAAAAAAAAAHHH\u003c/p\u003e","reblog":null,"account":{"id":"53351","username":"scream","acct":"scream@botsin.space","display_name":"Endless Screaming","locked":false,"bot":true,"discoverable":false,"group":false,"created_at":"2018-08-21T00:00:00.000Z","note":"\u003cp\u003eAHHH\u003c/p\u003e","url":"https://botsin.space/@scream","avatar":"https://assets.oldbytes.space/assets.oldbytes.space/accounts/avatars/000/053/351/original/d6cd4e4f7dfd06dc.jpeg","avatar_static":"https://assets.oldbytes.space/assets.oldbytes.space/accounts/avatars/000/053/351/original/d6cd4e4f7dfd06dc.jpeg","header":"https://assets.oldbytes.space/assets.oldbytes.space/accounts/headers/000/053/351/original/15a71550df68575e.jpeg","header_static":"https://assets.oldbytes.space/assets.oldbytes.space/accounts/headers/000/053/351/original/15a71550df68575e.jpeg","followers_count":639,"following_count":1,"statuses_count":31438,"last_status_at":"2022-06-01","emojis":[],"fi
[00:19:04] FNJSON::parse() - Could not parse JSON
[00:19:04] COMPLETE!
mozzwald commented
@billkendrick can you retest with the latest firmware? I increased the http timeout which was causing issues with some other apps.
billkendrick commented
Thanks @mozzwald. In FujiNet Flasher I see two recent Atari 8-bit firmware revisions right now: 2022-08-28 0.5.383dcb20 (so just over 2 weeks old) and (oddly, listed second in the pulldown) 2022-09-06 0.5.f3905ead-SIO2BT (1 week old).
I'm currently on 0.5.fd038d41 2022-06-05 01:37:38, so I'll give the 2022-09-06 version a whirl first, then the 2022-08-28 if I have trouble.
billkendrick commented
Still seems to fail. FujiNet logs showing "Could not parse JSON" when I hit the URL seen in this ticket (with "?limit=10") running the MASTODON.BAS example program.
[23:10:48] S: [{"id":"109000928182819000","created_at":"2022-09-15T06:08:58.000Z","in_reply_to_id":null,"in_reply_to_account_id":null,"sensitive":false,"spoiler_text":"","visibility":"public","language":"zh","uri":"https://pokemon.men/users/archlinux/statuses/109000926076044201","url":"https://pokemon.men/@archlinux/109000926076044201","replies_count":0,"reblogs_count":0,"favourites_count":0,"edited_at":null,"local_only":null,"content":"\u003cp\u003e\u003ca href=\"https://pokemon.men/tags/ArchLinuxPackageUpdate\" class=\"mention hashtag\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e#\u003cspan\u003eArchLinuxPackageUpdate\u003c/span\u003e\u003c/a\u003e libpng 1.6.38-1 x86_64 \u003ca href=\"https://archlinux.org/packages/testing/x86_64/libpng/\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e\u003cspan class=\"invisible\"\u003ehttps://\u003c/span\u003e\u003cspan class=\"ellipsis\"\u003earchlinux.org/packages/testing\u003c/span\u003e\u003cspan class=\"invisible\"\u003e/x86_64/libpng/\u003c/span\u003e\u003c/a\u003e \u003ca href=\"https://pokemon.men/tags/ArchLinux\" class=\"mention hashtag\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e#\u003cspan\u003eArchLinux\u003c/span\u003e\u003c/a\u003e \u003ca href=\"https://pokemon.men/tags/Linux\" class=\"mention hashtag\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e#\u003cspan\u003eLinux\u003c/span\u003e\u003c/a\u003e\u003c/p\u003e","reblog":null,"account":{"id":"85806","username":"archlinux","acct":"archlinux@pokemon.men","display_name":"ArchLinux :archlinux:","locked":false,"bot":true,"discoverable":true,"group":false,"created_at":"2019-08-08T00:00:00.000Z","note":"\u003cp\u003e\u003ca href=\"https://pokemon.men/tags/ArchLinux\" class=\"mention hashtag\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e#\u003cspan\u003eArchLinux\u003c/span\u003e\u003c/a\u003e \u003ca href=\"https://pokemon.men/tags/Linux\" class=\"mention hashtag\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e#\u003cspan\u003eLinux\u003c/span\u003e\u003c/a\u003e\u003c/p\u003e","url":"https://pokemon.men/@archlinux","avatar":"https://assets.oldbytes.space/assets.oldbytes.space/accounts/avatars/000/085/806/original/ba1e1fd92f8fe657.png","avatar_static":"https://assets.oldbytes.space/assets.oldbytes.space/accounts/avatars/000/085/806/original/ba1e1fd92f8fe657.png","header":"https://oldbytes.space/headers/original/missing.png","header_static":"https://oldbytes.space/headers/original/missing.png","followers_count":940,"following_count":3,"statuses_count":91268,"last_status_at":"2022-09-15","emojis":[{"shortcode":"archlinux","url":"https://assets.oldbytes.space/assets.oldbytes.space/custom_emojis/images/000/046/421/original/3a5dd094e9cd33b1.png","static_url":"https://assets.oldbytes.space/assets.oldbytes.space/custom_emojis/images/000/046/421/static/3a5dd094e9cd33b1.png","visible_in_picker":true}],"fields":[]},"media_attachments":[],"mentions":[],"tags":[{"name":"linux","url":"https://oldbytes.space/tags/linux"},{"name":"archlinux","url":"https://oldbytes.space/tags/archlinux"},{"name":"archlinuxpackageupdate","url":"https://oldbytes.space/tags/archlinuxpackageupdate"}],"emojis":[],"card":{"url":"https://archlinux.org/packages/testing/x86_64/libpng/","title":"Arch Linux - libpng 1.6.38-1 (x86_64)","description":"","type":"link","author_name":"","author_url":"","provider_name":"","provider_url":"","html":"","width":0,"height":0,"image":null,"embed_url":"","blurhash":null},"poll":null},{"id":"109000927485836781","created_at":"2022-09-15T06:09:04.000Z","in_reply_to_id":null,"in_reply_to_account_id":null,"sensitive":false,"spoiler_text":"","visibility":"public","language":"en","uri":"https://botsin.space/users/todayilearned/statuses/109000926475148775","url":"https://botsin.space/@todayilearned/109000926475148775","replies_count":0,"reblogs_count":0,"favourites_count":0,"edited_at":null,"local_only":null,"content":"\u003cp\u003eMargarine was illegal in Wisconsin until 1967. Before that, Wisconsinites would sometimes smuggle in margarine from the south.\u003cbr\u003e\u003ca href=\"https://onmilwaukee.com/articles/wisconsin-butter-laws\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e\u003cspan class=\"invisible\"\u003ehttps://\u003c/span\u003e\u003cspan class=\"ellipsis\"\u003eonmilwaukee.com/articles/wisco\u003c/span\u003e\u003cspan class=\"invisible\"\u003ensin-butter-laws\u003c/span\u003e\u003c/a\u003e\u003c/p\u003e\u003cp\u003eOriginal tweet : \u003ca href=\"https://twitter.com/tilbots/status/1570290042232242176\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e\u003cspan class=\"invisible\"\u003ehttps://\u003c/span\u003e\u003cspan class=\"ellipsis\"\u003etwitter.com/tilbots/status/157\u003c/span\u003e\u003cspan class=\"invisible\"\u003e0290042232242176\u003c/span\u003e\u003c/a\u003e\u003c/p\u003e","reblog":null,"account":{"id":"92024","username":"todayilearned","acct":"todayilearned@botsin.space","display_name":"Today I learned","locked":false,"bot":true,"discoverable":true,"group":false,"created_at":"2019-10-17T00:00:00.000Z","note":"\u003cp\u003e@tilbots twitter feed\u003cbr\u003e[Will be handed over to twitter account owner upon request]\u003c/p\u003e","url":"https://botsin.space/@todayilearned","avatar":"https://assets.oldbytes.space/assets.oldbytes.space/accounts/avatars/000/092/024/original/995df74a2aae5619.jpeg","avatar_static":"https://assets.oldbytes.space/assets.oldbytes.space/accounts/avatars/000/092/024/original/995df74a2aae5619.jpeg","header":"https://oldbytes.space/headers/original/missing.png","header_static":"https://oldbytes.space/headers/original/missing.png","followers_count":1250,"following_count":1,"statuses_count":44925,"last_status_at":"2022-09-15","emojis":[],"fields":[]},"media_attachments":[{"id":"109000927466743379","type":"image","url":"https://assets.oldbytes.space/assets.oldbytes.space/cache/media_attachments/files/109/000/927/466/743/379/original/dd26d52ab31c8f94.jpg","preview_url":"https://assets.oldbytes.space/assets.oldbytes.space/cache/media_attachments/files/109/000/927/466/743/379/small/dd26d52ab31c8f94.jpg","remote_url":"https://files.botsin.space/media_attachments/files/109/000/926/432/495/099/original/10afccb4db765290.jpg","preview_remote_url":null,"text_url":null,"meta":{"original":{"width":400,"height":300,"size":"400x300","aspect":1.3333333333333333},"small":{"width":400,"height":300,"size":"400x300","aspect":1.3333333333333333}},"description":null,"blurhash":"UCMOyXDl=}^%?F0MD+xslT-U?XX8VZ?ZkVI;"}],"mentions":[],"tags":[],"emojis":[],"card":{"url":"https://onmilwaukee.com/articles/wisconsin-butter-laws","title":"Wisconsin's wacky butter laws spread over centuries","description":"You once had to hide your margarine stash in Wisconsin.","type":"link","author_name":"","author_url":"","provider_name":"OnMilwaukee","provider_url":"","html":"","width":400,"height":300,"image":"https://assets.oldbytes.space/assets.oldbytes.space/cache/preview_cards/images/001/810/437/original/6fc79b5f7fd50376.jpg","embed_url":"","blurhash":"UCMOyXDl=}^%?F0ME2xsh0-U?XX8VZ?Zg2I;"},"poll":null},{"id":"109000927064477574","created_at":"2022-09-15T06:08:57.000Z","in_reply_to_id":null,"in_reply_to_account_id":null,"sensitive":false,"spoiler_text":"","visibility":"public","language":"zh","uri":"https://pokemon.men/users/archlinux/statuses/109000925966993681","url":"https://pokemon.men/@archlinux/109000925966993681","replies_count":0,"reblogs_count":0,"favourites_count":0,"edited_at":null,"local_only":null,"content":"\u003cp\u003e\u003ca href=\"https://pokemon.men/tags/ArchLinuxPackageUpdate\" class=\"mention hashtag\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e#\u003cspan\u003eArchLinuxPackageUpdate\u003c/span\u003e\u003c/a\u003e lib32-libpng 1.6.38-1 x86_64 \u003ca href=\"https://archlinux.org/packages/multilib-testing/x86_64/lib32-libpng/\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e\u003cspan class=\"invisible\"\u003ehttps://\u003c/span\u003e\u003cspan class=\"ellipsis\"\u003earchlinux.org/packages/multili\u003c/span\u003e\u003cspan class=\"invisible\"\u003eb-testing/x86_64/lib32-libpng/\u0
[23:10:48] FNJSON::parse() - Could not parse JSON
This example seems to be more data than before (NetworkProtocolFS::read_file(6283), vs ...(1388) when I opened the ticket.)
I tried "?limit=5" and received some data, then an ERROR 137 (Truncated Record: tried to read a record longer than allowed), but that's a different concern, I guess. :)
BTW, when either of these issues occur (Atari throws ERROR 136 or 137), the FujiNet seems to get into a bad state. I cannot successfully RUN the program even putting it back to "?limit=1", until rebooting the FujiNet device via the hardware button.
Hah, and now I'm at a situation where I cannot even read a single item from the feed, because it's also too long. Atari throws ERROR 136, and FujiNet log shows:
[23:16:46] S: [{"id":"109000956086712253","created_at":"2022-09-15T06:16:29.000Z","in_reply_to_id":null,"in_reply_to_account_id":null,"sensitive":false,"spoiler_text":"","visibility":"public","language":"en","uri":"https://berserker.town/users/thor/statuses/109000955601307481","url":"https://berserker.town/@thor/109000955601307481","replies_count":0,"reblogs_count":0,"favourites_count":0,"edited_at":null,"local_only":null,"content":"\u003cp\u003efundamentally speaking, a government is an organisation that promises to be violent on your behalf if you're attacked.\u003c/p\u003e","reblog":null,"account":{"id":"107832574582601862","username":"thor","acct":"thor@berserker.town","display_name":"Thorwegian 🇳🇴","locked":false,"bot":false,"discoverable":true,"group":false,"created_at":"2022-02-20T00:00:00.000Z","note":"\u003cp\u003eWe search for signs of intelligent life by pointing our telescopes AWAY from Earth.\u003c/p\u003e\u003cp\u003eMaintainer of \u003ca href=\"https://berserker.town\" rel=\"nofollow noopener noreferrer\" target=\"_blank\"\u003e\u003cspan class=\"invisible\"\u003ehttps://\u003c/span\u003e\u003cspan class=\"\"\u003eberserker.town\u003c/span\u003e\u003cspan class=\"invisible\"\u003e\u003c/span\u003e\u003c/a\u003e\u003c/p\u003e\u003cp\u003e\u003ca href=\"https://berserker.town/tags/nobot\" class=\"mention hashtag\" rel=\"nofollow noopener noreferrer\" target=\"_
[23:16:46] FNJSON::parse() - Could not parse JSON
[23:16:46] COMPLETE!
billkendrick commented
0.5.7eb55adb 2022-10-12 01:41:41 working better for me.
0 DIM A$(256),U$(256)
1 TRAP 91
2 POKE 756,204
10 OPEN #1,12,0,"N:HTTPS://OLDBYTES.SPACE/api/v1/timelines/public?limit=3"
20 XIO 252,#1,12,1,"N:"
30 XIO ASC("P"),#1,12,0,"N:"
40 U$="N:/0/account/display_name":GOSUB 1000
41 XIO ASC("Q"),#1,12,3,U$
50 INPUT #1,A$:? A$
60 U$="N:/0/creeated_at":GOSUB 1000
61 XIO ASC("Q"),#1,12,3,U$
70 INPUT #1,A$:? A$
80 U$="N:/0/content":GOSUB 1000
81 XIO ASC("Q"),#1,12,3,U$
90 GET #1,A:? CHR$(A);:GOTO 90
91 NNN=NNN+1:IF NNN<3 THEN TRAP 91:GOTO 40
92 CLOSE #1:? :? "DONE":END
1000 U$(4,4)=CHR$(NNN+48):RETURN
This fetches three items and prints the timestamp, creator, and content.
I think it's safe to close this ticket now, but I'll @mozzwald decide. Thanks!