error message with jsonio
Closed this issue · 19 comments
Hi,
Thanks for this great contribution.
I am trying to retrieve online data in the json format and turn it into stata 14.2.
I get the following error code :
java.lang.IndexOutOfBoundsException: Index: 10, Size: 10
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at org.paces.Stata.Input.FlatJSON.flatten(FlatJSON.java:205)
at org.paces.Stata.Input.FlatJSON.flatten(FlatJSON.java:116)
at org.paces.Stata.Input.InJSON.insheetUrl(InJSON.java:124)
My code:
jsonio kv, file("http://api.nightlights.io/months/1993.3-2013.12/villages/709000300016300")
I have installed the Jackson Java Library in the ado/jar folder of stata.
Would you happen to know where I am getting things wrong ?
Not sure where things are breaking down just yet, but Jackson is bound in the library already. If the version you put on the classpath is different it can create other issues. You shouldn't be fine with jsonio.jar in the classpath.
Thanks for your answer.
I have removed the jackson java library's .jar , but the issue still remains.
@guilhemc
Removing the other .jar wasn't going to solve the issue at hand, just eliminate a potential issue from creeping up down the road. It will probably take me a few days to get around to this, but feel free to pester me to make sure it stays on my radar in the meantime.
@wbuchanan
Following your suggestion, I am pestering you about this issue. Apologies if this reminder is coming earlier that is deemed reasonable in Github: I have little experience of that platform.
Not a problem at all. This did fall off my radar and I anticipating it still taking me a bit of time to get it higher in my queue (just starting to finish unpacking and getting settled into a new house).
No problem, hope you are settling well !
@wbuchanan
Sorry to pester you again, but would you have had a chance to look into that issue ?
Many thanks !
@guilhemc
I was looking at this a bit yesterday morning and came across a different issue, which is still important to address. Still trying to identify exactly where the bug you encountered is located.
@guilhemc
I think I may have fixed the issue you were running into while trying to address the simpler issue with the lineage for cases of an array of objects that don't contain any objects or arrays themselves (e.g., where each object in the array is essentially a row and each attribute a column). This might require a little manual work on your part to help me test, but if you download the updated jar from this link and replace the existing jar on your system with the newer one you should be able to run things without an issue.
Here is the simplified version of your data that I was using to test things :
[
{
"villagecode":"709000300016300",
"year":1993,
"month":3,
"satellite":"F10",
"count":18,
"vis_mean":"46.3361",
"vis_sd":"17.3630",
"vis_min":"15.3176",
"vis_median":"57.5855",
"vis_max":"59.6481"
},{
"villagecode":"709000300016300",
"year":1993,
"month":4,
"satellite":"F10",
"count":14,
"vis_mean":"49.2507",
"vis_sd":"17.2372",
"vis_min":"15.9540",
"vis_median":"58.8517",
"vis_max":"60.7931"
}, {
"villagecode":"709000300016300",
"year":1993,
"month":5,
"satellite":"F10",
"count":1,
"vis_mean":"9.8929",
"vis_sd":null,"vis_min":"9.8929",
"vis_median":"9.8929",
"vis_max":"9.8929"
}, {
"villagecode":"709000300016300",
"year":1993,
"month":8,
"satellite":"F10",
"count":5,
"vis_mean":"56.3648",
"vis_sd":"3.5260",
"vis_min":"52.5556",
"vis_median":"56.8077",
"vis_max":"60.2034"
}, {
"villagecode":"709000300016300",
"year":1993,
"month":9,
"satellite":"F10",
"count":17,
"vis_mean":"42.4296",
"vis_sd":"16.6513",
"vis_min":"11.6094",
"vis_median":"44.5833",
"vis_max":"60.5405"
}, {
"villagecode":"709000300016300",
"year":1993,
"month":10,
"satellite":"F10",
"count":22,
"vis_mean":"43.3510",
"vis_sd":"17.7220",
"vis_min":"7.0319",
"vis_median":"54.1882",
"vis_max":"60.0244"
}, {
"villagecode":"709000300016300",
"year":1993,
"month":11,
"satellite":"F10",
"count":23,
"vis_mean":"48.5610",
"vis_sd":"14.8306",
"vis_min":"17.1364",
"vis_median":"56.7097",
"vis_max":"60.3333"
}, {
"villagecode":"709000300016300",
"year":1993,
"month":12,
"satellite":"F10",
"count":23,
"vis_mean":"52.8305",
"vis_sd":"14.3656",
"vis_min":"1.1882",
"vis_median":"58.4805",
"vis_max":"60.8652"
}, {
"villagecode":"709000300016300",
"year":1994,
"month":1,
"satellite":"F10",
"count":24,
"vis_mean":"47.7586",
"vis_sd":"17.0738",
"vis_min":"4.6667",
"vis_median":"58.1544",
"vis_max":"61.2418"
}, {
"villagecode":"709000300016300",
"year":1994,
"month":2,
"satellite":"F10",
"count":24,
"vis_mean":"49.3809",
"vis_sd":"15.4064",
"vis_min":"17.2651",
"vis_median":"58.6961",
"vis_max":"60.5854"
}
]And here is what the output looks like:
jsonio kv, file(/Users/billy/Desktop/Programs/Java/Stata/issue2Test.json) nourl
. li
+-------------------------------------+
| key value |
|-------------------------------------|
1. | /id_0/villagecode 709000300016300 |
2. | /id_0/year 1993 |
3. | /id_0/month 3 |
4. | /id_0/satellite F10 |
5. | /id_0/count 18 |
|-------------------------------------|
6. | /id_0/vis_mean 46.3361 |
7. | /id_0/vis_sd 17.3630 |
8. | /id_0/vis_min 15.3176 |
9. | /id_0/vis_median 57.5855 |
10. | /id_0/vis_max 59.6481 |
|-------------------------------------|
11. | /id_1/villagecode 709000300016300 |
12. | /id_1/year 1993 |
13. | /id_1/month 4 |
14. | /id_1/satellite F10 |
15. | /id_1/count 14 |
|-------------------------------------|
16. | /id_1/vis_mean 49.2507 |
17. | /id_1/vis_sd 17.2372 |
18. | /id_1/vis_min 15.9540 |
19. | /id_1/vis_median 58.8517 |
20. | /id_1/vis_max 60.7931 |
|-------------------------------------|
21. | /id_2/villagecode 709000300016300 |
22. | /id_2/year 1993 |
23. | /id_2/month 5 |
24. | /id_2/satellite F10 |
25. | /id_2/count 1 |
|-------------------------------------|
26. | /id_2/vis_mean 9.8929 |
27. | /id_2/vis_sd |
28. | /id_2/vis_min 9.8929 |
29. | /id_2/vis_median 9.8929 |
30. | /id_2/vis_max 9.8929 |
|-------------------------------------|
31. | /id_3/villagecode 709000300016300 |
32. | /id_3/year 1993 |
33. | /id_3/month 8 |
34. | /id_3/satellite F10 |
35. | /id_3/count 5 |
|-------------------------------------|
36. | /id_3/vis_mean 56.3648 |
37. | /id_3/vis_sd 3.5260 |
38. | /id_3/vis_min 52.5556 |
39. | /id_3/vis_median 56.8077 |
40. | /id_3/vis_max 60.2034 |
|-------------------------------------|
41. | /id_4/villagecode 709000300016300 |
42. | /id_4/year 1993 |
43. | /id_4/month 9 |
44. | /id_4/satellite F10 |
45. | /id_4/count 17 |
|-------------------------------------|
46. | /id_4/vis_mean 42.4296 |
47. | /id_4/vis_sd 16.6513 |
48. | /id_4/vis_min 11.6094 |
49. | /id_4/vis_median 44.5833 |
50. | /id_4/vis_max 60.5405 |
|-------------------------------------|
51. | /id_5/villagecode 709000300016300 |
52. | /id_5/year 1993 |
53. | /id_5/month 10 |
54. | /id_5/satellite F10 |
55. | /id_5/count 22 |
|-------------------------------------|
56. | /id_5/vis_mean 43.3510 |
57. | /id_5/vis_sd 17.7220 |
58. | /id_5/vis_min 7.0319 |
59. | /id_5/vis_median 54.1882 |
60. | /id_5/vis_max 60.0244 |
|-------------------------------------|
61. | /id_6/villagecode 709000300016300 |
62. | /id_6/year 1993 |
63. | /id_6/month 11 |
64. | /id_6/satellite F10 |
65. | /id_6/count 23 |
|-------------------------------------|
66. | /id_6/vis_mean 48.5610 |
67. | /id_6/vis_sd 14.8306 |
68. | /id_6/vis_min 17.1364 |
69. | /id_6/vis_median 56.7097 |
70. | /id_6/vis_max 60.3333 |
|-------------------------------------|
71. | /id_7/villagecode 709000300016300 |
72. | /id_7/year 1993 |
73. | /id_7/month 12 |
74. | /id_7/satellite F10 |
75. | /id_7/count 23 |
|-------------------------------------|
76. | /id_7/vis_mean 52.8305 |
77. | /id_7/vis_sd 14.3656 |
78. | /id_7/vis_min 1.1882 |
79. | /id_7/vis_median 58.4805 |
80. | /id_7/vis_max 60.8652 |
|-------------------------------------|
81. | /id_8/villagecode 709000300016300 |
82. | /id_8/year 1994 |
83. | /id_8/month 1 |
84. | /id_8/satellite F10 |
85. | /id_8/count 24 |
|-------------------------------------|
86. | /id_8/vis_mean 47.7586 |
87. | /id_8/vis_sd 17.0738 |
88. | /id_8/vis_min 4.6667 |
89. | /id_8/vis_median 58.1544 |
90. | /id_8/vis_max 61.2418 |
|-------------------------------------|
91. | /id_9/villagecode 709000300016300 |
92. | /id_9/year 1994 |
93. | /id_9/month 2 |
94. | /id_9/satellite F10 |
95. | /id_9/count 24 |
|-------------------------------------|
96. | /id_9/vis_mean 49.3809 |
97. | /id_9/vis_sd 15.4064 |
98. | /id_9/vis_min 17.2651 |
99. | /id_9/vis_median 58.6961 |
100. | /id_9/vis_max 60.5854 |
+-------------------------------------+@wbuchanan
Many thanks !
This now works with kv.
Note however, that I get the following error message when I try the rv option:
java.lang.ArrayIndexOutOfBoundsException: 3
at org.paces.Stata.Input.InJSON.insheetUrlToVars(InJSON.java:214)
Code used:
jsonio rv, file("http://api.nightlights.io/months/1993.3-2013.12/villages/709000300016300")
Cool. Thanks for the info. I’ll try to look into the rv issue as I get time to dig into things.
Did you try without using the quotes? When I'm reading the data from a file it seems to work. I just recompiled after trying to address another potential place where an error could occur and now it looks like:
. jsonio rv , file(/Users/billy/Desktop/Programs/Java/Stata/issue2Test.json) nourl
. desc
Contains data
obs: 0
vars: 100
size: 0
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
storage display value
variable name type format label variable label
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
jsonvar1 str17 %17s /id_1/villagecode
jsonvar2 double %10.0g /id_1/year
jsonvar3 double %10.0g /id_1/month
jsonvar4 str15 %15s /id_1/satellite
jsonvar5 double %10.0g /id_1/count
jsonvar6 str14 %14s /id_1/vis_mean
jsonvar7 str12 %12s /id_1/vis_sd
jsonvar8 str13 %13s /id_1/vis_min
jsonvar9 str16 %16s /id_1/vis_median
jsonvar10 str13 %13s /id_1/vis_max
jsonvar11 str17 %17s /id_2/villagecode
jsonvar12 double %10.0g /id_2/year
jsonvar13 double %10.0g /id_2/month
jsonvar14 str15 %15s /id_2/satellite
jsonvar15 double %10.0g /id_2/count
jsonvar16 str14 %14s /id_2/vis_mean
jsonvar17 str12 %12s /id_2/vis_sd
jsonvar18 str13 %13s /id_2/vis_min
jsonvar19 str16 %16s /id_2/vis_median
jsonvar20 str13 %13s /id_2/vis_max
jsonvar21 str17 %17s /id_3/villagecode
jsonvar22 double %10.0g /id_3/year
jsonvar23 double %10.0g /id_3/month
jsonvar24 str15 %15s /id_3/satellite
jsonvar25 double %10.0g /id_3/count
jsonvar26 str14 %14s /id_3/vis_mean
jsonvar27 byte %8.0g /id_3/vis_sd
jsonvar28 str13 %13s /id_3/vis_min
jsonvar29 str16 %16s /id_3/vis_median
jsonvar30 str13 %13s /id_3/vis_max
jsonvar31 str17 %17s /id_4/villagecode
jsonvar32 double %10.0g /id_4/year
jsonvar33 double %10.0g /id_4/month
jsonvar34 str15 %15s /id_4/satellite
jsonvar35 double %10.0g /id_4/count
jsonvar36 str14 %14s /id_4/vis_mean
jsonvar37 str12 %12s /id_4/vis_sd
jsonvar38 str13 %13s /id_4/vis_min
jsonvar39 str16 %16s /id_4/vis_median
jsonvar40 str13 %13s /id_4/vis_max
jsonvar41 str17 %17s /id_5/villagecode
jsonvar42 double %10.0g /id_5/year
jsonvar43 double %10.0g /id_5/month
jsonvar44 str15 %15s /id_5/satellite
jsonvar45 double %10.0g /id_5/count
jsonvar46 str14 %14s /id_5/vis_mean
jsonvar47 str12 %12s /id_5/vis_sd
jsonvar48 str13 %13s /id_5/vis_min
jsonvar49 str16 %16s /id_5/vis_median
jsonvar50 str13 %13s /id_5/vis_max
jsonvar51 str17 %17s /id_6/villagecode
jsonvar52 double %10.0g /id_6/year
jsonvar53 double %10.0g /id_6/month
jsonvar54 str15 %15s /id_6/satellite
jsonvar55 double %10.0g /id_6/count
jsonvar56 str14 %14s /id_6/vis_mean
jsonvar57 str12 %12s /id_6/vis_sd
jsonvar58 str13 %13s /id_6/vis_min
jsonvar59 str16 %16s /id_6/vis_median
jsonvar60 str13 %13s /id_6/vis_max
jsonvar61 str17 %17s /id_7/villagecode
jsonvar62 double %10.0g /id_7/year
jsonvar63 double %10.0g /id_7/month
jsonvar64 str15 %15s /id_7/satellite
jsonvar65 double %10.0g /id_7/count
jsonvar66 str14 %14s /id_7/vis_mean
jsonvar67 str12 %12s /id_7/vis_sd
jsonvar68 str13 %13s /id_7/vis_min
jsonvar69 str16 %16s /id_7/vis_median
jsonvar70 str13 %13s /id_7/vis_max
jsonvar71 str17 %17s /id_8/villagecode
jsonvar72 double %10.0g /id_8/year
jsonvar73 double %10.0g /id_8/month
jsonvar74 str15 %15s /id_8/satellite
jsonvar75 double %10.0g /id_8/count
jsonvar76 str14 %14s /id_8/vis_mean
jsonvar77 str12 %12s /id_8/vis_sd
jsonvar78 str13 %13s /id_8/vis_min
jsonvar79 str16 %16s /id_8/vis_median
jsonvar80 str13 %13s /id_8/vis_max
jsonvar81 str17 %17s /id_9/villagecode
jsonvar82 double %10.0g /id_9/year
jsonvar83 double %10.0g /id_9/month
jsonvar84 str15 %15s /id_9/satellite
jsonvar85 double %10.0g /id_9/count
jsonvar86 str14 %14s /id_9/vis_mean
jsonvar87 str12 %12s /id_9/vis_sd
jsonvar88 str13 %13s /id_9/vis_min
jsonvar89 str16 %16s /id_9/vis_median
jsonvar90 str13 %13s /id_9/vis_max
jsonvar91 str18 %18s /id_10/villagecode
jsonvar92 double %10.0g /id_10/year
jsonvar93 double %10.0g /id_10/month
jsonvar94 str16 %16s /id_10/satellite
jsonvar95 double %10.0g /id_10/count
jsonvar96 str15 %15s /id_10/vis_mean
jsonvar97 str13 %13s /id_10/vis_sd
jsonvar98 str14 %14s /id_10/vis_min
jsonvar99 str17 %17s /id_10/vis_median
jsonvar100 str14 %14s /id_10/vis_max
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sorted by:
Note: Dataset has changed since last saved.@wbuchanan
I had not tried, but having now tried it, I get the same error message:
jsonio rv, file(http://api.nightlights.io/months/1993.3-2013.12/villages/709000300016300)
gives me:
java.lang.ArrayIndexOutOfBoundsException: 3
at org.paces.Stata.Input.InJSON.insheetUrlToVars(InJSON.java:214)
@guilhemc
Ok. Oddly enough it doesn't have that issue if the content is stored in a file, but I think I see the issue. Should be easier to fix. Can you create a new issue for this particular issue since it is something very different from the previous issue (even though it is a Java exception being thrown it is an issue with the syntax being used in Stata to call the method).
@wbuchanan
Done.
Thanks once again for your incredible responsiveness !
@guilhemc
Sorry it took me so long to get around to this. This is fixed in the dev branch now and I'll try to get things put together to push an update to the installation page over the coming weekend.
Are there any news on this? I have the same problem. I would like to load all country definitions from comtrade into Stata (from: https://comtrade.un.org/data/cache/partnerAreas.json).
If I run
jsonio kv, file("https://comtrade.un.org/data/cache/partnerAreas.json")
Stata returns the following:
java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at org.paces.Stata.Input.FlatJSON.flatten(FlatJSON.java:205)
at org.paces.Stata.Input.FlatJSON.flatten(FlatJSON.java:194)
at org.paces.Stata.Input.FlatJSON.flatten(FlatJSON.java:116)
at org.paces.Stata.Input.InJSON.insheetUrlToVars(InJSON.java:202)
r(5100);
I tried to save the json file and open it from the harddrive using the nourl option and the same error appears. Also using rv instead of kv does not help.
Thanks!
Jan
@JanDitzen
Can’t remember the specific open issue, but i believe someone else also had an issue with the comtrade API. In the instance I am thinking of, the problem is that the API is including control characters in the payload which breaks the parsing since they are not valid characters in the JSON specification. This might be unrelated, but it’d be helpful if you were able to respond to Issue #32 so I can find things more consistently as I have time to do maintenance work.
Apologies, I thought I replied here. I managed to solve the issue by re-installing jsonio and using the version from git hub rather than ssc. It seems that the jsonio.jar in the ado folder made the difference.
Thanks for the help.
Best,
Jan