louis-e/arnis

encountering various errors with other cities

Closed this issue · 8 comments

Catchears commented

Nice script - thought I'd test it with some other cities, but ran into 3 errors you might want to be aware of :)

Nortorf

>>> python arnis.py --city "Nortorf" --state "Schleswig Holstein" --country "Deutschland" ...
Fetching data...
Chosen server: https://overpass.kumi.systems/api/interpreter
Parsing data...
Processing data...
Traceback (most recent call last):
  File "C:\path\arnis.py", line 592, in <module>
    imgarray = processData(rawdata)
  File "C:\path\arnis.py", line 350, in processData
    if (imgLanduse[i[1]][i[0]] == 0): imgLanduse[i[1]][i[0]] = landuseType
IndexError: index 8846204 is out of bounds for axis 0 with size 4818

Gnutz

python arnis.py --city "Gnutz" --state "Schleswig Holstein" --country "Deutschland" ...
Fetching data...
Chosen server: https://maps.mail.ru/osm/tools/overpass/api/interpreter
Parsing data...
Processing data...
Traceback (most recent call last):
  File "C:\path\arnis.py", line 592, in <module>
    imgarray = processData(rawdata)
  File "C:\path\arnis.py", line 504, in processData
    if (str(img[y][x][0])[:1] != 5 and str(img[y][x][0])[:1] != 6 and str(img[y][x][0])[:1] != 7): img[i[1]][i[0]] = int("2" + str((wallHeight + 1)))
UnboundLocalError: local variable 'y' referenced before assignment

Kiel tries to request 590 GiB of memory - didn't happen with Hamburg, maybe it didn't get that far though (ran into the same IndexError as Nortorf).

python arnis.py --city "Kiel" --state "Schleswig Holstein" --country "Deutschland"  ...
Fetching data...
Chosen server: https://overpass-api.de/api/interpreter
Parsing data...
Traceback (most recent call last):
  File "C:\path\arnis.py", line 592, in <module>
    imgarray = processData(rawdata)
  File "C:\path\arnis.py", line 199, in processData
    img = np.zeros((round((greatestElementY - lowestElementY) / resDownScaler) + 5, round((greatestElementX - lowestElementX) / resDownScaler) + 5, 1), np.uint8)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 590. GiB for an array with shape (1113821, 568481, 1) and data type uint8
louis-e commented

Hi there, thanks for opening an issue! Just fixed the Gnutz and Kiel issue.
The Nortorf crash is most likely caused by two cities with the same name in the same state. I already had that problem during development and it took me quite a while to figure out haha! That's why I included the state tag, which had the purpose of minimizing the chance to get two different cities with the same name and a large distance between them returned. I'm not 100% sure tho if this really causes the crash in this case, but I think so. For now I can implement another check which should avoid that exact error to occur again. Will do that later!

sparr commented 
$ python3 arnis.py --city Arnis --state "Schleswig Holstein" --country "Deutschland" --path ~/.minecraft/saves/asdf
Fetching data...
Chosen server: https://maps.mail.ru/osm/tools/overpass/api/interpreter
Parsing data...
Traceback (most recent call last):
  File "/home/sparr/src/arnis/arnis.py", line 571, in <module>
    imgarray = processData(rawdata)
  File "/home/sparr/src/arnis/arnis.py", line 178, in processData
    img = np.zeros((round((greatestElementY - lowestElementY) / resDownScaler) + 5, round((greatestElementX - lowestElementX) / resDownScaler) + 5, 1), np.uint8)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 4.94 TiB for an array with shape (993759, 5466422, 1) and data type uint8

I get the memory allocation problem even with the documented and namesake city. Changing resDownScaler = 100 to resDownScaler = 1000 eliminates the problem by presumably reducing the memory allocation attempt by a factor of 10x10=100 to a barely manageable 49.4GiB, although I wasn't willing to wait for it to run to completion. Changing it to 3000 for another 9x reduction to fit in my available RAM made it run quickly, although I can't confirm it had any effect on my game world.

louis-e commented

That's definitely an unexpected behavior. I just tried several ways to reproduce this issue on multiple machines but none of them run into the same problem.

Can you run the script again with the --debug parameter and send me the arnis-debug-raw_data.json file over e.g. pastebin?

sparr commented

After update to current main I cannot reproduce that error.

  • 👍1
AndiLeni commented

I will leave my found issue here as well:

This error occurs for Augsburg, Bayern, Deutschland

Traceback (most recent call last):
  File "C:\Users\andil\Desktop\arnis\arnis.py", line 10, in <module>
    run()
  File "C:\Users\andil\Desktop\arnis\src\main.py", line 119, in run
    imgarray = processData(rawdata, args)
  File "C:\Users\andil\Desktop\arnis\src\processData.py", line 162, in processData
    and int(element["tags"]["building:levels"]) <= 8
ValueError: invalid literal for int() with base 10: '1.5'
louis-e commented

Just released a hotfix for this problem. Thanks for the report!

savvasenok commented

Wien, Österreich

Traceback (most recent call last):
File "/home/.../Desktop/arnis/arnis.py", line 10, in
run()
File "/home/.../Desktop/arnis/src/main.py", line 123, in run
imgarray = processData(rawdata, args)
File "/home/.../Desktop/arnis/src/processData.py", line 199, in processData
and int(float(element["tags"]["building:levels"])) <= 8
ValueError: could not convert string to float: 'EG'

louis-e commented

@savvasenok Pushed a hotfix, should be working now. I hope that Wien doesn't make a big problem with its size though. Thanks for the report!

  • 👍1