Dictionaries attributes not handled in io.load_graphml()
ncotie opened this issue · 5 comments
Contributing guidelines
- I understand the contributing guidelines
Documentation
- My problem is not addressed by the documentation or examples
Existing issues
- My problem does not appear in an existing issue
What operating system and Python version are you using?
Win11, Python 3.10.10
What OSMnx version are you using?
1.6.0
Environment packages and versions
#
# Name Version Build Channel
affine 2.4.0 pyhd8ed1ab_0 conda-forge
anyio 3.6.2 pyhd8ed1ab_0 conda-forge
argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py310h8d17308_3 conda-forge
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
attrs 23.1.0 pyh71513ae_1 conda-forge
babel 2.12.1 pyhd8ed1ab_1 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.12.2 pyha770c72_0 conda-forge
bleach 6.0.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.3 hdccc3a2_0 conda-forge
boost-cpp 1.78.0 h9f4b32c_3 conda-forge
branca 0.6.0 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 hcfcfb64_8 conda-forge
brotli-bin 1.0.9 hcfcfb64_8 conda-forge
brotlipy 0.7.0 py310h8d17308_1005 conda-forge
bzip2 1.0.8 h8ffe710_4 conda-forge
ca-certificates 2023.7.22 h56e8100_0 conda-forge
cairo 1.16.0 hdecc03f_1015 conda-forge
certifi 2023.7.22 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h628cb3f_3 conda-forge
cfitsio 4.2.0 h9ebe7e4_0 conda-forge
charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge
click 8.1.3 win_pyhd8ed1ab_2 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
comm 0.1.3 pyhd8ed1ab_0 conda-forge
contourpy 1.0.7 py310h232114e_0 conda-forge
cryptography 40.0.2 py310h6e82f81_0 conda-forge
curl 8.0.1 h68f0423_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
debugpy 1.6.7 py310h00ffb61_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
dill 0.3.7 pyhd8ed1ab_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
expat 2.5.0 h63175ca_1 conda-forge
fiona 1.9.3 py310h4a685fe_0 conda-forge
flit-core 3.8.0 pyhd8ed1ab_0 conda-forge
folium 0.14.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 hbde0cde_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.39.3 py310h8d17308_0 conda-forge
freetype 2.12.1 h546665d_1 conda-forge
freexl 1.0.6 h67ca5e6_1 conda-forge
gdal 3.6.4 py310h3ccc0e8_1 conda-forge
geodatasets 2023.3.0 pyhd8ed1ab_0 conda-forge
geopandas 0.13.0 pyhd8ed1ab_0 conda-forge
geopandas-base 0.13.0 pyha770c72_0 conda-forge
geos 3.11.2 h1537add_0 conda-forge
geotiff 1.7.1 h7222e44_8 conda-forge
gettext 0.21.1 h5728263_0 conda-forge
glpk 5.0 h8ffe710_0 conda-forge
hdf4 4.2.15 h1334946_6 conda-forge
hdf5 1.14.0 nompi_h918d9b7_103 conda-forge
icu 72.1 h63175ca_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
igraph 0.10.4 hc93da9c_2 conda-forge
importlib-metadata 6.6.0 pyha770c72_0 conda-forge
importlib_metadata 6.6.0 hd8ed1ab_0 conda-forge
importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge
intel-openmp 2023.1.0 h57928b3_46319 conda-forge
ipykernel 6.22.0 pyh025b116_0 conda-forge
ipython 8.13.2 pyh08f2357_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
jedi 0.18.2 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
json5 0.9.5 pyh9f0ad1d_0 conda-forge
jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge
jupyter_client 8.2.0 pyhd8ed1ab_0 conda-forge
jupyter_core 5.3.0 py310h5588dad_0 conda-forge
jupyter_events 0.6.3 pyhd8ed1ab_0 conda-forge
jupyter_server 2.5.0 pyhd8ed1ab_0 conda-forge
jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge
jupyterlab 3.5.3 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge
jupyterlab_server 2.22.1 pyhd8ed1ab_0 conda-forge
kealib 1.5.0 h2e80506_1 conda-forge
kiwisolver 1.4.4 py310h232114e_1 conda-forge
krb5 1.20.1 heb0366b_0 conda-forge
lcms2 2.15 h3e3b177_1 conda-forge
lerc 4.0.0 h63175ca_0 conda-forge
libaec 1.0.6 h63175ca_1 conda-forge
libblas 3.9.0 16_win64_mkl conda-forge
libbrotlicommon 1.0.9 hcfcfb64_8 conda-forge
libbrotlidec 1.0.9 hcfcfb64_8 conda-forge
libbrotlienc 1.0.9 hcfcfb64_8 conda-forge
libcblas 3.9.0 16_win64_mkl conda-forge
libcurl 8.0.1 h68f0423_0 conda-forge
libdeflate 1.18 hcfcfb64_0 conda-forge
libexpat 2.5.0 h63175ca_1 conda-forge
libffi 3.4.2 h8ffe710_5 conda-forge
libgdal 3.6.4 h69cf491_1 conda-forge
libglib 2.76.2 he8f3873_0 conda-forge
libhwloc 2.9.1 h51c2c0f_0 conda-forge
libiconv 1.17 h8ffe710_0 conda-forge
libjpeg-turbo 2.1.5.1 hcfcfb64_0 conda-forge
libkml 1.3.0 hf2ab4e4_1015 conda-forge
liblapack 3.9.0 16_win64_mkl conda-forge
liblapacke 3.9.0 16_win64_mkl conda-forge
libnetcdf 4.9.2 nompi_hc664c2b_104 conda-forge
libpng 1.6.39 h19919ed_0 conda-forge
libpq 15.2 ha9684e8_0 conda-forge
librttopo 1.1.0 he1da8c1_13 conda-forge
libsodium 1.0.18 h8d14728_1 conda-forge
libspatialindex 1.9.3 h39d44d4_4 conda-forge
libspatialite 5.0.1 h50a8ebb_25 conda-forge
libsqlite 3.40.0 hcfcfb64_1 conda-forge
libssh2 1.10.0 h9a1e1f7_3 conda-forge
libtiff 4.5.0 h6c8260b_6 conda-forge
libwebp-base 1.3.0 hcfcfb64_0 conda-forge
libxcb 1.13 hcd874cb_1004 conda-forge
libxml2 2.10.4 hc3477c8_0 conda-forge
libzip 1.9.2 h519de47_1 conda-forge
libzlib 1.2.13 hcfcfb64_4 conda-forge
lz4-c 1.9.4 hcfcfb64_0 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
mapclassify 2.5.0 pyhd8ed1ab_1 conda-forge
markupsafe 2.1.2 py310h8d17308_0 conda-forge
matplotlib-base 3.7.1 py310h51140c5_0 conda-forge
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
mistune 2.0.5 pyhd8ed1ab_0 conda-forge
mkl 2022.1.0 h6a75c08_874 conda-forge
mpir 3.0.0 he025d50_1002 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
multiprocess 0.70.14 py310h8d17308_3 conda-forge
munch 2.5.0 py_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
nbclassic 1.0.0 pyh8b2e9e2_0 conda-forge
nbclient 0.7.4 pyhd8ed1ab_0 conda-forge
nbconvert-core 7.3.1 pyhd8ed1ab_0 conda-forge
nbformat 5.8.0 pyhd8ed1ab_0 conda-forge
nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge
networkx 3.1 pyhd8ed1ab_0 conda-forge
notebook 6.5.4 pyha770c72_0 conda-forge
notebook-shim 0.2.3 pyhd8ed1ab_0 conda-forge
numpy 1.24.3 py310hd02465a_0 conda-forge
openjpeg 2.5.0 ha2aaf27_2 conda-forge
openssl 3.1.3 hcfcfb64_0 conda-forge
osmnx 1.6.0 pyhd8ed1ab_0 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pandas 2.0.1 py310h1c4a608_0 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pcre2 10.40 h17e33f8_0 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.5.0 py310h3dcae36_0 conda-forge
pip 23.1.2 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h8ffe710_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge
platformdirs 3.5.0 pyhd8ed1ab_0 conda-forge
pooch 1.7.0 pyha770c72_3 conda-forge
poppler 23.04.0 h934c637_1 conda-forge
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 15.2 hd87cd2b_0 conda-forge
proj 9.2.0 heca977f_0 conda-forge
prometheus_client 0.16.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.38 pyha770c72_0 conda-forge
prompt_toolkit 3.0.38 hd8ed1ab_0 conda-forge
psutil 5.9.5 py310h8d17308_0 conda-forge
pthread-stubs 0.4 hcd874cb_1001 conda-forge
pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.15.1 pyhd8ed1ab_0 conda-forge
pyopenssl 23.1.1 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyproj 3.5.0 py310ha4ef3e2_1 conda-forge
pyrsistent 0.19.3 py310h8d17308_0 conda-forge
pysocks 1.7.1 pyh0701188_6 conda-forge
python 3.10.10 h4de0772_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.16.3 pyhd8ed1ab_0 conda-forge
python-igraph 0.10.4 py310h4d9ddb3_0 conda-forge
python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge
python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge
python_abi 3.10 3_cp310 conda-forge
pytz 2023.3 pyhd8ed1ab_0 conda-forge
pywin32 304 py310h00ffb61_2 conda-forge
pywinpty 2.0.10 py310h00ffb61_0 conda-forge
pyyaml 6.0 py310h8d17308_5 conda-forge
pyzmq 25.0.2 py310hcd737a0_0 conda-forge
rasterio 1.3.6 py310h0534870_1 conda-forge
requests 2.29.0 pyhd8ed1ab_0 conda-forge
rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge
rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
rtree 1.0.1 py310h1cbd46b_1 conda-forge
scikit-learn 1.2.2 py310hd266714_1 conda-forge
scipy 1.10.1 py310h578b7cb_1 conda-forge
send2trash 1.8.2 pyh08f2357_0 conda-forge
setuptools 67.7.2 pyhd8ed1ab_0 conda-forge
shapely 2.0.1 py310h91617ff_1 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 hfb803bf_0 conda-forge
sniffio 1.3.0 pyhd8ed1ab_0 conda-forge
snuggs 1.4.7 py_0 conda-forge
sortedcollections 2.1.0 pyhd8ed1ab_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge
sqlite 3.40.0 hcfcfb64_1 conda-forge
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
suitesparse 5.4.0 h5d0cbe0_1 conda-forge
tbb 2021.9.0 h91493d7_0 conda-forge
terminado 0.15.0 py310h5588dad_0 conda-forge
texttable 1.7.0 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tiledb 2.13.2 h3132609_0 conda-forge
tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h8ffe710_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
tornado 6.3 py310h8d17308_0 conda-forge
traitlets 5.9.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.5.0 hd8ed1ab_0 conda-forge
typing_extensions 4.5.0 pyha770c72_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
ucrt 10.0.22621.0 h57928b3_0 conda-forge
unicodedata2 15.0.0 py310h8d17308_0 conda-forge
urllib3 1.26.15 pyhd8ed1ab_0 conda-forge
vc 14.3 hb25d44b_16 conda-forge
vc14_runtime 14.34.31931 h5081d32_16 conda-forge
vs2015_runtime 14.34.31931 hed1258a_16 conda-forge
wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 1.5.1 pyhd8ed1ab_0 conda-forge
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge
winpty 0.4.3 4 conda-forge
xerces-c 3.2.4 h63175ca_2 conda-forge
xorg-libxau 1.0.9 hcd874cb_0 conda-forge
xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge
xyzservices 2023.2.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h8d14728_0 conda-forge
yaml 0.2.5 h8ffe710_2 conda-forge
zeromq 4.3.4 h0e60522_1 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hcfcfb64_4 conda-forge
zstd 1.5.2 h12be248_6 conda-forgeHow did you install OSMnx?
Conda and conda-forge
Problem description
Hello Geoff,
Similar to issue #665 for bool types, _convert_node_attr_types() and _convert_edge_attr_types() can't interpret attributes of type dict stored as strings in a graphml object being loaded using load_graphml(). It throws a "ValueError: dictionary update sequence element #0 has length 1;...", when type 'dict' is provided in node_dtypes or edge_dtypes, e.g. {'amenity_counts': dict} for a custom attribute 'amenity_counts' of type dict.
_convert_edge_attr_types() already has code to handle lists, checking for '[' and ']' characters then using ast.literal_eval(), and it appears that a similar addition could be made for both functions that also included checks for '{' and '}'.
I am successfully using the workaround you suggested in #665, to use dtype ast.eval_literal instead of dtype dict for these attributes.
Thanks
Neil
Complete minimal reproducible example
import networkx as nx
import osmnx as ox
nodes = [1,2,3]
edges = [[1,2], [1,3]]
G = nx.MultiGraph()
G.add_nodes_from(nodes)
G.add_edges_from(edges)
G.edges[1,2, 0]['test_dict_attr'] = {'foo':'bar'}
# Comment the below line out to show error in edge loading
G.nodes[1]['test_dict_attr'] = {'foo':'bar'}
ox.save_graphml(G, filepath = 'testsave')
ox.load_graphml(filepath = 'testsave',
node_dtypes = {'test_dict_attr':dict},
edge_dtypes = {'test_dict_attr':dict})To save everything to GraphML, OSMnx stringifies the attribute values. When it reloads the GraphML, the node_dtypes argument (and its peers) are used to determine how to convert each stringified attribute value back to its correct, original type.
For example, if you wanted to have a floating point type for some custom attribute, you'd specify node_dtypes = {"custom_attr": float} because float("123.45") will convert that string value to a float value.
However, it works differently for a dictionary, because of how Python works. That is, dict("{'foo':'bar'}") will not produce a dictionary, but rather the ValueError you noted above. Instead, to convert a string-representation of a dictionary to a dictionary, you would run ast.literal_eval("{'foo':'bar'}").
So, in your code example, the converters you're looking for would be something like:
import ast
ox.load_graphml(filepath = 'testsave.graphml',
node_dtypes = {'test_dict_attr': ast.literal_eval},
edge_dtypes = {'test_dict_attr': ast.literal_eval})Exactly, that's the workaround that I'm using, that I referred to. For my purposes (custom attributes for processing metrics) it works fine.
I thought I should raise the issue, given that the description for load_graphml doesn't limit support to a specific subset of basic dtypes, so a user could assume that all should work.
I could readily imagine you might want to restrict code support to dtypes that exist in OSM, for example, if that would exclude dict, and I assume that that is why you have coded support for bool and list dtypes. Totally up to you, of course, how to approach this question, whether to extend code or just to document.
Regards
Neil
I could readily imagine you might want to restrict code support to dtypes that exist in OSM
That is the current logic, yes, in that we handle types that exist in OSM plus types we need to make OSMnx's graph building work (ie, Python lists to handle simplified edges' attributes). That said, it has always been a bit confusing regarding when a user needs to use literal_eval. It seems like handling dicts/sets (ie, any attribute value that starts with { and ends with }) might be a graceful compromise solution between simplicity and predictability.
It should be fairly simple to add this by just expanding the logic that already exists in the io module to handle list conversion. If you'd like to take a quick stab at a PR, I could take a look.
Geoff, I've opened a draft PR with proposed changes. This is my first 'real-world' PR so my apologies if I've gotten things messed up. I wasn't sure how far to get into the hooks and test runs as described in the contribution guideline, but I am assuming that is if I were to be actually executing the merge (which I certainly don't think I'm to do?). I did test the changes by installing from my github into an env and running simple tests of the sort I described here above.
All feedback on what I should have done differently most welcome...
Regards
Neil