`python setup.py install` doesn't copy over central metadata
JackKelly opened this issue · 8 comments
Oli emailed this bug report:
Hi Jack,
I'm trying to write the metadata for wikienergy at the moment, but have hit a problem when I try to read in the building1.yaml metadata for my dataset. When I don't include building1.yaml in the metadata, things seem to work just fine. However, when I do include it, it generates this stacktrace:
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
/usr/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
176 else:
177 filename = fname
--> 178 __builtin__.execfile(filename, *where)
/home/oli/nilmtk/nilmtk/dataset_converters/wikienergy/test_dataset_wikienergy.py in <module>()
10 print 'testing queries over single table'
11 houses = {26:('2014-05-02','2014-05-03')}
---> 12 download_wikienergy(database_username, database_password, output_directory, periods_to_load=houses)
13 print('')
14 '''
/home/oli/nilmtk/nilmtk/dataset_converters/wikienergy/download_wikienergy.py in download_wikienergy(database_username, database_password, hdf_filename, periods_to_load)
206
207 convert_yaml_to_hdf5(join(_get_module_directory(), 'metadata'),
--> 208 hdf_filename)
209
210 def _wikienergy_dataframe_to_hdf(wikienergy_dataframe, store, nilmtk_building_id):
/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/convert_yaml_to_hdf5.pyc in convert_yaml_to_hdf5(yaml_dir, hdf_filename)
53 _set_data_location(elec_meters, building)
54 _sanity_check_meters(elec_meters, meter_devices)
---> 55 _sanity_check_appliances(building_metadata)
56 group._f_setattr('metadata', building_metadata)
57
/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/convert_yaml_to_hdf5.pyc in _sanity_check_appliances(building_metadata)
105 """
106 appliances = building_metadata['appliances']
--> 107 appliance_types = get_appliance_types()
108 building_instance = building_metadata['instance']
109
/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/object_concatenation.pyc in get_appliance_types()
12 recursively resolved.
13 """
---> 14 appliance_types_from_disk = get_appliance_types_from_disk()
15 appliance_types = _concatenate_all_appliance_types(appliance_types_from_disk)
16 return appliance_types
/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.pyc in get_appliance_types_from_disk()
8
9 def get_appliance_types_from_disk():
---> 10 obj_filenames = _find_all_appliance_type_files()
11 obj_cache = {}
12 for filename in obj_filenames:
/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.pyc in _find_all_appliance_type_files()
20 def _find_all_appliance_type_files():
21 filenames = _find_all_files_with_suffix('.yaml',
---> 22 _get_appliance_types_directory())
23 return filenames
24
/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.pyc in _get_appliance_types_directory()
25
26 def _get_appliance_types_directory():
---> 27 return _path_to_directory('..', 'central_metadata', 'appliance_types')
28
29
/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.pyc in _path_to_directory(*args)
43 def _path_to_directory(*args):
44 path_to_directory = join(_get_module_directory(), *args)
---> 45 assert isdir(path_to_directory)
46 return path_to_directory
47
AssertionError:
> /usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/file_management.py(45)_path_to_directory()
44 path_to_directory = join(_get_module_directory(), *args)
---> 45 assert isdir(path_to_directory)
46 return path_to_directory
so it seems like nilm-metadata is breaking while asserting that the provided path is in fact a directory. Debugging a little further:
ipdb> path_to_directory
'/usr/local/lib/python2.7/dist-packages/nilm_metadata-0.2.0-py2.7.egg/nilm_metadata/../central_metadata/appliance_types'
and it turns out that the central_metadata directory doesn't exist beside the nilm_metadata directory. I downloaded the central_metadata from github which seems to have solved the problem. Is this something that should have copied over automatically when I did 'sudo python setup.py install' for nilm-metadata?
Sorry for the long email. Feel free to call me if it's easier!
Oli
Ah, thanks loads for digging into this. This is a bug. I think the problem is that I did all my developing by installing NILM Metadata using python setup.py develop instead of python setup.py install. The difference is that develop doesn't copy the package to /usr/local/lib/python2.7/dist-packages/. I imagine there's fairly easy fix by getting setup.py to copy the *.yaml files over... but, because I'm supposed to be focussing on my IBM work right now, this should fix the issue (I'll update the README)
sudo python setup.py develop
Great. I've just updated https://github.com/nilmtk/nilm_metadata/blob/master/README.md to include this line in the installation instructions rather than a duplicate of sudo python setup.py install
Ah, cool, I think we both did pretty much exactly the same thing at the same time! I think I've merged both our changes!
Hey @JackKelly.
cc: @nipunreddevil
For making a working conda recipe for nilm_metadata, python setup.py install option was necessary. Tried fixing the missing central_metadata with the package_data option in setuptools. However, problems arose as setuptools takes a specific package structure as specified in this example under the topic "Setting up your Directory Structure".
For now, the central_metadata folder is at a position relative to the inner nilm_metadata folder. But this would create problems with multiple platforms and packages. What do you suggest?
Thanks loads for looking at this.
sorry, I won't have time to look into this today.
What would you suggest? Moving the central_metadata folder?
Yes. The setuptools directory structure demands that all files required by the package have to reside in the inner folder named after the package taken into account. We could move central_metadata and have the setup.py file modified to make this work.
I'll fork nilm_metadata,try out the modifications and let you know if this works or not.
PS: The way in which nilm_metadata and central_metadata are being used have to be altered as well. Just basic path changes, but changes nevertheless.
cool. Yes, if you make the changes you feel necessary in your fork then
I'll take a look. Thanks for looking into this!
On 9 January 2015 at 09:46, Mridul Malpotra notifications@github.com
wrote:
Yes. The setuptools directory structure demands that all files required by
the package have to reside in the inner folder named after the package
taken into account. We could move central_metadata and have the setup.py
file modified to make this work.
I'll fork nilm_metadata,try out the modifications and let you know if
this works or not.PS: The way in which nilm_metadata and central_metadata are being used
have to be altered as well. Just basic path changes, but changes
nevertheless.—
Reply to this email directly or view it on GitHub
#6 (comment).