nismod/smif

results - Results.read_scenario_data reads in incorrect data

willu47 opened this issue · 1 comments

results.read_scenario_data reads in the wrong data, and the data changes as the list of timesteps passed changes. Somewhere, there is a misalignment between the Spec coordinates and the data.

For example, actual data:

timestep lad_uk_2016 residential_or_non floor_area
2030 E06000056 residential 10841750.0
2030 E06000056 non_residential 6525.1228990099

Using the api

>>> df = results.read_scenario_data(
        'socio-economic', 'arc_baseline', 'floor_area', [2030])
>>> df[df['lad_uk_2016'] == 'E06000056']

returns the incorrected data:

timestep lad_uk_2016 residential_or_non floor_area
2030 E06000056 residential 3424711.285
2030 E06000056 non_residential 4647666.550

Direct read from the store gives expected data:

>>> store = results._store
>>> df = store.read_scenario_variant_data('socio-economic', 'arc_baseline', 'floor_area', 2030).as_df().reset_index()
>>> df[df['lad_uk_2016'] == 'E06000056']
lad_uk_2016 residential_or_non floor_area
E06000056 residential 1.084175e+07
E06000056 non_residential 6.525123e+03

Associated changes also needed in the results API, which is now more thoroughly tested.