GSA/ckanext-geodatagov

Stop rolling up extras

avdata99 opened this issue · 2 comments

avdata99 commented

Each time we save a new dataset we group almost any extras in a special extra called extras_rollup (we do this for CKAN 2.3 and for CKAN 2.8).

We need to stop doing this and save extras as regular extras.
This will require:

  • Update code and test to fit this new extras structure
    • Update code and test in this extension
    • Update code and tests in the DCAT-US extension.
    • Update code and tests in the DataGovTheme extension.
  • Deal with the spatial extra. We already have Multi#332 for this.
amercader commented

@avdata99 Just a heads up that this was originally implemented to increase performance with a large number of datasets. When you reach the dataset count that data.gov has, the default CKAN model wasn't designed to cope with it, specially with large number of datasets with many extras like the harvested ones in data.gov. This was done back in the CKAN 2.3 days so a lot of things might have changed in the meantime but just flagging that you should check the performance impact of removing this. IIRC this wiki page had details about it.

  • 👍2
avdata99 commented

Thanks @amercader.
I couldn't understand why this was done and now I understand. We are going to have to make sure that this change does not have a bigger impact than expected.

Maybe also related to Multi#267

FYI @adborden @thejuliekramer