GSA/datagov-ckan-multi

ckanext-datajson Py3 + CKAN 2.9 support

thejuliekramer opened this issue · 9 comments

thejuliekramer commented

User Story

As a data.gov developer, I want ckanext-datajson running with CKAN 2.9 and Python 3 so that we can move out of CKAN 2.8 and reduce our technical debt and meet compliance standards.

Acceptance Criteria

  • WHEN I view CI
    THEN I see a successful job testing the extension under CKAN 2.9
    AND I see a successful job testing the extension under CKAN 2.8
    AND I see a successful job testing the extension under the legacy test environment
  • WHEN I remove the tests
    THEN I see CI fail due to test coverage being below a threshold

Background

CKAN 2.9 extension tracker.

Details / tasks

The goal is to support running this extension against CKAN 2.9 (Python 3) and CKAN 2.8 (Python 2) environments.
Please refer to the main CKAN documentation for Python 3 extensions migration. We're not following this guide exactly. Instead of creating a long-lived py3 branch, we should be merging often. New CI test suites don't need to required or to be fully passing in order to merge. It's only important that changes don't introduce new failures or break the existing test suites.

Below are a list of tasks. Depending on the extension's functionality, not all tasks will be actionable.

Tasks:

  • Update spreadsheet to indicate the extension upgrade is in progress
  • Rename default branch -> main.
  • Remove any CKAN 2.3 tests and functionality
  • Dependencies must be python 3 compatible (run caniusepython3 -r requirements.txt from within the virtualenv) and post the results to this issue (docs)
  • Add additional CI config for CKAN 2.8 and 2.9 Update CI config. These new CI jobs do not to be required yet (docs)
    • add code coverage reporting and required threshold (example)
  • Update documentation with compatibility table (docs)
    • Features
    • "Usage" (how to use/install/configure the extension) vs "Development" (how to test etc)
    • Required extensions and dependencies
    • "Weak" dependencies or how this code interacts with other extensions
  • Update/add docker-compose environment in order to run tests locally (example)
  • Update test suite (docs)
    • consider copying tests dir to tests/nose in order to preserve existing test environment (example)
    • replace nose with pytest (example)
  • Fix Python 3 issues / futurize code (docs)
    • Install future and run futurize --both-stages --write ckanext
  • Import from core ckan using toolkit (git grep -w 'paste\|pylons' should return no imports) (docs)
  • (optional) Create Mixin plugin implementations for Pylons / Flask (docs) (example)
  • Convert controllers to flask views/blueprints (docs)
  • Migrate Paster CLI commands to Click (docs)
  • Migrate js and css assets to Web assets (docs) (example)
  • Update templates (docs) (example)
    • you may want to split the template for CKAN >=2.9 in order to keep templates readable while maintaining backwards compatibility
    • update resource tags to asset for Web Assets
    • update controller='package' -> controller='dataset' or controller='resource'
  • Update spreadsheet to indicate the extension upgrade is complete

Additional resources:

adborden commented

Updated the issue description.

adborden commented

caniusepython3 output. Take this with a grain of salt because I don't think our requirements.txt is a complete view of the extensions' dependencies.

$ pipenv run caniusepython3 -r requirements.txt
Loading .env environment variables...
Finding and checking dependencies ...

You need 1 project to transition to Python 3.
Of that 1 project, 1 has no direct dependencies blocking its transition:

  lepl
adborden commented

Default branch has been renamed. Don't forget to run:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a
adborden commented

@FuhuXia I guess there's some flakiness in the CircleCI tests. It looks like main is mostly passing minus the test suites that we're planning on removing anyway.

adborden commented

@FuhuXia I've got 2 PRs for you:

Please check them out and merge them if ready. I'll sync up with you tomorrow.

adborden commented
  • add a test to inventory to make sure data.json and draft.json export is working
nickumia-reisys commented

On branch feature/py3, re-ran the caniusepython3 command and it returned this..

$ python3 -m  caniusepython3 -r requirements.txt
[WARNING] Skipping '-e git+https://github.com/ckan/ckanext-harvest.git': could not parse requirement
Finding and checking dependencies ...

You need 2 projects to transition to Python 3.
Of those 2 projects, 2 have no direct dependencies blocking their transition:

  ckantoolkit
  lepl
jbrown-xentity commented

We have decided that the 2 blocking tests are better implemented as integration tests on catalog and inventory respectively. We will remove these unit tests, and this should be ready for merging. Deployment to production will be dependent on implementing the integration tests:

nickumia-reisys commented

There was a bug found related to changes in this upgrade; however, that bug was not being explicitly tested for and we are considering ckanext-datajson py3 upgrade complete. It runs on py3 and will continue to have bugs pop up as every code does.