/ckanext-spc-pdh

CKAN repo for the Pacific Data Hub, an open data portal for datasets from the Pacific.

Primary LanguagePythonGNU Affero General Public License v3.0AGPL-3.0

ckanext-spc

Requirements

Installation

To install ckanext-spc:

  1. Activate your CKAN virtual environment, for example:

    . /usr/lib/ckan/default/bin/activate

  2. Install the ckanext-spc and its requirements into your virtual environment:

    pip install ckanext-spc
pip install -r dev-requirements.txt
  3. Add spc to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/production.ini).
  4. For the SpcNadaHarvester to work, install this branch of ckanext-ddi: github.com/roly97/ckanext-ddi/tree/nada_harvester. Alternatively, install the original ckanext-ddi extension github.com/liip/ckanext-ddi and then replace the files ddiharvester.py and metadata.py with the changed files found in ckanext-ddi_changes directory in this repo.

  5. Update SOLR schema:

    <field name="topic" type="string" indexed="true" stored="true" multiValued="true"/>

  6. Update DB schema:

    paster spc db-upgrade -c config.ini

  7. Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:

    sudo service apache2 reload

Config Settings

Config options:

scheming.dataset_schemas = ckanext.spc.schemas:dataset.json

Development Installation

To install ckanext-spc for development, activate your CKAN virtualenv and do:

git clone https://github.com/DataShades/ckanext-spc.git
cd ckanext-spc
python setup.py develop
pip install -r dev-requirements.txt
paster spc db-upgrade -c config.ini

Template macros

This extension provides new template macro _nested_field which is used for rendering fields in nested schemas. It can be easily added to any page that have access to dataset metadata via next snippet:

{{ nested_field(parent_field_name, index_of_current_field_amoung_other_siblings, current_field_name, parent_data_dict, parent_errors_dict) }}

More examples can be found in templates/macro/spc.html.

SPC specific examples

Recently updated dataset(within thematic area)::

http://ckan.url/api/action/package_search?q=extras_thematic_area_string:%22Climate%20Change%22&sort=metadata_modified+desc

Most popular dataset(within thematic area)::

http://site.url/api/action/package_search?q=extras_thematic_area_string:%22Climate%20Change%22&sort=extras_ga_view_count+desc

Interactive widgets

Develeopment server with the examples started via:

npm run dev-widgets

When development phase completed, build widgets with:

npm run build-widgets

Widgets registered inside SPCWidgets global JS variable. Use them form your CKAN JS modules as following:

var widget = new SPCWidgets.SearchForm({target: document.querySelector(selector)})

Search form

Running the Tests

To run the tests, do:

pytest --ckan-ini test.ini ckanext/spc

General Information

Harvesters

OAI-PMH

  • Name used in config: spc_oaipmh_harvester
  • Dataset type: publications
  • Sources:
Url Title Settings Organization
http://www.spc.int/DigitalLibrary/SPC/OAI Social Development Program {"set": "SDP_PDH", "topic": "Gender and Youth"}
spc-sdp
http://www.spc.int/DigitalLibrary/SPC/OAI Climate Change and Environmental Sustainability {"set": "CCES_PDH", "topic": "Climate Change"} spc-cces
http://www.spc.int/DigitalLibrary/SPC/OAI Fisheries, Aquaculture & Marine Ecosystems {"set": "FAME_PDH", "topic": "Fisheries"} spc-fame
http://www.spc.int/DigitalLibrary/SPC/OAI Geoscience, Energy and Maritime {"set": "GEM_PDH", "topic": "Geoscience"} spc-gem
http://www.spc.int/DigitalLibrary/SPC/OAI Land Resources Division {"set": "LRD_PDH", "topic": "Land Resources"} spc-lrd
http://www.spc.int/DigitalLibrary/SPC/OAI Public Health Division {"set": "PHD_PDH", "topic": "Health"} spc-phd
http://www.spc.int/DigitalLibrary/SPC/OAI Statistics for Development Division {"set": "SDD_PDH", "topic": "Official Statistics "} | spc-sdd

DKAN

  • Name used in config: spc_dkan_harvester
  • Dataset type: dataset
  • Sources:
Url Title Settings Organization

GBIF

  • Name used in config: spc_gbif_harvester
  • Dataset type: biodiversity_data
  • Sources:
Url Title Settings Organization
http://api.gbif.org GBIF SPREP published {"topic": "Fisheries", "hosting_org": "cd3512e7-886c-4873-b629-740abe8ae74e", "q": "-spc"} sprep
http://api.gbif.org GBIF SPC published {"topic": "Fisheries", "hosting_org": "cd3512e7-886c-4873-b629-740abe8ae74e", "q": "+spc"} spc-fame

PRDR Publications Harvester

  • Name used in config: spc_prdr_publications_harvester
  • Dataset type: publications
  • Sources:
Url Title Settings Organization
https://prdr-dev.spc.links.com.au/api/action/publications_list SPC PRDR Publications {"topic": "Energy"} spc-gem

PRDR Data(energy-resource) Harvester

  • Name used in config: spc_prdr_res_energy_harvester
  • Dataset type: dataset
  • Sources:
Url Title Settings Organization
https://prdr-dev.spc.links.com.au/api/action/energy_resources_list SPC PRDR Data {"topic": "Energy"} spc-gem

SPREP

  • Name used in config: spc_sprep_harvester
  • Dataset type: dataset
  • Sources:
Url Title Settings Organization

https://pacific-data.sprep.org

 Inform Regional Data Portal {"topic_mapping": {"Atmosphere and Climate": "Climate Change", "Info": null, "Land": "Land Resources", "Biodiversity": "Fisheries", "Build Environment": "Economic Development", "Coastal and Marine": "Fisheries", "Culture and Heritage": "Gender and Youth", "Inland Waters": "Geoscience"}}

sprep

Datasets types

  • Biodiversity data - EML schema
This Dataset type has multiple fields which includes subfields:
  • Creator
  • Metadata Provider
  • Associated Party
  • Keyword Set
  • Coverage
  • Maintenance
  • Contact
  • Methods
  • Project
  • Dataset - DCAT schema
  • Geographic data - ANZLIC schema
  • Publications - Dublin Core schema

CKAN Patches

curl https://patch-diff.githubusercontent.com/raw/ckan/ckan/pull/5546.diff | git apply curl https://patch-diff.githubusercontent.com/raw/ckan/ckan/pull/5594.diff | git apply