Magda connectors go out to external datasources and copy their metadata into the Registry, so that they can be searched and have other aspects attached to them. A connector is simply a docker-based microservice that is invoked as a job. It scans the target datasource (usually an open-data portal), then completes and shuts down.
Magda Ckan Connector is created for crawling data from Ckan data source.
It's recommanded to deploy connectors with as dependencies of a Magda helm deployment. Example can be found from here.
- Magda Helm Charts Repository Url: https://charts.magda.io
The helm chart for this connector is auto released when a Github Release is created for this repo.
- Add repository to helm:
helm repo add magda-io https://charts.magda.ioDocker image releases can be found from Docker Hub:
https://hub.docker.com/r/data61/magda-ckan-connector/
Development releases (per commit) are also available from GitHub Registry and accessible with access token.
Kubernetes: >= 1.14.0-0
| Repository | Name | Version |
|---|---|---|
| https://charts.magda.io | magda-common | 1.0.0-alpha.4 |
| Key | Type | Default | Description |
|---|---|---|---|
| config.id | string | "dga" |
|
| config.ignoreHarvestSources[0] | string | "*" |
|
| config.name | string | "data.gov.au" |
|
| config.pageSize | int | 1000 |
|
| config.sourceUrl | string | "https://data.gov.au/data/" |
|
| createConfigMap | bool | true |
|
| createFunction | bool | false |
|
| defaultImage.imagePullSecret | bool | false |
|
| defaultImage.pullPolicy | string | "IfNotPresent" |
|
| defaultImage.repository | string | "docker.io/data61" |
|
| defaultSettings.includeCronJobs | bool | true |
|
| defaultSettings.includeInitialJobs | bool | false |
|
| defaultTenantId | int | 0 |
|
| global.connectors.image | object | {} |
|
| global.image | object | {} |
|
| global.openfaas | object | {} |
|
| image.name | string | "magda-ckan-connector" |
|
| resources.limits.cpu | string | "100m" |
|
| resources.requests.cpu | string | "50m" |
|
| resources.requests.memory | string | "30Mi" |