dim

Data Installation Manager: Manage the open data in your project like a package manager.

Join community

We are looking for members to develop together as an open source community.

Slack

Features

📀 Record the source url and post-processing, etc., of downloaded open-data
🔧 Prepare all open data needed for the project in one command by using the dim.json recorded by someone else
🚀 General post-processing, such as unzip, encoding, etc., is available from the start
🔍 Search open-data from CKAN

Document

For more information about how to use it, please refer to this document.

Quick Start

Install the dim

Install the dim from binary files or Build the dim from Deno install

Install the dim from binary files

Download the dim from binary files.

aarch64-apple-darwin

curl -L https://github.com/c-3lab/dim/releases/latest/download/aarch64-apple-darwin-dim -o /usr/local/bin/dim

x86_64-apple-darwin

curl -L https://github.com/c-3lab/dim/releases/latest/download/x86_64-apple-darwin-dim -o /usr/local/bin/dim

x86_64-pc-windows-msvc

curl https://github.com/c-3lab/dim/releases/latest/download/x86_64-pc-windows-msvc-dim.exe -o C:\Users\user-name\dim.exe

x86_64-unknown-linux-gnu

curl -L https://github.com/c-3lab/dim/releases/latest/download/x86_64-unknown-linux-gnu-dim -o /usr/local/bin/dim

Grant user execution permission

chmod u+x /usr/local/bin/dim

New Project

init the project

Generate dim.json, dim-lock.json and data_files/ by init command.

$ dim init

Install a data

This command stores information about installed data in dim.json and dim-lock.json.

$ dim install https://example.com -n "example"

Installed data is saved in data_files/.

$ ls ./data_files

Install all data written to dim.json shared by members

Install all data written to dim.json shared by members.

Make sure existing the dim.json in current directory

$ ls ./

dim.json  ....

Install all data written in the dim.json

$ dim install

Installed data is saved in data_files/.

$ ls ./data_files

Command Usage

Init

Generate dim.json, dim-lock.json and data_files/.

$ dim init

Install

Install the all data.

Install all data written to dim.json.

$ dim install

Install from a specified local dim.json.

$ dim install -f ./path/dim.json

Install from a specified remote dim.json in internet.

$ dim install -f https://raw.githubusercontent.com/xxxx/xxxx/main/dim.json

Install the specified data.

$ dim install https://example.com -n "example"

Specify headers.

$ dim install https://example.com -n "example" -H "Authorization: 1234567890abc" -H "Fiware-Service: example"

Specify the installation post-process

Postprocess unzip

$ dim install https://example.com -n "example" -p unzip

Postprocess encoding

$ dim install https://example.com -n "example" -p "encode utf-8"

Postprocess xlsx-to-csv

$ dim install https://example.com -n "example" -p xlsx-to-csv

Postprocess custom command

You can specify a custom command after "cmd".

$ dim install https://example.com -n "example" -p "cmd ******"

The file path will be passed as an argument at the end of the specified command.

$ dim install https://example.com -n "example" -p "cmd python ./tests/test_custom_command.py"

Command to be executed during postprocessing.

$ python ./tests/test_custom_command.py ./data_files/***/***.xx

Forced execution

Forced install. Overwrite already exist data file.

$ dim install https://example.com -n "example" -F

Uninstall

Delete data with the specified name from dim.json, dim-lock.json and data_files/.

$ dim uninstall [name]

List

Display the information described in dim-lock.json.

$ dim list

Simple List

$ dim list -s

Update

Update the all data.

$ dim update

Update the specified data.

$ dim update [name]

Clean

Clean the project. Delete the data_files and init the project.

$ dim clean

Search

Search data from package_search CKAN API.

Use データカタログ横断検索システム by default to do the search.

$ dim search 避難所

Specify the number of data to get by option -n (default 10).

$ dim search 避難所 -n 3

Interactive installation

Write data information to dim.json from ckan.

Store the data to data_files.

$ dim search -i "東京 避難所"

131105_東京都_目黒区_大地震時における地域避難所
  - Catalog URL        : https://www.geospatial.jp/ckan/dataset/131105-002
  - Catalog Description: ####大地震時における地域避難所のデータです。
####東京都目黒区のオープンデータです。【リソース】大地震時における地域避難所 / ####大地震時における地域避難所のXLSXです。
【キーワード】東京都 / 目黒区 / 避難所
  - Catalog License    : クリエイティブ・コモンズ 表示
    1. 大地震時における地域避難所
      * Resource URL        : https://www.geospatial.jp/ckan/dataset/1e07b569-80a5-4c31-8a7b-be88d1e8f327/resource/8d8de117-2342-4c61-a98d-8f7a9c5b71a2/download/131105evacuationspace.xlsx
      * Resource Description: ####大地震時における地域避難所のXLSXです。
      * Created             : 2018-10-30T02:55:40.179726
      * Format              : XLSX

131059_東京都_文京区_緊急避難場所・避難所
  - Catalog URL        : https://www.geospatial.jp/ckan/dataset/131059-025
  - Catalog Description: ####緊急避難場所・避難所のデータです。
####東京都文京区のオープンデータです。【リソース】緊急避難場所・避難所 / ####文京区の避難所・緊急避難場所の一覧データのCSVです。####更新日:2018年10月23日 / ####文京区の避難所・緊急避難場所の一覧データのXLSXです。
####更新日:2018年10月23日【キーワード】文京区 / 東京都 / 避難場所 / 避難所
  - Catalog License    : CC-BY2.1
    2. 緊急避難場所・避難所
      * Resource URL        : https://www.geospatial.jp/ckan/dataset/b17c1f51-ce1c-4e6a-8ff9-5ff0203b1e43/resource/008d34ad-61a5-4dbd-8996-fa6d647c2986/download/kinkyuhinanbasyo-hinanjo.csv
      * Resource Description: ####文京区の避難所・緊急避難場所の一覧データのCSVです。
####更新日:2018年10月23日
      * Created             : 2018-10-30T05:44:44.623645
      * Format              : CSV
    3. 緊急避難場所・避難所
      * Resource URL        : https://www.geospatial.jp/ckan/dataset/b17c1f51-ce1c-4e6a-8ff9-5ff0203b1e43/resource/0c4942d4-a149-4091-a52f-69b7da8fa143/download/kinkyuhinanbasyo-hinanjo.xlsx
      * Resource Description: ####文京区の避難所・緊急避難場所の一覧データのXLSXです。
####更新日:2018年10月23日
      * Created             : 2018-10-30T05:44:46.127915
      * Format              : XLSX
...
? Enter the number of data to install > 1
? Enter the name. Enter blank if want to use CKAN resource name. > 
? Enter the post-processing you wish to add. Enter blank if not required. > xlsx-to-csv
? Is there a post-processing you would like to add next? (Y/n) > No
Convert xlsx to csv.
Installed to ./data_files/131105_東京都_目黒区_大地震時における地域避難所_大地震時における地域避難所/131105evacuationspace.xlsx

Help

$ dim help

Build the dim from Deno install

Install Deno

$ curl -fsSL https://deno.land/install.sh | sh
$ echo 'export DENO_INSTALL=~/.deno' >> ~/.bashrc
$ echo 'export PATH="$DENO_INSTALL/bin:$PATH"' >> ~/.bashrc
$ source ~/.bashrc

Clone the repository

$ git clone https://github.com/c-3lab/dim.git

$ cd dim

Install dim

$ deno install --unstable --allow-read --allow-write --allow-run --allow-net dim.ts

Upgrade the dim version

You need to be able to run Deno in a local environment.
If you don't have an execution environment of Deno, please re-install.
Install the dim from binary files

$ dim upgrade

Contributers

Made with contributors-img.

LICENSE

MIT LICENSE

nmatsui/dim

dim