- What is Cliboa
- Features
- Manual
- How to Contribute
- Quick Start
- Install Cliboa
- Configuration of a Simple ETL Processing
- Directory Tree
- Install python modules
- Write a Scenario of ETL Processing
- Set an environment
- Execute a scenario of ETL Processing
Cliboa is an application framework which can implement ETL processing. It eases the implementation of ETL processing. In this case, ETL Processing means the processings like fetch, transform and transfer of data between various databases, storages, and other services.
- Python based framework.
- ETL processing is executable by YAML based configuration.
- Additional modules for ETL processing can be implemented by only a few steps if not enough.
See MANUAL.md
See CONTRIBUTING.md
Available on any Linux distributions, like Debian, Ubuntu, CentOS, REL, or etc.
python version 3.4 or later and pipenv are required. In the environemnt which pip can be used, execute as below.
sudo pip3 install pipenv
sudo pip3 install cliboa
After installed cliboa, 'cliboadmin' can be used as an administrator command.
Create an executable environment of cliboa by using cliboadmin.
$ cd /usr/local
$ cliboadmin init sample
$ cd sample
$ cliboadmin create simple-etl
Directory tree which was created aforementioned commands is as below.
├── bin
│ └── clibomanager.py
├── common
│ ├── environment.py
│ ├── __init__.py
│ ├── scenario
│ └── scenario.yml
├── conf
├── logs
├── Pipfile
└── project
└── simple-etl
├── scenario
└── scenario.yml
$ cd sample
$ pipenv install --dev
or
$ cd sample
$ pipenv lock -r > requirments.txt
$ sudo pip3 install -r requirements.txt
As a simple etl processing, write scenario.yml in simple-etl as below.
The following example is just download a gzip file from the local sftp server, decompress it, and upload it to the local sftp server.
See Example1
To make the above scenario available, set a local machine as a sftp server according to respective environments. Also, put "test.csv.gz" under /usr/local.
After wrote scenario.yml and set the environment, execute a scenario by as below command.
cd sample
pipenv run python3 bin/clibomanager.py saimple-etl
or
cd sample
python3 bin/clibomanager.py saimple-etl