target-s3-parquet

target-s3-parquet is a Singer target for S3Parquet.

Metadata and Partition

By default we add a new sdc metadata called _sdc_started_at and defined this attribute as a partition to make easier to detect the latest extraction.

Initialize your Development Environment

pipx install poetry
poetry install

Create and Run Tests

Create tests within the target_s3_parquet/tests subfolder and then run:

poetry run pytest

You can also test the target-s3-parquet CLI interface directly using poetry run:

poetry run target-s3-parquet --help

Testing with Meltano

Note: This target will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Your project comes with a custom meltano.yml project file already created. Open the meltano.yml and follow any "TODO" items listed in the file.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd target-s3-parquet
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke target-s3-parquet --version
# OR run a test `elt` pipeline with the Carbon Intensity sample tap:
meltano elt tap-carbon-intensity target-s3-parquet

SDK Dev Guide