Target for Postgres.
Built with the Meltano SDK for Singer Taps and Targets. This target is in development, it probably doesn't work yet, stick with https://hub.meltano.com/loaders/target-postgres . Generally the goal here is to create a generalized target enough so that the SDK can automate >80% of testing for new targets, and potentially so taps can test very easily with a real local target.
about
stream-maps
schema-flattening
Setting | Required | Default | Description |
---|---|---|---|
host | False | None | Hostname for postgres instance. Note if sqlalchemy_url is set this will be ignored. |
port | False | 5432 | The port on which postgres is awaiting connection. Note if sqlalchemy_url is set this will be ignored. |
user | False | None | User name used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
sqlalchemy_url | False | None | SQLAlchemy connection string. This will override using host, user, password, port, dialect. Note that you must esacpe password special characters properly see https://docs.sqlalchemy.org/en/20/core/engines.html#escaping-special-characters-such-as-signs-in-passwords |
dialect+driver | False | postgresql+psycopg2 | Dialect+driver see https://docs.sqlalchemy.org/en/20/core/engines.html. Generally just leave this alone. Note if sqlalchemy_url is set this will be ignored. |
stream_maps | False | None | Config object for stream maps capability. For more information check out Stream Maps. |
stream_map_config | False | None | User-defined config values to be used within map expressions. |
flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
flattening_max_depth | False | None | The max depth to flatten schemas. |
A full list of supported settings and capabilities is available by running: target-postgres --about
-
Developer TODO:
Come back to this re #5
pipx install -e .
This Singer target will automatically import any environment variables within the working directory's
.env
if the --config=ENV
is provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the .env
file.
The database account provided must have access to:
- Create schemas
- Create tables (DDL)
- Push Data to tables (DML)
You can easily run target-postgres
by itself or in a pipeline using Meltano.
target-postgres --version
target-postgres --help
# Test using the "Carbon Intensity" sample:
pipx install git+https://gitlab.com/meltano/tap-carbon-intensity
tap-carbon-intensity | target-postgres --config /path/to/target-postgres-config.json
pipx install poetry
poetry install
Create tests within the target_postgres/tests
subfolder and
then run:
poetry run pytest
You can also test the target-postgres
CLI interface directly using poetry run
:
poetry run target-postgres --help
Testing with Meltano
Note: This target will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.
Your project comes with a custom meltano.yml
project file already created.
Next, install Meltano (if you haven't already) and any needed plugins:
# Install meltano
pipx install meltano
# Initialize meltano within this directory
meltano install
Now you can test and orchestrate using Meltano:
# Test invocation:
meltano invoke target-postgres --version
See the dev guide for more instructions on how to use the Meltano SDK to develop your own Singer taps and targets.