Unexpected `ValueError: Could not determine the key type` while testing the tap
Opened this issue · 2 comments
Just getting started with Meltano, and testing this Postgres tap.
So far, I have the following (redacted) meltano.yml config:
version: 1
default_environment: dev
project_id: ...
environments:
- name: dev
- name: staging
- name: prod
plugins:
extractors:
- name: tap-postgres
variant: meltanolabs
pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git
config:
database: ...
filter_schemas:
- ...
flattening_enabled: false
host: ...
ssh_tunnel:
enable: true
host: ...
port: ...
username: ...
user: ...As well as the following .env file
TAP_POSTGRES_PASSWORD='...'
TAP_POSTGRES_SSH_TUNNEL_PRIVATE_KEY='-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY——'
I took the SSH key file content and replaced all new lines with \n, just like in the example test case
tap-postgres/tests/test_ssh_tunnel.py
Line 14 in bcc0db1
Furthermore, I checked out the tap code locally and ran my key against the helper method
tap-postgres/tap_postgres/tap.py
Line 385 in bcc0db1
to be 100% sure that it's valid and detected as an RSA key type.
That's why I'm highly confused to see this when running meltano config tap-postgres test
2023-09-20T12:05:19.866936Z [info ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.
Plugin configuration is invalid
Catalog discovery failed: command ['/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/run/tap-postgres/tap.598e019d-53d3-4dab-99ff-e2240091d9d6.config.json', '--discover'] returned 1 with stderr:
Traceback (most recent call last):
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
sys.exit(TapPostgres.cli())
^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
with self.make_context(prog_name, args, **extra) as ctx:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
self.parse_args(ctx, args)
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
value, args = param.handle_parse_result(ctx, opts, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
value = self.process_value(ctx, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
value = self.callback(ctx, self, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 527, in cb_discover
tap.run_discovery()
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 287, in run_discovery
catalog_text = self.catalog_json_text
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 307, in catalog_json_text
return json.dumps(self.catalog_dict, indent=2)
^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 475, in catalog_dict
result["streams"].extend(self.connector.discover_catalog_entries())
^^^^^^^^^^^^^^
File "/Users/pranas/.pyenv/versions/3.11.3/lib/python3.11/functools.py", line 1001, in __get__
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 378, in connector
url = self.ssh_tunnel_connect(ssh_config=ssh_config, url=url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 429, in ssh_tunnel_connect
ssh_private_key=self.guess_key_type(ssh_config["private_key"]),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 414, in guess_key_type
raise ValueError(errmsg)
ValueError: Could not determine the key type.
ValueError: Could not determine the key type is certainly coming from the guess_key_type method even though I have validated it as explained above. Is this a red herring? How to diagnose further?
Thanks.
This is really a meltano question more than it is a tap question. From the tap perspective we need to focus just on the config file that's sent to the tap. My guess is the line feeds are mangled
Thanks for your response @visch.
I understand your perspective. I'll raise a pertinent issue in the Meltano repository. However, in my opinion, it's equally an implementation decision for the tap to manage SSH keys - which are inherently multi-line - as regular strings.
I wrote #100 (comment). FWIW, I try to avoid storing raw SSH keys as environment variables precisely for the reason that newline characters are treated differently across systems and tools -- I tend to reference file paths (to SSH keys) or, if needs be, use base64 encoding to get rid of the problematic characters. Just my 2 cents.