bug: Tap fails if stream maps config contains a key for an unselected stream
Opened this issue · 1 comments
edgarrmondragon commented
Singer SDK Version
0.31.1
Is this a regression?
- Yes
Python Version
NA
Bug scope
Mapping (stream maps, flattening, etc.)
Operating System
NA
Description
Consider the countries sample in this repo.
-
Generate the Singer catalog for the tap:
$ poetry run python samples/sample_tap_countries --discover > countries.catalog.json
-
Create a stream maps config for the tap. This one creates slugs for both the continent and country name:
$ jq < countries.stream_maps.json { "stream_maps": { "continents": { "slug": "str(name.lower().replace(' ', '-'))" }, "countries": { "slug": "str(name.lower().replace(' ', '-'))" } } }
-
Run the tap with the generated catalog and stream maps config:
$ poetry run python samples/sample_tap_countries --catalog countries.catalog.json --config countries.stream_maps.json > countries.singer.jsonl
This should work as expected.
-
Now, open
countries.catalog.json
and edit it to deselect thecountries
stream$ jq '.streams[1].metadata[-1]' < countries.catalog.json { "breadcrumb": [], "metadata": { "inclusion": "available", "selected": false, <----- this right here needs to be 'false' "selected-by-default": true, "table-key-properties": [ "code" ] } }
-
Run the tap again:
$ poetry run python samples/sample_tap_countries --catalog countries.catalog.json --config countries.stream_maps.json > countries.singer.jsonl
You'll end up with
singer_sdk.exceptions.StreamMapConfigError: Invalid key properties for 'countries': [code]. Property 'code' was not detected in schema.
Code
No response
stale commented
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen
label, or request that it be added.