meltano/sdk

bug: Tap fails if stream maps config contains a key for an unselected stream

Opened this issue · 1 comments

Singer SDK Version

0.31.1

Is this a regression?

  • Yes

Python Version

NA

Bug scope

Mapping (stream maps, flattening, etc.)

Operating System

NA

Description

Consider the countries sample in this repo.

  1. Generate the Singer catalog for the tap:

    $ poetry run python samples/sample_tap_countries --discover > countries.catalog.json
  2. Create a stream maps config for the tap. This one creates slugs for both the continent and country name:

    $ jq < countries.stream_maps.json
    {
        "stream_maps": {
            "continents": {
                "slug": "str(name.lower().replace(' ', '-'))"
            },
            "countries": {
                "slug": "str(name.lower().replace(' ', '-'))"
            }
        }
    }
  3. Run the tap with the generated catalog and stream maps config:

    $ poetry run python samples/sample_tap_countries --catalog countries.catalog.json --config countries.stream_maps.json > countries.singer.jsonl

    This should work as expected.

  4. Now, open countries.catalog.json and edit it to deselect the countries stream

    $ jq '.streams[1].metadata[-1]' < countries.catalog.json
    {
      "breadcrumb": [],
      "metadata": {
        "inclusion": "available",
        "selected": false,   <----- this right here needs to be 'false'
        "selected-by-default": true,
        "table-key-properties": [
          "code"
        ]
      }
    }
  5. Run the tap again:

    $ poetry run python samples/sample_tap_countries --catalog countries.catalog.json --config countries.stream_maps.json > countries.singer.jsonl

    You'll end up with

    singer_sdk.exceptions.StreamMapConfigError: Invalid key properties for 'countries': [code]. Property 'code' was not detected in schema.
    

Code

No response

stale commented

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.