Exception: No schema for resource ExternalStage(STAGE_NAME) found

[michael-lemiale]

Hello, love the project y'all have here!

Likely a mistake on my end, looking for some help - encountering the issue below trying to create external stage to S3 via Titan:

Error

Traceback (most recent call last):
  File "/Users/michaellemiale/Projects/tf-snowflake/titan/main.py", line 1211, in <module>
    plan_and_apply(
  File "/Users/michaellemiale/Projects/tf-snowflake/titan/main.py", line 1201, in plan_and_apply
    plan = blueprint.plan(session)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/michaellemiale/.pyenv/versions/3.11.6/lib/python3.11/site-packages/titan/blueprint.py", line 778, in plan
    manifest = self.generate_manifest(session_ctx)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/michaellemiale/.pyenv/versions/3.11.6/lib/python3.11/site-packages/titan/blueprint.py", line 764, in generate_manifest
    self._finalize(session_ctx)
  File "/Users/michaellemiale/.pyenv/versions/3.11.6/lib/python3.11/site-packages/titan/blueprint.py", line 755, in _finalize
    self._build_resource_graph(session_ctx)
  File "/Users/michaellemiale/.pyenv/versions/3.11.6/lib/python3.11/site-packages/titan/blueprint.py", line 647, in _build_resource_graph
    raise Exception(f"No schema for resource {repr(resource)} found")
Exception: No schema for resource ExternalStage(SNOWFLAKE_S3_STAGE) found

Versions

Python: 3.11.6
Titan: 0.8.9

Goal

Create the following resources:

CREATE STORAGE INTEGRATION SNOWFLAKE_S3_INTEGRATION
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'S3'
  STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::001234567890:role/myrole'
  ENABLED = TRUE
  STORAGE_ALLOWED_LOCATIONS = ('*');

CREATE STAGE SNOWFLAKE_S3_STAGE
  URL='s3://my-s3-bucket/'
  STORAGE_INTEGRATION = SNOWFLAKE_S3_INTEGRATION;

Code

from titan.blueprint import Blueprint, print_plan
from titan.resources import (
    ExternalStage,
    S3StorageIntegration,
)

snowflake_s3_integration = S3StorageIntegration(
    name="SNOWFLAKE_S3_INTEGRATION",
    comment="My comment",
    enabled=True,
    storage_allowed_locations=["*"],
    storage_aws_role_arn="arn:aws:iam::0123456789:role/myrole",
)

snowflake_s3_stage_prod_sources = ExternalStage(
    name="SNOWFLAKE_S3_STAGE",
    comment="My comment.",
    url="s3://my-s3-bucket/",
    storage_integration="SNOWFLAKE_S3_INTEGRATION",
    owner="ACCOUNTADMIN",
)

resources = [snowflake_s3_integration, snowflake_s3_stage_prod_sources]

connection_params = {
    "account": os.environ["SNOWFLAKE_ACCOUNT"],
    "user": os.environ["SNOWFLAKE_USER"],
    "password": os.environ["SNOWFLAKE_PASSWORD"],
    "role": "ACCOUNTADMIN",
}
session = snowflake.connector.connect(**connection_params)

blueprint = Blueprint(
    resources=resources,
)

plan = blueprint.plan(session)
    

[caineblood]

The above comment asking you to download a file is malware to steal your account; do not under any circumstances download or run it. The post needs to be removed. If you have attempted to run it please have your system cleaned and your account secured immediately.

[teej]

@caineblood thanks for looking out!

@michael-lemiale Stages in Snowflake are schema-level resources. Titan needs to know which database/schema to put your stage (SNOWFLAKE_S3_STAGE). A simple fix would be as follows, assuming you want the stage to live in MY_DATABASE.MY_SCHEMA.

snowflake_s3_stage_prod_sources = ExternalStage(
    name="SNOWFLAKE_S3_STAGE",
    comment="My comment.",
    url="s3://my-s3-bucket/",
    storage_integration="SNOWFLAKE_S3_INTEGRATION",
    owner="ACCOUNTADMIN",
    database="MY_DATABASE",
    schema="MY_SCHEMA",
)

[michael-lemiale]

Thanks @teej, that worked! Didn't realize you could leverage the database and schema fields from the ExternalStage docs