dagster-io/talks

Error running motherduck-dagster-hybrid-compute pipeline

Closed this issue · 0 comments

I've cloned the repository to wsl2 on my windows machine, I created a .env file containing the following:

# MotherDuck / DuckDb Connection String
#
# Can either be a simple <filename>.duckdb, or a connection string to MotherDuck.
#
# References:
# - https://motherduck.com/docs/authenticating-to-motherduck/#authentication-using-a-service-token
#
DUCKDB_DATABASE="local.duckdb"

# Evidence Connection Variables
#
# References
#  - https://motherduck.com/docs/integrations/evidence/
#  - https://docs.evidence.dev/cli#environment-variables
#
EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__TYPE="duckdb"
EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__FILENAME="local.duckdb"

I ran make install and then make dev, after which I clicked materialize all in the dagster webpage and get the following error:

Invalid Input Error: Python Object "birds" of type "AssetsDefinition" found on line "/home/maxisq/dev/dagster-project-demos/motherduck-dagster-hybrid-compute/motherduck_dagster_hybrid/assets.py:145" not suitable for replacement scans.
Make sure that "birds" is either a pandas.DataFrame, duckdb.DuckDBPyRelation, pyarrow Table, Dataset, RecordBatchReader, Scanner, or NumPy ndarrays with supported format

Line 145 in assets.py doesn't appear to mention a replacement scan at all and googling "dagster AssetDefinition type not suitable for replacement scan error" seems to show not relevant results.

Can anyone advise on how I can fix this error?