Cannot create tables out of SQL Select statements
Rahul-Gajula opened this issue · 14 comments
- I am interested to know what data types for the table creation the DBT-Synapse support.
- Attached are the screenshots of the error what was captured while doing the runs
Out of all this I was able to generate the views from the select SQL statements against the database, but not the tables.
Note: Under dbt.project.yml i clearly defined the materialization as table to perform the action.
what type of Azure synapse dedicated pools are you using: dedicated or serverless? My guess is you're using a serverless pool which do not support tables. You may want to try dbt-msft/dbt-synapse-serverless instead. Though huge caveat that serverless pools aren't that great of a solution for synapse today.
Hi swanderz , could not reply you immediately.
I am using a dedicated SQL pool. No luck with table creation.
weird! here's some questions:
- what version of
dbt-synapseare you using? - are you using seeds or sources for these transformations?
- what is the column type of
device_id? - can you share the info from the latest run in the log (
logs/dbt.log)?
Thanks @Rahul-Gajula. I’ll try to repro this error and see what I find
Looking at the error message I see:
Database error: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The statement failed. Column 'device_id' has a data type that cannot participate in a columnstore index.
Any chance you can change your column types to not use nvarchar(max)? Not only will it not work with dbt-synapse, but it's not a good idea as it isn't very optimized.
@swanderz you're absolutely right. Running this directly in Synapse validates that you cannot have nvarchar(max) as a datatype:
However, the below works with no errors:
Hi alieus and swanderz,
Tried creating table as per the comments. Failed yet again.
It's not even recognizing the Create Syntax
Hey @Rahul-Gajula , if you’d like to test this, you’d have to run it directly in your synapse workspace.
What you need to do is change the datatypes in your source tables from nvarchar(max) to nvarchar(n) and then run your select queries again in dbt.
Hi alieus,
Even with the data type nvarchar(n) i am not able to create a table.
CREATE TABLE telemetry_test(
device_id NVARCHAR(n),
IP_adress NVARCHAR(n)
);
where n is the size of the string in byte-pairs. For example, nvarchar(16).
Hi alieus,
Even that i tried. refer to the screen shot.
Hi alieus and swanderz,
i figured out the solution for it. i need to change my data type to make it work. Using convert operator in the sql syntax worked.
Thank you for your kind help. you may close the ticket.
Glad to hear it! I’m sure this issue will definitely help someone else in the future.