How to load parquet files from Azure Data Lake Gen2 to dedicated pool using dbt?
nvinhphuc opened this issue · 1 comments
nvinhphuc commented
Hi,
I want to load my parquet file from Azure Data Lake Gen2/Azure Blob Storage to the dedicated pool in Azure Synapse.
I tried to use
{{ config(materialized='table') }}
with my_table as (
SELECT
TOP 100 *
FROM
OPENROWSET(
BULK 'myparquetfile_url',
FORMAT = 'PARQUET'
) AS [result]
)
select * from my_table
but no luck.
It shows this error
('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Parse error at line: 7, column: 9: Incorrect syntax near 'OPENROWSET'. (103010) (SQLExecDirectW)")
So, how can I load parquets file from blob storage to dedicated pool using dbt?
sdebruyn commented
Hi @nvinhphuc
The OPENROWSET
syntax is not available in Synapse Dedicated SQL Pool. I would recommend raising an issue with your Microsoft partnership contact or with Microsoft Support if you would like to have this functionality in Dedicated SQL.
Take a look at dbt-external-tables for another solution that might help you.