snowflake-so/snowflake-sdk

Is there a way to execute the query but not fetch the results?

Closed this issue · 1 comments

Hello,
We are trying to optimize complex calculations where we need to first run one query and then use the RESULT_SCAN to perform additional work on top of the previous results.

However, I haven't found any way to execute the query and get queryId without fetching the results. The first query we do produces a huge result and the process just crashes hitting the RAM limit. I could increase the RAM limit but that seems wrong since I don't need the result in the first place. 🤔

Here's what I tried so far.
This way I get the statement ID after the query is executed successfully and I can use it right away but the results are still fetched and memory usage is huge.

const connection = await this.getConnection();
const statement = connection.execute({
    sqlText: preparedSql,
    binds: parameters,
    complete: (error, statement) => {
       const statementId = statement.getStatementId();
    },
});

This way is even worse. Not only the rows are still fetched but also the statementId appears before the query fully executed so there's still some gap before I can actually use it in RESULT_SCAN.

const connection = await this.getConnection();
const statement = connection.execute({
    sqlText: preparedSql,
    binds: parameters,
});

// Here I'm basically checking every second if statementId exists or not.
let statementId;
while (!statementId) {
    statementId = statement.getStatementId();
    await new Promise((resolve) => setTimeout(resolve, 1000));
}

I feel like there must be a way to execute the query without fetching results (especially since it's a use-case mentioned in the documentation: https://docs.snowflake.com/en/user-guide/querying-persisted-results#post-processing-query-results) but I can't find it.

Sorry, seems like I created this in the wrong repo. I wanted to report to the official SDK. 🤦‍♂️