googleapis/python-bigquery-magics

add an "engine" parameter and context property to support bigframes DataFrame output

Closed this issue · 2 comments

Sub-tasks:

  • Add a bigquery_magics.context.engine option which defaults to "pandas" but can be set to "bigframes" for %%bigquery to return a BigQuery DataFrames object.
  • The default option can also be overridden by specifying --engine="bigframes" in the first line of the %%bigquery magics.

Googlers, see go/bf-magics design.

Benefits for why the user might want to support this:

  • Support queries that return > 10 GB of results.
  • Hopefully some day: more natural "async" support so as not to block the current thread while the query runs.

Another thought: We should use a code path that doesn't immediately run the query at read_gbq / read_gbq_query time (internal issue 347050555). Otherwise the magic would end up running at least two queries: the original query & one or two to visualize the results from __repr__.