embulk/embulk-output-bigquery

Dynamic table creating

sakama opened this issue · 2 comments

sakama commented

create table when data load to BigQuery.

For example fluent-plugin-bigquery support 3method.

  1. List fields in config.yml
  2. Load a schema file in JSON. implemented
  3. Fetch a schema using BigQuery API implemented at v0.2.2

in embulk-output-bigquery...as follows?

load schema from json file implemented

out:
  type: bigquery
  auto_create_table: true
  table: target_table
  schema_path: /path/to/schema.json

load from config.yml

out:
  type: bigquery
  auto_create_table: true
  table: target_table
  schema: id:integer, name:string, time:timestamp

fetch_schema implemented at v0.2.2

out:
  type: bigquery
  fetch_schema: true

load from embulk schema?

But, FileOutputPlugin may not be received schema from embulk.
And support for NULLABLE/REQUIRED/REPEATED field is difficult.

out:
  type: bigquery
  auto_create_table: true
  table: target_table
  schema:
sakama commented

load schema from json file was implemented at v0.1.2

sakama commented

fetch_schema was implemented at v0.2.2.