Bigquery Source needs to set allowLargeResults to True when making requests to the API
elibixby opened this issue · 2 comments
elibixby commented
Currently the bigquery source fails even with quite small roles since the threshold for responseTooLarge will be exceeded by the unchangeable 10000 rows per request for even very small roles.
Not sure how anyone is using bigquery input this way...
chamikaramj commented
This shouldn't happen if you use DataflowPipelineRunner. I believe you are using DirectPipelineRunner which is not designed to handle large datasets.
See: #18