dbt-labs/dbt-bigquery

[Bug] `docs generate` does not find all schemas when there are more than 10K schemas

mikealfare opened this issue · 0 comments

Current Behavior

When there are more than 10K schemas, docs generate does not find all of the schemas, likely because of not paginating the results. This causes objects in those schemas to be missing from the catalog. This affects all docs generate runs for 1.6 and prior, and only docs generate --no-compile for 1.7 and forward. See #1216 for more information.

Expected Behavior

All objects should be included in the catalog.

Steps To Reproduce

  1. Create more than 10K schemas
  2. Create another schema (named test_schema for the sake of reference here)
  3. Run docs generate --no-compile
  4. test_schema will be empty in the catalog

Environment

- OS: all
- Python: all
- dbt-core: all
- dbt-bigquery: all (but differently for 1.7 and forward)

Additional Context

We actually found this during testing because our test schemas were building up for a test class, though this would affect any user who also has 10K schemas.