[Bug] `docs generate` does not find all schemas when there are more than 10K schemas
mikealfare opened this issue · 0 comments
mikealfare commented
Current Behavior
When there are more than 10K schemas, docs generate
does not find all of the schemas, likely because of not paginating the results. This causes objects in those schemas to be missing from the catalog. This affects all docs generate
runs for 1.6
and prior, and only docs generate --no-compile
for 1.7
and forward. See #1216 for more information.
Expected Behavior
All objects should be included in the catalog.
Steps To Reproduce
- Create more than 10K schemas
- Create another schema (named
test_schema
for the sake of reference here) - Run
docs generate --no-compile
test_schema
will be empty in the catalog
Environment
- OS: all
- Python: all
- dbt-core: all
- dbt-bigquery: all (but differently for 1.7 and forward)
Additional Context
We actually found this during testing because our test schemas were building up for a test class, though this would affect any user who also has 10K schemas.