tylertreat/BigQuery-Python

Not getting the full list of datasets when there are more than 50 datasets in the project

Bobain opened this issue · 1 comments

I think the code for get_datasets should be fixed with something like:

def get_datasets(self, project_id=None):
    project_id = self._get_project_id(project_id)

    try:            
        datasets = self.bigquery.datasets()
        request = datasets.list(projectId=project_id)
        result = request.execute(num_retries=self.num_retries)
        page_token = result.get('nextPageToken')
        while page_token:
            res = datasets.list(projectId=project_id, pageToken=page_token).execute(num_retries=self.num_retries)
            page_token = res.get('nextPageToken')
            result['datasets'] += res.get('datasets', [])
        return result.get('datasets', [])
    except HttpError as e:
        logger.error("Cannot list datasets: {0}".format(e))
        return None

Would you mind making a PR with the fix?