arcus-azure/arcus.azureml

When no files match the partition in AzureMLWorkEnvironment, an exception occurs

Closed this issue · 2 comments

Describe the bug
When calling this code (without a file matching), an exception happens

work_env = WorkEnvironment.Create(connected=True, datastore_path='arcus_partition_test')
partition_df = work_env.load_tabular_partition(partition_name= 'test-partitioning/BLALBA*', columns=['Close', 'High', 'Isin', 'ItemDate', 'Low', 'Market', 'Open', 'Ticker', 'Volume'])

To Reproduce
Execute the code above with a non matching partition_name

Expected behavior
This should return None, instead of failing with an exception

Additional context
Question open on stackoverflow.

This is currently handled in the following approach, but should hopefully be more gracefully handled:

        try:
            _header = PromoteHeadersBehavior.ALL_FILES_HAVE_SAME_HEADERS if first_row_header else False
            _aml_dataset = Dataset.Tabular.from_delimited_files(header=_header,
                path=DataPath(datastore, '/' + partition_name + '.csv')) #, set_column_types=columns
            _df = _aml_dataset.to_pandas_dataframe()
        except DatasetValidationError as dsvalex:
            if 'provided path is not valid' in str(dsvalex):
                return None
            else:
                raise

according to stackoverflow question, it's the only possibility by catching the exception