When no files match the partition in AzureMLWorkEnvironment, an exception occurs
Closed this issue · 2 comments
SamVanhoutte commented
Describe the bug
When calling this code (without a file matching), an exception happens
work_env = WorkEnvironment.Create(connected=True, datastore_path='arcus_partition_test')
partition_df = work_env.load_tabular_partition(partition_name= 'test-partitioning/BLALBA*', columns=['Close', 'High', 'Isin', 'ItemDate', 'Low', 'Market', 'Open', 'Ticker', 'Volume'])
To Reproduce
Execute the code above with a non matching partition_name
Expected behavior
This should return None, instead of failing with an exception
Additional context
Question open on stackoverflow.
SamVanhoutte commented
This is currently handled in the following approach, but should hopefully be more gracefully handled:
try:
_header = PromoteHeadersBehavior.ALL_FILES_HAVE_SAME_HEADERS if first_row_header else False
_aml_dataset = Dataset.Tabular.from_delimited_files(header=_header,
path=DataPath(datastore, '/' + partition_name + '.csv')) #, set_column_types=columns
_df = _aml_dataset.to_pandas_dataframe()
except DatasetValidationError as dsvalex:
if 'provided path is not valid' in str(dsvalex):
return None
else:
raise
SamVanhoutte commented
according to stackoverflow question, it's the only possibility by catching the exception