dbx doesn't like notebooks in artifact directories
Opened this issue · 0 comments
neil-katuna-at-mscience commented
Expected Behavior
I expect to be able to specify paths for notebooks in the artifact directory.
notebook_task:
notebook_path: "file://path/in/artifact/directory/notebook.py"
Current Behavior
The above line breaks with error:
Response from server:
{ 'error_code': 'INVALID_PARAMETER_VALUE',
'message': 'Invalid notebook_path: '
'dbfs:/dbx/my_project/98de20ce1cb84aeb9421368ee49e5872/art
ifacts/example_pipeline/do_one_last_thing.py. '
'Only absolute paths are currently supported. Paths must begin '
"with '/'."}
It's easy enough to see that removing the "dbfs:" part of the path would resolve the issue.
Steps to Reproduce (for bugs)
Context
Your Environment
Using cluster configurations:
new_cluster:
spark_version: "12.2.x-scala2.12"
driver_node_type_id: "c6gd.4xlarge"
node_type_id: "c6gd.4xlarge"
num_workers: 8
aws_attributes:
first_on_demand: 1
availability: "SPOT_WITH_FALLBACK"
zone_id: "auto"
instance_profile_arn: "{{ instance_profile_arn }}"
spot_bid_price_percent: 100
ebs_volume_count: 0
- dbx version used: 0.8.18
- Databricks Runtime version: 12.2