databrickslabs/dbx

dbx doesn't like notebooks in artifact directories

Opened this issue · 0 comments

Expected Behavior

I expect to be able to specify paths for notebooks in the artifact directory.

notebook_task:
  notebook_path: "file://path/in/artifact/directory/notebook.py"

Current Behavior

The above line breaks with error:

 Response from server: 
 { 'error_code': 'INVALID_PARAMETER_VALUE',
  'message': 'Invalid notebook_path: '
             'dbfs:/dbx/my_project/98de20ce1cb84aeb9421368ee49e5872/art
ifacts/example_pipeline/do_one_last_thing.py. '
             'Only absolute paths are currently supported. Paths must begin '
             "with '/'."}

It's easy enough to see that removing the "dbfs:" part of the path would resolve the issue.

Steps to Reproduce (for bugs)

Context

Your Environment

Using cluster configurations:

new_cluster:
              spark_version: "12.2.x-scala2.12"
              driver_node_type_id: "c6gd.4xlarge"
              node_type_id: "c6gd.4xlarge"
              num_workers: 8
              aws_attributes:
                first_on_demand: 1
                availability: "SPOT_WITH_FALLBACK"
                zone_id: "auto"
                instance_profile_arn: "{{ instance_profile_arn }}"
                spot_bid_price_percent: 100
                ebs_volume_count: 0
  • dbx version used: 0.8.18
  • Databricks Runtime version: 12.2