databricks-sync import is failing consistently

Question

databricks-sync import is failing consistently

pchinta opened this issue 3 years ago · 1 comments

Databricks sync has been failing consistently...with "value out of range error"....triggered from Jenkins. Any help here, please?

databricks-sync -v debug import --profile targetProfile --artifact-dir /home/jenkins/workspace/sandbytes-processing-dbx-workspace-syncup-prod@2/backend/ --backend-file /home/jenkins/workspace/sandbytes-processing-dbx-workspace-syncup-prod@2/backend/backendfile.json -l /home/jenkins/workspace/sandbytes-processing-dbx-workspace-syncup-prod@2/git-repo/ --plan --apply

2022-03-29 12:52:54 [INFO] ===USING LOCAL GIT DIRECTORY: /home/jenkins/workspace/sandbytes-processing-dbx-workspace-syncup-prod@2/git-repo/===
2022-03-29 12:52:54 [INFO] USING HOST: https://adb-xxxxxx.xx.azuredatabricks.net
2022-03-29 12:52:54 [INFO] Setting debug flags on.
2022-03-29 12:52:55 [INFO] Main TF File: {
.....
......
.....
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_dbfs_file.databricks_dbfs_files["/dbfs/certs/xxxxcert.zip"]: Refreshing state... [id=/dbfs/certs/xxxxx.zip][0m
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_instance_pool.databricks_instance_pool_0504_215013_boots354_pool_xxx: Refreshing state... [id=0324-045724-blend21-pool-xxxxx][0m
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_global_init_script.databricks_global_init_scripts["87A27C5F0BFF6A49"]: Refreshing state... [id=E6A6BAB2624D1984][0m
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_global_init_script.databricks_global_init_scripts["B991E778CE439611"]: Refreshing state... [id=B0FBF63CD0D3446A][0m
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_global_init_script.databricks_global_init_scripts["CFCAEB86A3CCC17F"]: Refreshing state... [id=750B7EA7A9458674][0m
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_global_init_script.databricks_global_init_scripts["E8E159F5874B5B1F"]: Refreshing state... [id=9E69055E76148828][0m
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_permissions.databricks_instance_pool_0504_213753_gamey331_pool_xxxxxx_permissions: Refreshing state... [id=/instance-pools/0324-045724-wader20-pool-xxxxx][0m
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_permissions.databricks_instance_pool_0504_214408_mufti352_pool_xxxxx_permissions: Refreshing state... [id=/instance-pools/0324-045724-duck19-pool-xxxx][0m
2022-03-29 12:53:00 [INFO] [0m[1mdatabricks_permissions.databricks_instance_pool_0504_214902_bless353_pool_xxxx_permissions: Refreshing state... [id=/instance-pools/0324-045724-clear22-pool-xxxxx][0m
2022-03-29 12:53:01 [INFO] [0m[1mdatabricks_permissions.databricks_instance_pool_0504_215013_boots354_pool_xxxxx_permissions: Refreshing state... [id=/instance-pools/0324-045724-blend21-pool-xxxx][0m
2022-03-29 12:53:01 [INFO] [0m[1mdatabricks_cluster.databricks_cluster_1124_081110_xxxxx: Refreshing state... [id=0324-045728-xxxxxx][0m
2022-03-29 12:53:01 [INFO] [0m[1mdatabricks_cluster.databricks_cluster_0514_130401_cxxxx: Refreshing state... [id=0324-045728-xxxxx][0m
2022-03-29 12:53:02 [INFO] [0m[1mdatabricks_job.databricks_job_6794856: Refreshing state... [id=504899988646013][0m
2022-03-29 12:53:02 [INFO] [0m[1mdatabricks_job.databricks_job_6745600: Refreshing state... [id=630883479491131][0m
2022-03-29 12:53:02 [INFO] [0m[1mdatabricks_job.databricks_job_6844849: Refreshing state... [id=337966858036044][0m
2022-03-29 12:53:02 [INFO] [0m[1mdatabricks_job.databricks_job_6718960: Refreshing state... [id=1100824357679556][0m
2022-03-29 12:53:02 [INFO] [0m[1mdatabricks_job.databricks_job_6745998: Refreshing state... [id=405210453678418][0m
2022-03-29 12:53:02 [INFO] [0m[1mdatabricks_job.databricks_job_6844157: Refreshing state... [id=722810208591835][0m
2022-03-29 12:53:02 [INFO] [0m[1mdatabricks_permissions.databricks_cluster_1124_081110_xxxxx_permissions: Refreshing state... [id=/clusters/0324-045728-xxxxx][0m
2022-03-29 12:53:02 [INFO] [0m[1mdatabricks_permissions.databricks_cluster_0514_130401_cxxxxx_permissions: Refreshing state... [id=/clusters/0324-045728-xxxxxx][0m
2022-03-29 12:53:02 [INFO] [31m
2022-03-29 12:53:02 [INFO] [1m[31mError: [0m[0m[1mstrconv.ParseInt: parsing "504899988646013": value out of range[0m
2022-03-29 12:53:02 [INFO]
2022-03-29 12:53:02 [INFO] [0m[0m[0m
2022-03-29 12:53:02 [INFO] [31m
2022-03-29 12:53:02 [INFO] [1m[31mError: [0m[0m[1mstrconv.ParseInt: parsing "722810208591835": value out of range[0m
2022-03-29 12:53:02 [INFO]
2022-03-29 12:53:02 [INFO] [0m[0m[0m
2022-03-29 12:53:02 [INFO] [31m
2022-03-29 12:53:02 [INFO] [1m[31mError: [0m[0m[1mstrconv.ParseInt: parsing "1100824357679556": value out of range[0m
2022-03-29 12:53:02 [INFO]
2022-03-29 12:53:02 [INFO] [0m[0m[0m
2022-03-29 12:53:02 [INFO] [31m
2022-03-29 12:53:02 [INFO] [1m[31mError: [0m[0m[1mstrconv.ParseInt: parsing "842600655414946": value out of range[0m
2022-03-29 12:53:02 [INFO]
2022-03-29 12:53:02 [INFO] [0m[0m[0m
2022-03-29 12:53:02 [INFO] [31m
2022-03-29 12:53:02 [INFO] [1m[31mError: [0m[0m[1mstrconv.ParseInt: parsing "337966858036044": value out of range[0m
2022-03-29 12:53:02 [INFO]
2022-03-29 12:53:02 [INFO] [0m[0m[0m
2022-03-29 12:53:02 [INFO] [31m
2022-03-29 12:53:02 [INFO] [1m[31mError: [0m[0m[1mstrconv.ParseInt: parsing "630883479491131": value out of range[0m
2022-03-29 12:53:02 [INFO]
2022-03-29 12:53:02 [INFO] [0m[0m[0m
2022-03-29 12:53:02 [INFO] [31m
2022-03-29 12:53:02 [INFO] [1m[31mError: [0m[0m[1mstrconv.ParseInt: parsing "405210453678418": value out of range[0m
2022-03-29 12:53:02 [INFO]
2022-03-29 12:53:02 [INFO] [0m[0m[0m
2022-03-29 12:53:02 [INFO] [31m
2022-03-29 12:53:02 [INFO] [1m[31mError: [0m[0m[1mstrconv.ParseInt: parsing "1073209794217287": value out of range[0m
2022-03-29 12:53:02 [INFO]
2022-03-29 12:53:02 [INFO] [0m[0m[0m
Traceback (most recent call last):
File "/usr/local/bin/databricks-sync", line 11, in
load_entry_point('databricks-sync==1.0.0', 'console_scripts', 'databricks-sync')()
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/databricks_cli/configure/config.py", line 55, in decorator
return function(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/databricks_sync/cmds/config.py", line 178, in modify_user_agent
return function(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/databricks_sync/cmds/config.py", line 161, in decorator
return function(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/databricks_sync/cmds/apply.py", line 53, in import_cli
te.execute()
File "/usr/local/lib/python3.7/site-packages/databricks_sync/sdk/sync/import.py", line 28, in wrapper
resp = func(self, stage_path=stage_path, **kwargs)
File "/usr/local/lib/python3.7/site-packages/databricks_sync/sdk/sync/import.py", line 55, in wrapper
resp = func(self_, repo_path=repo_path, **kwargs)
File "/usr/local/lib/python3.7/site-packages/databricks_sync/sdk/sync/import_.py", line 160, in execute
state_file_abs_path=state_loc)
File "/usr/local/lib/python3.7/site-packages/databricks_sync/sdk/terraform.py", line 154, in plan
return self._cmd(plan_cmd)
File "/usr/local/lib/python3.7/site-packages/databricks_sync/sdk/terraform.py", line 104, in _cmd
ret_code, ' '.join(cmds), out=out, err=out)
databricks_sync.sdk.terraform.TerraformCommandError: Command 'terraform plan -lock=false -out /home/jenkins/workspace/sandbytes-processing-dbx-workspace-syncup-prod@2/backend/plan.out -input=false' returned non-zero exit status 1.
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

Answer 1 · 2022-03-29T21:31:34.000Z

On Databricks workspace, we see few jobs with ID in 14 digits and over.... while REST API supports INT64... as per error, I believe, databricks-sync is expecting INT.