Failed to run katib on Kale, pipeline did not generate the corresponding run
Closed this issue · 1 comments
The error message is as follows：
I0927 06:34:52.337060 15 main.go:136] 2021-09-27 06:34:52 Kale kfputils:176 [INFO] Creating KFP experiment 'automltest-3z2ww'...
I0927 06:34:52.338298 15 main.go:136] Traceback (most recent call last):
I0927 06:34:52.338327 15 main.go:136] File "", line 1, in
I0927 06:34:52.338362 15 main.go:136] File "/usr/local/lib/python3.6/site-packages/kale/common/katibutils.py", line 152, in create_and_wait_kfp_run
I0927 06:34:52.338379 15 main.go:136] **kwargs)
I0927 06:34:52.338404 15 main.go:136] File "/usr/local/lib/python3.6/site-packages/kale/common/kfputils.py", line 177, in run_pipeline
I0927 06:34:52.338515 15 main.go:136] experiment = client.create_experiment(experiment_name)
I0927 06:34:52.338537 15 main.go:136] File "/usr/local/lib/python3.6/site-packages/kfp/_client.py", line 342, in create_experiment
I0927 06:34:52.338560 15 main.go:136] experiment = self._experiment_api.create_experiment(body=experiment)
I0927 06:34:52.338569 15 main.go:136] File "/usr/local/lib/python3.6/site-packages/kfp_server_api/api/experiment_service_api.py", line 187, in create_experiment
..............
I0927 06:34:52.340799 15 main.go:136] File "/usr/local/lib/python3.6/http/client.py", line 1264, in putheader
I0927 06:34:52.340825 15 main.go:136] if _is_illegal_header_value(values[i]):
I0927 06:34:52.340840 15 main.go:136] TypeError: expected string or bytes-like object
F0927 06:34:53.448167 15 main.go:365] Failed to wait for worker container: Training container is failed. Unable to read file /var/log/katib/8.pid for pid 8, error: open /var/log/katib/8.pid: no such file or directory
The create_and_wait_kfp_run method has an error. Thank you very much！