allegroai/clearml-agent

Cloned experiment ignores arguments

copper-ctl opened this issue · 5 comments

In the cloned experiment, arguments appear under the configuration tab but are not used during task run. Task data under the info tab also seems partial:
image

ainoam commented

@copper-ctl Missing some information here: What's the status of the task in question? What are you missing in the info tab? What configuration parameters were present originally and ignored? Which ClearML versions are you using?

Status of task is: completed.
What is missing in the info tab: all fields below ID. Example attached
image
ClearML version: 1.7.2

ainoam commented

@copper-ctl What about clearml server and agent versions? Any difference in console log between original and cloned execution?

clearml-agent: v1.6.1
clearml-server: not sure what you are referring to. api.version = 1.5
changes in console log: The main difference I see is in arguments not being present.

This problem is new for this server. Previously an agent (v1.5.1) was run on a different machine and no such problems were present. The main differences between the two cases:

  1. clearml-server ip has changed
  2. clearml-agent version changed
  3. Previously the agent was run on a host at the same local network as the server. Currently the agent is run on a machine in a different local network but has access to the server via VPN.

I recreated the virtual environment and re-ran the agent and the problem seems to be resolved. Still unsure what was the exact problem. Thanks.