allegroai/clearml-docs

Documentation: Error and lack of clarity in description of models

Make42 opened this issue · 6 comments

Make42 commented

In https://clear.ml/docs/latest/docs/clearml_sdk/model_sdk/ it says

  • Model - represents an experiment's output model (training results). An OutputModel is always connected to a task.
  • InputModel - represents an existing ClearML model to be used in an experiment.
  • OutputModel - represents a ClearML model, regardless of any task connection.

Where is OutputModel mentioned in the first point? There is some error - not sure what is wrong though. Maybe the texts for Model and OutputModel are switched?

ainoam commented

Thanks for spotting @Make42! Indeed the texts have been switched.

I'm moving this issue to the docs repo. If you like, you can issue a PR to fix :)

Make42 commented

Also, I am really not clear on what the difference between the model classes are. As far as I understand, the workflow is like this:

  1. I define some model. Then I register it as an OutputModel. Then I train it. During training I save snapshots (not idea how, though) and then I save the final model when training is finished. This way the Model is a) connected to the task and b) available in the model store of ClearML.
  2. Later, in a different task, I can load an already trained model with InputModel. This InputModel is read-only (regarding the ClearML model store), but I can make a copy on which I work and - e.g. - further train. So, what I would think, I get the model with InputModel from the model store, make a copy, register the copy with my new task as an OutputModel and then go on as I did in the last paragraph 1.

No idea, how Model comes into play here. Let me compare Model:

class Model()
Represent an existing model in the system, search by model id. The Model will be read-only and can be used to pre initialize a network

with InputModel

class InputModel()
Load an existing model in the system, search by model ID. The Model will be read-only and can be used to pre initialize a network. We can connect the model to a task as input model, then when running remotely override it with the UI.
Load a model from the Model artifactory, based on model_id (uuid) or a model name/projects/tags combination.

Sounds just the same to me. What is the difference?

I would like to ask for this to be documented in more detail. :-)

ainoam commented

Apologies for the ambiguity @Make42, the class hierarchy is indeed somewhat messy, and we have plans to simplify it in future versions.
In the mean time, let me try to clarify:

  • Model : This class is the basic interface to the ClearML model store Use it to programmatically access and manage the model store.
  • InputModel : Use this class when the model is used in the context of a ClearML experiment, as it's initial starting point.
  • OutputModel : Use this class for a model that is the result of your ClearML experiment.

We'll make a point to improve the docs in this regard :)

Make42 commented

@ainoam : I started fixing the issue with the pull request I just made. However, this only contributes to the swap mentioned at the beginning. Please add more clarity on the problem, as I have described it in my later posts.

ainoam commented

Thanks @Make42. You'll note #479 indeed addresses the extra details discussed, so this issue is taken care of?

Make42 commented

@ainoam: Are all the changes what I can see at https://github.com/allegroai/clearml-docs/pull/479/files?
This is helpful, because I see now, what the general idea is and when I am supposed to use each class.

It still leaves open the question, what the capability each of those have - in other words - why the architects decided to create three classes instead of only two. I somewhat got it after re-reading the text in https://clear.ml/docs/latest/docs/clearml_sdk/model_sdk and compared the available methods and their signatures in https://clear.ml/docs/latest/docs/references/sdk/model_model/ and https://clear.ml/docs/latest/docs/references/sdk/model_inputmodel, but maybe this can be made more explicit.