ssec-jhu/evolver-ng

Unbake calibration_file attr from calibration config.

Opened this issue · 3 comments

Relates to #174

See #174 (comment)

All configs can be loaded from file using model_validate, the calibration files are no exception.

See https://github.com/ssec-jhu/evolver-ng/blob/main/evolver/base.py#L52-L53

class _BaseConfig(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(extra="ignore", from_attributes=True)

    @classmethod
    def model_validate(cls, obj, *, strict=None, from_attributes=None, context=None):
        """Effectively the same as pydantic.BaseModel.model_validate() except that it automatically handles json, and
        conversion from instances of ``BaseConfig`` and ``BaseInterface`
        """
        if obj is None:
            return cls()
        elif isinstance(obj, os.PathLike):  #<------------------------
            return cls.load(file_path=obj)
        ...

It's quite possible that there are a few code paths that need patching or re-directing to ensure that this works from all nominal invocations, # can contain that if needed.

Relates to #165 as the calibration_file is CalibrationData.

One negative aspect of doing this would be the calibration data significantly bloating the overall Evolver.Config. We could special case the model conversion of this so that it always gets serialized to a filepath, that also gets saved to disk (on the PI).
A better solution, to avoid the special casing, could be to add such conversion for all and add a setting, e.g., MAX_CONFIG_SECTION_LINE_LIMIT, that when a nested section exceeds this number of lines/entries it is written to disk and "serialized" in the resultant config json/yaml as a filepath. Obviously, this isn't true serialization and would require some mechanism for copying said sub-files back and forth to the server & client.