CustomTarget for very specific load/dump methods in Luigi Extension

[gabrielgonzaga]

I am facing a problem where I need to save a model but it does not support the pickle process. The library that implements this model already provides a method to save the file, but it has some logic in it. One way to benefit from the luigi_extension's features for target load/dump would be to create a new Target with the provided logic. That way would work, but I think it would become messy with time, as more specific patterns for load/dump will appear in the future. Another way would be to not use luigi_extension's Target, but then I lose the benefits of using a luigi_extension's Target.

The proposal is to create a CustomTarget, since the LocalTarget already receives the Task related to the Target, and, in the Task, the user would define the load and dump methods for this CustomTarget that could be named target_load and target_dump, something like that. That way, there is a generic form of defining targets directly (and related to) the task's processes.

E.g.

class MyTask(Task):
TARGET = CustomLocalTarget
def easy_run(self, inputs):
...
return model

target_load(self):
    ... # logic for the loading process, very specific for a few use cases

target_dump(self, data):
    .... # logic for the dump process, very specific for a few use cases

Can I implement that or do you think there is an easier more reliable path? @rafonseca @rafarui

[rafonseca]

I think the proper way of doing this is to implement a new target. We have faced the same problem before. It will not become messy, if only add a new class (imports should be made inside init method)

[rafonseca]

You can either extend PyCarolTarget or LocalTarget, or both.

[gabrielgonzaga]

Yes, extending sounds better indeed, already implemented and seems nice. Thank you @rafonseca !