Please, make keras objects picklable
Closed this issue · 3 comments
Thanks a lot for Keras. Excellent modeling tool/API.
I need to be able to send keras trained models through a web service. No access to a filesystem, no permissions. I do this already for scikit-learn models flawlessly. The difference is that scikit-learn models can be pickled easily.
Pickle in python is not an optional feature. It is a very smart serialization protocol allowing saving/loading object states and can be extended so that any combination of picklable objects is also picklable.
Keras models can be saved as hd5 files. This is not an easily extendable feature. I mean, saving 10 models will need 10 hd5 files and a guy to control where these files are, assign names to them , have the permissions, necessary disk space, etc. not mentioning human errors and time cost (paycheck ;). piuckle can do that transparently.
Please note that this is only a software issue, nothing to do with machine learning etc. It is more an attempt to make keras objects behave like any "civilized" python object.
Adding pickle functionality, will make serialization of keras objects straightforward/transparent. Think of an object A containing an image, a keras model and some text. the object A is picklable if all its members/components are picklable. Without picklability, every user has to spend time on defining a way to save/load the object A.
I did not speak technical here, because it is rather a python philosophical issue. Some attempsts already exist, like this http://zachmoshe.com/2017/04/03/pickling-keras-models.html, but it is still not Ok for me (uses hd5 and creates a temp file for each keras object).
There is also a set of previous issues about saving/loading keras and, for me, the existence of these issues is just a result of non-picklability.
I know this doesn't address your philosophical desire to make Keras models pickle-able, but HDF5 can store many objects within a single file. You can have a HDF5 file that contains Keras models, text, and images.
I agree that HDF5 can contain many objects , but one has to define how to do that and in which order etc.
HDF5 is a data serialization format, not a universal serialization protocol like pickle. Think of serializing an object that has not yet been developed (next best model by lecun et al.). is it easy to store it in HDF5 ? For pickle, a lot of stuff is already available for this object. And in a lot of cases, nothing is to be done.