pickle protocol 5 is coming
irmen opened this issue · 6 comments
see https://www.python.org/dev/peps/pep-0574/
It seems that pickle will have a new protocol (5) that will appear in Python 3.8 (which right now has been released as a first beta version, so it's getting nearer)
We will have to support this protocol in the Unpickler at least.
The Pickler still writes protocol level 2 which is fine. Upgrading to 3 or 4 may be beneficial for some data types however that would lose the compatibility with Python 2. I think after 1 jan 2020 when Python 2.7 reaches end of life, it would be a good time to perhaps update the Pickler. Then again, it's not broken...
java support for this added in 8358ad4
dotnet support for this added in 3c7d519
@adamsitnik @stephentoub @eerhardt wondering if you will encounter these pickles later in Spark ? I guess it will eventually use Python 3.8 as well?
cc @airhorns @mengxr @JoshRosen I believe you used Pyrolite's java implementation with Spark. I have no idea what Spark's plans are regarding the python version it uses? Do you have any insights?
Sorry, haven't been down in the guts of Spark in a while. I know that they support Python 3 now and they just published this: https://spark.apache.org/news/plan-for-dropping-python-2-support.html , but I would suggest getting in touch with some of the core PySpark devs as they can likely share better info than me, or better yet, sponsor some of this development!
I've already got a first implementation done to support the new protocol 5 opcodes :)
version 4.30 has been released containing the unpickling support for this. Let's just wait and see if additional changes are required once Python 3.8 hits the streets