backwards and forwards compatibility, the exact contents of pickle files, and IntEnum
I'm not sure exactly how to phrase this inquiry, so please bear with me. What exactly does backwards compatibility mean as far as pickle goes? We have the various protocols, we have the contents of pickle files created at those protocols, and we have different versions of Python. Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a Python 3.3 system? Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4. I'm thinking the answer is yes, it should be. So how do we fix it? There are a couple different options: - modify IntEnum pickle methods to return the name only - modify IntEnum pickle methods to pickle just like the int they represent The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version. The second option has the advantage that the actual pickle contents are the same [1] as in previous versions. So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or is it enough if unpickling them returns the correct object? -- ~Ethan~
On 3/14/2015 10:52 PM, Ethan Furman wrote:
I'm not sure exactly how to phrase this inquiry, so please bear with me.
What exactly does backwards compatibility mean as far as pickle goes? We have the various protocols, we have the contents of pickle files created at those protocols, and we have different versions of Python.
Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a Python 3.3 system? Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4.
I'm thinking the answer is yes, it should be.
So how do we fix it? There are a couple different options:
- modify IntEnum pickle methods to return the name only
- modify IntEnum pickle methods to pickle just like the int they represent
The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version.
The second option has the advantage that the actual pickle contents are the same [1] as in previous versions.
So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or is it enough if unpickling them returns the correct object? In general, it would seem to me that each version of Python could add object types that are foreign to prior versions, and so as long as version N can unpickle anything produced by versions <=N, all would be well.
While I am _not_ an expert in pickle, this would just be my expectation based on years of experience with compatibility issues in non-Python non-pickle systems. Pickle may have stronger compatibility goals than I would expect, and so must you, if you are thinking the answer to the above is "yes, it should be". :)
On 03/14/2015 11:46 PM, Glenn Linderman wrote:
In general, it would seem to me that each version of Python could add object types that are foreign to prior versions, and so as long as version N can unpickle anything produced by versions <=N, all would be well.
The question to me is because we not so much adding something brand-new (socket.AF_INET has been there for ages), as changing the type of something already existing -- I'm just not sure if we need "end-result" compatibility (achieved by the first option) or if we need "bug for bug" compatibility (achieved by the second option). -- ~Ethan~
On 15.03.15 07:52, Ethan Furman wrote:
So how do we fix it? There are a couple different options:
- modify IntEnum pickle methods to return the name only
- modify IntEnum pickle methods to pickle just like the int they represent
The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version.
The second option has the advantage that the actual pickle contents are the same [1] as in previous versions.
So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or is it enough if unpickling them returns the correct object?
With the second option you lost the type even for 3.5+. This is a step back.
On Sat, 14 Mar 2015 22:52:17 -0700 Ethan Furman <ethan@stoneleaf.us> wrote:
I'm not sure exactly how to phrase this inquiry, so please bear with me.
What exactly does backwards compatibility mean as far as pickle goes? We have the various protocols, we have the contents of pickle files created at those protocols, and we have different versions of Python.
Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a Python 3.3 system? Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4.
I'm thinking the answer is yes, it should be.
I agree with you that this should not break.
So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or is it enough if unpickling them returns the correct object?
It's certainly enough if unpickling returns the correct object. Exact serialization is an implementation detail. Regards Antoine.
participants (4)
-
Antoine Pitrou -
Ethan Furman -
Glenn Linderman -
Serhiy Storchaka