enhancement request: make py3 read/write py2 pickle format
Robert Kern
robert.kern at gmail.com
Wed Jun 10 07:22:47 EDT 2015
On 2015-06-10 12:04, Neal Becker wrote:
> Chris Warrick wrote:
>
>> On Tue, Jun 9, 2015 at 8:08 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> One of the most annoying problems with py2/3 interoperability is that the
>>> pickle formats are not compatible. There must be many who, like myself,
>>> often use pickle format for data storage.
>>>
>>> It certainly would be a big help if py3 could read/write py2 pickle
>>> format. You know, backward compatibility?
>>
>> Don’t use pickle. It’s unsafe — it executes arbitrary code, which
>> means someone can give you a pickle file that will delete all your
>> files or eat your cat.
>>
>> Instead, use a safe format that has no ability to execute code, like
>> JSON. It will also work with other programming languages and
>> environments if you ever need to talk to anyone else.
>>
>> But, FYI: there is backwards compatibility if you ask for it, in the
>> form of protocol versions. That’s all you should know — again, don’t
>> use pickle.
>
> I believe a good native serialization system is essential for any modern
> programming language. If pickle isn't it, we need something else that can
> serialize all language objects. Or, are you saying, it's impossible to do
> this safely?
By the very nature of the stated problem: serializing all language objects.
Being able to construct any object, including instances of arbitrary classes,
means that arbitrary code can be executed. All I have to do is make a pickle
file for an object that claims that its constructor is shutil.rmtree().
This is fine in some use cases (e.g. wire format for otherwise-secured
communication between two endpoints under your complete control), but it is
worrying in others, like your use case of data storage (and presumably sharing).
Python 2/3 is also the least of your compatibility worries there. Refactor a
class to a different module, or did one of your third-party dependencies do
this? Poof! Your pickle files no longer work.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
More information about the Python-list
mailing list