On Tue, Nov 23, 2010 at 10:25 PM, Éric Araujo wrote:

Hi,

...

Recently I had the need to filter objects based on whether they're picklable or not:

http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-...

...

I'm not sure what's a good way to check for a specific object whether

it's

...

picklable.

< http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-... This led me to think: Maybe we should have an `Unpicklable` abstract base class in the `collections` module? Then various unpicklable classes, like locks, files or widgets, could inherit from this class to signify that they cannot be pickled.

What do you think?

This sounds useful. I’d rather spell the ABC pickle.Picklable, though.

Regards

(Spelling note: People told me that "pickleable" (with an "e" in the middle) makes more sense, so I'm using that now.) The best solution might be to have both a `Pickleable` class and an `Unpickleable` class. The reason to have the former is that `isinstance(thing, Pickleable)` is more natural, and the reason to have the latter is because we can't require people to inherit from `Pickleable` for every single class that they define. (Since pickleability is the rule and unpickleability is the exception.) So `Pickleable` could have a `__subclasshook__` that would do the real work, similarly to `Iterable`. Ram. -- Sincerely, Ram Rachum

Antoine Pitrou writes:

On Tue, 23 Nov 2010 22:46:51 +0200 cool-RR wrote:

...

(Spelling note: People told me that "pickleable" (with an "e" in the middle) makes more sense, so I'm using that now.)

The best solution might be to have both a `Pickleable` class and an `Unpickleable` class. The reason to have the former is that `isinstance(thing, Pickleable)` is more natural, and the reason to have the latter is because we can't require people to inherit from `Pickleable` for every single class that they define. (Since pickleability is the rule and unpickleability is the exception.)

So `Pickleable` could have a `__subclasshook__` that would do the real work, similarly to `Iterable`.

The problem is that "doing the real work" can be CPU-intensive, and I don't think we would like isinstance() calls to become arbitrarily expensive (even if technically it's possible).

An alternative is simply to catch the exception when trying to pickle a value. That implies that the type takes care to raise an exception if pickling would work but unpickling wouldn't, which is what file objects now do.

By the way, "Unpickleable" doesn't work since it's ambiguous: you don't know whether it means you can unpickle the thing, or you can't pickle it.

Regards

Antoine.

Sorry, I didn't explain well enough: I meant that only inherently unpickleable objects, like files and locks, will be instances of `Unpickleable`. So if we have a list containing a file or a lock, it will still be an instance of `Pickleable`, as well as an object which refers to a file/lock as an attribute. Maybe this is a good case for having only the negative `Unpickleable` and not the positive `Pickleable`. Ram.

On Tue, Nov 23, 2010 at 4:48 PM, Antoine Pitrou wrote:

Read what I said above about the ambiguity of the name, though.

Indeed. But there's always pickle.Me.Not. :-)   -Fred -- Fred L. Drake, Jr.    <fdrake at acm.org> "A storm broke loose in my mind."  --Albert Einstein

On 11/23/2010 6:32 PM, Greg Ewing wrote:

Antoine Pitrou wrote:

...

By the way, "Unpickleable" doesn't work since it's ambiguous: you don't know whether it means you can unpickle the thing, or you can't pickle it.

Nonpickleable?

Ephemeral? As an added bonus, twisted already uses this terminology, see: twisted.persisted.styles.Ephemeral. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

On 11/28/2010 4:11 AM, Antoine Pitrou wrote:

On Sat, 27 Nov 2010 21:31:02 -0500 Scott Dial wrote:

...

On 11/23/2010 6:32 PM, Greg Ewing wrote:

...

Nonpickleable?

Ephemeral?

As an added bonus, twisted already uses this terminology, see: twisted.persisted.styles.Ephemeral.

Twisted has a taste for "smart" obscure names (can you guess what Avatar and Portal are for?),

"pickle" is an ironic context to complain about "smart" obscure names. Anyways, yes, I do know what those classes are, but I have used them before. Analogously, I suspect that is also the only reason why "Nonpickleable" seems like an "obvious" choice to you. But, next you are gonna want a "NonMarshalable" and "NonJSONable" and "NonBananaable" and "NonJellyable" and every other version of persistence. Or, you pick a name that describes the property that you really want to describe (e.g., ephemeral). -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

On 11/28/2010 5:31 AM, Antoine Pitrou wrote:

Le dimanche 28 novembre 2010 à 05:24 -0500, Scott Dial a écrit :

...

On 11/28/2010 4:11 AM, Antoine Pitrou wrote:

...

On Sat, 27 Nov 2010 21:31:02 -0500 Scott Dial wrote:

...

Ephemeral?

As an added bonus, twisted already uses this terminology, see: twisted.persisted.styles.Ephemeral.

Twisted has a taste for "smart" obscure names (can you guess what Avatar and Portal are for?),

"pickle" is an ironic context to complain about "smart" obscure names. Anyways, yes, I do know what those classes are, but I have used them before. Analogously, I suspect that is also the only reason why "Nonpickleable" seems like an "obvious" choice to you.

Nonpickleable (spelling and casing notwithstanding) quite obviously means "which can't be pickled". I'm not sure what you're arguing about, or if you're just arguing for the sake of having an argument :)

I don't know why you snipped and ignored the part where I explained why "Ephemeral" was a better choice: On 11/28/2010 5:24 AM, Scott Dial wrote:

But, next you are gonna want a "NonMarshalable" and "NonJSONable" and "NonBananaable" and "NonJellyable" and every other version of persistence. Or, you pick a name that describes the property that you really want to describe (e.g., ephemeral).

The OPs problem happens to be using "pickle", but this is not a problem exclusive to pickling; there are a bunch of serialization methods in the stdlib and elsewhere, and his question generalizes to all of them. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

On Sun, Nov 28, 2010 at 2:24 AM, Scott Dial wrote:

On 11/28/2010 4:11 AM, Antoine Pitrou wrote:

...

On Sat, 27 Nov 2010 21:31:02 -0500 Scott Dial wrote:

...

On 11/23/2010 6:32 PM, Greg Ewing wrote:

...

Nonpickleable?

Ephemeral?

As an added bonus, twisted already uses this terminology, see: twisted.persisted.styles.Ephemeral.

Twisted has a taste for "smart" obscure names (can you guess what Avatar and Portal are for?),

"pickle" is an ironic context to complain about "smart" obscure names. Anyways, yes, I do know what those classes are, but I have used them before. Analogously, I suspect that is also the only reason why "Nonpickleable" seems like an "obvious" choice to you.

But, next you are gonna want a "NonMarshalable" and "NonJSONable" and "NonBananaable" and "NonJellyable" and every other version of persistence. Or, you pick a name that describes the property that you really want to describe (e.g., ephemeral).

I see your point but I still veto ephemeral -- as a non-native English speaker with a limited vocabulary it is too obscure for me (isn't it something to do with astrology? :-). Antoine is channeling me well today. :-) -- --Guido van Rossum (python.org/~guido)

Terry Reedy writes:

On 11/29/2010 12:10 AM, Guido van Rossum wrote:

...

I see your point but I still veto ephemeral -- as a non-native English speaker with a limited vocabulary it is too obscure for me (isn't it something to do with astrology? :-).

You are thinking of 'ephemeris':

I interpreted the Dutch smiley as meaning that Guido was aware of that difference :-)

"a tabular statement of the assigned places of a celestial body for regular intervals ". Such tables are produced and use by astronomers and used by navigators to determine ship position at night. And yes, astrologers piggyback their crap on such tables also ;-).

To be fair, ephemerides have been produced for many thousands of years, by people who were simultaneously practicing what we would today distinguish as both astrology and astronomy. The astrology crap was not piggybacked, but integral to the process and purpose of the ephemeris from its beginning, until we finally teased them apart. -- \ “One of the most important things you learn from the internet | `\ is that there is no ‘them’ out there. It's just an awful lot of | _o__) ‘us’.” —Douglas Adams | Ben Finney

On 30/11/2010 04:38, Carl M. Johnson wrote:

On Mon, Nov 29, 2010 at 11:27 AM, Ben Finney mailto:ben%2Bpython@benfinney.id.au> wrote:

> "a tabular statement of the assigned places of a celestial body for > regular intervals ". Such tables are produced and use by astronomers > and used by navigators to determine ship position at night. And yes, > astrologers piggyback their crap on such tables also ;-).

by people who were simultaneously practicing what we would today distinguish as both astrology and astronomy. The astrology crap was not piggybacked, but integral to the process and purpose of the ephemeris from its beginning, until we finally teased them apart.

"Astrology" and "astronomy" are clearly misnamed.

X-ology = the study of X. What do astronomers do? They study stars.

X+nomy = the laws of X. What do astrologers do? They "show" how the laws of the stars rule our lives.

Clearly, we need to fire up the time machine and fix this one, or astronomy will never get the increased userbase it deserves.

There's a theory that it's not possible to travel back in a time machine to before it was created. Since Guido wasn't even born then, it's not possible to fix that.

Am 28.11.2010 13:40, schrieb Masklinn:

...

Twisted has a taste for "smart" obscure names (can you guess what Avatar and Portal are for?), which is not in core Python's tradition. NonPickleable would be fine IMO.

Why not `pickle.Incompatible`? It's not like the ABC has anything to do with collections, so it would make more sense for it to live in `pickle` (which you've already imported if you're trying to pickle an object) rather than collections.

pickle.Indigestable? Georg

> Why not `pickle.Incompatible`? It's not like the ABC has anything

pickle.Indigestable?

Undigestable surely...

I agree with putting whatever in the pickle module. If there were ever demand for json.Incompatible or whatever, there we have it. -- Terry Jan Reedy

Ram Rachum

On 2010-11-30, at 17:52 , Dima Tisnek wrote:

Ahem, what happened to duck typing? Nothing.

To me, the whole isinstance(datum, type) appears no more elegant than the Java way (class implements this interface, implements that interface, ad infinitum) You're wrong. In fact, it could hardly be more different than what Java does. Python's ABCs are simply used to name ducks (and maybe one day tools can use them to implement a restricted structural static type system for python, who knows).

For instance, as of Python 2.7 these blocks of code are roughly equivalent (the first two are in fact identical in effect): if isinstance(obj, collections.Sized): doSomethingWith(len(obj)) if hasattr(obj, '__len__'): doSomethingWith(len(obj)) try: doSomethingWith(len(obj)) except TypeError: pass They all test for the same thing: that you can call len() on the object. They don't test anything about your type hierarchy or which interfaces you inherit, which would be the only thing doable in Java (barring abuses of reflection). `collections.Sized` is simply a name for "I can call len() on this object".

anyhow it's not pythonic. The inclusion of abcs in Python strongly seems to disagree with that statement.

how to test that object can do .lower(), .startswith(), .decode(), but not all methods of str/unicode? Depends if you can give a name to that structure. That's what ABCs do when they're used as "instance testers" (they can also be used as mixins which is very nice in that you don't have to reimplement every single method of `set` yourself to create a new and separate set type): they check that the instance provided has the right structure. Not that it inherits from the right class or implement the right methods. If you check what `isinstance(obj, collections.Iterator)` does for instance, it aliases to this:

hasattr(obj, "next") and hasattr(obj, "__iter__") the former looks much more readable than the latter, doesn't it? And they mean the exact same thing. It's not like ABCs remove your ability to EAFP

For instance, as of Python 2.7 these blocks of code are roughly equivalent (the first two are in fact identical in effect):

if isinstance(obj, collections.Sized): doSomethingWith(len(obj))

if hasattr(obj, '__len__'): doSomethingWith(len(obj))

try: doSomethingWith(len(obj)) except TypeError: pass

They all test for the same thing: that you can call len() on the object.

Not exactly: magic methods are looked up on the class, not on the instance. That’s why isintance+ABCs is the right test to use here. See http://docs.python.org/dev/reference/datamodel#special-method-names (The third block also suffers from an over-catching except clause. Put as little code as possible in the try block and use an else clause.) ABCs are not contradictory to duck typing at all:

...

...

import collections class Demo: ... def __len__(self): ... return 42 ... isinstance(Demo(), collections.Sized) True

Demo is not required to subclass Sized or register. ABCs are an optional mechanism that implements duck typing, so to speak. http://docs.python.org/dev/glossary http://docs.python.org/whatsnew/2.6#pep-3119-abstract-base-classes http://www.python.org/dev/peps/pep-3119/#abcs-vs-duck-typing Regards

On Wed, Dec 1, 2010 at 8:08 AM, Masklinn wrote:

...

 That’s why isintance+ABCs is the right test to use here.  See http://docs.python.org/dev/reference/datamodel#special-method-names Due to Python's lookup rule, looking up most (if not all) of the special method on the instance is going to give the same result.

It's the "most" in that sentence that makes the ABC the more accurate check of the examples given. Checking the instance does work most of the time (since most instances don't have magic methods attached) but fails abysmally when the instance in question is a class and the intent is to invoke the magic methods of the metaclass (usually 'type'). Deliberately bypassing the instance with either type(x) or x.__class__ is the closest pure Python code can get to correctly following the special method lookup rules: http://docs.python.org/reference/datamodel.html#special-method-lookup-for-ne... Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Éric Araujo wrote:

Not exactly: magic methods are looked up on the class, not on the instance. That’s why isintance+ABCs is the right test to use here.

That seems to be an exceedingly fine straw to split here. I'd be very surprised if someone deliberately gave an instance an attribute called "__len__" and relied on it both not being recognised as a special method and not being picked up by LYBL tests. -- Greg

On Sat, 4 Dec 2010 11:14:28 +0000 (UTC) Ram Rachum wrote:

Ram Rachum

I see a lot of discussion in this thread about use of `isinstance` in general. What about the `pickle.(In)compatible` proposal? Is it something I should write a PEP for?

You don't need to write a PEP. Just open an issue on http://bugs.python.org and propose a patch. Regards Antoine.

On Tue, Nov 23, 2010 at 10:54 AM, Antoine Pitrou wrote:

...

So `Pickleable` could have a `__subclasshook__` that would do the real work, similarly to `Iterable`.

The problem is that "doing the real work" can be CPU-intensive, and I don't think we would like isinstance() calls to become arbitrarily expensive (even if technically it's possible).

An alternative is simply to catch the exception when trying to pickle a value. That implies that the type takes care to raise an exception if pickling would work but unpickling wouldn't, which is what file objects now do.

Yes, but isn't the advantage of having an ABC that you only have to "do the real work" one time and after that the class will be registered as Pickleable/NonPickleable? So, the first time you check a random class it would see if it can call the Pickling methods and if so register that class so future isinstance checks are virtually free and don't have to be dealt with using try/except. Maybe I'm misunderstanding something.

On Tue, Nov 23, 2010 at 12:25 PM, Éric Araujo wrote:

Hi,

...

Recently I had the need to filter objects based on whether they're picklable or not:

http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-...

I'm not sure what's a good way to check for a specific object whether it's picklable.

http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-...This led me to think: Maybe we should have an `Unpicklable` abstract base class in the `collections` module? Then various unpicklable classes, like locks, files or widgets, could inherit from this class to signify that they cannot be pickled.

What do you think?

This sounds useful.  I’d rather spell the ABC pickle.Picklable, though.

Indeed; let's not turn the `collections` module into a repository of miscellaneous ABCs. I'm looking at you, collections.Callable! Cheers, Chris -- Burn the witch! http://blog.rebertia.com