[Python-ideas] iterable.__unpack__ method

[Alex Stewart]

On Sunday, February 24, 2013 7:43:58 PM UTC-8, Terry Reedy wrote:

> On 2/24/2013 5:07 PM, Alex Stewart wrote: 
> >  1. It is a fairly small change to the language 
>
> To me, taking something simple and elegant and complexifying it, to 
> little benefit I can see other than to solve a problem in the wrong 
> place, in no small matter.


I did not say it was a small matter.  I said it was a small change to the 
language.  From a purely practical perspective of what changes would be 
required to the language itself in order to support this proposal, in my 
opinion, it would not require that much to implement (it would probably 
only require a couple of lines of new code and some doc changes), and is 
very unlikely to affect any existing code or tools.

(Syntax changes are, in my opinion, far more disruptive changes to the 
language:  They require a change to the parser as well as the interpreter, 
and potentially changes to all manner of other utilities 
(debuggers/code-checkers/AST-manipulators/etc) out there in the wild.  They 
make it much harder to write Python code that supports older Python 
releases (with method changes, you can check for the existence of a feature 
in the interpreter and selectively use or not use it, but you typically 
can't use new syntax constructs in anything that might be run in an older 
Python release, because the parser will bomb out instead).  They also 
require programmers to learn new syntax (if nothing else, to understand 
what it means when they come across it in somebody else's code).  What's 
more, every syntax change potentially limits what can be done with the 
syntax in the future (for example, by using up special characters or 
constructs so they're not available for other things), and so on. 
 Personally, I think there should be a much higher bar for syntax changes 
than for adding a new special method, not the other way around.)

You also seem to be defining "in the wrong place" as "in a different way 
than I personally have a use for"..  As I pointed out (which you seem to 
have completely skipped over in your reply), there are use cases where the 
consumer side is the right (or only) place, and there are other different 
use cases where the producer side is the right (or only) place.  The two 
are not mutually-exclusive, and improving one does not really address the 
other issue at all.

I want to emphasize that I am not saying that we shouldn't extend unpacking 
notation to support partial-unpacking; in fact I think it might well be a 
good idea.  What I am saying, however, is that if we do, I do think it's a 
much bigger change to the language than what I proposed, and it still 
doesn't actually address most of the issues that __unpack__ was intended to 
deal with, so it's really not an either-or sort of thing anyway.

Certainly, adding a new special method to 
> some category of objects has not effect unless changes are made to the 
> interpreter elsewhere to make automatic use of that method. Otherwise, 
> you could just define a normal method to be used by user code.


My proposal does not include adding a new special method to any existing 
objects.  It adds the _ability_ for people to add a special method to 
objects *in the future* and have the interpreter take advantage of it.  It 
therefore should have no effect for any existing code.

>  2. There are (in my opinion) already pretty good parallels with this 
> >     sort of change that have already been implemented in Python (for 
> >     example, separating the concept of "iterable" from the concept of 
> >     "indexable"). 
>
> The related but distinct concepts of sequential access and random access 
> are basic to computation and were *always* distinct in Python.


Oh really?  Then how, prior to the development of the iterator protocol, 
did one define an object which was accessible sequentially but not randomly 
in the Python language?  If you couldn't do that, you can't claim that the 
concepts were really distinct in Python, in my opinion.

The truth is that the concept of an iterable object vs. an indexable object 
may have been distinct *in some people's minds*, but it was *not* originally 
distinct in the Python language itself.  People decided that that should 
change, so the language was extended to make that happen.  It should also 
be noted that the two concepts were not distinct in *everyone's* minds (I 
remember straightening out confusion on this point in numerous people's 
understanding (and still need to occasionally), particularly those who did 
not come from a classical computer-science background).  However, just 
because there were some people who didn't see them as distinct did not mean 
that it was an invalid way for other people to view them, or that they 
shouldn't ultimately be distinct in the language..

With all respect (and I do actually mean that), your primary argument 
against it seems to be one of intellectual complacency: "I've never 
considered that they might be different, so they shouldn't ever be 
separated, because then I don't have to change the way I think about 
things".

The addition of __iter__ along with __next__ 
> added more benefits than __next__ alone would have. 
>

You seem to be making my point for me.  In essence, __unpack__ is really 
just the equivalent for the unpacking operation to what __iter__ is for 
"for" loops.  Is it strictly required in order to support the basic 
behavior?  No (you could do for loops over simple iterables with just a 
variant of __next__, if you really wanted to), but it allows some more 
sophisticated objects to provide more useful behaviors by giving them more 
context about how they are being invoked (in the case of __iter__, by 
telling them ahead of time that they are going to be used in an iteration 
context, and allowing them to create or alter the initial object that the 
for loop interacts with).  This provides "more benefits than __next__ alone 
would have", so it was added to the language.

Logically, by the same argument, and in almost exactly the same way, 
__unpack__ provides more benefits than __iter__ alone would have (by 
telling them that they are going to be used in a particular unpacking 
context, and allowing them to alter the object used for that type of 
iteration), and therefore should arguably be part of the language as well.

>  3. It is consistent with the general idea in Python that an object can 
> >     choose to emulate whichever behaviors of other objects make sense, 
> >     and not emulate other ones that do not make sense.  In my opinion, 
> >     "iterating", and "unpacking" are two related-but-not-the-same 
> >     language concepts, which currently are unnecessarily entangled. 
>
> I do not see that at all. As near as I can tell, both mean 'sequential 
> access'.


Not true.  Iteration means "sequential access".  Unpacking means 
"sequential access with inherent length constraints", which is the bit you 
seem to be ignoring.  The most notable consequence of this (but definitely 
not the only one) is that a for loop (for example) will never raise 
ValueError, but the unpacking operation will, if it gets what it considers 
"wrong" results back from the unpacked object.  In this way the two 
operations do not interpret the data that the object in question provides 
in the same way, and it does not really have the same meaning.

Yes, from a purely mechanical perspective, they perform similar operations, 
but programming interfaces are not solely defined by mechanics.  The 
difference here is one of implied contracts.  In simple iteration, there is 
explicitly no minimum or maximum number of values that the iterable is 
required to produce, and in fact the entire protocol is based on the 
assumed perspective that returning a value is always the "success" 
condition and not being able to return one is the "failure" case, and in 
that context it makes sense to design iterators so that they should always 
return more data if they're able to do so.  In unpacking, however, when it 
gets to the end of the variable list, the conditions are reversed, and 
actually returning a value becomes a failure condition.  This change of 
meaning, however, is not communicated in any way to the iterator (this is 
just one example: there are similar contract issues inherent in extended 
unpacking as well).  This is, in my opinion, a flaw in the interface, 
because the language is misrepresenting (bounded) unpacking as (unbounded) 
iteration when calling the iterable object.

I admit I do not understand your __unpack__ proposal, since it seemed 
> vague and incomplete to me. But until I see your conceptual distinction, 
> the details do not matter to me.
>

I'm not exactly sure why you consider it vague or incomplete.  I provided a 
specific description of exactly what changes to the language I was 
proposing, with method signatures, a description of the new interpreter 
behavior and examples of data values in typical scenarios, and accompanied 
it with a list of several different use-cases which I believed it would 
provide significant utility.  I'm not really sure how I could have gotten 
more specific without providing a diff against the CPython sources (which 
seems a bit much for an initial proposal)..

>  4. It would help significantly with several types of problems that do 
> >     not currently have great solutions. 
>
> I obviously do not see the same pile of problems that would justify a 
> new special method.


Obviously, although I'm not really sure why not because I did explicitly 
state a bunch of them in my original post.  You seem to have just ignored 
all of them except for the issue of partial unpacking, which frankly was 
not even the most important or interesting use case I presented, in my 
opinion.  I would recommend you might want to go back and read the last 
part of my initial post where I presented several potential use cases for 
all this.

I do get the distinct impression, however, that you've already made up your 
mind and, right or wrong, there's nothing I could possibly say which would 
ever change it, so at this point I'm not sure whether it really matters 
(sigh)...

> no way to query an iterator to find out if it has more data available  

 

See how in my separate post: 
> "Add lookahead iterator (peeker) to itertools" 


I'm not necessarily opposed to that proposal, but I should point out that 
it does not actually address the problem I mentioned in this thread at all, 
so it's really kinda irrelevant to that discussion.

(Even if that is added to itertools, unpacking still won't use it, nor does 
it provide any way to safely query iterables which produce side-effects, 
even if there might be a way for them to provide that info without the 
side-effects, nor does it provide a way to test for end conditions without 
changing the underlying iterable (which might be used elsewhere in scopes 
outside the lookahead-wrapper), etc..)

--Alex

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130225/b6de1651/attachment.html>