On 6 July 2013 17:40, SpaghettiToastBook . wrote:

I like the PEP overall, and it seems to cover everything.

Ty.

I'm not sure that the function definition changes should be included due to backward incompatibility.

I decided when writing this that I'd just go and include everything, seeing as it's easier to remove things I've included than add things I haven't.

I do have one question about this line:

"- ``lambda *args, last: ...`` no longer requires ``last`` to be a keyword only argument"

What exactly is backward incompatible about this change?

To be particular, every extension to syntax is backward incompatible. Imagine any case where you have: try: exec("now-valid-syntax") except SyntaxError: expected_to_happen() That's obviously far-fetched, but the point is across. Any code that requires it to fail when "last" isn't passed in will break. For example, code that "builds" arguments to the function using input from the user (Ranger might do this, I'm not sure). It's also a bit far-fetched, but it might count. It might also change the behaviour of certain types of currying. Probably the most important change is that "(lambda *args, last=None: print(args, last))(1, 2, 3, 4, 5)" prints "(1, 2, 3, 4, 5) None", whereas with the PEP as it is will print "(1, 2, 3, 4) 5". This is quite a nonobvious change, so I should add it to the PEP.

Also, on line 34, "keywords" should be "keyword" instead.

Additionally, "A further extension to comprehensions is a logical and necessary extension." (line. 108) is quite a silly thing to write. I'll upload these fixes in-bulk when I update the PEP.

On 6 July 2013 21:38, Jan Kaliszewski wrote:

While I like all the literal/comprehension/etc.-related proposals, those related to function definitions seem to me to be dubious.

I am not convinced by the argument of unification with the assignment LHS syntax.

Assignment LHS and function parameter processing are inherently different in Python anyway:

Agreed, but similarity still lowers cognitive load.

Making possible to define some positional arguments after *args does not seem to be a big win, and would destroy nice simplicity of the way you specify keyword-only arguments.

My personal interpretation is that it simplifies things; if you want keyword-only arguments, you use a lone star. Otherwise you don't get them. I find that comprehensively simpler, especially as it is more in-tune with assignment. I'm not actually sure what levels of backward-incompatibility are deemed reasonable between releases, so I'm not sure whether there's any point arguing for this. I also seem to have magically created this - it seems not to be in the original implementation. Hence if no-one supports the idea, and since I'm not very attached to it, there's no loss in letting it go. It'd be easier if I had a running version of the implementation to test against (I wouldn't just make things up), but as I said above I'm finding it difficult to figure out.

Forbidding keyword arguments before *args in function calls does not seem so bad, but still it is a serious backwards incompatibility... And why would we actually want to forbid it?

I included it because my understanding is that it was in the original patch. I'm not sure why anyone would want to forbid it, other than it being easier to write the patch that way. Compatibility aside, I'm not sure why anyone would want to keep it either, though.

On 6 July 2013 23:06, Guido van Rossum wrote:

On Sat, Jul 6, 2013 at 2:35 PM, Joshua Landau wrote:

...

On 6 July 2013 21:38, Jan Kaliszewski wrote:

...

Forbidding keyword arguments before *args in function calls does not seem so bad, but still it is a serious backwards incompatibility... And why would we actually want to forbid it?

I included it because my understanding is that it was in the original patch.

I'm not sure why anyone would want to forbid it, other than it being easier to write the patch that way. Compatibility aside, I'm not sure why anyone would want to keep it either, though.

In this case, compatibility trumps everything, and we should keep it for sure.

But even if we had a choice, my experience tells me that it's a good thing to keep, because nobody can remember the rules of what goes before what.

Then should we expand to allow arbitrary mixing of keyword and positional arguments (which sounds reasonable if we want to allow keyword arguments before *args, and also treat *args like any positional argument)?

What do yo mean by Status Quo? No version of Python supports multiple *args or a regular positional arg after *args or after kw=arg. The only flexibility that was added "recently" (in 2.6) is that kw=arg may now follow *args. On Sat, Jul 6, 2013 at 4:03 PM, Joshua Landau wrote:

On 6 July 2013 23:20, Joshua Landau wrote:

...

Then should we expand to allow arbitrary mixing of keyword and positional arguments (which sounds reasonable if we want to allow keyword arguments before *args, and also treat *args like any positional argument)?

To give more hints as to what I am saying:

Original Proposal::

Function calls may accept an unbound number of ``*`` and ``**`` unpackings, which are allowed anywhere that positional and keyword arguments are allowed respectively. In approximate pseudo-notation::

function( argument or *args, argument or *args, ..., kwargument or **kwargs, kwargument or **kwargs, ... )

This has been rejected, primarily as it is not worth the backwards-incompatibility.

Status Quo::

Function calls may accept an unbound number of ``*`` and ``**`` unpackings. Keyword-arguments must follow positional arguments, and ``**`` unpackings must also follow ``*`` unpackings. In approximate pseudo-notation::

function( argument or *args, argument or *args, ..., kwargument or *args, kwargument or *args, ..., kwargument or **kwargs, kwargument or **kwargs, ... )

Looser rulings::

Function calls may accept an unbound number of ``*`` and ``**`` unpackings. Arguments can now occur in any position in a function call. As usual, keyword arguments always go to their respective keys and positional arguments are then placed into the remaining positional slots. In approximate pseudo-notation::

function( argument or keyword_argument or *args or **kwargs, argument or keyword_argument or *args or **kwargs, ... ) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- --Guido van Rossum (python.org/~guido)

I guess I have to read the whole PEP first. :-) That will be after July 9th (the Dropbox developer conference). On Sat, Jul 6, 2013 at 4:55 PM, Joshua Landau wrote:

On 7 July 2013 00:34, Guido van Rossum wrote:

...

What do yo mean by Status Quo? No version of Python supports multiple *args or a regular positional arg after *args or after kw=arg.

The only flexibility that was added "recently" (in 2.6) is that kw=arg may now follow *args.

It's the one that changes the least -- given that people seem to have accepted [1] multiple unpacking, and that you seem to have already (in the issue tracker) accepted [2] "foo(*a, b, c)"¹, should we continue with the restriction that "Keyword-arguments must follow positional arguments, and ``**`` unpackings must also follow ``*`` unpackings"?

That has the fewest changes, but I believe that given [1] and [2] that these restrictions are either insufficient (hence the rejected "Original Proposal" from the PEP) or confusing (hence the additional "Looser rulings").

aka. I didn't mean Status Quo as "change nothing", but "change nothing other than those two things we already seem to like".

¹ And, by extension, one could also assume you support "foo(**a, b=...)"

-- --Guido van Rossum (python.org/~guido)

On 8 July 2013 01:56, Jan Kaliszewski wrote:

On 07.07.2013 01:03, Joshua Landau wrote:

...

Function calls may accept an unbound number of ``*`` and ``**`` unpackings. Arguments can now occur in any position in a function call. As usual, keyword arguments always go to their respective keys and positional arguments are then placed into the remaining positional slots. In approximate pseudo-notation::

function( argument or keyword_argument or *args or **kwargs, argument or keyword_argument or *args or **kwargs, ...

What do you exactly mean by "remaining positional slots"? Please note that the current behaviour is to raise TypeError when several (more than 1) arguments match the same parameter slot. IMHO it must be kept.

You're right -- I've never gotten that error before, so this is actually new to me. That is a nicer solution, and it keeps things clean.

Another question is related to this matter as well: if we adopt the idea of more than one **kwargs in function call -- what about key duplication? I.e. whether:

fun(**{'a': 1}, **{'a': 2})

...should raise TypeError as well, or should it be equivalent to fun(a=2)?

My first thought was that it should raise TypeError -- prohibition of parameter duplication is a simple and well settled rule for Python function calls. On second thought: it could be relaxed a bit if we agreed about another rule that would be simple enough, e.g.: "for anything *after* the first '**kwargs' (or maybe also bare '**,'?) another rule is applied: later arguments override earlier (looking from left to right), as in dict(...)/.update(...) or as in {**foo, **bar} in literals (if the rest of the PEP is accepted).

My first opinion would be that if relaxation is something people find useful, it would be suited to a separate proposal; it seems outside of this PEP's scope á mon avis. Given:

...

...

{1:"original", 1:"override"} {1: 'override'}

the most consistent behaviours would be what are in the PEP already, and I think that's worth keeping. --- Thinking about examples, the two cases ("status quo" rules [1] and relaxed rules [2]) would allow things like: def f(a, b, c=0, d=0, e=0): ... "Status Quo" rules: f(a, e=e, d=d, *[b, c]) Relaxed rules only: f(a, e=e, d=d, b, c) I brought up the idea for the Relaxed rules because the priority rules for arguments are somewhat complicated when you add in the ability to have multiple *args and **kwargs, and remove the restriction of *args after positionals and **kwargs after positionals. However, considering that the Relaxed rules are never actually useful AFAICT (there's no real reason to define positionals after keywords), this would be a simplification to the specification alone. That'll make it easier to learn the rules, I believe, but simply saying "write your arguments in a sane order" should do more than enough to cover it anyway. Personally, the rule from the issue itself (positionals, then keywords) is the simplest, but I agree with Guido that it's not worth breaking backwards compatibility. In a sense, then, the best way to describe the "Status Quo" as: Positionals, then Keywords -- but *if you must* you are allowed to put "*args" after keywords. I'm still undecided, so I'll leave this for others to comment on. An updated version of the PEP that removes the changes to function definitions and discusses the alternatives for function calls is attached. I haven't double-checked it, so it may be a bit rougher around the edges.

A blessing from the Gods has resulted in http://www.python.org/dev/peps/pep-0448/! See what you think; it's not too changed from before but it's mighty pretty now. Still up for discussion are the specifics of function call syntax, the full details of which should already be in the PEP. If you come up with a better suggestion or want argue for one of the choices, go ahead.

On 14 July 2013 12:04, Guido van Rossum wrote:

On Fri, Jul 12, 2013 at 5:25 PM, Joshua Landau wrote:

...

A blessing from the Gods has resulted in http://www.python.org/dev/peps/pep-0448/! See what you think; it's not too changed from before but it's mighty pretty now.

Still up for discussion are the specifics of function call syntax, the full details of which should already be in the PEP. If you come up with a better suggestion or want argue for one of the choices, go ahead.

I like it.

Finally read it myself - looks promising.

I note that we now end up with new ways for concatenating sequences (e.g. [*a, *b]) and also for merging dicts (e.g. {**a, **b}). I think it would be good to prepare an implementation in time for inclusion in Python 3.4a1 to avoid the same issue with this we had before -- I could imagine that there might be some implementation problems and I don't want to accept an unimplementable PEP. Also it would be good to know that code not using the new syntax won't run any slower (especially for function calls this is very important).

I believe we should be able to confine those changes to the bytecode generation, which would mean existing code would be unaffected.

Regarding the decision about the allowable syntax for argument lists, I prefer to keep the existing restriction (making *args after a keyword argument basically an exception) since, as you point out, placing regular positional arguments after regular keyword arguments looks plain silly.

Agreed. One interesting point I see is that the "*expr" syntax in comprehensions is getting close to a nested "yield from":

...

...

list((yield from x) for x in ([1], [2, 3], [4, 5, 6])) [1, None, 2, 3, None, 4, 5, 6, None]

The reason those "None" results show up is that this still emits the standard implied "yield value" for the comprehension, and the result of "(yield from x)" is None So the translation for star unpacking in generator expressions would be along the lines of a straightforward replacement of the implied "yield" with an implied "yield from": # Expansion of existing generator expression g = (x for x in iterable) def _g(outermost_iterable): for x in outermost_iterable: yield x g = _g() # Flattening generator expression g = (*x for x in iterable) def _g(outermost_iterable): for x in outermost_iterable: yield from x g = _g(iterable) The meaning for list and set comprehensions then follows from the generator expression semantics. Dictionary comprehensions would remain a unique snowflake, as they would be the only form which permitted the doublestar unpacking (they're already unique, since they rely on the embedded "k:v" notation to distinguish themselves from set comprehensions and set displays in general). As with existing doublestar unpacking in function calls, the semantics of what is acceptable would be driven by http://docs.python.org/3/c-api/dict.html#PyDict_Update (Note that those docs are currently inaccurate, as they imply it also accepts an iterable of key, value 2-tuples like dict.update, which is not the case: http://bugs.python.org/issue18456) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 13 July 2013 01:25, Joshua Landau wrote:

A blessing from the Gods has resulted in http://www.python.org/dev/peps/pep-0448/! See what you think; it's not too changed from before but it's mighty pretty now.

I definitely like the general colour of this shed but would probably repaint one side of it: while unpacking can create tuples, sets, lists and dicts there's no way to create an iterator. I would like it if the unpacking syntax could somehow be used for iterators. For example: first_line = next(inputfile) # inspect first_line for line in chain([first_line], inputfile): # process line could be rewritten as first_line = next(inputfile): for line in first_line, *inputfile: pass without reading the whole file into memory. Using the tuple syntax is probably confusing but it would be great if there were some way to spell this and get an iterator instead of a concrete collection. Also this may be outside the scope of this PEP but since unpacking is likely to be overhauled I'd like to put forward a previous suggestion by Greg Ewing that there be a way to unpack some items from an iterator without consuming the whole thing e.g.: a, ... = iterable could be roughly equivalent to: try: a = next(iter(iterable)) except StopIteration: raise ValueError('Need more than 0 items to unpack') I currently write code like: def parsefile(inputfile): inputfile = iter(inputfile) try: first_line = next(inputfile) except StopIteration: raise ValueError('Empty file') # Inspect first_line for line in chain([first_line], inputfile): # Process line But with the changes above I could do def parsefile(inputfile): inputfile = iter(inputfile) first_line, ... = inputfile # Inspect first_line for line in first_line, *inputfile: # Process line Oscar

On 15 July 2013 12:08, Joshua Landau wrote:

On 15 July 2013 11:40, Oscar Benjamin wrote:

...

Also this may be outside the scope of this PEP but since unpacking is likely to be overhauled I'd like to put forward a previous suggestion by Greg Ewing that there be a way to unpack some items from an iterator without consuming the whole thing e.g.:

a, ... = iterable

That's definitely outside of this PEP's scope ;). Also, I think you oversimplified your last version -- you still need a try-except AFAICT.

Where? The point is that next() raises StopIteration which is not an acceptable type of Error. Leaking the StopIteration makes the function not "generator-safe" i.e. if you call it from a generator the StopIteration could terminate an outer loop. That's why I have the try/except. As long as a, ... = iterator gives me a ValueError I'm happy to let the error propagate upwards. Oscar

On 15 July 2013 12:08, Joshua Landau wrote:

On 15 July 2013 11:40, Oscar Benjamin wrote:

In fact, I'd much like it if there was an iterable "unpacking" method for functions, too, so "chain.from_iterable()" could use the same interface as "chain" (and str.format with str.format_map, etc.). I feel we already have a good deal of redundancy due to this.

I've also considered this before. I don't know what a good spelling would be but lets say that it uses *args* so that you have a function signature like: def chain(*iterables*): for iterable in iterables: yield from iterable And then if the function is called with for line in chain(first_line, *inputfile): # do stuff then iterables would be bound to a lazy generator that chains [first_line] and inputfile. Then you could create the unpacking iterator I wanted by just using chain e.g.: chain(prepend, *iterable, append) Oscar

On 15 July 2013 21:06, Guido van Rossum wrote:

On Mon, Jul 15, 2013 at 1:01 PM, Oscar Benjamin wrote:

...

On 15 July 2013 12:08, Joshua Landau wrote:

...

On 15 July 2013 11:40, Oscar Benjamin wrote:

In fact, I'd much like it if there was an iterable "unpacking" method for functions, too, so "chain.from_iterable()" could use the same interface as "chain" (and str.format with str.format_map, etc.). I feel we already have a good deal of redundancy due to this.

I've also considered this before. I don't know what a good spelling would be but lets say that it uses *args* so that you have a function signature like:

def chain(*iterables*): for iterable in iterables: yield from iterable

And then if the function is called with

for line in chain(first_line, *inputfile): # do stuff

then iterables would be bound to a lazy generator that chains [first_line] and inputfile. Then you could create the unpacking iterator I wanted by just using chain e.g.:

chain(prepend, *iterable, append)

But how could you do this without generating different code depending on how the function you are calling is declared? Python's compiler doesn't have access to that information.

Good point. Maybe you'd have to spell it that way at both ends: chain(prepend, *iterable*, append) Oscar

On Mon, Jul 15, 2013 at 3:40 AM, Oscar Benjamin wrote:

I definitely like the general colour of this shed but would probably repaint one side of it: while unpacking can create tuples, sets, lists and dicts there's no way to create an iterator. I would like it if the unpacking syntax could somehow be used for iterators. For example:

first_line = next(inputfile) # inspect first_line for line in chain([first_line], inputfile): # process line

could be rewritten as

first_line = next(inputfile): for line in first_line, *inputfile: pass

without reading the whole file into memory. Using the tuple syntax is probably confusing but it would be great if there were some way to spell this and get an iterator instead of a concrete collection.

I think this is going down a slippery slope that could jeopardize the whole PEP, which is nice and non-controversial so far. The problem is that "tuples" (more precisely, things separated by commas) are already overloaded to the point where both the parser and most human readers are strained to the max to tell the different cases apart. For example, this definitely creates a tuple: a = 1, 2, 3 Now consider this: b = 2, 3 a = 1, *b Why would that not create the same tuple? In general, all other uses of *x and **xx create concrete objects (there is nothing "iterator-like" about an argument list). I think overloading these same operators to return iterators in some contexts would just cause too much confusion, and discontinuities in edge cases, making it harder to reason about the equivalency of different ways to write the same thing. (The use of *x in a generator expression is an exception -- a generator expression is *already* an iterator, so here there is no confusion.)

Also this may be outside the scope of this PEP but since unpacking is likely to be overhauled I'd like to put forward a previous suggestion by Greg Ewing that there be a way to unpack some items from an iterator without consuming the whole thing e.g.:

a, ... = iterable

Definitely a different PEP. -- --Guido van Rossum (python.org/~guido)

On Mon, Jul 15, 2013 at 12:53 PM, Oscar Benjamin wrote:

On 15 July 2013 18:39, Guido van Rossum wrote:

...

On Mon, Jul 15, 2013 at 3:40 AM, Oscar Benjamin wrote:

...

I would like it if the unpacking syntax could somehow be used for iterators. For example:

first_line = next(inputfile) # inspect first_line for line in chain([first_line], inputfile): # process line

could be rewritten as

first_line = next(inputfile): for line in first_line, *inputfile: pass

without reading the whole file into memory. Using the tuple syntax is probably confusing but it would be great if there were some way to spell this and get an iterator instead of a concrete collection.

I think this is going down a slippery slope that could jeopardize the whole PEP, which is nice and non-controversial so far. The problem is that "tuples" (more precisely, things separated by commas) are already overloaded to the point where both the parser and most human readers are strained to the max to tell the different cases apart. For example, this definitely creates a tuple:

a = 1, 2, 3

Now consider this:

b = 2, 3 a = 1, *b

Why would that not create the same tuple?

Yeah, this is what I mean that tuple syntax is confusing. I think, though, that it would be good if some way of creating iterators evolved from this.

Consider that while list, set and dict comprehensions can create lists, sets and dicts. Generator expressions can create tuples, OrderedDicts, blists, deques and many more. They can also be used with folds like min, max, sum, any, all and many more. Creating an iterator provides a much more general tool then creating a particular concrete type.

But the point remains that I see no way to creatively reuse the *x notation to serve both purposes. So I recommend that you think of a different way to obtain your goal, proposing a different PEP, which can be discussed independently from PEP 448. (And I don't mean this in the sense of "go away, I don't want to listen to you". I do want to hear your ideas. I just think that it is better for PEP 448 to be more limited in scope.) Regarding the importance of more general/abstract tools, I have just started reading Seymour Papert's Mindstorms, and one of his early insights about learning programming seems to be that the learner's path goes from more concrete things to more abstract things. In this context it feels appropriate that Python's syntax has notations to create concrete objects such as lists, tuples, dicts but that the more general concepts like iterators must be created without much syntactic help (generator expressions notwithstanding). -- --Guido van Rossum (python.org/~guido)

On 15 July 2013 22:34, Jan Kaliszewski wrote:

15.07.2013 12:40, Oscar Benjamin wrote:

...

first_line = next(inputfile) # inspect first_line for line in chain([first_line], inputfile): # process line

could be rewritten as

first_line = next(inputfile): for line in first_line, *inputfile: pass

without reading the whole file into memory.

Please note, that with PEP 448 syntax you could express it by:

first_line = next(inputfile) for line in (*it for it in ([first_line], inputfile)): ...

I had realised that but I probably prefer chain() to the above. Thinking about it now what really bothers me about writing that kind of code is the need to trap StopIteration around calls to next() (or to supply and check for a default value). The chain part isn't so bad.

Event now, in Python 3.3, you can[*] write:

first_line = next(inputfile) for line in [(yield from it) for it in [[first_line], inputfile]]: ...

[*] Please note that it is `yield from` within a *list comprehension*, not a generator expression... And that this list cimprehension still evaluates to a *generator*, not a list! (a [None, None] list is set as StopIteration's value when the generator is exhausted)

Where exactly is the above defined/discussed? I looked through PEP 380 (yield from) but I can't find any mention of comprehensions or generator expressions. I guess that it unrolls as def _func(): tmp = [] for it in [[first_line], inputfile]: tmp.append(yield from it) # Now _func is a generator function return tmp # becomes raise StopIteration(tmp) for line in _func(): ... but I hadn't considered the fact that using yield from in the expression would turn a list comprehension into a generator function according to the unrolling logic. Oscar

On 7 July 2013 07:26, Ronald Oussoren wrote:

On 6 Jul, 2013, at 6:30, Joshua Landau wrote:

...

The PEP is attached. I'm not sure if I've covered the basics, but it's a try.

If anyone knows how to get the patch (from the bug report) working, or where to find http://code.python.org/python/users/twouters/starunpack after code.python.org was deleted in favour of hg.python.org (which seems not to have it), it'd be nice to know.

As you already noted in your proposal the proposed changes to function definitions are not backward compatible:

I wrote the PEP, but I tried not to add any ideas of my own (other than clarifications). Whilst I failed at that -- function signature changes weren't actually in the original implementation -- it's worth keeping me separate from those who proposed the ideas and did the bulk of the work for them. I just liked the idea so am trying to nudge it a bit with this.

def func(*args, foo): pass

Currently 'foo' is a required keyword argument, with your change it would be just another positional only argument. How would you define keyword-only arguments with your proposal? The only alternative I could come up with is an extension of how you currently define keyword arguments without having a '*args' argument:

def func(*args, *, foo): pass

This however is currently not valid (SyntaxError) and would therefore make it a lot harder to write functions with keyword-only arguments that work both before and after your proposed change.

As I replied to someone else, "I also seem to have magically created this - it seems not to be in the original implementation. Hence if no-one supports the idea, and since I'm not very attached to it, there's no loss in letting it go." That's a good counter-argument, and no-one seems to support changes to function definitions, so I'll go with the flow and remove it. It's not needed for the rest of the PEP to make sense. I'll probably update the PEP tomorrow, for some skewed definition of tomorrow (I don't really have a sleep-cycle ATM) but how to define function-call grammar is still undefined so I'll lay out the two alternatives that currently make sense to me.