> But that's what this list is for, proposing changes.
That's wordplay. The list is for proposing changes, but it's also for
discussing them. I think making a suggestion (simply: using kw
restraints to prevent regression) for how to implement proposed
changes (kwargs as ordered dict) is not simply "not a big deal", but
on-topic.
My continued defense of it is because I don't agree with the reasons
put forth for it being *impossible*. For example, the claims that it
is *necessary* to allow loss of order upon addition to a kw-optimized
dict. I myself gave a bunch of reasons why it MIGHT not be faster
(biggest so far being "str-optimization may already be implemented"),
but I'm not convinced that those reasons given by others can't be
worked around.
I'm not opposed to criticism of the idea. But I believe we're going to
talk a lot more than necessary if people respond as if I'm convinced
it WILL work, or that I am pushing for it to happen.
I'm thinking of writing up pseudo-code for it later, so I can at least
share a common reference point, and solidify the concept. Or I might
find that the already-implemented str-dict optimizations can't be
improved upon, and report back.
Would string interning help?
As in, all keywords in a **pack have to be strings. All parameter names can
be interned (if they're not already). The **pack can be a special dict
optimized for strings (more specifically: identifiers), with an
order-preserving overhead. (And then **unpacking would be smart about the
type of thing being unpacked.)
I understand that there would need to be a C proof-of-concept that this
could act like a dict on the Python level, and maybe even beat a dict for
**kwargs performance. I'm not suggesting that it's viable, but I haven't
seen a mention of taking advantage of the restrictions on the keys, and
while I personally got stuck trying to flesh out an algorithm and structure
for it (unnecessary(?) overhead on extracting some keys and passing on the
rest while preserving order), maybe someone more clever can figure out how
to work it.
Hi,
what do you think about officially supporting Semantic Versioning?
( http://semver.org )
Semantic Versioning has gained a lot of attention in other programming
languages. Even though not officially supported, many python libraries use it
as well (just search the pypi listing for things like "-alpha" "-beta" "-dev"
"-pre" to get a rough idea). There even is a class handling Semantic Versioning
in pip already: pip.util.version.SemanticVersion .
I'd speak in favor of officially adding support for semantic versioning to the
python module versioning specs as a follow up to PEP 440
( http://legacy.python.org/dev/peps/pep-0440/ ).
I want to avoid the fruitless discussion about personal versioning scheme
preference.
My main point is: both schemes are widely used (even in the python world).
As far as i can see both schemes can just co-exist in the python world giving
us the benefits of both without hurting us.
Short re-cap of both versioning schemes
=======================================
The current version format specified in PEP 440 [2] follows this pseudo format:
[N:]N(.N)*[{a|b|c|rc}N][.postN][.devN]
So python module version numbers look like (taken from [2]):
1.0.dev456
1.0a1
1.0a2.dev456
1.0a12.dev456
1.0a12
1.0b1.dev456
1.0b2
1.0b2.post345.dev456
1.0b2.post345
1.0c1.dev456
1.0c1
1.0
1.0.post456.dev34
1.0.post456
1.1.dev1
Semantic Versioning follows this pseudo format:
N.N.N(-((N)|([0-9A-Za-z-]+))(.((N)|([0-9A-Za-z-])+))*)?
Some examples in order (taken from http://semver.org ):
1.0.0-alpha
1.0.0-alpha.1
1.0.0-alpha.beta
1.0.0-beta
1.0.0-beta.2
1.0.0-beta.11
1.0.0-rc.1
1.0.0
Key differences
===============
Semantic Versioning supports
- free-text pre-releases without version number such as '-alpha', '-pre' and
the very widely used '-dev' (after a release the next commit increases the
version number and appends the '-dev', which is only removed for the release).
- always has MAJOR.MINOR.PATCH, (so 3 relevant) numbers as version number and
offers guidelines which to change when.
Semantic Versioning does not support
- post-releases (a post release would be an increase of the PATCH number).
- special handling for '-dev', '-alpha', '-beta', '-gamma' or 'rc'.
Ideas to solve (cross scheme) comparisons
=========================================
A version comparator could first try parsing both versions to be compared
according to the current scheme and if that fails try parsing them both as
SemanticVersions.
Switching from one version naming scheme to another should be discouraged at
least within the leading N parts at the change boundary staying the same. That
way comparisons between the naming schemes could be settled by comparing the
leading numerical parts of the version.
Cheers,
Jörn
On Apr 14, 2014 4:44 PM, "Franklin? Lee" <leewangzhong+python(a)gmail.com>
wrote:
[...]
> This is similar to keyword arguments, which don't have to quote their
first parts. This can be generalized to dicts, to keep consistency.
> d = {'hello':1, 'world':2} #original
> d = {hello=1, world=2} #new
This not working is unexpected, if considering the expectation literals:
{..} and [...]
...translate to:
dict(...) and list(...) [int(..) and str(..)]
Why isn't/can't this be true? I discovered this awhile back while
attempting to change the default constructor with:
dict = OtherDict
...equiv:
module.dict = OtherDict
...and:
__builtins__['dict'] = OtherDict
...but none work :-(
--
C Anthony
(Attempting to reply to
https://mail.python.org/pipermail/python-ideas/2014-April/027443.html)
> In JS, if you want to define a simple object, you can write it in two
ways:
> {'spam': 'ham'}
> or
> {spam: 'ham'}
> But when you have a key name defined in a variable, you'll need to do
> key = 'spam'
> o = {}
> o[key] = 'ham'
> Where in Python, you'd simply write {key: 'ham'}.
> So for Python, I think that having unquoted keys in literals is a bad
idea.
I agree with this (even if you no longer do). There's no precedence (that I
can think of) for NOT evaluating things before a colon, and nothing like
using attr names (rather than objects that hold attribute names) with a
colon.
As long as we're just fooling around with ideas, here's an alternative
proposal for namedtuple literal syntax[0]. I don't mean this as an actual
suggestion, but I think it is better than unquoted keys before colons.
('color':c, 'position':c) #quoted, or with variables
(color=c, position=c) #unquoted, with literals
Example:
p = (x=1, y=5) #type is something new, like `ntuple`, or just
`namedtuple`.
This is similar to keyword arguments, which don't have to quote their first
parts. This can be generalized to dicts, to keep consistency.
d = {'hello':1, 'world':2} #original
d = {hello=1, world=2} #new
and the earlier-proposed "ordered dict literal"[1]
od = ['hello':1, 'world':2] #new
od = [hello=1, world=2] #new new
Or alternatively, with `=` just appearing in ntuples, this can be used:
od = OrderedDict((hello=1, world=2))
which is interesting and also somehow horrifying.
Some arguments against this:
- Introducing `=` in yet another context. Ugly.
- Tuples are ordered, and **kwargs are unordered[2], so there's a
conceptual discontinuity there.
- Accidentally writing `:` when you mean `=` can lead to silent bugs.
- Extending the syntax to dictionaries possibly violates TOOWTDI, and not
doing so feels inconsistent to me. I think about `(var_holding_spam: 1,
literally_eggs=2)` and I'm like, "This is obviously two ways of doing it."
- People might expect `p = (x=1) to be a 1-ntuple. I don't want it to be,
and I don't like that it wouldn't be.
I like the taste of the syntax (and the generalizations, and the
OrderedDict literal), but I see how it isn't necessarily necessary (or
welcome).
[0] Found a mention of this here:
https://mail.python.org/pipermail/python-ideas/2010-October/008489.html
[1] Guido hated that:
https://mail.python.org/pipermail/python-ideas/2009-June/004924.html
[2] Making it ordered is obstructed by concerns about performance (
https://mail.python.org/pipermail/python-ideas/2013-February/019699.html).
Currently being discussed here:
https://mail.python.org/pipermail/python-ideas/2014-April/027454.html
There are several problems with Python's implicit variable declaration.
Just one tiny mistake, like assigning to SpamEggs instead of spamEggs will
create a new variable SpamEggs, and spamEggs will hold a stale value.
Two statements should be added to Python: (1) a statement that specifies
that implicit declaration by assignment should be disabled, like Fortran's
"implicit none", and (2) a variable declaration statement, a la "var
spamEggs". Trying to assign to a non-existent variable would, of course,
cause an error. If the non-existent variable's name is close enough to an
existing variable's name, the interpreter might suggest the correct name:
"Did you mean "spamEggs"?".
This idea could even open Python to static typed variables, declared like
"var spamEggs as int". This would prevent the stupid but very possible
error of assigning "clientName" (a string) to "doorStatus" (a boolean).
-rb
I propose to allow to define keyword-only arguments for __getitem__.
The main use case is to be able to set a default value for indexing. I
though about it after reading PEP 463 (Exception-catching expressions),
this would be an alternative for one of the problem this PEP is trying
to solve.
Compare these solutions:
>>> lst = [1, 2, 3]
>>> value = lst[3]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>> try: # Python
.... value = lst[3]
.... except IndexError:
.... value = 0
>>> value = lst[3] except IndexError: 0 # PEP 463
>>> value = lst[3, default=0] # My proposal
Default value is not the only use case. One could for example define a
subclass of list computing an interpolation if a float is given as a
parameter instead of truncating it, and the keword argument would define
which kind of interpolation (nearest, linear, spline...). Or it could
define if the value returned are in degrees or radians. (Yes, I have a
scientific background.)
I use something similar for a dictionary-like interface where the keys
are strings and only numbers are stored. The second argument, if given,
is the default:
>>> data["key", 0]
It would be clearer to use:
>>> data["key", default=0]
(Of course I could use a get method for this purpose, but bracket for
indexing seems always more clear).
The keyword would be keywords-only because right now when many arguments
are passed to __getitem__, they are grouped in a tuple and this tuple is
passed to __getitem__ as a unique argument. Also it is more clear:
ordered arguments are the indexes, keyword arguments are the options.
To be perfectly honest I don't think that this proposal is that great,
it's main selling point is that a "default" keyword in indexing would be
a good alternative for some of the problems raised in PEP 463.
Joseph ML
---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com
Hi!
Sorry if someone has already talked about it (my simple search did not show
any results).
What do you think about adding map, flatmap, filter and reduce methods to
generator type ?
I must admit I've seen and I like Java 8 notation and I think it might be
more readable than Python way in a few occasions.
I would like to be able to write:
range(100).\
filter( f1 ).\
map( f2 ).\
filter( f3 ).\
map( f4 ).\
reduce(operator.add)
in addition to current Pythonic way of
reduce( operator.add, f4(x) for x in
( f2(y) for y in range(100) if f1(y))
if f3(x) )
Though longer - Java way seems to be a bit more readable as the notation
follows the data flow sequence.
Do you think it is worth a PEP?
BR,
Jacek
Here's the proper proposal I promised. Hopefully it reflects our
discussions and the past ones too. My goal here is to focus in on
what I consider to be the most viable approach, which capturing the
gist of the various concerns and alternatives. There are still a few
small gaps that I will work on as time permits. Feedback is welcome.
FYI, I also hope to have a basic implementation up in time for the
language summit at PyCon (Wednesday) so that related discussions there
might have something more concrete to support them. :)
-eric
=====================================================================
PEP: XXX
Title: Preserving the order of \*\*kwargs in a function.
Version: $Revision$
Last-Modified: $Date$
Author: Eric Snow <ericsnowcurrently(a)gmail.com>
Discussions-To: python-ideas(a)python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 5-Apr-2014
Python-Version: 3.5
Post-History:
Resolution:
Abstract
========
The \*\*kwargs syntax in a function definition indicates that the
interpreter should collect all keyword arguments that do not correspond
to other named parameters. However,
Python does not preserved the order in which those collected keyword
arguments were passed to the function. In some contexts the order
matters. This PEP introduces a mechanism by which the passed order of
collected keyword arguments will now be preserved.
Motivation
==========
Python's \*\*kwargs syntax in function definitions provides a powerful
means of dynamically handling keyword arguments. In some applications
of the syntax (see _`Use Cases`), the semantics applied to the
collected keyword arguments requires that order be preserved.
Unsurprisingly, this is similar to how OrderedDict is related to dict.
Currently to preserved the order you have to do so manually and
separately from the actual function call. This involves building an
ordered mapping, whether an OrderedDict or an iterable of 2-tuples,
which is then passed as a single argument to the function.
[#arg_unpacking]_
With the capability described in this PEP, that boilerplate would no
longer be required.
For comparision, currently::
kwargs = OrderedDict()
kwargs['eggs'] = ...
...
def spam(a, kwargs):
...
and with this proposal::
def spam(a, **kwargs):
...
Nick Coglan, speaking of some of the uses cases, summed it up well
[#nick_obvious]_::
These *can* all be done today, but *not* by using keyword arguments.
In my view, the problem to be addressed is that keyword arguments
*look* like they should work for these cases, because they have a
definite order in the source code. The only reason they don't work
is because the interpreter throws that ordering information away.
It's a textbook case of a language feature becoming an attractive
nuisance in some circumstances: the simple and obvious solution for
the above use cases *doesn't actually work* for reasons that aren't
obviously clear if you don't have a firm grasp of Python's admittedly
complicated argument handling.
This observation is supported by the appearance of this proposal over
the years and the numerous times that people have been confused by the
constructor for OrderedDict. [#past_threads]_ [#loss_of_order]_
[#compact_dict]_
Use Cases
=========
As Nick noted, the current behavior of \*\*kwargs is unintuitive in
cases where one would expect order to matter. Aside from more specific
cases outlined below, in general "anything else where you want to
control the iteration order *and* set field names and values in a single
call will potentially benefit." [#nick_general]_ That matters in the
case of factories (e.g. __init__()) for ordered types.
Serialization
-------------
Obviously OrderedDict would benefit (both __init__() and update()) from
ordered kwargs. However, the benefit also extends to serialization
APIs [#nick_obvious]_::
In the context of serialisation, one key lesson we have learned is
that arbitrary ordering is a problem when you want to minimise
spurious diffs, and sorting isn't a simple solution.
Tools like doctest don't tolerate spurious diffs at all, but are
often amenable to a sorting based answer.
The cases where it would be highly desirable to be able use keyword
arguments to control the order of display of a collection of key
value pairs are ones like:
* printing out key:value pairs in CLI output
* mapping semantic names to column order in a CSV
* serialising attributes and elements in particular orders in XML
* serialising map keys in particular orders in human readable formats
like JSON and YAML (particularly when they're going to be placed
under source control)
Debugging
---------
In the words of Raymond Hettinger [#raymond_debug]_::
It makes it easier to debug if the arguments show-up in the order
they were created. AFAICT, no purpose is served by scrambling them.
Other Use Cases
---------------
* Mock objects. [#mock]_
* Controlling object presentation.
* Alternate namedtuple() where defaults can be specified.
* Specifying argument priority by order.
Concerns
========
Performance
-----------
As already noted, the idea of ordered keyword arguments has come up on
a number of occasions. Each time it has been met with the same
response, namely that preserving keyword arg order would have a
sufficiently adverse effect on function call performance that it's not
worth doing. However, Guido noted the following [#guido_open]_::
Making **kwds ordered is still open, but requires careful design and
implementation to avoid slowing down function calls that don't benefit.
As will be noted below, there are ways to work around this at the
expense of increased complication. Ultimately the simplest approach is
the one that makes the most sense: pack collected key word arguments
into an OrderedDict. However, without a C implementation of OrderedDict
there isn't much to discuss. That should change in Python 3.5.
[#c_ordereddict]_
In some cases the difference of performance between dict and OrderedDict
*may* be of significance. For instance: when the collected
kwargs has an extended lifetime outside the originating function or the
number of collected kwargs is massive. However, the difference in
performance (both CPU and memory) in those cases should not be
significant. Furthermore, the performance of the C OrderedDict
implementation is essentially identical with dict for the non-mutating
API. A concrete representation of the difference in performance will be
a part of this proposal before its resolution.
Other Python Implementations
----------------------------
Another important issue to consider is that new features must be
cognizant of the multiple Python implementations. At some point each of
them would be expected to have implemented ordered kwargs. In this
regard there doesn't seem to be an issue with the idea. [#ironpython]_
Each of the major Python implementations will be consulted regarding
this proposal before its resolution.
Specification
=============
Starting in version 3.5 Python will preserve the order of keyword
arguments as passed to a function. This will apply only to functions
for which the definition uses the \*\*kwargs syntax for collecting
otherwise unspecified keyword arguments. Only the order of those
keyword arguments will be preserved.
Relationship to **-unpacking syntax
-----------------------------------
The ** unpacking syntax in function calls has no special connection with
this proposal. Keyword arguments provided by unpacking will be treated
in exactly the same way as they are now: ones that match defined
parameters are gather there and the remainder will be collected into the
ordered kwargs (just like any other unmatched keyword argument).
Note that unpacking a mapping with undefined order, such as dict, will
preserve its iteration order like normal. It's just that the order will
remain undefined. The OrderedDict into which the unpacked key-value
pairs will then be packed will not be able to provide any alternate
ordering. This should not be surprising.
There have been brief discussions of simply passing these mappings
through to the functions kwargs without unpacking and repacking them,
but that is both outside the scope of this proposal and probably a bad
idea regardless. (There is a reason those discussions were brief.)
Relationship to inspect.Signature
---------------------------------
Signature objects should need no changes. The `kwargs` parameter of
inspect.BoundArguments (returned by Signature.bind() and
Signature.bind_partial()) will change from a dict to an OrderedDict.
C-API
-----
TBD
Syntax
------
No syntax is added or changed by this proposal.
Backward-Compatibility
----------------------
The following will change:
* type(kwargs)
* iteration order of kwargs will now be consistent (except of course in
the case described above)
* as already noted, performance will be marginally different
None of these should be an issue. However, each will be carefully
considered while this proposal is under discussion.
Reference Implementation
========================
TBD
Implementation Notes
--------------------
TBD
Alternate Approaches
====================
Opt-out Decorator
-----------------
This is identical to the current proposal with the exception that Python
would also provide a decorator in functools that would cause collected
keyword arguments to be packed into a normal dict instead of an
OrderedDict.
Prognosis:
This would only be necessary if performance is determined to be
significantly different in some uncommon cases or that there are other
backward-compatibility concerns that cannot be resolved otherwise.
Opt-in Decorator
----------------
The status quo would be unchanged. Instead Python would provide a
decorator in functools that would register or mark the decorated
function as one that should get ordered keyword arguments. The
performance overhead to check the function at call time would be
marginal.
Prognosis:
The only real down-side is in the case of function wrappers factories
(e.g. functools.partial and many decorators) that aim to perfectly
preserve keyword arguments by using kwargs in the wrapper definition
and kwargs unpacking in the call to the wrapped function. Each wrapper
would have to be updated separately, though having functools.wraps() do
this automaticallywould help.
__kworder__
-----------
The order of keyword arguments would be stored separately in a list at
call time. The list would be bound to __kworder__ in the function
locals.
Prognosis:
This likewise complicates the wrapper case.
Compact dict with faster iteration
----------------------------------
Raymond Hettinger has introduced the idea of a dict implementation that
would result in preserving insertion order on dicts (until the first
deletion). This would be a perfect fit for kwargs. [#compact_dict]_
Prognosis:
The idea is still uncertain in both viability and timeframe.
***kwargs
---------
This would add a new form to a function's signature as a mutually
exclusive parallel to \*\*kwargs. The new syntax, ***kwargs (note that
there are three asterisks), would indicate that kwargs should preserve
the order of keyword arguments.
Prognosis:
New syntax is only added to Python under the most *dire* circumstances.
With other available solutions, new syntax is not justifiable.
Furthermore, like all opt-in solutions, the new syntax would complicate
the pass-through case.
annotations
-----------
This is a variation on the decorator approach. Instead of using a
decorator to mark the function, you would use a function annotation on
\*\*kwargs.
Prognosis:
In addition to the pass-through complication, annotations have been
actively discouraged in Python core development. Use of annotations to
opt-in to order preservation runs the risk of interfering with other
application-level use of annotations.
dict.__order__
--------------
dict objects would have a new attribute, `__order__` that would default
to None and that in the kwargs case the interpreter would use in the
same way as described above for __kworder__.
Prognosis:
It would mean zero impact on kwargs performance but the change would be
pretty intrusive (Python uses dict a lot). Also, for the wrapper case
the interpreter would have to be careful to preserve `__order__`.
KWArgsDict.__order__
--------------------
This is the same as the `dict.__order__` idea, but kwargs would be an
instance of a new minimal dict subclass that provides the `__order__`
attribute. dict would instead be unchanged.
Prognosis:
Simply switching to OrderedDict is a less complicated and more intuitive
change.
Acknowledgements
================
Thanks to Andrew Barnert for helpful feedback and to the participants of
all the past email threads.
Footnotes
=========
.. [#arg_unpacking]
Alternately, you could also replace ** in your function definition
with * and then pass in key/value 2-tuples. This has the advantage
of not requiring the keys to be valid identifier strings. See
https://mail.python.org/pipermail/python-ideas/2014-April/027491.html.
References
==========
.. [#nick_obvious]
https://mail.python.org/pipermail/python-ideas/2014-April/027512.html
.. [#past_threads]
https://mail.python.org/pipermail/python-ideas/2009-April/004163.htmlhttps://mail.python.org/pipermail/python-ideas/2010-October/008445.htmlhttps://mail.python.org/pipermail/python-ideas/2011-January/009037.htmlhttps://mail.python.org/pipermail/python-ideas/2013-February/019690.htmlhttps://mail.python.org/pipermail/python-ideas/2013-May/020727.htmlhttps://mail.python.org/pipermail/python-ideas/2014-March/027225.htmlhttp://bugs.python.org/issue16276http://bugs.python.org/issue16553http://bugs.python.org/issue19026http://bugs.python.org/issue5397#msg82972
.. [#loss_of_order]
https://mail.python.org/pipermail/python-dev/2007-February/071310.html
.. [#compact_dict]
https://mail.python.org/pipermail/python-dev/2012-December/123028.htmlhttps://mail.python.org/pipermail/python-dev/2012-December/123105.htmlhttps://mail.python.org/pipermail/python-dev/2013-May/126327.htmlhttps://mail.python.org/pipermail/python-dev/2013-May/126328.html
.. [#nick_general]
https://mail.python.org/pipermail/python-dev/2012-December/123105.html
.. [#raymond_debug]
https://mail.python.org/pipermail/python-dev/2013-May/126327.html
.. [#mock]
https://mail.python.org/pipermail/python-ideas/2009-April/004163.htmlhttps://mail.python.org/pipermail/python-ideas/2009-April/004165.htmlhttps://mail.python.org/pipermail/python-ideas/2009-April/004175.html
.. [guido_open]
https://mail.python.org/pipermail/python-dev/2013-May/126404.html
.. [#c_ordereddict]
http://bugs.python.org/issue16991
.. [#ironpython]
https://mail.python.org/pipermail/python-dev/2012-December/123100.html
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: