Can Python guarantee the order of keyword-only parameters?
First, a thirty-second refresher, so we're all using the same terminology: A *parameter* is a declared input variable to a function. An *argument* is a value passed into a function. (*Arguments* are stored in *parameters.*) So in the example "def foo(clonk): pass; foo(3)", clonk is a parameter, and 3 is an argument. ++ Keyword-only arguments were conceived of as being unordered. They're stored in a dictionary--by convention called **kwargs--and dictionaries didn't preserve order. But knowing the order of arguments is occasionally very useful. PEP 468 proposed that Python preserve the order of keyword-only arguments in kwargs. This became easy with the order-preserving dictionaries added to Python 3.6. I don't recall the order of events, but in the end PEP 468 was accepted, and as of 3.6 Python guarantees order in **kwargs. But that's arguments. What about parameters? Although this isn't as directly impactful, the order of keyword-only parameters *is* visible to the programmer. The best way to see a function's parameters is with inspect.signature, although there's also the deprecated inspect.getfullargspec; in CPython you can also directly examine fn.__code__.co_varnames. Two of these methods present their data in a way that preserves order for all parameters, including keyword-only parameters--and the third one is deprecated. Python must (and does) guarantee the order of positional and positional-or-keyword parameters, because it uses position to map arguments to parameters when the function is called. But conceptually this isn't necessary for keyword-only parameters because their position is irrelevant. I only see one place in the language & library that addresses the ordering of keyword-only parameters, by way of omission. The PEP for inspect.signature (PEP 362) says that when comparing two signatures for equality, their positional and positional-or-keyword parameters must be in the same order. It makes a point of *not* requiring that the two functions' keyword-only parameters be in the same order. For every currently supported version of Python 3, inspect.signature and fn.__code__.co_varnames preserve the order of keyword-only parameters. This isn't surprising; it's basically the same code path implementing those as the two types of positional-relevant parameters, so the most straightforward implementation would naturally preserve their order. It's just not guaranteed. I'd like inspect.signature to guarantee that the order of keyword-only parameters always matches the order they were declared in. Technically this isn't a language feature, it's a library feature. But making this guarantee would require that CPython internally cooperate, so it's kind of a language feature too. Does this sound reasonable? Would it need a PEP? I'm hoping for "yes" and "no", respectively. Three final notes: * Yes, I do have a use case. I'm using inspect.signature metadata to mechanically map arguments from an external domain (command-line arguments) to a Python function. Relying on the declaration order of keyword-only parameters would elegantly solve one small problem. * I asked Armin Rigo about PyPy's support for Python 3. He said it should already maintain the order of keyword-only parameters, and if I ever catch it not maintaining them in order I should file a bug. I assert that making this guarantee would be nearly zero effort for any Python implementation--I bet they all already behave this way, all they need is a test case and some documentation. * One can extend this concept to functools.partial and inspect.Signature.bind: should its transformations of keyword-only parameters also maintain order in a consistent way? I suspect the answer there is much the same--there's an obvious way it should behave, it almost certainly already behaves that way, but it doesn't guarantee it. I don't think I need this for my use case. //arry/ ++ Yes, that means "Argument Clinic" should really have been called "Parameter Clinic". But the "Parameter Clinic" sketch is nowhere near as funny.
Plus 1 from me. I'm not 100% sure the signature / inspect backport does this, but as you say, it should be trivial to do, to whatever extent the python version we're hosted on does it. Rob On 28 Nov. 2017 07:14, "Larry Hastings" <larry@hastings.org> wrote:
First, a thirty-second refresher, so we're all using the same terminology:
A *parameter* is a declared input variable to a function. An *argument* is a value passed into a function. (*Arguments* are stored in *parameters.*)
So in the example "def foo(clonk): pass; foo(3)", clonk is a parameter, and 3 is an argument. ++
Keyword-only arguments were conceived of as being unordered. They're stored in a dictionary--by convention called **kwargs--and dictionaries didn't preserve order. But knowing the order of arguments is occasionally very useful. PEP 468 proposed that Python preserve the order of keyword-only arguments in kwargs. This became easy with the order-preserving dictionaries added to Python 3.6. I don't recall the order of events, but in the end PEP 468 was accepted, and as of 3.6 Python guarantees order in **kwargs.
But that's arguments. What about parameters?
Although this isn't as directly impactful, the order of keyword-only parameters *is* visible to the programmer. The best way to see a function's parameters is with inspect.signature, although there's also the deprecated inspect.getfullargspec; in CPython you can also directly examine fn.__code__.co_varnames. Two of these methods present their data in a way that preserves order for all parameters, including keyword-only parameters--and the third one is deprecated.
Python must (and does) guarantee the order of positional and positional-or-keyword parameters, because it uses position to map arguments to parameters when the function is called. But conceptually this isn't necessary for keyword-only parameters because their position is irrelevant. I only see one place in the language & library that addresses the ordering of keyword-only parameters, by way of omission. The PEP for inspect.signature (PEP 362) says that when comparing two signatures for equality, their positional and positional-or-keyword parameters must be in the same order. It makes a point of *not* requiring that the two functions' keyword-only parameters be in the same order.
For every currently supported version of Python 3, inspect.signature and fn.__code__.co_varnames preserve the order of keyword-only parameters. This isn't surprising; it's basically the same code path implementing those as the two types of positional-relevant parameters, so the most straightforward implementation would naturally preserve their order. It's just not guaranteed.
I'd like inspect.signature to guarantee that the order of keyword-only parameters always matches the order they were declared in. Technically this isn't a language feature, it's a library feature. But making this guarantee would require that CPython internally cooperate, so it's kind of a language feature too.
Does this sound reasonable? Would it need a PEP? I'm hoping for "yes" and "no", respectively.
Three final notes:
- Yes, I do have a use case. I'm using inspect.signature metadata to mechanically map arguments from an external domain (command-line arguments) to a Python function. Relying on the declaration order of keyword-only parameters would elegantly solve one small problem. - I asked Armin Rigo about PyPy's support for Python 3. He said it should already maintain the order of keyword-only parameters, and if I ever catch it not maintaining them in order I should file a bug. I assert that making this guarantee would be nearly zero effort for any Python implementation--I bet they all already behave this way, all they need is a test case and some documentation. - One can extend this concept to functools.partial and inspect.Signature.bind: should its transformations of keyword-only parameters also maintain order in a consistent way? I suspect the answer there is much the same--there's an obvious way it should behave, it almost certainly already behaves that way, but it doesn't guarantee it. I don't think I need this for my use case.
*/arry*
++ Yes, that means "Argument Clinic" should really have been called "Parameter Clinic". But the "Parameter Clinic" sketch is nowhere near as funny.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ robertc%40robertcollins.net
On 11/27/2017 12:19 PM, Robert Collins wrote:
Plus 1 from me. I'm not 100% sure the signature / inspect backport does this, but as you say, it should be trivial to do, to whatever extent the python version we're hosted on does it.
I'm not sure exactly what you mean when you say "signature / inspect backport". If you mean backporting inspect.signature to Python 2, this topic is irrelevant, as Python 2 doesn't have keyword-only parameters. //arry/
On Mon, Nov 27, 2017 at 10:13 AM Larry Hastings <larry@hastings.org> wrote:
First, a thirty-second refresher, so we're all using the same terminology:
A *parameter* is a declared input variable to a function. An *argument* is a value passed into a function. (*Arguments* are stored in *parameters.*)
So in the example "def foo(clonk): pass; foo(3)", clonk is a parameter, and 3 is an argument. ++
Keyword-only arguments were conceived of as being unordered. They're stored in a dictionary--by convention called **kwargs--and dictionaries didn't preserve order. But knowing the order of arguments is occasionally very useful. PEP 468 proposed that Python preserve the order of keyword-only arguments in kwargs. This became easy with the order-preserving dictionaries added to Python 3.6. I don't recall the order of events, but in the end PEP 468 was accepted, and as of 3.6 Python guarantees order in **kwargs.
But that's arguments. What about parameters?
Although this isn't as directly impactful, the order of keyword-only parameters *is* visible to the programmer. The best way to see a function's parameters is with inspect.signature, although there's also the deprecated inspect.getfullargspec; in CPython you can also directly examine fn.__code__.co_varnames. Two of these methods present their data in a way that preserves order for all parameters, including keyword-only parameters--and the third one is deprecated.
Python must (and does) guarantee the order of positional and positional-or-keyword parameters, because it uses position to map arguments to parameters when the function is called. But conceptually this isn't necessary for keyword-only parameters because their position is irrelevant. I only see one place in the language & library that addresses the ordering of keyword-only parameters, by way of omission. The PEP for inspect.signature (PEP 362) says that when comparing two signatures for equality, their positional and positional-or-keyword parameters must be in the same order. It makes a point of *not* requiring that the two functions' keyword-only parameters be in the same order.
For every currently supported version of Python 3, inspect.signature and fn.__code__.co_varnames preserve the order of keyword-only parameters. This isn't surprising; it's basically the same code path implementing those as the two types of positional-relevant parameters, so the most straightforward implementation would naturally preserve their order. It's just not guaranteed.
I'd like inspect.signature to guarantee that the order of keyword-only parameters always matches the order they were declared in. Technically this isn't a language feature, it's a library feature. But making this guarantee would require that CPython internally cooperate, so it's kind of a language feature too.
Does this sound reasonable? Would it need a PEP? I'm hoping for "yes" and "no", respectively.
Seems reasonable to me. I'm in the "yes" and "no" respectively "just do it" camp on this if want to see it happen. The groundwork was already laid for this by using the order preserving dict in 3.6. Having the inspect module behave in a similar manner follows naturally from that. -gps
Three final notes:
- Yes, I do have a use case. I'm using inspect.signature metadata to mechanically map arguments from an external domain (command-line arguments) to a Python function. Relying on the declaration order of keyword-only parameters would elegantly solve one small problem. - I asked Armin Rigo about PyPy's support for Python 3. He said it should already maintain the order of keyword-only parameters, and if I ever catch it not maintaining them in order I should file a bug. I assert that making this guarantee would be nearly zero effort for any Python implementation--I bet they all already behave this way, all they need is a test case and some documentation. - One can extend this concept to functools.partial and inspect.Signature.bind: should its transformations of keyword-only parameters also maintain order in a consistent way? I suspect the answer there is much the same--there's an obvious way it should behave, it almost certainly already behaves that way, but it doesn't guarantee it. I don't think I need this for my use case.
*/arry*
++ Yes, that means "Argument Clinic" should really have been called "Parameter Clinic". But the "Parameter Clinic" sketch is nowhere near as funny. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
On Mon, Nov 27, 2017 at 12:05 PM, Larry Hastings <larry@hastings.org> wrote: [..]
The PEP for inspect.signature (PEP 362) says that when comparing two signatures for equality, their positional and positional-or-keyword parameters must be in the same order. It makes a point of *not* requiring that the two functions' keyword-only parameters be in the same order.
Yes, and I believe Signature.__eq__ should stay that way.
For every currently supported version of Python 3, inspect.signature and fn.__code__.co_varnames preserve the order of keyword-only parameters. This isn't surprising; it's basically the same code path implementing those as the two types of positional-relevant parameters, so the most straightforward implementation would naturally preserve their order. It's just not guaranteed.
I'd like inspect.signature to guarantee that the order of keyword-only parameters always matches the order they were declared in. Technically this isn't a language feature, it's a library feature. But making this guarantee would require that CPython internally cooperate, so it's kind of a language feature too.
We can update the documentation and say that we preserve the order in simple cases: def foo(*, a=1, b=2): pass s = inspect.signature(foo) assert list(s.parameters.keys()) == ['a', 'b'] We can't say anything about the order if someone passes a partial object, or sets custom Signature objects to func.__signature__. Yury
On 11/27/2017 03:58 PM, Yury Selivanov wrote:
We can't say anything about the order if someone passes a partial object
Sure we could. We could ensure that functools.partial behaves in a sane way, then document and guarantee that behavior.
or sets custom Signature objects to func.__signature__.
Consenting Adults rule applies here. Obviously we should honor the signature they set. //arry/
On 28 November 2017 at 15:42, Larry Hastings <larry@hastings.org> wrote:
On 11/27/2017 03:58 PM, Yury Selivanov wrote:
We can't say anything about the order if someone passes a partial object
Sure we could. We could ensure that functools.partial behaves in a sane way, then document and guarantee that behavior.
Right, I think the main implication here would be that we need to ensure that any signature manipulation operations *we* provide are order preserving. Fortunately for Larry, we kinda cheat on that front: all the logic for dealing with this problem is in the inspect module itself, which knows about all the different ways we manipulate signatures in the standard library. That means that if we want to declare that the inspect module will be order preserving, we can, and it shouldn't require changes to anything else. Cheers, Nick. P.S. Note that inspect.getfullargspec() was actually undeprecated a while back - enough folks that didn't need access to the function annotations were reimplementing it for themselves "because the standard library API is deprecated" that the most logical course of action was to just declare it as being supported again. I don't think that changes the argument here though - it just means guaranteed order preservation in that API will only happen if we declare dicts to be insertion ordered in general. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 28 Nov 2017 at 03:33 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 November 2017 at 15:42, Larry Hastings <larry@hastings.org> wrote:
On 11/27/2017 03:58 PM, Yury Selivanov wrote:
We can't say anything about the order if someone passes a partial object
Sure we could. We could ensure that functools.partial behaves in a sane way, then document and guarantee that behavior.
Right, I think the main implication here would be that we need to ensure that any signature manipulation operations *we* provide are order preserving.
Fortunately for Larry, we kinda cheat on that front: all the logic for dealing with this problem is in the inspect module itself, which knows about all the different ways we manipulate signatures in the standard library. That means that if we want to declare that the inspect module will be order preserving, we can, and it shouldn't require changes to anything else.
Cheers, Nick.
P.S. Note that inspect.getfullargspec() was actually undeprecated a while back - enough folks that didn't need access to the function annotations were reimplementing it for themselves "because the standard library API is deprecated" that the most logical course of action was to just declare it as being supported again. I don't think that changes the argument here though - it just means guaranteed order preservation in that API will only happen if we declare dicts to be insertion ordered in general.
OT for this thread, but is there an issue number tracking the un-deprecating? Basically I want to make sure there is an issue tracking deprecating it again when we stop worrying about any Python 2/3 support.
I was going to suggest the final DeprecationWarning should be raised unconditionally (subject to whatever silencing rules are agreed) by the final 2.7 release to recommend migration to Python 3. "This is an ex-language ... it has ceased to be ... it is no more ... it has returned to the great heap whence it came ..." Steve Holden On Tue, Nov 28, 2017 at 5:13 PM, Brett Cannon <brett@python.org> wrote:
On Tue, 28 Nov 2017 at 03:33 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 November 2017 at 15:42, Larry Hastings <larry@hastings.org> wrote:
On 11/27/2017 03:58 PM, Yury Selivanov wrote:
We can't say anything about the order if someone passes a partial object
Sure we could. We could ensure that functools.partial behaves in a sane way, then document and guarantee that behavior.
Right, I think the main implication here would be that we need to ensure that any signature manipulation operations *we* provide are order preserving.
Fortunately for Larry, we kinda cheat on that front: all the logic for dealing with this problem is in the inspect module itself, which knows about all the different ways we manipulate signatures in the standard library. That means that if we want to declare that the inspect module will be order preserving, we can, and it shouldn't require changes to anything else.
Cheers, Nick.
P.S. Note that inspect.getfullargspec() was actually undeprecated a while back - enough folks that didn't need access to the function annotations were reimplementing it for themselves "because the standard library API is deprecated" that the most logical course of action was to just declare it as being supported again. I don't think that changes the argument here though - it just means guaranteed order preservation in that API will only happen if we declare dicts to be insertion ordered in general.
OT for this thread, but is there an issue number tracking the un-deprecating? Basically I want to make sure there is an issue tracking deprecating it again when we stop worrying about any Python 2/3 support.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ steve%40holdenweb.com
On 29 November 2017 at 03:13, Brett Cannon <brett@python.org> wrote:
On Tue, 28 Nov 2017 at 03:33 Nick Coghlan <ncoghlan@gmail.com> wrote:
P.S. Note that inspect.getfullargspec() was actually undeprecated a while back - enough folks that didn't need access to the function annotations were reimplementing it for themselves "because the standard library API is deprecated" that the most logical course of action was to just declare it as being supported again. I don't think that changes the argument here though - it just means guaranteed order preservation in that API will only happen if we declare dicts to be insertion ordered in general.
OT for this thread, but is there an issue number tracking the un-deprecating? Basically I want to make sure there is an issue tracking deprecating it again when we stop worrying about any Python 2/3 support.
The issue is https://bugs.python.org/issue27172, but the undeprecation isn't a Python 2/3 issue, it's a "tuples, lists and dicts are really handy representations of things, and Python developers often prefer them to more structured objects" issue. The modern inspect.getfullargspec implementation is a relatively thin wrapper around inspect.signature, and the only lossy part of the output transformation is that you can't tell the difference between positional-only and positional-or-keyword parameters. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Nov 27, 2017 at 10:05 AM, Larry Hastings <larry@hastings.org> wrote:
I'd like inspect.signature to guarantee that the order of keyword-only parameters always matches the order they were declared in. Technically this isn't a language feature, it's a library feature. But making this guarantee would require that CPython internally cooperate, so it's kind of a language feature too.
Does this sound reasonable? Would it need a PEP? I'm hoping for "yes" and "no", respectively.
+1 There is definitely significant information in source code text that gets thrown away in some cases, which I'm generally in favor of preserving (see PEP 468 and PEP 520). The use case here is unclear to me, but the desired guarantee is effectively the status quo and it is a minor guarantee as well, so I don't see the harm. Furthermore, I don't see a need for a PEP given the small scale and impact. -eric
participants (8)
-
Brett Cannon -
Eric Snow -
Gregory P. Smith -
Larry Hastings -
Nick Coghlan -
Robert Collins -
Steve Holden -
Yury Selivanov