signature.object, argument clinic and grouped parameters
In the midst of work on the issue #17481, it became apparent that we need a way of specifying optional/grouped parameters. One good example of grouped parameters in python is the `type` function. Basically, it has two different signatures: * type(name, bases, dict) * type(object) Which we can combine in one, if we define a notion of grouped parameters: * type(object_or_name, [bases, dict]) Another good example, is 'itertools.repeat'. Its signature is "(elem[, n])". If "n" argument is passed, then it's how many times the "elem" will be repeated, and if it is not passed at all, then "elem" will be repeated endlessly. One way of emulating this behavior in pure python is to define a special private marker object and use it as a default value: _optional = object() def repeat(elem, n=_optional): if n is _optional: # `n` wasn't passed, repeat indefinitely else: # we have something for `n` One of the problems with the above approach is how to represent its signature, how to document it clearly. Another one, is that there is no common marker, so whenever this is needed, a new marker is invented. Now, the more I think about having a concept of grouped parameters, the more different things to consider and take care of appear: * In issue #17481 Larry proposed that parameters will have a group id (arbitrary), and perhaps parent group id, to make it possible to have nested groups. * API for retrieving grouped parameters for the Signature objects. Something akin to what we have for ``regex.Match`` objects, probably. * An accepted and documented method of declaring groups for pure python function would be a nice thing to have. * Will we have groups for keyword-only parameters? Can we combine ``*args`` and a keyword-only parameter in a group? etc. That seems to be a lot of work (some of it is maybe enough for a PEP.) So before committing to the parameters groups idea, I'd like to propose somewhat simpler, but powerful enough to solve our todays problems solution. What if we add a notion of "optional" parameters? * ``Parameter.__init__ `` will receive one more keyword-only argument: ``optional``, ``False`` by default. * We add a special marker ``Parameter.optional`` (or some other name, like ``inspect.optional`` or ``functools.optional``), and teach ``inspect.signature`` to recognize it. So for pure-python functions, if you want to define an optional parameter, you would write: ``def mytype(name_or_obj, bases=Parameter.optional, dict=Parameter.optional)`` * Argument Clinic may get a new syntax for specifying if parameter is optional. * We standardize how optional parameters should look like in documentation and ``Signature.__str__``. In PEP 362 we used <optional> notation for optional parameters: ``foo(param=<optional>)``, but we also can use square brackets for that: ``bar([spam][, ham])``. With this approach, a signature of the ``type`` function would look like: ``type(object_or_name[, bases][, dict])``. The main downside is that it's not immediately apparent, that you can only pass either one argument "(object)", or all three arguments "(name, bases, dict)". But that's something, that a good documentation (and meaningful exceptions) could help with. The advantages if this approach, is that it works for all types of parameters, and that the implementation is going to be simpler than groups (and we will need fewer new APIs). Yury
Guido, Larry and I thrashed out the required semantics for parameter groups at PyCon US last year (and I believe the argument clinic PEP describes those accurately). They're mainly needed to represent oddball signatures like range() and slice(). However, I'm inclined to say that the affected functions should simply not support introspection until Python 3.5. It's not just a matter of the data model, there's also the matter of defining the string representation. Cheers, Nick.
On 01/19/2014 08:30 PM, Nick Coghlan wrote:
Guido, Larry and I thrashed out the required semantics for parameter groups at PyCon US last year (and I believe the argument clinic PEP describes those accurately).
They're mainly needed to represent oddball signatures like range() and slice().
However, I'm inclined to say that the affected functions should simply not support introspection until Python 3.5.
It's not just a matter of the data model, there's also the matter of defining the string representation.
Au contraire!, says the man writing patches. How it works right now: Argument Clinic publishes the signature information for builtins as the first line of the builtin's docstring. It gets clipped off before __doc__ returns it, and it's made available as __text_signature__. inspect.Signature retrieves this string and passes it in to ast.parse(). It then walks the resulting tree, producing the Function and Parameter objects as it goes. Argument Clinic has the information about positional-only parameters and optional groups, but it can't express it in this text signature. The text signature is parsed by ast.parse(), and ast.parse() only understands Python syntax. So the information is thrown away. Boo hoo. I was trying to change this with PEP 457: http://www.python.org/dev/peps/pep-0457/ PEP 457 proposed new Python syntax for positional-only parameters and optional groups. The syntax looks like Argument Clinic's syntax for these features, but with commas replacing newlines: Positional-only parameters are delimited with a single slash. In "def foo(self, /, a)", self is positional-only and a is not. Optional groups are parameters wrapped with square brackets, with the brackets never appearing between a parameter and a comma. In "def foo(a, [b, c])", b and c are in an optional group together. Groups can be nested with some restrictions. To be clear: I wasn't proposing we actually add this syntax to Python! Just that, *if* we added support for positional-only parameters and optional groups, it would be this syntax. I did propose that documentation and tools like Argument Clinic could use it now. PEP 457 didn't get any support. People said: "We don't need to define syntax if we're not going to use it. If you really need this you can do it privately." Okay! I'll do it privately! Hopefully this week! Step 1: Extend the format of the __text_signature__ to use PEP 457 syntax. Then *strip* that extra syntax in inspect.Signature, producing a sanitized string that is parseable by ast.parse(), and storing the extra information on the side. Pass the sanitized string in to ast.parse(), walk the parse tree, and merge the positional-only / optional group information into the Parameter objects as I go. inspect.Signature objects can already represent positional-only parameters, and I plan to add a new member ("group", see below) that lets them represent optional groups too. Hooray! inspect.Signature objects can now represent almost every Python callable! Step 2: make pydoc print the optional groups information, aka the PEP 457 square brackets, because that's uncontroversial. pydoc on builtins has printed things like "range([start], stop, [step])" for decades. Everybody intuitively understands it, everybody likes it. Step 3, if people want it: Also make pydoc display which parameters are positional-only. I was going to propose that in a couple of days, when I was getting near ready to implement it. But I guess we're discussing it now. Unsurprisingly, I propose to use the PEP 457 syntax here. -------------------- Regarding Step 3 above, here's something to consider. str(inspect.signature(foo)) produces a very nice-looking string, which currently is (almost!) always parseable by ast.parse() like so: sig = inspect.signature(foo) if sig: ast.parse("def foo" + str(sig) + ": pass") On the one hand, this mechanical round-trip ability is kind of a nice property. On the other hand, it does restrict the format of the string--it can't have our square brackets, it can't have our "/,". Meanwhile __str__ makes no guarantee that the string it returns is a valid Python expression. We could do both. Make inspect.Signature.__str__ only return Python compatible syntax, and allow another mechanism for pydoc to produce something more informative (but not Python compatible) for the user, like so: >>> str(inspect.signature(_pickle.Pickler.dump)) '(self, obj)' >>> pydoc.plain(pydoc.render_doc(pickle.Pickler.dump)) "...\n_pickle.Pickler.dump(self, /, obj)\n..." Or we could do it the other way around. So I guess I'm proposing four alternatives: 1) inspect.Signature.__str__() must always be parsable by ast.parse(). 2) inspect.Signature.__str__() must always be parsable by ast.parse(). Add another method to inspect.Signature that can use PEP 457 syntax, use that from pydoc. 3) inspect.Signature.__str__() produces PEP 457 syntax. Add another method to inspect.Signature producing a text representation that is parseable by ast.parse(). 4) inspect.Signature.__str__() produces PEP 457 syntax. -------------------- In case you thought we were done, there's one more wrinkle: str(inspect.signature(foo)) *already* marks positional-only parameters! I discovered that in the last day or two. Late last week I noticed that "self" is *always* positional-only for builtins. It doesn't matter if the builtin is METH_KEYWORDS. So, in my burgeoning "add support for more builtins" patch, in inspect.Signature I marked self parameters on builtins as positional-only. I was rewarded with this: >>> str(inspect.signature(_pickle.Pickler.dump)) '(<self>, obj)' Yes, inspect.Signature.__str__() uses angle brackets to denote positional-only parameters. I think this behavior needs to be removed. It's undocumented and way non-obvious. I'm not aware of this syntax getting support from much of anybody--the only reason it's survived this long is because nobody besides me has ever seen it in the wild. -------------------- To address Yury's proposal:
So before committing to the parameters groups idea, I'd like to propose somewhat simpler, but powerful enough to solve our todays problems solution.
What if we add a notion of "optional" parameters?
Your proposal gets a "no, absolutely not" vote from me. 1. We already have a notion of "optional parameters". Parameters with default values are optional. 2. Your proposed syntax doesn't mention how we'd actually establish default values for parameters. So it's insufficient to handle existing code. 3. Your syntax/semantics, as described, can't convey the concept of optional groups. So it's insufficient to solve the problem it sets out to solve. 4. Your proposed syntax changes 80% of existing code--any parameter with a default value. I don't know how you concluded this was "simpler". Here's my counter-proposal. 1. We add a new class to inspect named "ParameterGroup". This class will have only one public member, "parent". ParameterGroup.parent will always be either None or a different inspect.ParameterGroup instance. 2. We add a new member to inspect.Parameter named "group". "group" will be either None or an instance of inspect.ParameterGroup. 3. We add a new method on Parameter named "is_optional()". "is_optional()" returns True if the function can be called without providing the parameter. Here's the implementation: def is_optional(self): return (self.default is not self._empty) or (self.group is not None) 4. Textual representations intended for displaying to the user are permitted to use square brackets to denote optional groups. They might also be permitted to use "/" to delimit positional-only parameters from other types of parameters, if the community accepts this. 5. We don't change any Python language semantics and we don't break any existing code. Under my proposal: bytearray([source, [encoding, [errors]]]) source.group != encoding.group encoding.group != errors.group source.group.parent == None encoding.group.parent == source.group errors.group.parent == encoding.group source.is_optional() == encoding.is_optional() == errors.is_optional() == True curses.window.addch([x, y,] ch, [attr]) x.group == y.group x.group != attr.group x.group.parent == attr.group.parent == None x.is_optional() == y.is_optional() == attr.is_optional() == True ch.is_optional() == False Sorry about the length of this email, //arry/
On 20 January 2014 20:16, Larry Hastings <larry@hastings.org> wrote:
On 01/19/2014 08:30 PM, Nick Coghlan wrote:
Guido, Larry and I thrashed out the required semantics for parameter groups at PyCon US last year (and I believe the argument clinic PEP describes those accurately).
They're mainly needed to represent oddball signatures like range() and slice().
However, I'm inclined to say that the affected functions should simply not support introspection until Python 3.5.
It's not just a matter of the data model, there's also the matter of defining the string representation.
Au contraire!, says the man writing patches.
When I wrote that, I was thinking we had made inspect.Signature.__repr__ produce a nice string format, but then I noticed in the REPL today that we never got around to doing that - I think because we didn't know how to handle positional-only arguments, which already can't be expressed as Python syntax. (I haven't checked if we have an RFE filed anywhere) However, while I know you're keen to finally make introspection work for all C level callables in 3.4, even the ones with signatures that can't be expressed as Python function signatures, I'd like to strongly encourage you to hold off on that last part until Python 3.5. We're already in beta, we're already introducing a lot of code churn to get the C level callables that *can* have their signatures expressed as Python syntax converted, so where's the harm to users in saying that C level callables with non-Python signatures still don't support introspection in Python 3.4? Almost no C level callables support programmatic introspection in Python 3.3, so even what inspect.signature will already provide in beta 3 is a big step forward. While the text string used to communicate between Argument Clinic and inspect.signature will be private, the representation on inspect.Signature objects will be a new *public* API. As the discussions between you, me and Yury show, I don't think there's an immediately obvious best answer of how to do that. Your suggestion of just adding the group numbers to the Parameter objects would *work*, but it's not very Pythonic - we have container types that support nesting, which seems like a more natural structure for indicating parameter groups at the Python level. Essentially, the group number proposal feels like the kind of low level interface returned by getfullargspec(), not the kind of high level interface defined for inspect.Signature in PEP 362. It's going to take a while to come up with a public API for this aspect of C level signatures that feels right to at least you, me and Yury, and the beta period *really* isn't the right time to be doing that. If other changes like the binary interpolation proposals and adding the PEP 451 based target attributes to runpy can wait until Python 3.5 due to feature freeze, then I think adding full C level signature support to inspect.Signature can also wait. That way, you can resurrect PEP 457, recast it as proposing an *output* format for inspect.Signature.__repr__(), add an inspect.Signature.fromstr() API that can use it to create a signature object from __text_signature__ attributes (rather than relying on ast.parse), add the optional group support and do it *right*, rather than trying to squeeze it in as a new public API during the beta period, which may lock us in to supporting an introspection API we later regret. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Larry, Nick, On January 20, 2014 at 8:00:35 AM, Nick Coghlan (ncoghlan@gmail.com) wrote:
Your proposal gets a "no, absolutely not" vote from me.
1. We already have a notion of "optional parameters". Parameters with default values are optional. 2. Your proposed syntax doesn't mention how we'd actually establish default values for parameters. So it's insufficient to handle existing code. 3. Your syntax/semantics, as described, can't convey the concept of optional groups. So it's insufficient to solve the problem it sets out to solve. 4. Your proposed syntax changes 80% of existing code--any parameter with a default value. I don't know how you concluded this was "simpler".
I withdraw my proposal in its current form. Turns out I’ve missed groups definition in PEP 436. But, just want to add a few things. My proposal is still good for the other problem — having support for optional parameters (where “optional” is in a sense of itertools.repeat last parameter). Again, my example from the first email in this thread. Suppose you want to write itertools.repeat in python (where “n” can not be None, what matters is *was is passed or not*): _marker = object() def repeat(elem, n=_marker): pass Now, if you do 'str(inspect.signature(repeat))’ you’ll have something like: '(elem, n=<object object at 0x108650080>)’ However, if we choose to have a special marker object defined in the stdlib, you’d write: def repeat(elem, n=functools.optional): if n is functools.optional: # no param was passed and str of signature would look like ‘(elem[, n])’. Yury
On 1/20/2014 7:59 AM, Nick Coghlan wrote:
However, while I know you're keen to finally make introspection work for all C level callables in 3.4, even the ones with signatures that can't be expressed as Python function signatures, I'd like to strongly encourage you to hold off on that last part until Python 3.5. ... That way, you can resurrect PEP 457, recast it as proposing an *output* format for inspect.Signature.__repr__(), add an inspect.Signature.fromstr() API that can use it to create a signature object from __text_signature__ attributes (rather than relying on ast.parse), add the optional group support and do it *right*, rather than trying to squeeze it in as a new public API during the beta period, which may lock us in to supporting an introspection API we later regret.
I agree. What we can do with the API we have already is a great advance. -- Terry Jan Reedy
On 21 Jan 2014 06:26, "Terry Reedy" <tjreedy@udel.edu> wrote:
On 1/20/2014 7:59 AM, Nick Coghlan wrote:
However, while I know you're keen to finally make introspection work for all C level callables in 3.4, even the ones with signatures that can't be expressed as Python function signatures, I'd like to strongly encourage you to hold off on that last part until Python 3.5.
...
That way, you can resurrect PEP 457, recast it as proposing an *output* format for inspect.Signature.__repr__(), add an inspect.Signature.fromstr() API that can use it to create a signature object from __text_signature__ attributes (rather than relying on ast.parse), add the optional group support and do it *right*, rather than trying to squeeze it in as a new public API during the beta period, which may lock us in to supporting an introspection API we later regret.
I agree. What we can do with the API we have already is a great advance.
It also occurred to me last night that PEP 457 could define a "functools.textsignature" decorator to permit describing a particular signature on arbitrary callables (using the attribute already added for Argument Clinic, but extended to arbitrary types). That would allow signature overrides without needing to import the inspect module at startup. Cheers, Nick.
-- Terry Jan Reedy
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On 01/20/2014 04:59 AM, Nick Coghlan wrote:
When I wrote that, I was thinking we had made inspect.Signature.__repr__ produce a nice string format, but then I noticed in the REPL today that we never got around to doing that - I think because we didn't know how to handle positional-only arguments, which already can't be expressed as Python syntax. (I haven't checked if we have an RFE filed anywhere)
I don't know what you had intended to do, but right now inspect.Signature inherits the standard repl from object. inspect.Signature.__str__ produces something that looks like a Python function signature, starting and ending with parentheses. (For those of you unfamiliar with inspect.Signature: A signature is agnostic about the name of the function. So it doesn't include the name.)
However, while I know you're keen to finally make introspection work for all C level callables in 3.4, even the ones with signatures that can't be expressed as Python function signatures, I'd like to strongly encourage you to hold off on that last part until Python 3.5.
If we hold off on all of this until 3.5, the signatures for most builtins will be wrong in 3.4, because most builtins take positional-only parameters. I had higher hopes for Python 3.4 than that. To be honest I'd rather not have the feature at all than have it be wrong most of the time. I think it's fair to summarize your argument as "there could be monsters lurking in CPython with signatures that can't be expressed in PEP 457 syntax". To me this smacks of FUD. Let me open my kimono and tell you all the counter-examples we know of so far. * socket.sendto() has an optional group in the middle of required parameters. (This signature is from 1993.) PEP 457 could support this just by relaxing one requirement. I know what's needed here, but given that PEP 457 was such a dud I haven't bothered to update it. Regardless, Argument Clinic, and the syntax used for text signatures, could (and I expect will soon) support this. The inspect.Parameter.group proposal from my email last night supports this just fine. * itertools.repeat() has a parameter that behaves differently if it's passed by keyword vs passed by position. Guido already ruled that this signature must be changed so it is representable with Python syntax--this behavior is a "bug". * Many functions have default values that are not representable in Python, chiefly a NULL pointer. Guido has already ruled that these signatures should be changed so that they're representable in Python. The best approach is often accepting None, which is inconvenient for non-pointer arguments like integers. Right now Argument Clinic gives you no assistance in this area, but I plan to add explicit support making it easy (via "nullable ints"). In short, there's a clear trend: functions must have signatures representable in Python syntax, with the exception of optional groups which are a legacy feature we can't get rid of but won't support in Python syntax. Any functions whose signatures are not representable in Python syntax shall be tweaked until they are. Any new monsters we discover lurking in CPython will be slain, not supported. ----- We could split the difference, and not add a feature to the inspect module to support optional groups. We could still support marking positional-only parameters, as inspect currently supports that. That would mean nearly all signatures for builtins would be correct. Personally I'd rather go the extra distance and support optional groups too. There are important callables that can only be expressed with optional groups (range, type). Given the trend above, Parameter arguments with optional groups should be sufficient to express every signature available in Python. We've come this far... or, as the British say, in for a penny, in for a pound. Let's hash it out right now and get it done.
While the text string used to communicate between Argument Clinic and inspect.signature will be private, the representation on inspect.Signature objects will be a new *public* API. As the discussions between you, me and Yury show, I don't think there's an immediately obvious best answer of how to do that. Your suggestion of just adding the group numbers to the Parameter objects would *work*, but it's not very Pythonic - we have container types that support nesting,
Apparently you didn't read my proposal in the email you replied to. I didn't propose that "group" contain a number, I proposed it contain a ParameterGroup object that supports nesting. We could take another approach, one you seem to be suggesting, where the nesting is outside the Parameter objects. In this alternate approach, the Signature.parameters array can contain either Parameter objects or OrderedDicts. The nested OrderedDicts themselves can contain either Parameter objects or more nested OrderedDicts. The API would specify that the nested OrderedDicts of parameters are optional en masse. This works fine too. The chief difference between these proposals: if you ignore the complexity of optional groups, the failure mode with ".group" is that it kind of works except when it doesn't, whereas with having OrderedDicts in .parameters the failure mode is that your code blows up with missing attributes (like "couldn't find an attribute called name on this OrderedDict object"). That's probably a vote in favor of the nested OrderedDicts. //arry/
On 21 Jan 2014 09:26, "Larry Hastings" <larry@hastings.org> wrote:
On 01/20/2014 04:59 AM, Nick Coghlan wrote:
When I wrote that, I was thinking we had made inspect.Signature.__repr__ produce a nice string format, but then I noticed in the REPL today that we never got around to doing that - I think because we didn't know how to handle positional-only arguments, which already can't be expressed as Python syntax. (I haven't checked if we have an RFE filed anywhere)
I don't know what you had intended to do, but right now inspect.Signature
inherits the standard repl from object. inspect.Signature.__str__ produces something that looks like a Python function signature, starting and ending with parentheses. (For those of you unfamiliar with inspect.Signature: A signature is agnostic about the name of the function. So it doesn't include the name.)
However, while I know you're keen to finally make introspection work for all C level callables in 3.4, even the ones with signatures that can't be expressed as Python function signatures, I'd like to strongly encourage you to hold off on that last part until Python 3.5.
If we hold off on all of this until 3.5, the signatures for most builtins
will be wrong in 3.4, because most builtins take positional-only parameters. I had higher hopes for Python 3.4 than that. To be honest I'd rather not have the feature at all than have it be wrong most of the time. Positional only is fine - PEP 362 already handles those. It only doesn't handle things like range(), and those callables should continue to not support introspection at all rather than reporting an incorrect signature.
I think it's fair to summarize your argument as "there could be monsters lurking in CPython with signatures that can't be expressed in PEP 457 syntax".
No. I am saying there *are* signatures that the *inspect module* cannot express in its public API. You already *know* that, since you are proposing to add a new feature (group support) to inspect.Signature late in the beta cycle in order to handle those cases. I am saying that's a gross violation of our established processes. The argument clinic conversions can be defended as internal implementation details. A new public feature in the inspect module cannot. Please turn the question around and look at it with your release manager hat on rather than your creator of Argument Clinic hat: if I came to you and said I wanted to add a new public API to the inspect module after the second beta release, what would you say? Can you honestly say that if *someone else* was proposing the inclusion of a new public API this late in the release cycle, you would say yes? If I can wait until 3.5 to add PEP 451 "target" parameters to runpy because I was too busy to land that before feature freeze, and Eric can wait to fully support PEP 451 for builtin and extension modules, and Ethan can wait to restore binary interpolation, and Antoine can wait a full release cycle between adding qualified names in 3.3 and actually seeing them used in an updated pickle protocol in 3.4, there's no good reason to rush adding introspection support for oddball legacy signatures to the inspect module. Regards, Nick.
On 01/20/2014 03:53 PM, Nick Coghlan wrote:
Please turn the question around and look at it with your release manager hat on rather than your creator of Argument Clinic hat: if I came to you and said I wanted to add a new public API to the inspect module after the second beta release, what would you say? Can you honestly say that if *someone else* was proposing the inclusion of a new public API this late in the release cycle, you would say yes?
You're right. Optional group information won't be a public API in 3.4. //arry/
On 21 Jan 2014 13:49, "Larry Hastings" <larry@hastings.org> wrote:
On 01/20/2014 03:53 PM, Nick Coghlan wrote:
Please turn the question around and look at it with your release manager
hat on rather than your creator of Argument Clinic hat: if I came to you and said I wanted to add a new public API to the inspect module after the second beta release, what would you say? Can you honestly say that if *someone else* was proposing the inclusion of a new public API this late in the release cycle, you would say yes?
You're right. Optional group information won't be a public API in 3.4.
Thanks for that. I agree it's a shame we missed the 3.4 feature deadline (as I would also like to see full C level signature support done and dusted), but once 3.4 is out the door we can hopefully resurrect PEP 457 as a direct successor to the original PEP 362, and update the inspect module to also handle these quirkier variants. Perhaps we can even add that public API for building Signature objects from string definitions, since that has proved quite handy internally :) Cheers, Nick.
/arry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
participants (4)
-
Larry Hastings -
Nick Coghlan -
Terry Reedy -
Yury Selivanov