[Python-Dev] signature.object, argument clinic and grouped parameters
larry at hastings.org
Mon Jan 20 11:16:47 CET 2014
On 01/19/2014 08:30 PM, Nick Coghlan wrote:
> Guido, Larry and I thrashed out the required semantics for parameter
> groups at PyCon US last year (and I believe the argument clinic PEP
> describes those accurately).
> They're mainly needed to represent oddball signatures like range() and
> However, I'm inclined to say that the affected functions should simply
> not support introspection until Python 3.5.
> It's not just a matter of the data model, there's also the matter of
> defining the string representation.
Au contraire!, says the man writing patches.
How it works right now: Argument Clinic publishes the signature
information for builtins as the first line of the builtin's docstring.
It gets clipped off before __doc__ returns it, and it's made available
as __text_signature__. inspect.Signature retrieves this string and
passes it in to ast.parse(). It then walks the resulting tree,
producing the Function and Parameter objects as it goes.
Argument Clinic has the information about positional-only parameters and
optional groups, but it can't express it in this text signature. The
text signature is parsed by ast.parse(), and ast.parse() only
understands Python syntax. So the information is thrown away. Boo hoo.
I was trying to change this with PEP 457:
PEP 457 proposed new Python syntax for positional-only parameters and
optional groups. The syntax looks like Argument Clinic's syntax for
these features, but with commas replacing newlines:
Positional-only parameters are delimited with a single slash. In
"def foo(self, /, a)", self is positional-only and a is not.
Optional groups are parameters wrapped with square brackets, with
the brackets never appearing between a parameter and a comma. In
"def foo(a, [b, c])", b and c are in an optional group together.
Groups can be nested with some restrictions.
To be clear: I wasn't proposing we actually add this syntax to Python!
Just that, *if* we added support for positional-only parameters and
optional groups, it would be this syntax. I did propose that
documentation and tools like Argument Clinic could use it now.
PEP 457 didn't get any support. People said: "We don't need to define
syntax if we're not going to use it. If you really need this you can do
Okay! I'll do it privately! Hopefully this week!
Step 1: Extend the format of the __text_signature__ to use PEP 457
syntax. Then *strip* that extra syntax in inspect.Signature, producing
a sanitized string that is parseable by ast.parse(), and storing the
extra information on the side. Pass the sanitized string in to
ast.parse(), walk the parse tree, and merge the positional-only /
optional group information into the Parameter objects as I go.
inspect.Signature objects can already represent positional-only
parameters, and I plan to add a new member ("group", see below) that
lets them represent optional groups too. Hooray! inspect.Signature
objects can now represent almost every Python callable!
Step 2: make pydoc print the optional groups information, aka the PEP
457 square brackets, because that's uncontroversial. pydoc on builtins
has printed things like "range([start], stop, [step])" for decades.
Everybody intuitively understands it, everybody likes it.
Step 3, if people want it: Also make pydoc display which parameters are
positional-only. I was going to propose that in a couple of days, when
I was getting near ready to implement it. But I guess we're discussing
it now. Unsurprisingly, I propose to use the PEP 457 syntax here.
Regarding Step 3 above, here's something to consider.
str(inspect.signature(foo)) produces a very nice-looking string, which
currently is (almost!) always parseable by ast.parse() like so:
sig = inspect.signature(foo)
ast.parse("def foo" + str(sig) + ": pass")
On the one hand, this mechanical round-trip ability is kind of a nice
On the other hand, it does restrict the format of the string--it can't
have our square brackets, it can't have our "/,". Meanwhile __str__
makes no guarantee that the string it returns is a valid Python expression.
We could do both. Make inspect.Signature.__str__ only return Python
compatible syntax, and allow another mechanism for pydoc to produce
something more informative (but not Python compatible) for the user,
"...\n_pickle.Pickler.dump(self, /, obj)\n..."
Or we could do it the other way around.
So I guess I'm proposing four alternatives:
1) inspect.Signature.__str__() must always be parsable by ast.parse().
2) inspect.Signature.__str__() must always be parsable by ast.parse().
Add another method to inspect.Signature that can use PEP 457 syntax, use
that from pydoc.
3) inspect.Signature.__str__() produces PEP 457 syntax. Add another
method to inspect.Signature producing a text representation that is
parseable by ast.parse().
4) inspect.Signature.__str__() produces PEP 457 syntax.
In case you thought we were done, there's one more wrinkle:
str(inspect.signature(foo)) *already* marks positional-only parameters!
I discovered that in the last day or two.
Late last week I noticed that "self" is *always* positional-only for
builtins. It doesn't matter if the builtin is METH_KEYWORDS. So, in my
burgeoning "add support for more builtins" patch, in inspect.Signature I
marked self parameters on builtins as positional-only. I was rewarded
Yes, inspect.Signature.__str__() uses angle brackets to denote
I think this behavior needs to be removed. It's undocumented and way
non-obvious. I'm not aware of this syntax getting support from much of
anybody--the only reason it's survived this long is because nobody
besides me has ever seen it in the wild.
To address Yury's proposal:
> So before committing to the parameters groups idea, I'd like to propose
> somewhat simpler, but powerful enough to solve our todays problems
> What if we add a notion of "optional" parameters?
Your proposal gets a "no, absolutely not" vote from me.
1. We already have a notion of "optional parameters". Parameters with
default values are optional.
2. Your proposed syntax doesn't mention how we'd actually establish
default values for parameters. So it's insufficient to handle existing
3. Your syntax/semantics, as described, can't convey the concept of
optional groups. So it's insufficient to solve the problem it sets out
4. Your proposed syntax changes 80% of existing code--any parameter with
a default value. I don't know how you concluded this was "simpler".
Here's my counter-proposal.
1. We add a new class to inspect named "ParameterGroup". This class
will have only one public member, "parent". ParameterGroup.parent will
always be either None or a different inspect.ParameterGroup instance.
2. We add a new member to inspect.Parameter named "group". "group" will
be either None or an instance of inspect.ParameterGroup.
3. We add a new method on Parameter named "is_optional()".
"is_optional()" returns True if the function can be called without
providing the parameter. Here's the implementation:
return (self.default is not self._empty) or (self.group is not
4. Textual representations intended for displaying to the user are
permitted to use square brackets to denote optional groups. They might
also be permitted to use "/" to delimit positional-only parameters from
other types of parameters, if the community accepts this.
5. We don't change any Python language semantics and we don't break any
Under my proposal:
bytearray([source, [encoding, [errors]]])
source.group != encoding.group
encoding.group != errors.group
source.group.parent == None
encoding.group.parent == source.group
errors.group.parent == encoding.group
source.is_optional() == encoding.is_optional() ==
errors.is_optional() == True
curses.window.addch([x, y,] ch, [attr])
x.group == y.group
x.group != attr.group
x.group.parent == attr.group.parent == None
x.is_optional() == y.is_optional() == attr.is_optional() == True
ch.is_optional() == False
Sorry about the length of this email,
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-Dev