[Python-Dev] signature.object, argument clinic and grouped parameters

Larry Hastings larry at hastings.org
Mon Jan 20 11:16:47 CET 2014

On 01/19/2014 08:30 PM, Nick Coghlan wrote:
> Guido, Larry and I thrashed out the required semantics for parameter 
> groups at PyCon US last year (and I believe the argument clinic PEP 
> describes those accurately).
> They're mainly needed to represent oddball signatures like range() and 
> slice().
> However, I'm inclined to say that the affected functions should simply 
> not support introspection until Python 3.5.
> It's not just a matter of the data model, there's also the matter of 
> defining the string representation.

Au contraire!, says the man writing patches.

How it works right now: Argument Clinic publishes the signature 
information for builtins as the first line of the builtin's docstring.  
It gets clipped off before __doc__ returns it, and it's made available 
as __text_signature__.  inspect.Signature retrieves this string and 
passes it in to ast.parse().  It then walks the resulting tree, 
producing the Function and Parameter objects as it goes.

Argument Clinic has the information about positional-only parameters and 
optional groups, but it can't express it in this text signature.  The 
text signature is parsed by ast.parse(), and ast.parse() only 
understands Python syntax.  So the information is thrown away.  Boo hoo.

I was trying to change this with PEP 457:


PEP 457 proposed new Python syntax for positional-only parameters and 
optional groups.  The syntax looks like Argument Clinic's syntax for 
these features, but with commas replacing newlines:

    Positional-only parameters are delimited with a single slash.  In
    "def foo(self, /, a)", self is positional-only and a is not.
    Optional groups are parameters wrapped with square brackets, with
    the brackets never appearing between a parameter and a comma.  In
    "def foo(a, [b, c])", b and c are in an optional group together.
    Groups can be nested with some restrictions.

To be clear: I wasn't proposing we actually add this syntax to Python!  
Just that, *if* we added support for positional-only parameters and 
optional groups, it would be this syntax.  I did propose that 
documentation and tools like Argument Clinic could use it now.

PEP 457 didn't get any support.  People said: "We don't need to define 
syntax if we're not going to use it.  If you really need this you can do 
it privately."

Okay!  I'll do it privately!  Hopefully this week!

Step 1: Extend the format of the __text_signature__ to use PEP 457 
syntax.  Then *strip* that extra syntax in inspect.Signature, producing 
a sanitized string that is parseable by ast.parse(), and storing the 
extra information on the side.  Pass the sanitized string in to 
ast.parse(), walk the parse tree, and merge the positional-only / 
optional group information into the Parameter objects as I go.  
inspect.Signature objects can already represent positional-only 
parameters, and I plan to add a new member ("group", see below) that 
lets them represent optional groups too.  Hooray! inspect.Signature 
objects can now represent almost every Python callable!

Step 2: make pydoc print the optional groups information, aka the PEP 
457 square brackets, because that's uncontroversial.  pydoc on builtins 
has printed things like "range([start], stop, [step])" for decades.  
Everybody intuitively understands it, everybody likes it.

Step 3, if people want it: Also make pydoc display which parameters are 
positional-only.  I was going to propose that in a couple of days, when 
I was getting near ready to implement it.  But I guess we're discussing 
it now.  Unsurprisingly, I propose to use the PEP 457 syntax here.


Regarding Step 3 above, here's something to consider. 
str(inspect.signature(foo)) produces a very nice-looking string, which 
currently is (almost!) always parseable by ast.parse() like so:

    sig = inspect.signature(foo)
    if sig:
         ast.parse("def foo" + str(sig) + ": pass")

On the one hand, this mechanical round-trip ability is kind of a nice 

On the other hand, it does restrict the format of the string--it can't 
have our square brackets, it can't have our "/,".  Meanwhile __str__ 
makes no guarantee that the string it returns is a valid Python expression.

We could do both.   Make inspect.Signature.__str__ only return Python 
compatible syntax, and allow another mechanism for pydoc to produce 
something more informative (but not Python compatible) for the user, 
like so:

     >>> str(inspect.signature(_pickle.Pickler.dump))
    '(self, obj)'

     >>> pydoc.plain(pydoc.render_doc(pickle.Pickler.dump))
    "...\n_pickle.Pickler.dump(self, /, obj)\n..."

Or we could do it the other way around.

So I guess I'm proposing four alternatives:
1) inspect.Signature.__str__() must always be parsable by ast.parse().
2) inspect.Signature.__str__() must always be parsable by ast.parse().  
Add another method to inspect.Signature that can use PEP 457 syntax, use 
that from pydoc.
3) inspect.Signature.__str__() produces PEP 457 syntax.  Add another 
method to inspect.Signature producing a text representation that is 
parseable by ast.parse().
4) inspect.Signature.__str__() produces PEP 457 syntax.


In case you thought we were done, there's one more wrinkle: 
str(inspect.signature(foo)) *already* marks positional-only parameters!  
I discovered that in the last day or two.

Late last week I noticed that "self" is *always* positional-only for 
builtins.  It doesn't matter if the builtin is METH_KEYWORDS.  So, in my 
burgeoning "add support for more builtins" patch, in inspect.Signature I 
marked self parameters on builtins as positional-only.  I was rewarded 
with this:

     >>> str(inspect.signature(_pickle.Pickler.dump))
    '(<self>, obj)'

Yes, inspect.Signature.__str__() uses angle brackets to denote 
positional-only parameters.

I think this behavior needs to be removed.  It's undocumented and way 
non-obvious.  I'm not aware of this syntax getting support from much of 
anybody--the only reason it's survived this long is because nobody 
besides me has ever seen it in the wild.


To address Yury's proposal:
> So before committing to the parameters groups idea, I'd like to propose
> somewhat simpler, but powerful enough to solve our todays problems
> solution.
> What if we add a notion of "optional" parameters?

Your proposal gets a "no, absolutely not" vote from me.

1. We already have a notion of "optional parameters".  Parameters with 
default values are optional.
2. Your proposed syntax doesn't mention how we'd actually establish 
default values for parameters.  So it's insufficient to handle existing 
3. Your syntax/semantics, as described, can't convey the concept of 
optional groups.  So it's insufficient to solve the problem it sets out 
to solve.
4. Your proposed syntax changes 80% of existing code--any parameter with 
a default value.  I don't know how you concluded this was "simpler".

Here's my counter-proposal.

1. We add a new class to inspect named "ParameterGroup".  This class 
will have only one public member, "parent".  ParameterGroup.parent will 
always be either None or a different inspect.ParameterGroup instance.
2. We add a new member to inspect.Parameter named "group".  "group" will 
be either None or an instance of inspect.ParameterGroup.
3. We add a new method on Parameter named "is_optional()". 
"is_optional()" returns True if the function can be called without 
providing the parameter.  Here's the implementation:

    def is_optional(self):
         return (self.default is not self._empty) or (self.group is not

4. Textual representations intended for displaying to the user are 
permitted to use square brackets to denote optional groups.  They might 
also be permitted to use "/" to delimit positional-only parameters from 
other types of parameters, if the community accepts this.
5. We don't change any Python language semantics and we don't break any 
existing code.

Under my proposal:

    bytearray([source, [encoding, [errors]]])

        source.group != encoding.group
        encoding.group != errors.group
        source.group.parent == None
        encoding.group.parent == source.group
        errors.group.parent == encoding.group
        source.is_optional() == encoding.is_optional() ==
        errors.is_optional() == True

    curses.window.addch([x, y,] ch, [attr])

        x.group == y.group
        x.group != attr.group
        x.group.parent == attr.group.parent == None
        x.is_optional() == y.is_optional() == attr.is_optional() == True
        ch.is_optional() == False

Sorry about the length of this email,

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140120/19d7598e/attachment-0001.html>

More information about the Python-Dev mailing list