On Mon, Feb 3, 2014 at 8:04 AM, Larry Hastings wrote:

On 02/03/2014 07:08 AM, Barry Warsaw wrote:

On Feb 03, 2014, at 06:43 AM, Larry Hastings wrote:

But that only fixes part of the problem. Our theoretical extension that wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can they support signatures? They can't give PyMethodDefEx structures to 3.3, it will blow up. But if they don't use PyMethodDefEx, they can't have signatures.

Can't an extension writer #ifdef around this? Yeah, it's ugly, but it's a pretty standard approach for making C extensions multi-version compatible.

For source compatibility, yes. But I thought the point of the binary ABI was to allow compiling a single extension that worked unmodified with multiple versions of Python. If we simply don't support that, then an ifdef would be fine.

Wouldn't your proposal to extend the PyMethodDef structure would require ifdef's and make it impossible to include the type information in something compiled against the 3.3 headers that you want to use in 3.4 without recompiling? If you don't like seeing an sig= at the front of the docstring couldn't you just move it to the end of the docstring. I don't think messiness in docstrings when running something read for 3.4 under 3.3 is a big deal. [side note] I consider it CRAZY for anyone to load a binary extension module compiled for one version in a later version of Python. People do it, I know, but they're insane. I wish we didn't bother trying to support that crap. I know this isn't going to change in 3.4. Just ranting. [/side note] -gps

On 4 February 2014 02:04, Larry Hastings wrote:

On 02/03/2014 07:08 AM, Barry Warsaw wrote:

On Feb 03, 2014, at 06:43 AM, Larry Hastings wrote:

But that only fixes part of the problem. Our theoretical extension that wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can they support signatures? They can't give PyMethodDefEx structures to 3.3, it will blow up. But if they don't use PyMethodDefEx, they can't have signatures.

Can't an extension writer #ifdef around this? Yeah, it's ugly, but it's a pretty standard approach for making C extensions multi-version compatible.

For source compatibility, yes. But I thought the point of the binary ABI was to allow compiling a single extension that worked unmodified with multiple versions of Python. If we simply don't support that, then an ifdef would be fine.

Then the solution appears straightforward to me: Python 3.4 will not support providing introspection information through the stable ABI. If you want to provide signature info for your C extension without an odd first line in your 3.3 docstring, you must produce version specific binaries (which allows #ifdef hackery). Then PEP 457 can address this properly for 3.5 along with the other issues it needs to cover. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 02/03/2014 08:05 AM, Stefan Krah wrote:

I think we may slowly get into PEP territory here. Just imagine that we settle on X, then decide at a later point to have a standard way of adding type annotations, then find that X does not work because of (unknown).

I'm mentioning this because signatures get really interesting for me if they contain type information.

I simultaneously share your interest, and also suspect that maybe Python is the wrong language for that. After all, Python has always been about duck-typing. Even if it did happen, it won't be for quite a while yet. The logical mechanism for type information in pure Python is annotations, and afaik they're not getting any large-scale real-world use for type annotating. (If I'm misinformed I'd love to hear counterexamples.) //arry/

On 02/03/2014 09:46 AM, Guido van Rossum wrote:

Can you summarize why neither of the two schemes you tried so far worked?

Certainly. In the first attempt, the signature looked like this: <name-of-function>(arguments)\n The "(arguments)" part of the string was 100% compatible with Python syntax. So much so that I didn't write my own parser. Instead, I would take the whole line, strip off the \n, prepend it with "def ", append it with ": pass", and pass in the resulting string to ast.parse(). This had the advantage of looking great if the signature was not mechanically separated from the rest of the docstring: it looked like the old docstring with the handwritten signature on top. The problem: false positives. This is also exactly the traditional format for handwritten signatures. The function in C that mechanically separated the signature from the rest of the docstring had a simple heuristic: if the docstring started with "<name-of-function>(", it assumed it had a valid signature and separated it from the rest of the docstring. But most of the functions in CPython passed this test, which resulted in complaints like "help(open) eats first line": http://bugs.python.org/issue20075 I opened an issue, writing a long impassioned plea to change this syntax: http://bugs.python.org/issue20326 Which we did. In the second attempt, the signature looked like this: sig=(arguments)\n In other words, the same as the first attempt, but with "sig=" instead of the name of the function. Since you never see docstrings that start with "sig=" in the wild, the false positives dropped to zero. I also took the opportunity to modify the signature slightly. Signatures were a little inconsistent about whether they specified the "self" parameter or not, so there were some complicated heuristics in inspect.Signature about when to keep or omit the first argument. In the new format I made this more explicit: if the first argument starts with a dollar sign ("$"), that means "this is a special first argument" (self for methods, module for module-level callables, type for class methods and __new__). That removed all the guesswork from inspect.Signature; now it works great. (In case you're wondering: I still use ast.parse to parse the signature, I just strip out the "$" first.) I want to mention: we anticipate modifying the syntax further in 3.5, adding square brackets around parameters to indicate "optional groups". This all has caused no problems so far. But my panicky email last night was me realizing a problem we may see down the road. To recap: if a programmer writes a module using the binary ABI, in theory they can use it with different Python versions without modification. If this programmer added Python 3.4+ compatible signatures, they'd have to insert this "sig=(" line at the top of their docstring. The downside: Python 3.3 doesn't understand that this is a signature and would happily display it to the user as part of help().

How bad would it be if we decided to just live with it or if we added a new flag bit (only recognized by 3.4) to disambiguate corner-cases?

A new flag might solve the problem cheaply. Let's call it METH_SIG, set in the flags portion of the PyMethodDef. It would mean "This docstring contains a computer-readable signature". One could achieve source compatibility with 3.3 easily by adding "#ifndef METH_SIG / #define METH_SIG 0 / #endif"; the next version of 3.3 could add that itself. We could then switch back to the original approach of "<name-of-function>(", so the signature would look presentable when displayed to the user. It would still have the funny dollar-sign, a la "$self" or "$module" or "$type", but perhaps users could live with that. Though perhaps this time maybe the end delimiter should be two newlines in a row, so that we can text-wrap long signature lines to enhance their readability if/when they get shown to users. I have two caveats: A: for binary compatibility, would Python 3.3 be allergic to this unfamiliar flag in PyMethodDef? Or does it ignore flags it doesn't explicitly look for? B: I had to modify four (or was it five?) different types in Python to add support for mechanically separating the __text_signature__. Although all of them originally started with a PyMethodDef structure, I'm not sure that all of them carry the "flags" parameter around with them. We might have to add a "flags" to a couple of these. Fortunately I believe they're all part of Py_LIMITED_API. //arry/

On 02/03/2014 08:19 PM, Guido van Rossum wrote:

But why do you even need a flag? Reading issue 20075 where the complaint started, it really feels that the change was an overreaction to a very minimal problem.

I'll cop to that. I'm pretty anxious about trying to "get it right". My worry was (and is) that this hiding-the-signature-in-the-docstring approach is a cheap hack, and it will have unexpected and undesirable side-effects that will in retrospect seem obvious. This is FUD I admit. But it seems to me if we did it "the right way", with a "PyMethodDefEx", we'd be able to do a lot better job predicting the side-effects.

A few docstrings appear truncated. Big deal. We can rewrite the ones that are reported as broken (either by adjusting the docstring to not match the patter or by adjusting it to match the pattern better, depending on the case). Tons of docstrings contain incorrect info, we just fix them when we notice the issue, we don't declare the language broken.

I don't think #20075 touches on it, but my biggest concern was third-party modules. If you maintain a Python module, you very well might compile for 3.4 only to find that the first line of your docstrings have mysteriously vanished. You'd have to be very current on changes in Python 3.4 to know what was going on. It seemed like an overly efficient way of pissing off external module maintainers. (Why would they vanish? The mechanical separator for __doc__ vs __text_signature__ would accept them, but unless they're 100% compatible Python ast.parse will reject them. So they'd get stripped from your docstring, but you wouldn't get a valid signature in return.) I'd feel much better with an explicit flag--explicit is better than implicit, after all. But here's a reminder, to make it easier for you to say "no". That would mean adding an explicit flag to all the objects which support a signature hidden in the docstring: * PyTypeObject (has tp_flags, we've used 18 of 32 bits by my count) * PyMethodDef (has ml_flags, 7 of 32 bits in use) * PyMethodDescrObject (reuses PyMethodDef) * PyWrapperDescrObject (has d_base->flags, 1 of 32 bits in use) * wrapperobject (reuses PyWrapperDescrObject) Argument Clinic would write the PyMethodDefs, so we'd get those for free. The last three all originate from a PyMethodDef, so when we copied out the docstring pointer we could also propagate the flag. But we'd have to add the flag by hand to the PyTypeObjects. If you won't let me have a flag, can I at least have a more-clever marker? How about this: <name-of-function>(...)\n \n Yes, the last four characters are right-parenthesis, newline, space, and newline. Benefits: * The odds of finding *that* in the wild seem remote. * If this got displayed as help in 3.3 the user would never notice the space. For the record, here are things that may be in the signature that aren't legal Python syntax and therefore might be surprising: * "self" parameters (and "module" and "type") are prefixed with '$'. * Positional-only parameters will soon be delimited by '/', just as keyword-only parameters are currently delimited by '*'. (Hasn't happened yet. Needs to happen for 3.4, in order for inspect.Signature to be accurate.) //arry/

On 02/04/2014 01:41 AM, Georg Brandl wrote:

Clever, but due to the "hidden" space it also increases the frustration factor for people trying to find out "why is this accepted as a signature and not this".

I don't think a well-chosen visible separator is worse off, such as "--\n".

I could live with that. To be explicit: the signature would then be of the form

Am 04.02.2014 13:14, schrieb Antoine Pitrou:

On Tue, 04 Feb 2014 02:21:51 -0800 Larry Hastings wrote:

...

On 02/04/2014 01:41 AM, Georg Brandl wrote:

...

Clever, but due to the "hidden" space it also increases the frustration factor for people trying to find out "why is this accepted as a signature and not this".

I don't think a well-chosen visible separator is worse off, such as "--\n".

I could live with that. To be explicit: the signature would then be of the form

The scanning function would look for "<name-of-function>(" at the front. If it found it it'd scan forwards in the docstring for ")\n--\n". If it found *that*, then it would declare success.

This would have to be checked for layout regressions. If the docstring is formatted using a ReST-to-HTML converter, what will be the effect?

The "--" will be added after the signature in the same paragraph. However, I don't think this is a valid concern: if you process signatures as ReST you will already have to deal with lots of markup errors (e.g. due to unpaired "*" and "**"). Tools that extract the docstrings and treat them specially (such as Sphinx) will adapt anyway. Georg

Sounds like a good compromise to me - and we certainly have a lot of interesting experience to feed into the design of PEP 457 for 3.5 ;) Cheers, Nick.

Larry Hastings writes:

In the second attempt, the signature looked like this:

sig=(arguments)\n

[...]

This all has caused no problems so far. But my panicky email last night was me realizing a problem we may see down the road. To recap: if a programmer writes a module using the binary ABI, in theory they can use it with different Python versions without modification. If this programmer added Python 3.4+ compatible signatures, they'd have to insert this "sig=(" line at the top of their docstring. The downside: Python 3.3 doesn't understand that this is a signature and would happily display it to the user as part of help().

I think this is not a bug, it's a feature. Since 3.3 users don't have the special signature parser either, this gives them exactly the information they need and without any duplication. The only drawback is in the cosmetic "sig=" prefix -- but that's the right amount of non-intrusive, kind nudging to get people to eventually update.

...

How bad would it be if we decided to just live with it or if we added a new flag bit (only recognized by 3.4) to disambiguate corner-cases?

A new flag might solve the problem cheaply. Let's call it METH_SIG, set in the flags portion of the PyMethodDef. It would mean "This docstring contains a computer-readable signature". One could achieve source compatibility with 3.3 easily by adding "#ifndef METH_SIG / #define METH_SIG 0 / #endif"; the next version of 3.3 could add that itself. We could then switch back to the original approach of "<name-of-function>(", so the signature would look presentable when [...]

That much effort to fix a purely cosmetic problem showing up only in older releases? Note that it's going to be a while until machine generated signatures have actually trickled down to end-users, so it's not as if every 3.3 installation would suddenly show different docstrings for all modules. Just my $0.02 of course. Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.«

On 02/03/2014 12:55 PM, Terry Reedy wrote:

I think the solution adopted should be future-oriented (ie, clean in the future) even if the cost is slight awkwardness in 3.3.

Just a minor point: I keep saying 3.3, but I kind of mean "3.2 through 3.3". I believe the binary ABI shipped with 3.2. However, in practice I suspect there are few installations that * are still on 3.2, and * will ever use binary-ABI-clean third-party modules compiled against 3.4+ that contain signatures. //arry/

On 02/03/2014 01:10 PM, Zachary Ware wrote:

What about just choosing a marker value that is somewhat less unsightly? "signature = (", or "parameters: (", or something (better) to that effect? It may not be beautiful in 3.3, but we can at least make it make sense.

It's a reasonable enough idea, and we could consider it if we stick with something like "sig=". However, see later in the thread where Guido says to return to the old somewhat-ambiguous form with the function name. ;-) //arry/

On 02/03/2014 02:26 PM, Antoine Pitrou wrote:

How do you create an array that mixes PyMethodDefs and PyMethodDefExs together?

You're right, you wouldn't be able to. For my PyMethodDefEx proposal, the entire array would have to be one way or the other.

It sounds like METH_SIGNATURE is the wrong mechanism. Instead, you may want a tp_methods_ex as well as as a Py_TPFLAGS_HAVE_METHODS_EX.

You may well be right. We'd already need a flag on the type object anyway, to indicate "tp_doc start with a signature". So if we had such a flag, it could do double-duty to also indicate "tp_methods points to PyMethodDefEx structures". My only concern here: __text_signature__ is supported on five internal objects: PyCFunctionObject, PyTypeObject, PyMethodDescr_Type, _PyMethodWrapper_Type, and PyWrapperDescr_Type. I'm not certain that all of those carry around their own pointer back to their original type object. If you went off the "self" parameter, you wouldn't have one if you were an unbound method. And you might get the wrong answer if the user bound you to a different class, or if you were accessed through a subclass. (I say "might" not to mean "it could happen sometimes", but rather "I don't know what the correct answer is".)

Note that this constrains future growth to only add pointer fields, unless you also add a couple of long fields. But at least it sounds workable.

Ah, in the back of my mind I meant to say "add some unused union {void *p; long i;} fields". Though in practice I don't think we support any platforms where sizeof(long) > sizeof(void *).

Uh... If you write a "conversion function", you may as well make it so it converts the "sig=" line to a plain signature line in 3.3, which avoids the issue entirely.

Yeah, that's an improvement, though it makes the conversion function a lot more complicated, and presumably uses more memory.

(and how would that conversion function be shipped to the user anyway? Python 3.3 and the stable ABI don't have it)

As a C function in a text file, that they'd have to copy into their program. I know it's ugly. //arry/

Am 05.02.2014 14:52, schrieb "Martin v. Löwis":

Am 03.02.14 15:43, schrieb Larry Hastings:

...

A: We create a PyMethodDefEx structure with an extra field: "const char *signature". We add a new METH_SIGNATURE (maybe just METH_SIG?) flag to the flags, indicating that this is an extended structure. When iterating over the PyMethodDefs, we know how far to advance the pointer based on this flag.

B: Same as A, but we add three unused pointers (void *reserved1 etc) to PyMethodDefEx to give us some room to grow.

C: Same as A, but we add two fields to PyMethodDefEx. The second new field identifies the "version" of the structure, telling us its size somehow. Like the lStructSize field of the OPENFILENAME structure in Win32. I suspect YAGNI.

D: Add a new type slot for method signatures. This would be a tp_signature field, along with a new slot id Py_tp_signature. The signature field itself could be

struct PyMethodSignature { char *method_name; char *method_signature; };

Mostly unrelated question while seeing the "char *" here: do we (or do we want to) support non-ASCII names for functions implemented in C? Georg

Am 05.02.14 17:04, schrieb Georg Brandl:

Mostly unrelated question while seeing the "char *" here: do we (or do we want to) support non-ASCII names for functions implemented in C?

I didn't try, but I think it should work. methodobject.c:meth_get__name__ uses PyUnicode_FromString, which in turn decodes from UTF-8. Regards, Martin