Mailman 3 Status of the Argument Clinic DSL - Python-Dev

Status of the Argument Clinic DSL

older
Method signatures in the datetime...

Alexander Belopolsky

4 Aug 2016 4 Aug '16

9:41 p.m.

What is the current status of the Argument Clinic DSL? The clinic preprocessor was released [1] with Python 3.4 with a promise [2] "that signature metadata for programmatic introspection will be added to additional callables implemented in C as part of Python 3.4 maintenance releases." We are now close to 3.6 beta and clinic coverage in stdlib is sparse at best. For example, the math module makes no use of the clinic (but interestingly a few methods in cmath do). The documentation is very sparse as well. While one would expect PEP 436 to serve as *the* documentation while clinic is being developed, but the PEP has the "Draft" status and has not been updated since 2013. Furthermore, the 3.4 release notes contain a warning saying "The Argument Clinic PEP is not fully up to date with the state of the implementation." It does not look like this situation has improved since. The practical question is: does it make sense to invest time in learning the clinic DSL and converting more modules (e.g. datetime) to it or it is likely that the whole approach will be revised in the future? [1] http://bugs.python.org/issue16612 [2] https://docs.python.org/3.6/whatsnew/3.4.html?highlight=clinic#pep-436-argum...

Show replies by date

Brett Cannon

4 Aug 4 Aug

11:39 p.m.

On Thu, 4 Aug 2016 at 09:14 Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

...

What is the current status of the Argument Clinic DSL? The clinic preprocessor was released [1] with Python 3.4 with a promise [2] "that signature metadata for programmatic introspection will be added to additional callables implemented in C as part of Python 3.4 maintenance releases." We are now close to 3.6 beta and clinic coverage in stdlib is sparse at best. For example, the math module makes no use of the clinic (but interestingly a few methods in cmath do). The documentation is very sparse as well. While one would expect PEP 436 to serve as *the* documentation while clinic is being developed, but the PEP has the "Draft" status and has not been updated since 2013. Furthermore, the 3.4 release notes contain a warning saying "The Argument Clinic PEP is not fully up to date with the state of the implementation." It does not look like this situation has improved since.

I always read https://docs.python.org/3/howto/clinic.html as the documentation and not the PEP.

...

The practical question is: does it make sense to invest time in learning the clinic DSL and converting more modules (e.g. datetime) to it

Yes. Part of the issue is there was a big push to convert modules for the 3.4 release and some of us helped convert however many we can, but then people stopped putting the time in to convert code a module at a time once 3.4 was released. Serhiy has actually been working on a patch to speed up keyword parsing which will automatically be applied to code using Argument Clinic: http://bugs.python.org/issue27574 .

...

or it is likely that the whole approach will be revised in the future?

I don't see it being revised, but maybe Larry has something to say on the matter.

...

[1] http://bugs.python.org/issue16612 [2] https://docs.python.org/3.6/whatsnew/3.4.html?highlight=clinic#pep-436-argum... _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

Larry Hastings

11:49 p.m.

On 08/04/2016 09:11 AM, Alexander Belopolsky wrote:

...

Furthermore, the 3.4 release notes contain a warning saying "The Argument Clinic PEP is not fully up to date with the state of the implementation." It does not look like this situation has improved since.

It hasn't. The PEP lags behind the implementation. The HOWTO is up-to-date though.

...

The practical question is: does it make sense to invest time in learning the clinic DSL and converting more modules (e.g. datetime) to it or it is likely that the whole approach will be revised in the future?

AFAIK the Clinic DSL can handle all of Python's C extensions. I have no plans to "revise the whole approach"; if someone else does I haven't heard about it. //arry/

Alexander Belopolsky

5 Aug 5 Aug

4:15 a.m.

On Thu, Aug 4, 2016 at 2:19 PM, Larry Hastings wrote:

...

AFAIK the Clinic DSL can handle all of Python's C extensions. I have no plans to "revise the whole approach"; if someone else does I haven't heard about it.

I was just wondering that with so much effort to bring typing to the mainstream, a more "pythonic" DSL may emerge for describing the signatures of functions in C modules. BTW, is there any document describing the syntax of "text signatures" such as:

...

...
...
os.rename.__text_signature__ '($module, /, src, dst, *, src_dir_fd=None, dst_dir_fd=None)'

? What does the "$module, /," part mean?

Brett Cannon

4:30 a.m.

On Thu, 4 Aug 2016 at 15:47 Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

...

On Thu, Aug 4, 2016 at 2:19 PM, Larry Hastings wrote:

...
AFAIK the Clinic DSL can handle all of Python's C extensions. I have no plans to "revise the whole approach"; if someone else does I haven't heard about it.

I was just wondering that with so much effort to bring typing to the mainstream, a more "pythonic" DSL may emerge for describing the signatures of functions in C modules.

BTW, is there any document describing the syntax of "text signatures" such as:

...
...
...
os.rename.__text_signature__ '($module, /, src, dst, *, src_dir_fd=None, dst_dir_fd=None)'

?

What does the "$module, /," part mean?

Don't remember what the $module means but the / means positional-only arguments.

Alexander Belopolsky

4:34 a.m.

On Thu, Aug 4, 2016 at 7:00 PM, Brett Cannon wrote:

...

...
...
...
...
os.rename.__text_signature__ '($module, /, src, dst, *, src_dir_fd=None, dst_dir_fd=None)'

?

What does the "$module, /," part mean?

Don't remember what the $module means but the / means positional-only arguments.

Not in this case. Apparently the / applies to the $module thingy because src and dst are not positional only (and the / would be after them if they were, I think.)

Larry Hastings

4:42 a.m.

On 08/04/2016 03:45 PM, Alexander Belopolsky wrote:

...

On Thu, Aug 4, 2016 at 2:19 PM, Larry Hastings mailto:larry@hastings.org> wrote:

AFAIK the Clinic DSL can handle all of Python's C extensions. I have no plans to "revise the whole approach"; if someone else does I haven't heard about it.

I was just wondering that with so much effort to bring typing to the mainstream, a more "pythonic" DSL may emerge for describing the signatures of functions in C modules.

BTW, is there any document describing the syntax of "text signatures" such as:

...
...
...
os.rename.__text_signature__ '($module, /, src, dst, *, src_dir_fd=None, dst_dir_fd=None)'

?

What does the "$module, /," part mean?

/ is the delimiter between positional-only parameters and positional-or-keyword arguments. It's not actual Python syntax, but Guido said (somewhere) that *if* Python ever sprouted a syntax for positional-only parameters, that was as good a delimiter as any. I think he picked it because / is the inverse of *. The "$" in "$module" means it's what we called a "bound parameter", a parameter which gets bound to a value before Python ever sees it. C extension functions get the module passed in automatically, but this is done internally and from the Python level you can't see it. So it's accurate to present it there, but we suppress it before we compute the inspect.signature. For example, os_chdir_impl in Modules/posixmodule.c takes two arguments, the first being the module, the second being the path; inspect.signature(os.chdir) only shows one parameter, the path. __text_signature__, although user-visible, is not considered information for users. It's an internal implementation detail like co_code. However, with the exception of things like "$module" and "/" it's just a Python function declaration. Literally we remove the funny bits like "$module" and "/", prepend that string with "def foo", append that string with ": pass", and parse the result with ast.parse. If you have more questions about __text_signature__, I recommend reading the implementation of inspect.signature, since that's the one and only consumer of it. //arry/

Alexander Belopolsky

5:03 a.m.

On Thu, Aug 4, 2016 at 7:12 PM, Larry Hastings wrote:

...

C extension functions get the module passed in automatically, but this is done internally and from the Python level you can't see it.

Always something new to learn! This was not so in Python 2.x - self was passed as NULL to the C module functions. When did this change?

eryk sun

5:57 a.m.

On Thu, Aug 4, 2016 at 11:33 PM, Alexander Belopolsky wrote:

...

On Thu, Aug 4, 2016 at 7:12 PM, Larry Hastings wrote:

...
C extension functions get the module passed in automatically, but this is done internally and from the Python level you can't see it.

Always something new to learn! This was not so in Python 2.x - self was passed as NULL to the C module functions. When did this change?

In 2.x this is the `self` parameter (actually named "passthrough" in the source) of Py_InitModule4 [1, 2]. You probably use the Py_InitModule or Py_InitModule3 macros, which pass NULL for this parameter: #define Py_InitModule(name, methods) \ Py_InitModule4(name, methods, (char *)NULL, (PyObject *)NULL, \ PYTHON_API_VERSION) #define Py_InitModule3(name, methods, doc) \ Py_InitModule4(name, methods, doc, (PyObject *)NULL, \ PYTHON_API_VERSION) Python 3's PyModule_Create2 [3-5] API makes this a reference to the module. It's currently implemented in PyModule_AddFunctions [6, 7]. [1]: https://docs.python.org/2/c-api/allocation.html#c.Py_InitModule4 [2]: https://hg.python.org/cpython/file/v2.7.12/Python/modsupport.c#l31 [3]: https://docs.python.org/3/c-api/module.html#c.PyModule_Create2 [4]: https://hg.python.org/cpython/file/v3.5.2/Objects/moduleobject.c#l133 [5]: https://hg.python.org/cpython/file/v3.0b1/Objects/moduleobject.c#l63 [6]: https://docs.python.org/3/c-api/module.html#c.PyModule_AddFunctions [7]: https://hg.python.org/cpython/file/v3.5.2/Objects/moduleobject.c#l387

Nick Coghlan

12:28 p.m.

On 5 August 2016 at 09:12, Larry Hastings wrote:

...

/ is the delimiter between positional-only parameters and positional-or-keyword arguments. It's not actual Python syntax, but Guido said (somewhere) that *if* Python ever sprouted a syntax for positional-only parameters, that was as good a delimiter as any. I think he picked it because / is the inverse of *.

I was thinking Guido picked the "/" symbol when we were discussing the Argument Clinic DSL at PyCon 2014 and you had also been working on https://www.python.org/dev/peps/pep-0457/ However, re-reading the latter shows it dates back earlier than that: https://www.python.org/dev/peps/pep-0457/#guido

...

If you have more questions about __text_signature__, I recommend reading the implementation of inspect.signature, since that's the one and only consumer of it.

I occasionally wonder if we should document the "/" notation in https://docs.python.org/3/library/inspect.html#introspecting-callables-with-... as it can sometimes show up in the text representation of signature objects: >>> print(inspect.signature(format)) (value, format_spec='', /) Likewise for describing the option of using __text_signature__ to override the default inspect.signature() output (and perhaps adding a public "inspect.Signature.from_text" alternate constructor at the same time). However, either notion is a bit tricky while PEP 457 is still in draft rather than accepted - even if it's "just" a syntax for overriding inspect.signature, we'd still be effectively locking this in as *the* syntax for positional-only arguments (if they're ever added) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Larry Hastings

6:06 p.m.

On 08/04/2016 11:58 PM, Nick Coghlan wrote:

...

I occasionally wonder if we should document the "/" notation in https://docs.python.org/3/library/inspect.html#introspecting-callables-with-... as it can sometimes show up in the text representation of signature objects:

>>> print(inspect.signature(format)) (value, format_spec='', /)

I think we probably should. After all, this same string is used for pydoc: >>> import os >>> help(os.execv) Help on built-in function execv in module posix: execv(path, argv, /) Execute an executable path with arguments, replacing current process. path Path of executable file. argv Tuple or list of strings. so it's easily user-visible. I've always found it a little strange that the signatures for functions using Py_ArgParseTuple() had this / in them that wasn't Python syntax. On the other hand, it accurately reflects the fact that these functions have signatures that you can't write in Python. (And, FWIW, I wasn't the person who added the code that made "/" start showing up in the text representations of signatures. I was waffling on it, then someone else JFDI, to quote Barry.) //arry/

2815

Age (days ago)

2816

Last active (days ago)

List overview

Download

10 comments

5 participants

participants (5)

Alexander Belopolsky
Brett Cannon
eryk sun
Larry Hastings
Nick Coghlan

Status of the Argument Clinic DSL

tags

participants (5)