<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Dec 3, 2012 at 2:29 PM, Larry Hastings <span dir="ltr"><<a href="mailto:larry@hastings.org" target="_blank">larry@hastings.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Say there, the Python core development community! Have I got<br>
a question for you!<br>
<br>
*ahem*<br>
<br>
Which of the following four options do you dislike least? ;-)<br>
<br>
1) CPython continues to provide no "function signature"<br>
objects (PEP 362) or inspect.getfullargspec() information<br>
for any function implemented in C.<br>
<br></blockquote><div><br></div><div>yuck on #1, though this is what happens by default if we don't do anything nice.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
2) We add new hand-coded data structures representing the<br>
metadata necessary for function signatures for builtins.<br>
Which means that, when defining arguments to functions in C,<br>
we'd need to repeat ourselves *even more* than we already do.<br>
<br></blockquote><div><br></div><div>yuck on #2.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
3) Builtin function arguments are defined using some seriously<br>
uncomfortable and impenetrable C preprocessor macros, which<br>
produce all the various types of output we need (argument<br>
processing code, function signature metadata, possibly<br>
the docstrings too).<br></blockquote><div><br></div><div>Likely painful to maintain. C++ templates would likely be easier.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
4) Builtin function arguments are defined in a small DSL; these<br>
are expanded to code and data using a custom compile-time<br>
preprocessor step.</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
<br>
All the core devs I've asked said "given all that, I'd prefer the<br>
hairy preprocessor macros". But by the end of the conversation<br>
they'd changed their minds to prefer the custom DSL. Maybe I'll<br>
make a believer out of you too--read on!<br></blockquote><div><br></div><div>It always strikes me that C++ could be such a DSL that could likely be used for this purpose rather than defining and maintaining our own "yet another C preprocessor" step. But I don't have suggestions and we're not allowing C++ so... nevermind. :)</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I've named this DSL preprocessor "Argument Clinic", or Clinic<br>
for short**. Clinic works similarly to Ned Batchelder's brilliant<br>
"Cog" tool:<br>
<a href="http://nedbatchelder.com/code/cog/" target="_blank">http://nedbatchelder.com/code/<u></u>cog/</a><br>
<br>
You embed the input to Clinic in a comment in your C file,<br>
and the output is written out immediately after that comment.<br>
The output's overwritten every time the preprocessor is run.<br>
In short it looks something like this:<br>
<br>
/*[clinic]<br>
input to the DSL<br>
[clinic]*/<br>
<br>
... output from the DSL, overwritten every time ...<br>
<br>
/*[clinic end:<checksum>]*/<br>
<br>
The input to the DSL includes all the metadata about the<br>
function that we need for the function signature:<br>
<br>
* the name of the function,<br>
* the return annotation (if any),<br>
* each parameter to the function, including<br>
* its name,<br>
* its type (in C),<br>
* its default value,<br>
* and a per-parameter docstring;<br>
* and the docstring for the function as a whole.<br>
<br>
The resulting output contains:<br>
<br>
* the docstring for the function,<br>
* declarations for all your parameters,<br>
* C code handling all argument processing for you,<br>
* and a #define'd methoddef structure for adding the<br>
function to the module.<br>
<br>
<br>
I discussed this with Mark "HotPy" Shannon, and he suggested we break<br>
our existing C functions into two. We put the argument processing<br>
into its own function, generated entirely by Clinic, and have the<br>
implementation in a second function called from the first. I like<br>
this approach simply because it makes the code cleaner. (Note that<br>
this approach should not cause any overhead with a modern compiler,<br>
as both functions will be "static".)<br>
<br>
But it also provides an optimization opportunity for HotPy: it could<br>
read the metadata, and when generating the JIT'd code it could skip<br>
building the PyObjects and argument tuple (and possibly keyword<br>
argument dict), and the subsequent unpacking/decoding, and just call<br>
the implementation function directly, giving it a likely-measurable<br>
speed boost.<br>
<br>
And we can go further! If we add a new extension type API allowing<br>
you to register both functions, and external modules start using it,<br>
sophisticated Python implementations like PyPy might be able to skip<br>
building the tuple for extension type function calls--speeding those<br>
up too!<br>
<br>
Another plausible benefit: alternate implementations of Python could<br>
read the metadata--or parse the input to Clinic themselves--to ensure<br>
their reimplementations of the Python standard library conform to the<br>
same API!<br>
<br>
<br>
Clinic can also run general-purpose Python code ("/*[python]").<br>
All output from "print" is redirected into the output section<br>
after the Python code.<br>
<br>
<br>
As you've no doubt already guessed, I've made a prototype of<br>
Argument Clinic. You can see it--and some sample conversions of<br>
builtins using it for argument processing--at this BitBucket repo:<br>
<br>
<a href="https://bitbucket.org/larry/python-clinic" target="_blank">https://bitbucket.org/larry/<u></u>python-clinic</a><br>
<br>
I don't claim that it's fabulous, production-ready code. But it's<br>
a definite start!<br>
<br>
<br>
To save you a little time, here's a preview of using Clinic for<br>
dbm.open(). The stuff at the same indent as a declaration are<br>
options; see the "clinic.txt" in the repo above for full documentation.<br>
<br>
/*[clinic]<br>
dbm.open -> mapping<br>
basename=dbmopen<br>
<br>
const char *filename;<br>
The filename to open.<br>
<br>
const char *flags="r";<br>
How to open the file. "r" for reading, "w" for writing, etc.<br>
<br>
int mode=0666;<br>
default=0o666<br>
If creating a new file, the mode bits for the new file<br>
(e.g. os.O_RDWR).<br>
<br>
Returns a database object.<br>
<br>
[clinic]*/<br>
<br>
PyDoc_STRVAR(dbmopen__doc__,<br>
"dbm.open(filename[, flags=\'r\'[, mode=0o666]]) -> mapping\n"<br>
"\n"<br>
" filename\n"<br>
" The filename to open.\n"<br>
"\n"<br>
" flags\n"<br>
" How to open the file. \"r\" for reading, \"w\" for writing, etc.\n"<br>
"\n"<br>
" mode\n"<br>
" If creating a new file, the mode bits for the new file\n"<br>
" (e.g. os.O_RDWR).\n"<br>
"\n"<br>
"Returns a database object.\n"<br>
"\n");<br>
<br>
#define DBMOPEN_METHODDEF \<br>
{"open", (PyCFunction)dbmopen, METH_VARARGS | METH_KEYWORDS, dbmopen__doc__}<br>
<br>
static PyObject *<br>
dbmopen_impl(PyObject *self, const char *filename, const char *flags, int mode);<br>
<br>
static PyObject *<br>
dbmopen(PyObject *self, PyObject *args, PyObject *kwargs)<br>
{<br>
const char *filename;<br>
const char *flags = "r";<br>
int mode = 0666;<br>
static char *_keywords[] = {"filename", "flags", "mode", NULL};<br>
<br>
if (!PyArg_ParseTupleAndKeywords(<u></u>args, kwargs,<br>
"s|si", _keywords,<br>
&filename, &flags, &mode))<br>
return NULL;<br>
<br>
return dbmopen_impl(self, filename, flags, mode);<br>
}<br>
<br>
static PyObject *<br>
dbmopen_impl(PyObject *self, const char *filename, const char *flags, int mode)<br>
/*[clinic end:<u></u>eddc886e542945d959b44b483258bf<u></u>038acf8872]*/<br>
<br>
<br>
As of this writing, I also have sample conversions in the following files<br>
available for your perusal:<br>
Modules/_cursesmodule.c<br>
Modules/_dbmmodule.c<br>
Modules/posixmodule.c<br>
Modules/zlibmodule.c<br>
Just search in C files for '[clinic]' and you'll find everything soon<br>
enough.<br>
<br>
As you can see, Clinic has already survived some contact with the<br>
enemy. I've already converted some tricky functions--for example,<br>
os.stat() and curses.window.addch(). The latter required adding a<br>
new positional-only processing mode for functions using a legacy<br>
argument processing approach. (See "clinic.txt" for more.) If you<br>
can suggest additional tricky functions to support, please do!<br>
<br>
<br>
Big unresolved questions:<br>
<br>
* How would we convert all the builtins to use Clinic? I fear any<br>
solution will involve some work by hand. Even if we can automate<br>
big chunks of it, fully automating it would require parsing arbitrary<br>
C. This seems like overkill for a one-shot conversion.<br>
(Mark Shannon says he has some ideas.)<br></blockquote><div><br></div><div>A lot of hand work. Sprints at pycon. etc. Automating nice chunks of it could be partially done for some easy cases such as things that only use ParseTuple today.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
* How do we create the Signature objects? My current favorite idea:<br>
Clinic also generates a new, TBD C structure defining all the<br>
information necessary for the signature, which is also passed in to<br>
the new registration API (you remember, the one that takes both the<br>
argument-processing function and the implementation function). This<br>
is secreted away in some new part of the C function object. At<br>
runtime this is converted on-demand into a Signature object. Default<br>
values for arguments are represented in C as strings; the conversion<br>
process attempts eval() on the string, and if that works it uses the<br>
result, otherwise it simply passes through the string.<br></blockquote><div><br></div><div>I think passing on the string if that doesn't work is wrong. It could lead to a behavior change not realized until runtime due to some other possibly unrelated thing causing the eval to fail. A failure to eval() one of these strings should result in an ImportError from the extension module's init or a fatal failure if it is a builtin. (I'm assuming these would be done at extension module import time at or after the end of the module init function)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
* Right now Clinic paves over the PyArg_ParseTuple API for you.<br>
If we convert CPython to use Clinic everywhere, theoretically we<br>
could replace the parsing API with something cleaner and/or faster.<br>
Does anyone have good ideas (and time, and energy) here?<br></blockquote><div><br></div><div>By "paves over" do you mean that Clinic is currently using the ParseTuple API in its generated code? Yes, we should do better. But don't hold Clinic up on that. In fact allowing a version of Clinic to work stand alone as a PyPI project and generate Python 2.7 and 3.2/3.3 extension module boilerplate could would increase its adoption and improve the quality of some existing extension modules that choose to use it.</div>
<div><br></div><div>My first take on this would be to do the obvious and expand the code within the case/switch statement in the loop that ParseTuple ends up in directly so that we're just generating raw parameter validation and acceptance code based on the clinic definition. I've never liked things in C that parse a string at runtime to determine behavior. (please don't misinterpret that to suggest I don't like Python ;)</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
* There's actually a fifth option, proposed by Brett Cannon. We<br>
constrain the format of docstrings for builtin functions to make<br>
them machine-readable, then generate the function signature objects<br>
from that. But consider: generating *everything* in the signature<br>
object may get a bit tricky (e.g. Parameter.POSITIONAL_ONLY), and<br>
this might gunk up the docstring.<br>
<br>
<br>
But the biggest unresolved question... is this all actually a terrible<br>
idea?<br></blockquote><div><br></div><div>No it is not. I like it.</div><div><br></div><div>I don't _like_ adding another C preprocessor but I think if we keep it very limited it is a perfectly reasonable thing to do as part of our build process.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<br>
//arry/<br>
<br>
<br>
** "Is this the right room for an argument?"<br>
"I've told you once...!"<br>
______________________________<u></u>_________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="http://mail.python.org/mailman/listinfo/python-dev" target="_blank">http://mail.python.org/<u></u>mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="http://mail.python.org/mailman/options/python-dev/greg%40krypto.org" target="_blank">http://mail.python.org/<u></u>mailman/options/python-dev/<u></u>greg%40krypto.org</a><br>
</blockquote></div><br></div>