[Python-Dev] The Return Of Argument Clinic

[Larry Hastings]

On 08/05/2013 02:55 AM, Nick Coghlan wrote:
> On 5 August 2013 18:48, Larry Hastings<larry at hastings.org>  wrote:
>> Question 0: How should we integrate Clinic into the build process?
> Isn't solving the bootstrapping problem the reason for checking in the
> clinic-generated output? If there's no Python available, we build what
> we have (without the clinic step), then we build it again *with* the
> clinic step.

It solves the bootstrapping problem, but that's not the only problem 
Clinic presents to the development workflow.

If you modify some Clinic DSL in a C file in the CPython tree, then run 
"make", should the Makefile re-run Clinic over that file?  If you say 
"no", then there's no problem.  If you say "yes", then we have the 
problem I described.


>> ___________________________________________________________________
>> Question 1: Which C function nomenclature?
> Consider this from the client side, and I believe it answers itself: 
> other code in the module will be expected the existing signature, so 
> that signature needs to stay with the existing name, while the new C 
> implementation function gets the new name.

One vote for "os_stat_impl".  Bringing the sum total of votes up to 1!  ;-)


>> ___________________________________________________________________
>> Question 2: Emit code for modules and classes?
>>
>> There are some complications to this, one of which I'll
>> discuss next.  But I put it to you, gentle reader: how
>> much boilerplate should Argument Clinic undertake to
>> generate, and how much more class and module metadata
>> should be wired in to it?
> I strongly recommend deferring this. Incremental development is good,
> and getting this bootstrapped at all is going to be challenging enough
> without trying to do everything at once.

I basically agree.  But you glossed over an important part of that 
question, "how much more class and module metadata should be wired in 
right now?".

Originally Clinic didn't ask for full class and module information, you 
just specified the full dotted path and that was that.  But that's 
ambiguous; Clinic wouldn't be able to infer what was a module vs what 
was a class.  And in the future, if/when it generates module and class 
boilerplate, obviously it'll need to know the distinction.  I figure, 
specifying the classes and modules doesn't add a lot of additional cost, 
but it'll very likely save us a lot of time in the long run, so I made 
it a requirement.  (WAGNI!)

Anyway, I guess what I was really kind of trying to get at here was:
a) are there any other obvious bits of metadata Clinic should require 
right now for functions,
b) what other metadata might Clinic take in the future--not because I 
want to add it, but just so we can figure out the next question,
c) to what degree can we future-proof Clinic 1.0 so extension authors 
can more easily straddle versions.

Thinking about it more with a fresh perspective, maybe all we need is a 
Clinic version number directive.  This would declare the minimum Clinic 
version--which would really just track the Python version it shipped 
with--that you may use to process this file. Like so:

    /*[clinic]
    clinic 3.5
    [clinic]*/

As long as the code Clinic generates is backwards compatible for Python 
3.4, I think this will has it covered.  We may at times force developers 
to use fresher versions of Python to process Clinic stuff, but I don't 
think that's a big deal.


>> ___________________________________________________________________
>> Question 4: Return converters returning success/failure?
>>
>> Can we live with PyErr_Occurred() here?
> Armin's suggestion of a valid return value (say, -1) that indicates
> "error may have occurred" sounds good to me.

Yes indeed, and thanks Armin for pointing it out.  This works perfectly 
in Clinic, as each individual return converter controls the code 
generated for cleanup afterwards.  So it can be a completely local 
policy per-return-converter what the magic value is.  Heck, you could 
have multiple int converters with different magic return values (not 
that that seems like a good idea).


>> ___________________________________________________________________
>> Question 5: Keep too-magical class decorator Converter.wrap?
>>
>> I'd like to keep it in, and anoint it as the preferred way
>> of declaring Converter subclasses.  Anybody else have a strong
>> opinion on this either way?
> Can't you get the same effect without the magic by having a separate
> "custom_init" method that the main __init__ method promises to call
> with the extra keyword args after finishing the other parts of the
> initialization? Them a custom converter would just look like:
>
>      class path_t_converter(Converter):
>          def custom_init(self, *, allow_fd=False, nullable=False):
>              ...

I can get the same effect without reusing the name __init__, but I 
wouldn't say I can do it "without the magic".  The whole point of the 
decorator is magic.

Let's say I go with your proposal.  What happens if someone makes a 
Converter, and wraps it with Converter.wrap, and defines their own 
__init__?  It would never get called.  Silently, by default, which is 
worse--though I could explicitly detect such an __init__ and throw an 
exception I guess.  Still, now we have a class where you can't use the 
name __init__, you have to use this funny other name, for arbitrary 
"correctness" reasons.

My metaphor for why I prefer my approach is the set of "os" module 
functions that allow "specifying a file descriptor":

    http://docs.python.org/3/library/os.html#path-fd

Taking the example of os.chdir(), yes, it would have been more correct 
to require specifying the file descriptor as a separate argument, like

    os.chdir(None, fd=my_dir_fd)

But this would have meant that when using "fd" the first parameter would 
always be None.  And the first parameter and the "fd" do the same thing, 
just with different types.  So while normally we eschew polymorphic 
parameters (and with good reason) in this case I think practicality beat 
purity.

And I think the same holds here.  Since class instance initialization 
functions in Python are called __init__, and this is a class instance 
initialization function, I think it should be called __init__.  By 
decorating with Converter.wrap, you're signing a contract that says "yes 
it's a fake __init__, that's what I want".  A real __init__ in this 
class would never be called, which is the whole point, so we might as 
well reuse the name for our slightly-fake __init__.


Let me put it this way: Which is more surprising to the person 
unfamiliar with the code?  That this __init__ doesn't get all the 
parameters, and the base class __init__ is getting called 
automatically?  Or that this funny function "custom_init" is what gets 
called, and this class is not allowed to have a function called __init__?


In case I didn't make it clear: the actual call site for constructing 
these objects is buried deep in clinic.py.  Users don't create them 
directly, or at least I don't know why they'd ever need to do so.  
Instead, they're created inside Clinic preprocessor blocks in your 
source files, where they look like tis:

    parameter_name: Converter(argument=value, argument2=value) = default

Using the funny magic of Converter.wrap makes this and the 
implementation look a great deal more alike.  So I remain a fan of 
Converter.wrap and calling the initialization function __init__.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130805/bc2fd1ba/attachment.html>