PEP 671 proof-of-concept: A less invasive implementation
Hi One of the motives for PEP 671 is that the signature of a function fn, and hence the associated help(fn) is sometimes opaque regarding default values. I won't repeat the excellent examples already given. In the current implementation default values are handled outside the compiled code of the function, which is available at fn.__code__. Instead they are stored in 'metadata' associated with the function. Here's one way to see this. from inspect import signature as sig def fn(a, b=1, c=2): return a, b, c sig(fn) # Gives <Signature (a, b=1, c=2)> fn.__defaults__ = ('hi', 'there') sig(fn) # Gives <Signature (a, b='hi', c='there')> We can also change the __code__ object, but care is needed here. def gn(): return (1, 2, 3) fn.__code__ = gn.__code__ fn() # Gives (1, 2, 3). sig(fn) # Gives <Signature ()> fn.__defaults__ # Gives ('hi', 'there') The signature of fn, together with the arguments actually supplied, is used to initialise the code frame which is put on the top of the stack, and in which fn.__code__ executes. I suggest that an implementation which provides additional flexibility in the manner in which the code frame is initialised would be less invasive. Necessarily, PEP 671 allows programmer supplied code to be used in the 'initialisation phase'. The previous attempts place that code in fn.__code__. I suggest that the implementation adds a new attribute fn.__wibble__ to fn, which can either be None or a code object. And if fn.__wibble__ is not None, then it is used to initialise the code frame in which fn.__code__ executes. It would as before take as input the arguments actually supplied by the user. I stop here, saying nothing for now about two important questions. First, what is the programmer syntax for creating such a 'two-part' function fn. Second, what does the user see as a result of help(fn). Or in other words, how to extend the semantics of inspect.signature. with kind regards Jonathan
On Sat, Oct 30, 2021 at 10:32 PM Jonathan Fine <jfine2358@gmail.com> wrote:
I suggest that an implementation which provides additional flexibility in the manner in which the code frame is initialised would be less invasive. Necessarily, PEP 671 allows programmer supplied code to be used in the 'initialisation phase'. The previous attempts place that code in fn.__code__.
I suggest that the implementation adds a new attribute fn.__wibble__ to fn, which can either be None or a code object. And if fn.__wibble__ is not None, then it is used to initialise the code frame in which fn.__code__ executes. It would as before take as input the arguments actually supplied by the user.
I stop here, saying nothing for now about two important questions. First, what is the programmer syntax for creating such a 'two-part' function fn. Second, what does the user see as a result of help(fn). Or in other words, how to extend the semantics of inspect.signature.
The code has to be executed in the context of the called function. How would fn.__wibble__ be different from checks at the top of fn.__code__? And, seeing something in help(fn) largely necessitates that the source code be retained. I don't know of any other way to do it. If you say that the default argument is "len(a)", then that's what help() should say. ChrisA
Hi Chris I like your questions. You ask: How would fn.__wibble__ be different from checks at the top of fn.__code__? They'd be in two different code blocks. I see a function call going as follows. 1. Process the supplied arguments in the usual way. 2. Create a new frame object and place it on the stack. 3. In that new frame execute fn.__wibble. 4. When fn.__wibble__ is done, execute fn.__code__ IN THE SAME FRAME. I think step 4 is a tail call, as in https://en.wikipedia.org/wiki/Tail_call, which includes the concept of tail recursion. Your other question is: And, seeing something in help(fn) largely necessitates that the source code be retained. Yes, this is true whatever syntax is used. In help(fn) inspect.signature repr() is used to produce help text. There's no extra storage overhead for that. Both your implementation and mine will require source text to be stored (unless the module is compiled as optimised). Oh, but I've made a mistake. If the module is compiled non-optimised then the compile code contains points to the source file. These are used in traceback when an exception occurs. I'm not to say at this point which approach is best for the person who reads help(fn), except the lawyer's answer "it just depends". At this point my focus is on designing a less invasive implementation. Your good questions have led me to rethink. The tail call in my proposed implementation can be removed and then fn.__wibble__ would not be needed. It would be the same as checks at the top of fn.__code__. But instead of fn.__wibble__ we have a pointer (as in fn.__code__) to some location in the body of fn. (Or as fn.__code__ is already well equipped with pointers, we equip fn with a pointer to one of these pointers.) So all that's required now is 1. A syntax in source files that allows the author of fn to specify the end of the 'preamble extra help' in the body of fn. 2. An addition to help(fn) that provides the 'preamble' of fn as an extra help message. with kind regards Jonathan
On 10/30/2021 10:40 AM, Chris Angelico wrote:
And, seeing something in help(fn) largely necessitates that the source code be retained. I don't know of any other way to do it. If you say that the default argument is "len(a)", then that's what help() should say.
"from __future__ import annotations" works by decompiling the AST into a string. From PEP 563: "The string form is obtained from the AST during the compilation step, which means that the string form might not preserve the exact formatting of the source." I'm not saying this is a better way to do it, but it is another way. And it might save some memory. I don't think the slight differences mentioned at the end of that sentence would make a difference in practice. Eric
Hi I've just had a brainwave that may give an even less invasive implementation of PEP 671. It relies on every function having a dict, as provided by https://www.python.org/dev/peps/pep-0232/. Consider: def fn(a, b, c): pass fn.__wibble__ = 123 fn.__wibble__ # Give 123, of course. Now consider: @wibble def fn(a, b, c=None): '''Usual docstring goes here.''' if c is None: c = [] '#wibble' return c.extend([a, b]) The @wibble decorator sets fn.__wibble__ to point to the end of the line just before '#wibble'. We can now define Help(fn) to produce something like: fn(a, b, c=None) Usual docstring goes here. -- if c is None: c = [] There we go. The functionality asked for, in a way that works with existing Python. If nothing else, we can use this for experiments to explore the idea. Often in Python there are functions whose source is self-explanatory. Example: @wibble def fahr_to_cent(fahr): return (fahr - 32) * 5 /9 '#wibble' And now for Help(fahr_to_cent) we get: fahr_to_cent(fahr) -- return (fahr - 32) * 5 /9 Final word. Does anyone use PEP 232, function attributes? If not, perhaps remove it as a waste of space. with kind regards Jonathan
On Sun, Oct 31, 2021 at 7:01 PM Jonathan Fine <jfine2358@gmail.com> wrote:
Hi
I've just had a brainwave that may give an even less invasive implementation of PEP 671. It relies on every function having a dict, as provided by https://www.python.org/dev/peps/pep-0232/.
Consider: def fn(a, b, c): pass fn.__wibble__ = 123 fn.__wibble__ # Give 123, of course.
Now consider: @wibble def fn(a, b, c=None): '''Usual docstring goes here.''' if c is None: c = [] '#wibble'
return c.extend([a, b])
The @wibble decorator sets fn.__wibble__ to point to the end of the line just before '#wibble'.
Interesting. How's it going to do that? What's it going to look at in the function to find a bare string? They get completely optimized out during compilation.
We can now define Help(fn) to produce something like: fn(a, b, c=None) Usual docstring goes here. -- if c is None: c = []
There we go. The functionality asked for, in a way that works with existing Python. If nothing else, we can use this for experiments to explore the idea. Often in Python there are functions whose source is self-explanatory.
That's one small part of it. You haven't improved the signature, you haven't removed the need for sentinels, but you have at least added some source code to help(fn). Assuming that the source code is indeed available, that is; wouldn't work if the function came from a .pyc file. ChrisA
Hi Chris Again you ask good questions. Q: How to find the bare string '#wibble'? It's optimised out during compilation. A: Very good. I didn't know that. For current Python we'll have to use a different marker. For future Python we could change the compiler so that it directly sets fn.__wibble__ without needing the @wibble decorator. Q: You have at least added some source code to help(fn), if it's available. But what if the function came from a .pyc file? A: It will work the same as tracebacks work, and in exactly the same circumstances. I think that's exactly what we want. Any other solution would increase the size of .pyo files, which is to be avoided. Thank you for your interest. I hope this helps. -- Jonathan
On Sun, Oct 31, 2021 at 4:50 AM Eric V. Smith <eric@trueblade.com> wrote:
On 10/30/2021 10:40 AM, Chris Angelico wrote:
And, seeing something in help(fn) largely necessitates that the source code be retained. I don't know of any other way to do it. If you say that the default argument is "len(a)", then that's what help() should say.
"from __future__ import annotations" works by decompiling the AST into a string. From PEP 563: "The string form is obtained from the AST during the compilation step, which means that the string form might not preserve the exact formatting of the source."
I'm not saying this is a better way to do it, but it is another way. And it might save some memory. I don't think the slight differences mentioned at the end of that sentence would make a difference in practice.
Thank you Eric for pointing this out, and a big thank you to Lukasz and everyone else who implemented this. I allocated a couple of hours to figuring out how to get source representations of late-bound defaults, started up a livestream, and in about fifteen minutes, had deployed and tested something that uses _PyAST_ExprAsUnicode to do the whole job. Was able to settle that and then move on to other things. Thank you for making my life easier! ChrisA
participants (3)
-
Chris Angelico -
Eric V. Smith -
Jonathan Fine