Proposal for special name and qualname symbols
This is an new thread inspired by the "Quick idea: defining variables from functions that take the variable name" thread. For the benefit of those who haven't read that thread, in a nutshell, there are some functions or classes that find it useful or necessary to know their own name or qualified name. At the moment, apart from a few special cases where the compiler does the work behind the scenes (function and class definitions), the only way to do this is to hard-code the name/qualname as a string. The classic example from the standard library is namedtuple: Record = namedtuple('Record', fields) Proposal: we introduce two special symbols to represent the name and qualname at any point. They can be used anywhere an identifier can be used. We could use special dunder names, let's say __NAME__ and __QNAME__, but for now I'm going to use two special symbols: @ for the unqualified bare name; @@ for the qualified name. The namedtuple example would become: Record = namedtuple(@, fields) If your function takes its name parameter somewhere other than the first argument, that's not a problem: MyClass = make_a_class(spam, eggs, name=@, fnord=True) '@' takes on the name, or dotted name, on the left-hand side of an assignment: x = function(@) # @ == 'x' spam.x = function(@) # @ == 'spam.x' If there are multiple targets, and they are all simple or dotted names, @ is a tuple of names: a, b, c.d = function(@) # @ == ('a', 'b', 'c.d') a = b = c.d = function(@) # @ == ('a', 'b', 'c.d') Those two cases are indistinguishable by @. I'm not sure if that's a bug or a feature :-) For now, any other assignment target will be a syntax error. That restriction may be relaxed in the future, if necessary. spam['key'] = function(@) # SyntaxError @ is not restricted to appear inside function calls: element = mapping[@] # like mapping['element'] The qualified name @@ gets the module name: x = function(@@) # @@ = 'module.x' or, inside a class or function, the class/function name: class K: attr = @@ # 'module.K.attr' def method(self): x = @@ # 'module.K.method.x' Whether dunder names or symbols, assignment to them will be illegal: @ = "myname" # SyntaxError Questions and answers: Q: Why @ rather than $? A: $ is too similar to S for my liking, and I believe that Guido has said that $ will never be used for syntax in Python. Q: What's the use-case for qualified names? A: Classes need to know their qualified name for pickling. Q: Isn't this rather magical? A: Of course it is, but the `def` and `class` statements already make use of that compile magic, this just extends it to other things. Q: Are you sure about the module being part of the qualified name? A: I'm not wedded to that idea. If it's not necessary, take it out. Q: Wait a minute, isn't @ used for matrix multiplication now? And @@ may be used for matrix exponentiation in the future? A: Oh yeah, I forgot that. Damn. Guess we'll have to use the dunder names then. Q: But the dunder names are too long to type. A: Then think of your own symbols and propose them :-) -- Steve
On Jun 8, 2016 11:09 AM, "Steven D'Aprano" <steve@pearwood.info> wrote:
Wouldn't it still be unambiguous, since they're both binary operators?
Q: But the dunder names are too long to type.
A: Then think of your own symbols and propose them :-)
Well, since I'll probably get killed if I mention `?`... What about & or ! (the exclamation point)? IIRC the latter isn't used at all right now. It also reminds me (for better or worse) of hashbangs. Only other symbols I can think of would be \ and ` (the backtick), but I'm not sure what the heck the idea behind them is (although \ could probably be referenced to lambdas or something?), and the backtick is kind of unreadable...
-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/
On 2016-06-08 09:09, Steven D'Aprano wrote:
Do you mean "anywhere an identifier can be used on the right-hand side of an assignment statement"? Presumably these would not be allowed outside of assignment statements. Would they be allowed on augmented assignment statements? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Jun 09 2016, Steven D'Aprano <steve-iDnA/YwAAsAk+I/owrrOrA@public.gmane.org> wrote:
This is so similar (in both semantics and syntax) to the Record = namedtuple($lhs, fields) proposal that I've made several times in that thread that I am a little offended. Either you've not read that thread all that thorougly, or you have a rather selective memory. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Jun 08 2016, Nikolaus Rath <Nikolaus-BTH8mxji4b0@public.gmane.org> wrote:
...or there might have been another reason. Sorry, that wasn't meant to come out so aggressively. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Jun 08, 2016, at 12:43 PM, Nikolaus Rath wrote:
It's of course quite easy to miss a specific suggestion in the various millithreads on this list, or not quite connect the dots to see the similarities on the initial read. Just be happy that great minds think alike. :) On to the topic at hand, when I saw your suggestion and Steve's I immediately thought I'd like to see a special symbol like __LHS__, e.g. Record = namedtuple(__LHS__, fields) which would at least have the benefit of not introducing a special character, and one that could be confusing in other contexts (e.g. PEP 292 strings). Cheers, -Barry
On Wed, Jun 8, 2016 at 5:10 PM Barry Warsaw <barry@python.org> wrote:
For what it's worth, __LHS__ seems best to me, of the suggestions so far. The fact that it's all caps suggests even more magic than most dunders. To recap the problems this might solve: 1. pass identifier as string arg (seems trivial, status quo is fine) 2. get name and qualname for pickling (instead of sys._getframe) 3. @public for constants (as per Barry's email) I don't know what that last one was referring to. Do you mind clarifying, Barry (or anyone else)? Any other problems this could help with?
On Jun 08, 2016, at 10:57 PM, Michael Selik wrote:
Sure! I propose that it's difficult to keep __all__ up to date. Witness the plethora of bugs open on similar issues in the stdlib. Furthermore, the definition of __all__ is usually far removed from the object it names. And it's not obvious when looking at the object whether it is intended to be exported or not. @public solves these problems and I contend that its meaning is pretty obvious, given that (unknown to me at the time) there have been at least two other independent inventions of nearly exactly the same decorator. My tracker issue proposing to add `public` to builtins is here: http://bugs.python.org/issue26632 I gave a lightning talk at the Pycon 2016 language summit about it but it got a rather frosty reception. ;) OTOH, other people like it, I've begun to use it in my own code, and I like it too. Now you don't have to wait for Python to catch up <wink>: http://public.readthedocs.io/en/latest/ https://pypi.io/project/atpublic/ https://gitlab.com/warsaw/public The issue in this context is that @public as a decorator only works for things with an __name__, e.g. functions and classes. There have been different proposals to make it work for constants, and my approach supports calling public() with keyword arguments, e.g. public(SEVEN=7) public(a_bar=Bar()) Admittedly this is rather un-Pythonic since it not only modifies the module's __all__ but it pokes a variable binding into the module globals. It's yucky and I'd rather be able to do something like: @public SEVEN = 7 @public a_bar = Bar() Thus the relevance to this thread. Cheers, -Barry
On Jun 08, 2016, at 09:30 PM, Eric V. Smith wrote:
SEVEN = public(__LHS__, 7) a_bar = public(__LHS__, Bar())
Yes, if we had __LHS__ that's how I'd write it.
Although this might be a different version of public (or not!).
It would be, because public() wouldn't then need to poke values into the module globals. It would just have to return the second argument and put the first argument in __all__. Tools like pyflakes would stop complaining too. Win! -Barry
(API bikeshedding follows.) In `@public` => `public(__LHS__, value)`, I'm a little bothered by __LHS__ being the first argument. The first argument in a regular decorator is the value. Instead, how about this? SEVEN = public(7, <some_keyword>=__LHS__) ... where the keyword is, for example, `name` or `target`. I thought about `__name__`, too, but that sort of implies the name will be attached to the object, which is true for `namedtuple` but not for `public`. Now the problem with that is, the assignment is now reversed. (I'm not so bothered, somehow, that __LHS__ is magically substituted in.) ---- By the way, should these LHSs also have API? - `indexable[index] = value` - `obj.attrname = value` - `first, *middle, last = value` - ... and more, if we get unpacking generalizations like dict unpacking and pattern-match. (`obj` and `indexable` can be expressions, but that's not an issue.) How should these be passed? Or should they just be disallowed? Here's an ugly straw man proposal, using `decorator(name, value)`: - `decorator((indexable.__setitem__, index), value)` - `decorator((obj.__setattr__, attrname), value)` - `decorator(['first', ..., 'middle', 'last'], value)` Another (ugly) possibility, using kw params and `functools.partial`: - `decorator(value, target=partial(indexable.__setitem__, index))` - `decorator(value, target=partial(obj.__setattr__, attrname))` - `decorator(value, name=['first', 'middle', 'last'], splat=1)` Possible usecases: ??? On Fri, Jun 10, 2016 at 12:42 PM, Barry Warsaw <barry@python.org> wrote:
On Thu, Jun 09, 2016 at 03:29:07AM +0100, MRAB wrote:
Ooooh! I like that! Except that instead of a scripting updating __all__, it's a unit test that fails. I don't know if this should go into unittest or test.support. https://docs.python.org/3/library/test.html#module-test.support Off the top of my head, something like: class TestAll(unittest.CheckAll): module = name_of_module_to_check skip = [names, to, skip] which scans the source of module, extracts all the public names, and fails if any of them (other than those listed in skip) are missing from __all__. If the source is unavailable, the test is skipped rather than passing. -- Steve
On Thu, Jun 09, 2016 at 11:30:46AM +0100, Rob Cliffe <rob.cliffe@btinternet.com> wrote:
@public is just a special syntax for that.
Rob Cliffe
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
2016-06-08 14:10 GMT-07:00 Barry Warsaw <barry@python.org>:
I implemented Barry's variation of this idea at https://github.com/JelleZijlstra/cpython/tree/lhs in case somebody wants to play with it.
Record = namedtuple(__LHS__, 'a b')
Record <class '__main__.Record' at 0x7fb12af03028>
On 6/13/2016 5:25 AM, Barry Warsaw wrote:
Very insane, and very useless! Preserving the whitespace would be hard, and of dubious value. I suggest that if we do this, we limit it to a single identifier on the lhs. In the unlikely event that we need it later, we can relax the restriction. Eric.
I can't imagine why that would ever come up. More likely would be mydict = { name : Record[name] for name in standard_symbol_names } and the whole issue is just gone, this is TOOWTDI for such iterative cases. It's only when you want to inject a name (such as "mydict" itself) into a namespace normally maintained by the compiler that you need this. Barry Warsaw writes:
That's a good question. How insane/useless would it be to return in this case the string "mydict[ index ]"?
Given the assumption that this kind of anaphorism is Pythonic at all, it would either be that or a canonical flattening of the AST. It looks quite useless to me. But IMO this is unacceptably ugly given that its use cases are so small. A dunder keyword that is used by design even in main programs? C'mon, Barry, you can't mean you're going to litter Mailman code with this! I don't like (-0.9) Steven d'Aprano's "->" operator either, but it's the best of a bad lot so far. Note that it can't be used iteratively either unless its signature is changed from <id> -> <constructor> to <str> -> <constructor>, so that instead of writing x -> Symbol() you'd write 'x' -> Symbol() (thus instead of reading the name of an identifier from the namespace and passing the string to Symbol, it would find or inject an identifier with that name in the namespace, and assign the value of the RHS to it). Then you could write things like for name in standard_symbol_names: name -> Symbol() which I don't like but I suspect advocates will. Steve
On Jun 8, 2016 11:09 AM, "Steven D'Aprano" <steve@pearwood.info> wrote:
Wouldn't it still be unambiguous, since they're both binary operators?
Q: But the dunder names are too long to type.
A: Then think of your own symbols and propose them :-)
Well, since I'll probably get killed if I mention `?`... What about & or ! (the exclamation point)? IIRC the latter isn't used at all right now. It also reminds me (for better or worse) of hashbangs. Only other symbols I can think of would be \ and ` (the backtick), but I'm not sure what the heck the idea behind them is (although \ could probably be referenced to lambdas or something?), and the backtick is kind of unreadable...
-- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/
On 2016-06-08 09:09, Steven D'Aprano wrote:
Do you mean "anywhere an identifier can be used on the right-hand side of an assignment statement"? Presumably these would not be allowed outside of assignment statements. Would they be allowed on augmented assignment statements? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Jun 09 2016, Steven D'Aprano <steve-iDnA/YwAAsAk+I/owrrOrA@public.gmane.org> wrote:
This is so similar (in both semantics and syntax) to the Record = namedtuple($lhs, fields) proposal that I've made several times in that thread that I am a little offended. Either you've not read that thread all that thorougly, or you have a rather selective memory. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Jun 08 2016, Nikolaus Rath <Nikolaus-BTH8mxji4b0@public.gmane.org> wrote:
...or there might have been another reason. Sorry, that wasn't meant to come out so aggressively. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Jun 08, 2016, at 12:43 PM, Nikolaus Rath wrote:
It's of course quite easy to miss a specific suggestion in the various millithreads on this list, or not quite connect the dots to see the similarities on the initial read. Just be happy that great minds think alike. :) On to the topic at hand, when I saw your suggestion and Steve's I immediately thought I'd like to see a special symbol like __LHS__, e.g. Record = namedtuple(__LHS__, fields) which would at least have the benefit of not introducing a special character, and one that could be confusing in other contexts (e.g. PEP 292 strings). Cheers, -Barry
On Wed, Jun 8, 2016 at 5:10 PM Barry Warsaw <barry@python.org> wrote:
For what it's worth, __LHS__ seems best to me, of the suggestions so far. The fact that it's all caps suggests even more magic than most dunders. To recap the problems this might solve: 1. pass identifier as string arg (seems trivial, status quo is fine) 2. get name and qualname for pickling (instead of sys._getframe) 3. @public for constants (as per Barry's email) I don't know what that last one was referring to. Do you mind clarifying, Barry (or anyone else)? Any other problems this could help with?
On Jun 08, 2016, at 10:57 PM, Michael Selik wrote:
Sure! I propose that it's difficult to keep __all__ up to date. Witness the plethora of bugs open on similar issues in the stdlib. Furthermore, the definition of __all__ is usually far removed from the object it names. And it's not obvious when looking at the object whether it is intended to be exported or not. @public solves these problems and I contend that its meaning is pretty obvious, given that (unknown to me at the time) there have been at least two other independent inventions of nearly exactly the same decorator. My tracker issue proposing to add `public` to builtins is here: http://bugs.python.org/issue26632 I gave a lightning talk at the Pycon 2016 language summit about it but it got a rather frosty reception. ;) OTOH, other people like it, I've begun to use it in my own code, and I like it too. Now you don't have to wait for Python to catch up <wink>: http://public.readthedocs.io/en/latest/ https://pypi.io/project/atpublic/ https://gitlab.com/warsaw/public The issue in this context is that @public as a decorator only works for things with an __name__, e.g. functions and classes. There have been different proposals to make it work for constants, and my approach supports calling public() with keyword arguments, e.g. public(SEVEN=7) public(a_bar=Bar()) Admittedly this is rather un-Pythonic since it not only modifies the module's __all__ but it pokes a variable binding into the module globals. It's yucky and I'd rather be able to do something like: @public SEVEN = 7 @public a_bar = Bar() Thus the relevance to this thread. Cheers, -Barry
On Jun 08, 2016, at 09:30 PM, Eric V. Smith wrote:
SEVEN = public(__LHS__, 7) a_bar = public(__LHS__, Bar())
Yes, if we had __LHS__ that's how I'd write it.
Although this might be a different version of public (or not!).
It would be, because public() wouldn't then need to poke values into the module globals. It would just have to return the second argument and put the first argument in __all__. Tools like pyflakes would stop complaining too. Win! -Barry
(API bikeshedding follows.) In `@public` => `public(__LHS__, value)`, I'm a little bothered by __LHS__ being the first argument. The first argument in a regular decorator is the value. Instead, how about this? SEVEN = public(7, <some_keyword>=__LHS__) ... where the keyword is, for example, `name` or `target`. I thought about `__name__`, too, but that sort of implies the name will be attached to the object, which is true for `namedtuple` but not for `public`. Now the problem with that is, the assignment is now reversed. (I'm not so bothered, somehow, that __LHS__ is magically substituted in.) ---- By the way, should these LHSs also have API? - `indexable[index] = value` - `obj.attrname = value` - `first, *middle, last = value` - ... and more, if we get unpacking generalizations like dict unpacking and pattern-match. (`obj` and `indexable` can be expressions, but that's not an issue.) How should these be passed? Or should they just be disallowed? Here's an ugly straw man proposal, using `decorator(name, value)`: - `decorator((indexable.__setitem__, index), value)` - `decorator((obj.__setattr__, attrname), value)` - `decorator(['first', ..., 'middle', 'last'], value)` Another (ugly) possibility, using kw params and `functools.partial`: - `decorator(value, target=partial(indexable.__setitem__, index))` - `decorator(value, target=partial(obj.__setattr__, attrname))` - `decorator(value, name=['first', 'middle', 'last'], splat=1)` Possible usecases: ??? On Fri, Jun 10, 2016 at 12:42 PM, Barry Warsaw <barry@python.org> wrote:
On Thu, Jun 09, 2016 at 03:29:07AM +0100, MRAB wrote:
Ooooh! I like that! Except that instead of a scripting updating __all__, it's a unit test that fails. I don't know if this should go into unittest or test.support. https://docs.python.org/3/library/test.html#module-test.support Off the top of my head, something like: class TestAll(unittest.CheckAll): module = name_of_module_to_check skip = [names, to, skip] which scans the source of module, extracts all the public names, and fails if any of them (other than those listed in skip) are missing from __all__. If the source is unavailable, the test is skipped rather than passing. -- Steve
On Thu, Jun 09, 2016 at 11:30:46AM +0100, Rob Cliffe <rob.cliffe@btinternet.com> wrote:
@public is just a special syntax for that.
Rob Cliffe
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
2016-06-08 14:10 GMT-07:00 Barry Warsaw <barry@python.org>:
I implemented Barry's variation of this idea at https://github.com/JelleZijlstra/cpython/tree/lhs in case somebody wants to play with it.
Record = namedtuple(__LHS__, 'a b')
Record <class '__main__.Record' at 0x7fb12af03028>
On 6/13/2016 5:25 AM, Barry Warsaw wrote:
Very insane, and very useless! Preserving the whitespace would be hard, and of dubious value. I suggest that if we do this, we limit it to a single identifier on the lhs. In the unlikely event that we need it later, we can relax the restriction. Eric.
I can't imagine why that would ever come up. More likely would be mydict = { name : Record[name] for name in standard_symbol_names } and the whole issue is just gone, this is TOOWTDI for such iterative cases. It's only when you want to inject a name (such as "mydict" itself) into a namespace normally maintained by the compiler that you need this. Barry Warsaw writes:
That's a good question. How insane/useless would it be to return in this case the string "mydict[ index ]"?
Given the assumption that this kind of anaphorism is Pythonic at all, it would either be that or a canonical flattening of the AST. It looks quite useless to me. But IMO this is unacceptably ugly given that its use cases are so small. A dunder keyword that is used by design even in main programs? C'mon, Barry, you can't mean you're going to litter Mailman code with this! I don't like (-0.9) Steven d'Aprano's "->" operator either, but it's the best of a bad lot so far. Note that it can't be used iteratively either unless its signature is changed from <id> -> <constructor> to <str> -> <constructor>, so that instead of writing x -> Symbol() you'd write 'x' -> Symbol() (thus instead of reading the name of an identifier from the namespace and passing the string to Symbol, it would find or inject an identifier with that name in the namespace, and assign the value of the RHS to it). Then you could write things like for name in standard_symbol_names: name -> Symbol() which I don't like but I suspect advocates will. Steve
participants (18)
-
Barry Scott -
Barry Warsaw -
Bernardo Sulzbach -
Brendan Barnwell -
Eric V. Smith -
Ethan Furman -
Franklin? Lee -
Jelle Zijlstra -
Michael Selik -
MRAB -
Nikolaus Rath -
Oleg Broytman -
Pavol Lisy -
Rob Cliffe -
Robert Collins -
Ryan Gonzalez -
Stephen J. Turnbull -
Steven D'Aprano