Ability to set precedence of classdict (returned from __prepare__) for a metaclass type
When using a custom classdict to implement a DSL or use in the body of a class definition, from what I can tell by experiment, the classdict takes priority, and the surrounding context is only referenced if trying to get an item from the classdict raises `KeyError`. There is at least one case in which I would like to do the reverse, and have the classdict be secondary to (masked by) any variables defined in the context surrounding the execution of the class body. I have been able to at least unmask builtins by having the classdict object first try to get a result from `__builtins__` and then fall back to itself. Also, in the actual class body, declaring a variable as `global` seems to be a workaround for global variables, but that has to be done explicitly in the body of any subclass that needs it. Also, that does not handle non-globals in its surrounding context. I'm not sure what the best way to deal with this would be, but my first thought is to maybe have a property that can be set on the metaclass type such as `metacls.relictant_classdict = True`.
On Sun, Oct 20, 2019 at 1:58 PM Steve Jorgensen <stevej@stevej.name> wrote:
Not entirely sure I follow here; are you talking about unadorned names during the execution of the class body itself, or the way attribute lookup is done on the resulting object? If you want to be able to mess with attribute lookup even when the attribute does exist, __getattribute__ is your friend: https://docs.python.org/3/reference/datamodel.html#object.__getattribute__ For changing the behaviour of class body execution, it's a bit harder. Do you actually need the class members to be available as unadorned names, but such that the surrounding context takes precedence? That could be awkward, since there's no way (as far as I know) to get a "hook" whenever NameError would have been raised. But you can return any dict-like object from the metaclass's __prepare__ method, and if its __getitem__ and __setitem__ are disconnected, you could get the effect you want. Here's one possibility that gets kinda close: class BlackBox(dict): def __getitem__(self, item): if item == "v": return self raise KeyError __getattr__ = dict.__getitem__ class Meta(type): @classmethod def __prepare__(meta, cls, bases): return BlackBox() var = "global" class Demo(metaclass=Meta): var = "local" print("var =", var) print("v.var =", v.var) Accessing local names can be done with the "v." prefix, global names are directly accessible, but missing globals won't be looked up locally. That said, though, I'm not sure that this was even what you were asking about :) ChrisA
That said, though, I'm not sure that this was even what you were asking about :)
It would be significantly more helpful if the OP had an example for the specific case where they wanted contextually defined variables to take priority over the classdict, and explained _why_ this behavior might be desirable:
Describing the specific use case in detail and providing an example would also help with determining the practical value of adding the described property. I'm not certain that a property would be the best way to do it (a new optional parameter for the __prepare__ attribute might be an alternative). But before that, we have to determine whether or not the feature would have enough practical value to justify implementing in any capacity. This is of course assuming that what the OP wants doesn't already exist in some other form (if Chris's example isn't what the OP is looking for) or that it's even reasonably possible to implement in the first place. On Sat, Oct 19, 2019 at 11:39 PM Chris Angelico <rosuav@gmail.com> wrote:
On 10/19/2019 07:56 PM, Steve Jorgensen wrote:
When using a custom classdict to implement a DSL or use in the body of a class definition, from what I can tell by experiment, the classdict takes priority, and the surrounding context is only referenced if trying to get an item from the classdict raises `KeyError`.
An example would work wonders for explaining what you are asking. It sounds like an interesting problem, but I can't tell exactly what you need. -- ~Ethan~
Several requests for a specific use case. This is something I have been working on just to practice getting a deeper understanding of Python metaprogramming, so whether this is something that someone should be trying to do is, of course, a matter of opinion. The idea is to use a class as a singleton collection of choices where every choice appears both as an attribute of the class/singleton and as a value/label pair when treating it as a sequence. Usage example: class Colors(Choices): with choices(): RED GREEN BLUE Colors.RED -> 'Red' Colors.GREEN -> 'Green' Colors.BLUE -> 'Blue' tuple(Colors) -> (('RED', 'Red'), ('GREEN', 'Green'), ('BLUE', 'Blue')) This can be implemented using a mapping object (classdict) returned from the metaclass' `__prepare__` class method. The classdict is checked prior to checking the surrounding context and takes priority unless its `__getitem__` method raises `KeyError`. The problem in this case is that the classdict generates results dynamically, and it has no way of knowing whether a variable reference would have been resolved in the surrounding context or not. Therefore, it will ALWAYS take precedence, even masking builtins like `print`, `range`, etc. This is surprising behavior and generally undesirable. As I mentioned already, it is possible to work around the problem for builtins by having the classdict object specifically test for those, but that doesn't solve the problem of masking other variables from the surrounding context. Masking those is also surprising behavior. Steve Jorgensen wrote:
We try to do the same thing in various libraries. We've settled on using existing python and end up with syntax like: class MyForm(Form): field = Field() or in your case class Colors(TokenContainer): red = Token() green = Token() blue = Token() (this is using tri.token). The discussion on creating namespaces seem to match this type of usage more than what you've suggested here.
Anders Hovmöller wrote:
Here's another interesting possible pattern/DSL that I've come up with for my value/label case. This pattern doesn't require getting fancy with the classdict at all. It does create choices implicitly via name access, but it's the name of an attribute of a context object, not the name of a variable in the namespace. from choices import choices def test_constructs_with_auto_valued_choices(): # Get choices context for Food choices. fcc = choices() # Optionally replace fcc.base with a subclass of fcc.base or # fcc.mixin here if additional behavior is desirable to have # within the class body below. class Food(fcc.base): # Add a new instance of fcc.base w/ value of 'APPLE' and label # of None, and create an APPLE class attribute with a value # of 'APPLE' fcc.APPLE # Just as for APPLE above. fcc.GREEK_YOGURT # Add an EGG_MCMUFFIN choice and class value-attribute, then # replace it with a choice that has the same value and a label # of 'Egg McMuffin' (since instances are immutable). fcc.EGG_MCMUFFIN %= 'Egg McMuffin' # Add a choice with value of '~OTHER~' and label of # not associated with any attribute 'Something else' fcc.append('~OTHER~', 'Something else') # After class is defined, unlabeled instances of fcc.base have been # auto-labeled and upgraded to instances of Food by passing through # Food.coerce_from(obj). # The __new__ method behavior of Food has now been locked, and raises # TypeError for any subsequent attempts to instantiate. # The default fcc.base class has a value/label namedtuple as a base # class, so this is valid/successful. assert tuple(Food) == ( ('RED', 'Apple'), ('GREEN', 'Greek Yogurt'), ('EGG_MCMUFFIN', 'Egg McMuffin'), ) # Members are instances of Food. assert tuple(type(food) for food in Food) == (Food, Food)
Oops. Obviously, the above is pseudo-code because I left `('~OTHER~', 'Something else')` the first assert, and I have `(Food, Food)` instead of `(Food,) * 4` for the second assert. Possibly some other mistakes as well.
Steve Jorgensen wrote:
Umm, like the fact that I'm testing for colors instead of foods since I changed the kind of example case. :) Here's an updated example, minus the comments — still pseudocode with possible mistakes: from choices import ChoicesContext # Get choices context for Food choices. fcc = ChoicesContext() class Food(fcc.base): fcc.APPLE fcc.GREEK_YOGURT fcc.EGG_MCMUFFIN %= 'Egg McMuffin' fcc.append('~OTHER~', 'Something else') assert ( (Food.Apple, Food.GREEK_YOGURT, fcc.EGG_MCMUFFIN) == ('APPLE', 'GREEK_YOGURT', 'EGG_MCMUFFIN')) assert tuple(Food) == ( ('APPLE', 'Apple'), ('GREEK_YOGURT', 'Greek Yogurt'), ('EGG_MCMIFFIN', 'Egg McMuffin'), ('~OTHER~', 'Something else'), ) assert tuple(type(food) for food in Food) == (Food,) * 4
On Oct 19, 2019, at 22:57, Steve Jorgensen <stevej@stevej.name> wrote:
The idea is to use a class as a singleton collection of choices where every choice appears both as an attribute of the class/singleton and as a value/label pair when treating it as a sequence.
What you’re describing is just enum.Enum, minus half its features, but a sequence instead of a mapping. I’m not sure why you’d ever want this, but if you did, you could just subclass EnumMeta and add/override the sequence methods. At any rate, almost all of that seems irrelevant to your issue. Making it act like both a singleton with members and like a sequence is already trivial to do today. What you’re looking for is a way to auto-assign attributes on lookup during class definition. Which is an idea that was actually brought up for enums, and specifically rejected in PEP 435, but there’s a proof of concept implementation of it anyway if you want it. It’s also worth looking at the implementation of the various alternative enum implementations on PyPI (both the ones that pre-existed 3.4, and the ones that build on it).
Andrew Barnert wrote:
Actually, it's a bit different. This is for building a collection of value/label pairs where the values are as likely to be strings as numbers, and its sequence behavior is as a list of value/label pairs (e.g. for use as the choices for a Django field).
OK. I'm not at all surprised that was rejected. In that case, perhaps the answer to my proposal is — nobody should be doing this kind of thing, so there's no reason to work around the limitations of trying to do this kind of thing. :)
Yes. I'll take a look. My code and this DSL was probably never going to be used in production anyway because it primarily a learning exercise. Possibly actually using it is a distant second priority. Reading those alternative implementations is, of course, also a good learning exercise. :)
On Oct 20, 2019, at 03:36, Steve Jorgensen <stevej@stevej.name> wrote:
But the values of an Enum can be strings. And there are multiple PyPI projects that build Django choices on top of Enum, which show that the only thing you need is to override __getitem__ to replace its mapping behavior with sequence behavior. I believe this was also one of the examples of what you can build for yourself on top of EnumMeta in Nick Coghlan’s blog post explaining the design of enum. Maybe the additional functionality in Enum gets in your way for some reason I don’t understand, that doesn’t affect other people who want Django choices. But otherwise, I don’t see how it really is any different. And again, none of this different-from-Enum behavior is relevant to your problem anyway. The fact that you think it is, and think it can’t be done with Enum, makes it harder to understand where the actual problem is. Maybe you’re not just trying to do the same magic that all the autoenum/simplenum/etc. packages on PyPI and the PoC code from the PEP and the section in Nick’s blog post all do (in a couple different ways), or maybe it’s insufficiently magic for your purposes, but without knowing how and why it’s different or insufficient it’s impossible to know whether there’s an actual limitation to fix in Python, or just a problem in your design or implementation that could be easily fixed to work in existing Python.
Andrew Barnert wrote:
On Oct 20, 2019, at 03:36, Steve Jorgensen stevej@stevej.name wrote:
So… You're exactly right. I just had a very limited understanding of what enum was about and had not dug into it deeply enough to find out how flexible it really is. Here's an implementation that I came up with in a short hacking session using enum. Tests: from choices import ChoiceEnum, Label as L def test_creates_auto_valued_auto_labeled_choices(): class Food(ChoiceEnum): APPLE = () ICED_TEA = () assert tuple(tuple(fc) for fc in Food) == ( ('APPLE', 'Apple'), ('ICED_TEA', 'Iced Tea'), ) def test_creates_auto_labeled_choices_with_given_values(): class Food(ChoiceEnum): APPLE = 1 ICED_TEA = 2 assert tuple(tuple(fc) for fc in Food) == ( (1, 'Apple'), (2, 'Iced Tea'), ) def test_creates_choices_with_given_values_and_labels(): class Food(ChoiceEnum): CHICKEN_MCNUGGETS = (1, 'Chicken McNuggets') CHICKEN_PROVENCAL = (2, 'Chicken Provençal') assert tuple(tuple(fc) for fc in Food) == ( (1, 'Chicken McNuggets'), (2, 'Chicken Provençal'), ) def test_creates_auto_valued_choices_with_given_values(): class Food(ChoiceEnum): CHX_MN = L('Chicken McNuggets') CHX_PV = L('Chicken Provençal') assert tuple(tuple(fc) for fc in Food) == ( ('CHX_MN', 'Chicken McNuggets'), ('CHX_PV', 'Chicken Provençal'), ) Implementation: from enum import Enum class ChoiceEnum(Enum): def __init__(self, src=None, label=None): super().__init__() if isinstance(src, Label): value = None label = str(src) else: value = src self._value_ = self.name if value is None else value self.label = label or _label_from_name(self.name) def __getitem__(self, idx): if idx == 0: return self.value elif idx == 1: return self.label else: raise IndexError('Index value must be 0 or 1') def __len__(self): return 2 class Label(str): pass def _label_from_name(name): return name.replace('_', ' ').title()
On 10/21/2019 10:33 PM, Steve Jorgensen wrote:
Experimenting is good! However, you'll want to either build your own metaclass and/or prepared dict, or do some work on your `__new__`/`__init__` methods for building enum members. Currently, you are reassigning `_value_` in `__init__`, which leaves some internal structures not matching the Enum. --> class Food(ChoiceEnum): ... APPLE = () ... ICED_TEA = () ... --> Food['APPLE'] <Food.APPLE: 'APPLE'> --> Food.APPLE <Food.APPLE: 'APPLE'> --> Food('APPLE') Traceback (most recent call last): ... ValueError: 'APPLE' is not a valid Food -- ~Ethan~
Ethan Furman wrote:
Thanks for that info. Per the example in https://docs.python.org/3/library/enum.html#when-to-use-new-vs-init, it looks like I can properly set the `_value_` property in the `__new__` method of the `Enum` subclass without needing to subclass `EnumMeta`. Am I understanding that correctly?
On 10/22/2019 03:13 PM, Steve Jorgensen wrote:
Yes, you are. The fun part for you is that the value can sometimes be the name, and the name is not passed into `__new__`. The name /is/ passed into `_generate_next_value_`, but if any value is supplied then `_generate_next_value_` isn't called. I haven't had enough space cycles to either find a solution or rule any out, but I know the limitations above are baked into EnumMeta and _EnumDict, so if you rolled your own you could simply not put them in. -- ~Ethan~
On 10/19/2019 11:25 PM, Andrew Barnert via Python-ideas wrote:
It is actually implemented in the `aenum`* library. The code in question is in _EnumDict, and my workaround for the problem was another variable (IIRC, _ignore_) that listed any names that were outside the Enum's scope. -- ~Ethan~ * disclosure: I'm the one to hold responsible for the stdlib enum, enum34, and aenum.
On Sun, Oct 20, 2019 at 02:56:22AM -0000, Steve Jorgensen wrote:
I've read the entire thread, and I don't understand what builtins has to do with your problem. But regardless, you shouldn't touch `__builtins__`, it is a private implementation detail of CPython. The right way is: import builtins # Python 3 import __builtin__ as builtins # Python 2 and then use `builtins`. -- Steven
Steven D'Aprano wrote:
I've read the entire thread, and I don't understand what builtins has to do with your problem.
Just that, as a workaround, I can have the classdict object raise KeyError for anything that's a member of builtins so that the builtins are used instead of being masked.
Thanks for setting me straight on this. Much appreciated.
On 10/19/2019 07:56 PM, Steve Jorgensen wrote:
When using a custom classdict to implement a DSL or use in the body of a class definition, from what I can tell by experiment, the classdict takes priority, and the surrounding context is only referenced if trying to get an item from the classdict raises `KeyError`.
Thank you for the examples. As you have discovered, the issue is that when Python is processing a class and gets a name (whether a def, a property, a variable, etc., is irrelevant) it tries to look that name up in the class_dict that was returned by __prepare__, and that class_dict has one shot at it -- either return an object for the name, or raise KeyError -- in which case Python will keep looking up the scope in an attempt to find a match. Having said all that, I'm now wondering if wrapping a super() call in a try/except inside class_dict.__get_item__ might solve the problem -- although, even if that worked, you would have the reverse problem of wanting a name to be new in your class, but already existing outside. The solution I use in aenum.Enum is an extra variable, _ignore_, that lists all names that shouldn't be shadowed in the new dict; it starts as being set to property, classmethod, and staticmethod, and the user can replace that list with whatever they like. -- ~Ethan~
On 20/10/2019 03:56:22, Steve Jorgensen wrote:
When using a custom classdict to implement a DSL or use in the body of a class definition, from what I can tell by experiment, the classdict takes priority, and the surrounding context is only referenced if trying to get an item from the classdict raises `KeyError`.
Can I make a plea that posters to this (and other) lists do not introduce acronyms, abbreviations or jargon that are not commonly used Python terminology and may not be familiar to everyone who reads them? (I say "_*introduce*_". I have no problem with a term explained in one post being used without explanation in subsequent posts in the same thread. If I can't be bothered to read the whole thread, that's my problem.) Googling DSL Wikipedia offers Digital Subscriber Line Damn Small Linux, a very small Linux distribution Danish Sign Language Definitive software library Design Science License, a copyleft license for intellectual property Diagnostic Systems Laboratories, a company now part of Beckman-Coulter Dictionary of the Scots Language Domain-specific language, a specialized computer language Dominican Summer League of baseball Artists signed to Ed Banger Records Deutsche Schule Lagos, or the German School Lagos I don't know which, if any, of these is the OP's intended meaning (my best guess is Domain-specific language, but that's all it is - a guess). I (and no doubt others) am less likely to engage with a thread if I have to spend more time than necessary trying to understand it. Thanks Rob Cliffe
Rob Cliffe via Python-ideas writes:
I (and no doubt others) am less likely to engage with a thread if I have to spend more time than necessary trying to understand it.
You write as if that's a bad thing. ;-) I consider that I often make the largest contribution to this list by reading more posts and stifling the impulse to reply to any of them. :-) YMMV, of course.
I don't see why this would apply to examples such as "DSL". If you don't know what "DSL" means from context, you're probably not interested in that example in the first place. On the other hand, if the jargon in question is Python-specific (and that is usually apparent from context), maybe spending the time to look it up and then read the language manual or the stdlib manual is worthwhile. I can say it always has been for me, YMMV, as usual. Steve -- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
On Sun, Oct 20, 2019 at 1:58 PM Steve Jorgensen <stevej@stevej.name> wrote:
Not entirely sure I follow here; are you talking about unadorned names during the execution of the class body itself, or the way attribute lookup is done on the resulting object? If you want to be able to mess with attribute lookup even when the attribute does exist, __getattribute__ is your friend: https://docs.python.org/3/reference/datamodel.html#object.__getattribute__ For changing the behaviour of class body execution, it's a bit harder. Do you actually need the class members to be available as unadorned names, but such that the surrounding context takes precedence? That could be awkward, since there's no way (as far as I know) to get a "hook" whenever NameError would have been raised. But you can return any dict-like object from the metaclass's __prepare__ method, and if its __getitem__ and __setitem__ are disconnected, you could get the effect you want. Here's one possibility that gets kinda close: class BlackBox(dict): def __getitem__(self, item): if item == "v": return self raise KeyError __getattr__ = dict.__getitem__ class Meta(type): @classmethod def __prepare__(meta, cls, bases): return BlackBox() var = "global" class Demo(metaclass=Meta): var = "local" print("var =", var) print("v.var =", v.var) Accessing local names can be done with the "v." prefix, global names are directly accessible, but missing globals won't be looked up locally. That said, though, I'm not sure that this was even what you were asking about :) ChrisA
That said, though, I'm not sure that this was even what you were asking about :)
It would be significantly more helpful if the OP had an example for the specific case where they wanted contextually defined variables to take priority over the classdict, and explained _why_ this behavior might be desirable:
Describing the specific use case in detail and providing an example would also help with determining the practical value of adding the described property. I'm not certain that a property would be the best way to do it (a new optional parameter for the __prepare__ attribute might be an alternative). But before that, we have to determine whether or not the feature would have enough practical value to justify implementing in any capacity. This is of course assuming that what the OP wants doesn't already exist in some other form (if Chris's example isn't what the OP is looking for) or that it's even reasonably possible to implement in the first place. On Sat, Oct 19, 2019 at 11:39 PM Chris Angelico <rosuav@gmail.com> wrote:
On 10/19/2019 07:56 PM, Steve Jorgensen wrote:
When using a custom classdict to implement a DSL or use in the body of a class definition, from what I can tell by experiment, the classdict takes priority, and the surrounding context is only referenced if trying to get an item from the classdict raises `KeyError`.
An example would work wonders for explaining what you are asking. It sounds like an interesting problem, but I can't tell exactly what you need. -- ~Ethan~
Several requests for a specific use case. This is something I have been working on just to practice getting a deeper understanding of Python metaprogramming, so whether this is something that someone should be trying to do is, of course, a matter of opinion. The idea is to use a class as a singleton collection of choices where every choice appears both as an attribute of the class/singleton and as a value/label pair when treating it as a sequence. Usage example: class Colors(Choices): with choices(): RED GREEN BLUE Colors.RED -> 'Red' Colors.GREEN -> 'Green' Colors.BLUE -> 'Blue' tuple(Colors) -> (('RED', 'Red'), ('GREEN', 'Green'), ('BLUE', 'Blue')) This can be implemented using a mapping object (classdict) returned from the metaclass' `__prepare__` class method. The classdict is checked prior to checking the surrounding context and takes priority unless its `__getitem__` method raises `KeyError`. The problem in this case is that the classdict generates results dynamically, and it has no way of knowing whether a variable reference would have been resolved in the surrounding context or not. Therefore, it will ALWAYS take precedence, even masking builtins like `print`, `range`, etc. This is surprising behavior and generally undesirable. As I mentioned already, it is possible to work around the problem for builtins by having the classdict object specifically test for those, but that doesn't solve the problem of masking other variables from the surrounding context. Masking those is also surprising behavior. Steve Jorgensen wrote:
We try to do the same thing in various libraries. We've settled on using existing python and end up with syntax like: class MyForm(Form): field = Field() or in your case class Colors(TokenContainer): red = Token() green = Token() blue = Token() (this is using tri.token). The discussion on creating namespaces seem to match this type of usage more than what you've suggested here.
Anders Hovmöller wrote:
Here's another interesting possible pattern/DSL that I've come up with for my value/label case. This pattern doesn't require getting fancy with the classdict at all. It does create choices implicitly via name access, but it's the name of an attribute of a context object, not the name of a variable in the namespace. from choices import choices def test_constructs_with_auto_valued_choices(): # Get choices context for Food choices. fcc = choices() # Optionally replace fcc.base with a subclass of fcc.base or # fcc.mixin here if additional behavior is desirable to have # within the class body below. class Food(fcc.base): # Add a new instance of fcc.base w/ value of 'APPLE' and label # of None, and create an APPLE class attribute with a value # of 'APPLE' fcc.APPLE # Just as for APPLE above. fcc.GREEK_YOGURT # Add an EGG_MCMUFFIN choice and class value-attribute, then # replace it with a choice that has the same value and a label # of 'Egg McMuffin' (since instances are immutable). fcc.EGG_MCMUFFIN %= 'Egg McMuffin' # Add a choice with value of '~OTHER~' and label of # not associated with any attribute 'Something else' fcc.append('~OTHER~', 'Something else') # After class is defined, unlabeled instances of fcc.base have been # auto-labeled and upgraded to instances of Food by passing through # Food.coerce_from(obj). # The __new__ method behavior of Food has now been locked, and raises # TypeError for any subsequent attempts to instantiate. # The default fcc.base class has a value/label namedtuple as a base # class, so this is valid/successful. assert tuple(Food) == ( ('RED', 'Apple'), ('GREEN', 'Greek Yogurt'), ('EGG_MCMUFFIN', 'Egg McMuffin'), ) # Members are instances of Food. assert tuple(type(food) for food in Food) == (Food, Food)
Oops. Obviously, the above is pseudo-code because I left `('~OTHER~', 'Something else')` the first assert, and I have `(Food, Food)` instead of `(Food,) * 4` for the second assert. Possibly some other mistakes as well.
Steve Jorgensen wrote:
Umm, like the fact that I'm testing for colors instead of foods since I changed the kind of example case. :) Here's an updated example, minus the comments — still pseudocode with possible mistakes: from choices import ChoicesContext # Get choices context for Food choices. fcc = ChoicesContext() class Food(fcc.base): fcc.APPLE fcc.GREEK_YOGURT fcc.EGG_MCMUFFIN %= 'Egg McMuffin' fcc.append('~OTHER~', 'Something else') assert ( (Food.Apple, Food.GREEK_YOGURT, fcc.EGG_MCMUFFIN) == ('APPLE', 'GREEK_YOGURT', 'EGG_MCMUFFIN')) assert tuple(Food) == ( ('APPLE', 'Apple'), ('GREEK_YOGURT', 'Greek Yogurt'), ('EGG_MCMIFFIN', 'Egg McMuffin'), ('~OTHER~', 'Something else'), ) assert tuple(type(food) for food in Food) == (Food,) * 4
On Oct 19, 2019, at 22:57, Steve Jorgensen <stevej@stevej.name> wrote:
The idea is to use a class as a singleton collection of choices where every choice appears both as an attribute of the class/singleton and as a value/label pair when treating it as a sequence.
What you’re describing is just enum.Enum, minus half its features, but a sequence instead of a mapping. I’m not sure why you’d ever want this, but if you did, you could just subclass EnumMeta and add/override the sequence methods. At any rate, almost all of that seems irrelevant to your issue. Making it act like both a singleton with members and like a sequence is already trivial to do today. What you’re looking for is a way to auto-assign attributes on lookup during class definition. Which is an idea that was actually brought up for enums, and specifically rejected in PEP 435, but there’s a proof of concept implementation of it anyway if you want it. It’s also worth looking at the implementation of the various alternative enum implementations on PyPI (both the ones that pre-existed 3.4, and the ones that build on it).
Andrew Barnert wrote:
Actually, it's a bit different. This is for building a collection of value/label pairs where the values are as likely to be strings as numbers, and its sequence behavior is as a list of value/label pairs (e.g. for use as the choices for a Django field).
OK. I'm not at all surprised that was rejected. In that case, perhaps the answer to my proposal is — nobody should be doing this kind of thing, so there's no reason to work around the limitations of trying to do this kind of thing. :)
Yes. I'll take a look. My code and this DSL was probably never going to be used in production anyway because it primarily a learning exercise. Possibly actually using it is a distant second priority. Reading those alternative implementations is, of course, also a good learning exercise. :)
On Oct 20, 2019, at 03:36, Steve Jorgensen <stevej@stevej.name> wrote:
But the values of an Enum can be strings. And there are multiple PyPI projects that build Django choices on top of Enum, which show that the only thing you need is to override __getitem__ to replace its mapping behavior with sequence behavior. I believe this was also one of the examples of what you can build for yourself on top of EnumMeta in Nick Coghlan’s blog post explaining the design of enum. Maybe the additional functionality in Enum gets in your way for some reason I don’t understand, that doesn’t affect other people who want Django choices. But otherwise, I don’t see how it really is any different. And again, none of this different-from-Enum behavior is relevant to your problem anyway. The fact that you think it is, and think it can’t be done with Enum, makes it harder to understand where the actual problem is. Maybe you’re not just trying to do the same magic that all the autoenum/simplenum/etc. packages on PyPI and the PoC code from the PEP and the section in Nick’s blog post all do (in a couple different ways), or maybe it’s insufficiently magic for your purposes, but without knowing how and why it’s different or insufficient it’s impossible to know whether there’s an actual limitation to fix in Python, or just a problem in your design or implementation that could be easily fixed to work in existing Python.
Andrew Barnert wrote:
On Oct 20, 2019, at 03:36, Steve Jorgensen stevej@stevej.name wrote:
So… You're exactly right. I just had a very limited understanding of what enum was about and had not dug into it deeply enough to find out how flexible it really is. Here's an implementation that I came up with in a short hacking session using enum. Tests: from choices import ChoiceEnum, Label as L def test_creates_auto_valued_auto_labeled_choices(): class Food(ChoiceEnum): APPLE = () ICED_TEA = () assert tuple(tuple(fc) for fc in Food) == ( ('APPLE', 'Apple'), ('ICED_TEA', 'Iced Tea'), ) def test_creates_auto_labeled_choices_with_given_values(): class Food(ChoiceEnum): APPLE = 1 ICED_TEA = 2 assert tuple(tuple(fc) for fc in Food) == ( (1, 'Apple'), (2, 'Iced Tea'), ) def test_creates_choices_with_given_values_and_labels(): class Food(ChoiceEnum): CHICKEN_MCNUGGETS = (1, 'Chicken McNuggets') CHICKEN_PROVENCAL = (2, 'Chicken Provençal') assert tuple(tuple(fc) for fc in Food) == ( (1, 'Chicken McNuggets'), (2, 'Chicken Provençal'), ) def test_creates_auto_valued_choices_with_given_values(): class Food(ChoiceEnum): CHX_MN = L('Chicken McNuggets') CHX_PV = L('Chicken Provençal') assert tuple(tuple(fc) for fc in Food) == ( ('CHX_MN', 'Chicken McNuggets'), ('CHX_PV', 'Chicken Provençal'), ) Implementation: from enum import Enum class ChoiceEnum(Enum): def __init__(self, src=None, label=None): super().__init__() if isinstance(src, Label): value = None label = str(src) else: value = src self._value_ = self.name if value is None else value self.label = label or _label_from_name(self.name) def __getitem__(self, idx): if idx == 0: return self.value elif idx == 1: return self.label else: raise IndexError('Index value must be 0 or 1') def __len__(self): return 2 class Label(str): pass def _label_from_name(name): return name.replace('_', ' ').title()
On 10/21/2019 10:33 PM, Steve Jorgensen wrote:
Experimenting is good! However, you'll want to either build your own metaclass and/or prepared dict, or do some work on your `__new__`/`__init__` methods for building enum members. Currently, you are reassigning `_value_` in `__init__`, which leaves some internal structures not matching the Enum. --> class Food(ChoiceEnum): ... APPLE = () ... ICED_TEA = () ... --> Food['APPLE'] <Food.APPLE: 'APPLE'> --> Food.APPLE <Food.APPLE: 'APPLE'> --> Food('APPLE') Traceback (most recent call last): ... ValueError: 'APPLE' is not a valid Food -- ~Ethan~
Ethan Furman wrote:
Thanks for that info. Per the example in https://docs.python.org/3/library/enum.html#when-to-use-new-vs-init, it looks like I can properly set the `_value_` property in the `__new__` method of the `Enum` subclass without needing to subclass `EnumMeta`. Am I understanding that correctly?
On 10/22/2019 03:13 PM, Steve Jorgensen wrote:
Yes, you are. The fun part for you is that the value can sometimes be the name, and the name is not passed into `__new__`. The name /is/ passed into `_generate_next_value_`, but if any value is supplied then `_generate_next_value_` isn't called. I haven't had enough space cycles to either find a solution or rule any out, but I know the limitations above are baked into EnumMeta and _EnumDict, so if you rolled your own you could simply not put them in. -- ~Ethan~
On 10/19/2019 11:25 PM, Andrew Barnert via Python-ideas wrote:
It is actually implemented in the `aenum`* library. The code in question is in _EnumDict, and my workaround for the problem was another variable (IIRC, _ignore_) that listed any names that were outside the Enum's scope. -- ~Ethan~ * disclosure: I'm the one to hold responsible for the stdlib enum, enum34, and aenum.
On Sun, Oct 20, 2019 at 02:56:22AM -0000, Steve Jorgensen wrote:
I've read the entire thread, and I don't understand what builtins has to do with your problem. But regardless, you shouldn't touch `__builtins__`, it is a private implementation detail of CPython. The right way is: import builtins # Python 3 import __builtin__ as builtins # Python 2 and then use `builtins`. -- Steven
Steven D'Aprano wrote:
I've read the entire thread, and I don't understand what builtins has to do with your problem.
Just that, as a workaround, I can have the classdict object raise KeyError for anything that's a member of builtins so that the builtins are used instead of being masked.
Thanks for setting me straight on this. Much appreciated.
On 10/19/2019 07:56 PM, Steve Jorgensen wrote:
When using a custom classdict to implement a DSL or use in the body of a class definition, from what I can tell by experiment, the classdict takes priority, and the surrounding context is only referenced if trying to get an item from the classdict raises `KeyError`.
Thank you for the examples. As you have discovered, the issue is that when Python is processing a class and gets a name (whether a def, a property, a variable, etc., is irrelevant) it tries to look that name up in the class_dict that was returned by __prepare__, and that class_dict has one shot at it -- either return an object for the name, or raise KeyError -- in which case Python will keep looking up the scope in an attempt to find a match. Having said all that, I'm now wondering if wrapping a super() call in a try/except inside class_dict.__get_item__ might solve the problem -- although, even if that worked, you would have the reverse problem of wanting a name to be new in your class, but already existing outside. The solution I use in aenum.Enum is an extra variable, _ignore_, that lists all names that shouldn't be shadowed in the new dict; it starts as being set to property, classmethod, and staticmethod, and the user can replace that list with whatever they like. -- ~Ethan~
On 20/10/2019 03:56:22, Steve Jorgensen wrote:
When using a custom classdict to implement a DSL or use in the body of a class definition, from what I can tell by experiment, the classdict takes priority, and the surrounding context is only referenced if trying to get an item from the classdict raises `KeyError`.
Can I make a plea that posters to this (and other) lists do not introduce acronyms, abbreviations or jargon that are not commonly used Python terminology and may not be familiar to everyone who reads them? (I say "_*introduce*_". I have no problem with a term explained in one post being used without explanation in subsequent posts in the same thread. If I can't be bothered to read the whole thread, that's my problem.) Googling DSL Wikipedia offers Digital Subscriber Line Damn Small Linux, a very small Linux distribution Danish Sign Language Definitive software library Design Science License, a copyleft license for intellectual property Diagnostic Systems Laboratories, a company now part of Beckman-Coulter Dictionary of the Scots Language Domain-specific language, a specialized computer language Dominican Summer League of baseball Artists signed to Ed Banger Records Deutsche Schule Lagos, or the German School Lagos I don't know which, if any, of these is the OP's intended meaning (my best guess is Domain-specific language, but that's all it is - a guess). I (and no doubt others) am less likely to engage with a thread if I have to spend more time than necessary trying to understand it. Thanks Rob Cliffe
Rob Cliffe via Python-ideas writes:
I (and no doubt others) am less likely to engage with a thread if I have to spend more time than necessary trying to understand it.
You write as if that's a bad thing. ;-) I consider that I often make the largest contribution to this list by reading more posts and stifling the impulse to reply to any of them. :-) YMMV, of course.
I don't see why this would apply to examples such as "DSL". If you don't know what "DSL" means from context, you're probably not interested in that example in the first place. On the other hand, if the jargon in question is Python-specific (and that is usually apparent from context), maybe spending the time to look it up and then read the language manual or the stdlib manual is worthwhile. I can say it always has been for me, YMMV, as usual. Steve -- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
participants (10)
-
Anders Hovmöller -
Andrew Barnert -
Chris Angelico -
Ethan Furman -
Kyle Stanley -
Rhodri James -
Rob Cliffe -
Stephen J. Turnbull -
Steve Jorgensen -
Steven D'Aprano