Ezio Melotti added the comment: Can you upload the patch as a diff against the CPython repo? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Changes by Eli Bendersky <eliben@gmail.com>: Removed file: http://bugs.python.org/file30191/ref435.py _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Eli Bendersky added the comment: OK, I sent another batch of reviews through the code review system - Ethan you should've gotten an email about it. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Changes by Alex Gaynor <alex.gaynor@gmail.com>: ---------- nosy: +alex _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Changes by Ethan Furman <ethan@stoneleaf.us>: ---------- hgrepos: -189 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: More adjustments due to review. ---------- Added file: http://bugs.python.org/file30202/pep-0435_3.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Eli Bendersky added the comment: By the way, if anyone is willing to take on the documentation, that could be of great help. It shouldn't be very hard since PEP 435 is extremely detailed. The stdlib .rst doc would be the descriptions from PEP 435 plus a reference of the exposed classes which is pretty minimal. We'll need the documentation patch available by the time the code is done reviewing. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Nick Coghlan added the comment: The accepted PEP states that the frame hack should be removed: "To support pickling of these enums, the module name can be specified using the module keyword-only argument." Ergo, if you don't specify it, they cannot be pickled. Explicit is better than implicit, and this hack should not propagate beyond namedtuple (it should actually be deprecated in namedtuple as well). As far as the second point goes, I had already reviewed the PEP implementation before making the comment (that's why I am reasonably sure you can already do it just by overriding _get_mixins). I see it as similar to the changes that were already made to support autonumbered subtypes. However, also note that I said we should wait before doing anything about providing a supported mechanism for that customisation. I've now created issue 17954 to cover a possible refactoring and documentation of that part of the implementation. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Nick Coghlan added the comment: Why we need two ways to do it? If "module=__name__" is available, what's the rationale for providing the option of leaving it out when pickling support is required? (The only times the frame hack will work to enable pickling, the explicit fix will also work). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Eli Bendersky added the comment: Also, your test must run within the regrtest framework. Currently I get: ./python -mtest.regrtest test_enum [1/1] test_enum test test_enum failed -- Traceback (most recent call last): File "/home/eliben/python-src/default/Lib/test/test_enum.py", line 245, in test_pickle_enum_function_with_module self.assertIs(Question.who, loads(dumps(Question.who))) _pickle.PicklingError: Can't pickle <enum 'Question'>: attribute lookup __main__.Question failed 1 test failed: test_enum ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Benjamin Peterson added the comment: I copied the wrong bug number. should have been #17927. sorry ---------- nosy: +benjamin.peterson _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Changes by Benjamin Peterson <benjamin@python.org>: ---------- Removed message: http://bugs.python.org/msg188954 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Changes by Benjamin Peterson <benjamin@python.org>: ---------- Removed message: http://bugs.python.org/msg188956 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: regrtest framework now supported (had to change the module name I was passing in from '__main__' to __name__). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Nick Coghlan added the comment: I also created issue 17959 to track a legitimate concern with the current aliasing design, without impacting incorporation of the accepted PEP. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: After more thought, I think we should leave the shortened version of _StealthProperty as is. The reasoning being that if _StealthProperty does the __getattr__ lookup itself, and fails, it will return an AttributeError... and then Python will use __getattr__ again to try and find the attribute. So I'll add a comment into the shortened version and leave it at that. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Eli Bendersky added the comment: Ethan, I pasted the minimized version to point out the problem; I fully agree it should do the getattr lookup because otherwise it's very difficult to understand what's going on. Let's keep exceptions for actual exceptional situations and explicitly code the normal path. Also, _MemberOrProperty is probably a more descriptive name. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: Here's the latest patch. Note that the functional API portion is broken, and I'll get back to that this evening. Please only comment on the working code. ;) I'll make that _MemborOrProperty name change then as well. ---------- Added file: http://bugs.python.org/file30231/pep-0435.05.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Guido van Rossum added the comment: Here's the promised explanation why I want to keep the getframe hack. I'm sure it won't satisfy everyone, but this will have to do. There are two parts to my argument. TL;DR: (a) by implementing this hack, we will maximize user happiness; (b) I expect that all Python implementations can provide the functionality needed. So why does this maximize user happiness? First of all, enums are such simple objects that it's a disgrace to have an enum that isn't picklable. But most users are lazy and lack imagination, so if they find they have to do extra work to ensure that something is picklable, they will make a judgement call -- is the extra work to make it picklable worth it? Unfortunately they will try to make this judgment call in a fraction of a second, and they limited imagination they may well say "I can't imagine ever pickling this", save themselves the work, and move on. If you think more than a second about the decision, you've wasted more time than it takes to type "module=__name__", so it's going to be a split-second decision. But nevertheless, having to think about it and typing it is a distraction, and when you're "in the zone" you may prefer to spend your time thinking about the perfect names for your enum class and values rather than the red tape of making in picklable. So, in my mind, it's a given that there will be many enums that are missing the module=__name__ keyword argument. Moreover, it's likely that most of the time this will be fine -- you can get through most days just fine without ever reading or writing a pickle. (For example, I very much doubt that I've ever pickled one of the flag values used by the socket module to define e.g. the address family, socket type, or protocol.) But now you enter a different phase of your project, or one of your collaborators does, or perhaps you've released your code on PyPI and one of your users does. So someone tries to pickle some class instance that happens to contain an unpicklable enum. That's not a great experience. Pickling and unpickling errors are often remarkably hard to debug. (Especially the latter, so I have privately admonished Ethan to ensure that if the getframe hack doesn't work, the pickle failure should happen at pickling time, not at unpickle time.) Once you've tracked down the source, you have to figure out the fix -- hopefully just typing the error message into Google will link back to a StackOverflow answer explaining the need to say "module=__name__". But the damage is done, especially if the person encountering the pickling error is not the original author of the code defining the enum. (Often the person downloading and using a package from PyPI has less advanced Python knowledge than the package author, so they may have a hard time debugging the situation.) You can see how having the getframe hack in place makes life more pleasant for many people -- the package user won't have to debug the pickling problem, and the package author won't have to deal with the bug report and fix. But what about other Python implementations? Well, TBH, they have plenty of other issues. The reality is that you can't take a large package that hasn't been tested on Jython, or IronPython, or PyPy, and expect it to just work on any of those. Sure, things are getting better. But there are still tons of differences between the various Python implementations (as there are between different versions of CPython), and whether you like it or not, CPython is still the Python version of choice for most people. The long and short of it is that porting any significant package to another implementation is a bit of work, and keeping the port working probably requires adding some set of additional line items to the style guide used by its developers -- don't use feature X, don't depend on behavior Y, always use pattern Z... However, I don't expect that "always pass module=__name__ when using the enum convenience API" won't have to be added to that list. sys._getframe() is way more powerful than what's needed. Even on platforms where sys._getframe() is unimplementable (or disabled by default in favor of a 10% speedup), it should still be possible to provide an API that *just* gets the module name of the caller, at least in case the call site is top-level module code (and for anything else, the getframe hack doesn't work anyway). After all, we're talking about a full Python implementation, right? That's a dynamic language with lots of introspection APIs, and any implementation worth its salt will have to deal with that, even if full-fledged sys._getframe() is explicitly excluded. So I propose that we use sys._getframe() for the time being, and the authors of Jython, IronPython and PyPy can get together and figure out how to implement something like sys.get_calling_module_name(). They have plenty of time -- at least until they pledge support for Python 3.4. To make it easy for them we should probably add that API to CPython 3.4. And it's fine with me if the function only works if the caller is top-level code in a module. Which reminds me. Nick offered another use case where using sys._getframe() breaks down: a wrapper function that constructs an enum for its caller. First of all, I think this is a pretty rare use case. But additionally, that wrapper could just use sys.get_calling_module_name(), and everything would be fine. PS. Whoever adds sys.get_calling_module_name() to CPython, please pick a shorter name. :-) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Alex Gaynor added the comment:

From PyPy's perspective we don't really care what you name this particular bikeshed, and it's probably not that important to us (in this particular case).

As far as I know IronPython is the only Python VM that doesn't have _getframe() support by default (you need a CLI flag at startup, IIRC). In PyPy calling _getframe (right now) has a negative performance impact, but it's local to wherever you're calling it, so unless you create Enum in frequent loops (as opposed to just once at module level), this doesn't really matter to us. If someone wants to invent a more narrow, tailored API for stack introspection, logging is a much better target (right now each call to logging.{info,error,etc.} calls _getframe(), which does have a real performance impact. (We'll eventually invent enough engineering to fix this ourselves, but in the meantime if somewhere were to pain this bikeshed, we certainly wouldn't object ;)). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: Got the pickle issues worked out. Added super to the metaclass' __new__. Checking for illegal names of members and raising ValueError if any are found (I know, I know, safety checks! But such an enum is broken from the getgo so I see no reason to allow those names through). Thanks everyone for the excellent feed back. I really appreciate it. Hopefully we're almost done! :) ---------- assignee: -> ethan.furman Added file: http://bugs.python.org/file30265/pep-0435.06.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Donald Stufft added the comment: Small nitpick, weakref is imported but not used in the latest patch. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Eli Bendersky added the comment: I tweaked the code a bit (no functionality changes, mostly cleanups and a bit of refactoring). Figured it will be easier to just send an updated patch than another review. The diff from patch 06 can be seen via the Rietveld interface. Also, after reading the code more carefully, I think we're doing a mistake by over-complicating it for the sake of custom enum metaclasses and over-customization (like auto numbering). The original point Guido raised against auto-numbering was too much magic in the implementation. Well, we already have that in Lib/enum.py - the code is so complex it seems fragile because of the tight coupling with many class and metaclass related protocols. Just defining a wholly new enum implementation that does something very specific seems simpler than customizing the existing one. I'd suggest we stick to the existing Enum + IntEnum, giving up the more complex customizations for now. It can always be added in the future if we see it's very important. ---------- Added file: http://bugs.python.org/file30384/pep-0435.07.eliben.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Eli Bendersky added the comment: Thanks Ethan :)

From my point of view this is LGTM, as long as:

* There's ReST documentation * You remove the code to support extensions and customizations not mandated by PEP 435. As I mentioned before, this seems to be a YAGNI that complicates the code needlessly. It's find to keep the changes in some external repo and at a later point discuss their gradual addition similarly to the way Nick has been pushing enhancements through additional issues. We can always add more capabilities to Enum. We can "never" take them away once added, and this complicated code will remain with us forever even if no one ends up using it. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Guido van Rossum added the comment: Eli, what's wrong with having a backdoor? Python is literally *full* of backdoors. I have a feeling that somehow you are trying to build an Enum class that is unpythonic in its desire to enforce some kind of "ideal enum" behavior. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Nick Coghlan added the comment: Eli, remember that TOOWTDI stands for "There's one *obvious* way to do it" rather than "There's *only* one way to do it". The latter interpretation leads to insanely complex APIs that attempt to solve everyone's problems, while the former favours 80% solutions that cover most use cases, with extension hooks that let people handle the other 20% as they see fit. The point of the enum standardisation is to have a conventional way that enums *behave*, and then allow variations on that theme for those cases where the stdlib implementation is "close, but not quite what I need or want". The whole metaclass machinery is built around this concept of letting people create domain specific behaviour, that is still somewhat unified due to conventions like the descriptor protocol. You can do a *lot* with just descriptors, so if you don't need a custom metaclass, you shouldn't use one. PEP 422's class initialisation hook is aimed specifically at certain cases that currently need a metaclass and providing a simpler way to do them that lets you just use "type" as the metaclass instead. It's the same with enums - if you don't need to customise the metaclass, you shouldn't. But there are some use cases (such as syncing the Python level enum definition with a database level one) where additional customisation will be needed. We also want to give people the freedom they need to experiment with different forms of definition time syntactic sugar to see if they can come up with one we like enough to add to the standard library in 3.5. Does documenting these definition time extension points constrain what we're allowed to do in the future? Yes, it does. But, at the same time, it takes a lot of pressure off us to add more features to the standard enum type over time - if people have niche use cases that aren't handled well by the standard solution (and we already know they do), we can point them at the supported extension interface and say "go for it". For the majority of users though, the standard enum type will work fine, just as ordinary classes are adequate for the vast majority of object oriented code. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: Working on documentation... ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Changes by Ethan Furman <ethan@stoneleaf.us>: Removed file: http://bugs.python.org/file30504/pep-0435.08.stoneleaf.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Changes by Ethan Furman <ethan@stoneleaf.us>: Removed file: http://bugs.python.org/file30505/pep-0435.08.stoneleaf.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: So, which is better? To have a @unique class decorator as part of the module, or to have a UniqueEnum recipe in the docs? A decorator is immediately usable, but requires remembering an extra line of code. An example requires being put into a local utility module, but also serves as a guide to customising Enums. (Took be about five tries to get it right. :/ ) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: Good idea, thanks. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: Doc updates are in. I removed the 'unique, constant' from the first line of the intro, as neither of those things are necessarily true. ---------- Added file: http://bugs.python.org/file30511/pep-0435.10.stoneleaf.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Ethan Furman added the comment: Hopefully the last update. :) ---------- Added file: http://bugs.python.org/file30512/pep-0435.11.stoneleaf.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Zachary Ware added the comment: enum.rst will need to be added to a table of contents page somewhere, I would guess possibly Development Tools (Doc/library/development.rst) or maybe Data Types (Doc/library/datatypes.rst). I would trust almost anybody else's opinion over mine on where it should go, though :) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Roundup Robot added the comment: New changeset fae92309c3be by Ethan Furman in branch 'default': Closes issue 17947. Adds PEP-0435 (Enum, IntEnum) to the stdlib. http://hg.python.org/cpython/rev/fae92309c3be ---------- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Roundup Robot added the comment: New changeset e7a01c7f69fe by Ethan Furman in branch 'default': Closes issue 17947. Adds PEP-0435 (Adding an Enum type to the Python standard library). http://hg.python.org/cpython/rev/e7a01c7f69fe ---------- resolution: -> fixed stage: commit review -> committed/rejected status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________

Nick Coghlan added the comment: Nicely done - you can also mark the PEP as Final now :) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue17947> _______________________________________