Mailman 3 April 2013 - Python-Dev

PEP 435 -- Adding an Enum type to the Python standard library
by Eli Bendersky 30 Apr '13

30 Apr '13

Hello python-dev, We're happy to present the revised PEP 435, collecting valuable feedback from python-ideas discussions as well as in-person discussions and decisions made during the latest PyCon language summit. We believe the proposal is now better than the original one, providing both a wider set of features and more convenient ways to use those features. Link to the PEP: http://www.python.org/dev/peps/pep-0435/ [it's also pasted fully below for convenience]. Reference implementation is available as the recently released flufl.enum version 4.0 - you can get it either from PyPi or https://launchpad.net/flufl.enum. flufl.enum 4.0 was developed in parallel with revising PEP 435. Comments welcome, Barry and Eli ---------------------------------- PEP: 435 Title: Adding an Enum type to the Python standard library Version: $Revision$ Last-Modified: $Date$ Author: Barry Warsaw <barry(a)python.org>, Eli Bendersky <eliben(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2013-02-23 Python-Version: 3.4 Post-History: 2013-02-23 Abstract ======== This PEP proposes adding an enumeration type to the Python standard library. Specifically, it proposes moving the existing ``flufl.enum`` package by Barry Warsaw into the standard library. Much of this PEP is based on the "using" [1]_ document from the documentation of ``flufl.enum``. An enumeration is a set of symbolic names bound to unique, constant values. Within an enumeration, the values can be compared by identity, and the enumeration itself can be iterated over. Decision ======== TODO: update decision here once pronouncement is made. Status of discussions ===================== The idea of adding an enum type to Python is not new - PEP 354 [2]_ is a previous attempt that was rejected in 2005. Recently a new set of discussions was initiated [3]_ on the ``python-ideas`` mailing list. Many new ideas were proposed in several threads; after a lengthy discussion Guido proposed adding ``flufl.enum`` to the standard library [4]_. During the PyCon 2013 language summit the issue was discussed further. It became clear that many developers want to see an enum that subclasses ``int``, which can allow us to replace many integer constants in the standard library by enums with friendly string representations, without ceding backwards compatibility. An additional discussion among several interested core developers led to the proposal of having ``IntEnum`` as a special case of ``Enum``. The key dividing issue between ``Enum`` and ``IntEnum`` is whether comparing to integers is semantically meaningful. For most uses of enumerations, it's a **feature** to reject comparison to integers; enums that compare to integers lead, through transitivity, to comparisons between enums of unrelated types, which isn't desirable in most cases. For some uses, however, greater interoperatiliby with integers is desired. For instance, this is the case for replacing existing standard library constants (such as ``socket.AF_INET``) with enumerations. This PEP is an attempt to formalize this decision as well as discuss a number of variations that were discussed and can be considered for inclusion. Motivation ========== *[Based partly on the Motivation stated in PEP 354]* The properties of an enumeration are useful for defining an immutable, related set of constant values that have a defined sequence but no inherent semantic meaning. Classic examples are days of the week (Sunday through Saturday) and school assessment grades ('A' through 'D', and 'F'). Other examples include error status values and states within a defined process. It is possible to simply define a sequence of values of some other basic type, such as ``int`` or ``str``, to represent discrete arbitrary values. However, an enumeration ensures that such values are distinct from any others including, importantly, values within other enumerations, and that operations without meaning ("Wednesday times two") are not defined for these values. It also provides a convenient printable representation of enum values without requiring tedious repetition while defining them (i.e. no ``GREEN = 'green'``). Module and type name ==================== We propose to add a module named ``enum`` to the standard library. The main type exposed by this module is ``Enum``. Hence, to import the ``Enum`` type user code will run:: >>> from enum import Enum Proposed semantics for the new enumeration type =============================================== Creating an Enum ---------------- Enumerations are created using the class syntax, which makes them easy to read and write. An alternative creation method is described in `Convenience API`_. To define an enumeration, derive from the ``Enum`` class and add attributes with assignment to their integer values:: >>> from enum import Enum >>> class Colors(Enum): ... red = 1 ... green = 2 ... blue = 3 Enumeration values have nice, human readable string representations:: >>> print(Colors.red) Colors.red ...while their repr has more information:: >>> print(repr(Colors.red)) <EnumValue: Colors.red [value=1]> The enumeration value names are available through the class members:: >>> for member in Colors.__members__: ... print(member) red green blue Let's say you wanted to encode an enumeration value in a database. You might want to get the enumeration class object from an enumeration value:: >>> cls = Colors.red.enum >>> print(cls.__name__) Colors Enums also have a property that contains just their item name:: >>> print(Colors.red.name) red >>> print(Colors.green.name) green >>> print(Colors.blue.name) blue The str and repr of the enumeration class also provides useful information:: >>> print(Colors) <Colors {red: 1, green: 2, blue: 3}> >>> print(repr(Colors)) <Colors {red: 1, green: 2, blue: 3}> The ``Enum`` class supports iteration. Iteration is defined as the sorted order of the item values:: >>> class FiveColors(Enum): ... pink = 4 ... cyan = 5 ... green = 2 ... blue = 3 ... red = 1 >>> [v.name for v in FiveColors] ['red', 'green', 'blue', 'pink', 'cyan'] Enumeration values are hashable, so they can be used in dictionaries and sets:: >>> apples = {} >>> apples[Colors.red] = 'red delicious' >>> apples[Colors.green] = 'granny smith' >>> apples {<EnumValue: Colors.green [value=2]>: 'granny smith', <EnumValue: Colors.red [value=1]>: 'red delicious'} To programmatically access enumeration values, use ``getattr``:: >>> getattr(Colors, 'red') <EnumValue: Colors.red [value=1]> Comparisons ----------- Enumeration values are compared by identity:: >>> Colors.red is Colors.red True >>> Colors.blue is Colors.blue True >>> Colors.red is not Colors.blue True >>> Colors.blue is Colors.red False Ordered comparisons between enumeration values are *not* supported. Enums are not integers (but see `IntEnum`_ below):: >>> Colors.red < Colors.blue Traceback (most recent call last): ... NotImplementedError >>> Colors.red <= Colors.blue Traceback (most recent call last): ... NotImplementedError >>> Colors.blue > Colors.green Traceback (most recent call last): ... NotImplementedError >>> Colors.blue >= Colors.green Traceback (most recent call last): ... NotImplementedError Equality comparisons are defined though:: >>> Colors.blue == Colors.blue True >>> Colors.green != Colors.blue True Comparisons against non-enumeration values will always compare not equal:: >>> Colors.green == 2 False >>> Colors.blue == 3 False >>> Colors.green != 3 True >>> Colors.green == 'green' False Extending enumerations by subclassing ------------------------------------- You can extend previously defined Enums by subclassing:: >>> class MoreColors(Colors): ... pink = 4 ... cyan = 5 When extended in this way, the base enumeration's values are identical to the same named values in the derived class:: >>> Colors.red is MoreColors.red True >>> Colors.blue is MoreColors.blue True However, these are not doing comparisons against the integer equivalent values, because if you define an enumeration with similar item names and integer values, they will not be identical:: >>> class OtherColors(Enum): ... red = 1 ... blue = 2 ... yellow = 3 >>> Colors.red is OtherColors.red False >>> Colors.blue is not OtherColors.blue True These enumeration values are not equal, nor do they and hence may exist in the same set, or as distinct keys in the same dictionary:: >>> Colors.red == OtherColors.red False >>> len(set((Colors.red, OtherColors.red))) 2 You may not define two enumeration values with the same integer value:: >>> class Bad(Enum): ... cartman = 1 ... stan = 2 ... kyle = 3 ... kenny = 3 # Oops! ... butters = 4 Traceback (most recent call last): ... ValueError: Conflicting enums with value '3': 'kenny' and 'kyle' You also may not duplicate values in derived enumerations:: >>> class BadColors(Colors): ... yellow = 4 ... chartreuse = 2 # Oops! Traceback (most recent call last): ... ValueError: Conflicting enums with value '2': 'green' and 'chartreuse' Enumeration values ------------------ The examples above use integers for enumeration values. Using integers is short and handy (and provided by default by the `Convenience API`_), but not strictly enforced. In the vast majority of use-cases, one doesn't care what the actual value of an enumeration is. But if the value *is* important, enumerations can have arbitrary values. The following example uses strings:: >>> class SpecialId(Enum): ... selector = '$IM($N)' ... adaptor = '~$IM' ... >>> SpecialId.selector <EnumValue: SpecialId.selector [value=$IM($N)]> >>> SpecialId.selector.value '$IM($N)' >>> a = SpecialId.adaptor >>> a == '~$IM' False >>> a == SpecialId.adaptor True >>> print(a) SpecialId.adaptor >>> print(a.value) ~$IM Here ``Enum`` is used to provide readable (and syntactically valid!) names for some special values, as well as group them together. While ``Enum`` supports this flexibility, one should only use it in very special cases. Code will be most readable when actual values of enumerations aren't important and enumerations are just used for their naming and comparison properties. IntEnum ------- A variation of ``Enum`` is proposed where the enumeration values also subclasses ``int`` - ``IntEnum``. These values can be compared to integers; by extension, enumerations of different types can also be compared to each other:: >>> from enum import IntEnum >>> class Shape(IntEnum): ... circle = 1 ... square = 2 ... >>> class Request(IntEnum): ... post = 1 ... get = 2 ... >>> Shape == 1 False >>> Shape.circle == 1 True >>> Shape.circle == Request.post True However they still can't be compared to ``Enum``:: >>> class Shape(IntEnum): ... circle = 1 ... square = 2 ... >>> class Colors(Enum): ... red = 1 ... green = 2 ... >>> Shape.circle == Colors.red False ``IntEnum`` values behave like integers in other ways you'd expect:: >>> int(Shape.circle) 1 >>> ['a', 'b', 'c'][Shape.circle] 'b' >>> [i for i in range(Shape.square)] [0, 1] For the vast majority of code, ``Enum`` is strongly recommended. Since ``IntEnum`` breaks some semantic promises of an enumeration (by being comparable to integers, and thus by transitivity to other unrelated enumerations), it should be used only in special cases where there's no other choice; for example, when integer constants are replaced with enumerations and backwards compatibility is required with code that still expects integers. Pickling -------- Enumerations created with the class syntax can also be pickled and unpickled:: >>> from enum.tests.fruit import Fruit >>> from pickle import dumps, loads >>> Fruit.tomato is loads(dumps(Fruit.tomato)) True Convenience API --------------- The ``Enum`` class is callable, providing the following convenience API:: >>> Animals = Enum('Animals', 'ant bee cat dog') >>> Animals <Animals {ant: 1, bee: 2, cat: 3, dog: 4}> >>> Animals.ant <EnumValue: Animals.ant [value=1]> >>> Animals.ant.value 1 The semantics of this API resemble ``namedtuple``. The first argument of the call to ``Enum`` is the name of the enumeration. The second argument is a source of enumeration value names. It can be a whitespace-separated string of names, a sequence of names or a sequence of 2-tuples with key/value pairs. The last option enables assigning arbitrary values to enumerations; the others auto-assign increasing integers starting with 1. A new class derived from ``Enum`` is returned. In other words, the above assignment to ``Animals`` is equivalent to:: >>> class Animals(Enum): ... ant = 1 ... bee = 2 ... cat = 3 ... dog = 4 Examples of alternative name/value specifications:: >>> Enum('Animals', ['ant', 'bee', 'cat', 'dog']) <Animals {ant: 1, bee: 2, cat: 3, dog: 4}> >>> Enum('Animals', (('ant', 'one'), ('bee', 'two'), ('cat', 'three'), ('dog', 'four'))) <Animals {dog: four, ant: one, cat: three, bee: two}> The second argument can also be a dictionary mapping names to values:: >>> levels = dict(debug=10, info=20, warning=30, severe=40) >>> Enum('Levels', levels) <Levels {debug: 10, info: 20, warning: 30, severe: 40}> Proposed variations =================== Some variations were proposed during the discussions in the mailing list. Here's some of the more popular ones. Not having to specify values for enums -------------------------------------- Michael Foord proposed (and Tim Delaney provided a proof-of-concept implementation) to use metaclass magic that makes this possible:: class Color(Enum): red, green, blue The values get actually assigned only when first looked up. Pros: cleaner syntax that requires less typing for a very common task (just listing enumeration names without caring about the values). Cons: involves much magic in the implementation, which makes even the definition of such enums baffling when first seen. Besides, explicit is better than implicit. Using special names or forms to auto-assign enum values ------------------------------------------------------- A different approach to avoid specifying enum values is to use a special name or form to auto assign them. For example:: class Color(Enum): red = None # auto-assigned to 0 green = None # auto-assigned to 1 blue = None # auto-assigned to 2 More flexibly:: class Color(Enum): red = 7 green = None # auto-assigned to 8 blue = 19 purple = None # auto-assigned to 20 Some variations on this theme: #. A special name ``auto`` imported from the enum package. #. Georg Brandl proposed ellipsis (``...``) instead of ``None`` to achieve the same effect. Pros: no need to manually enter values. Makes it easier to change the enum and extend it, especially for large enumerations. Cons: actually longer to type in many simple cases. The argument of explicit vs. implicit applies here as well. Use-cases in the standard library ================================= The Python standard library has many places where the usage of enums would be beneficial to replace other idioms currently used to represent them. Such usages can be divided to two categories: user-code facing constants, and internal constants. User-code facing constants like ``os.SEEK_*``, ``socket`` module constants, decimal rounding modes and HTML error codes could require backwards compatibility since user code may expect integers. ``IntEnum`` as described above provides the required semantics; being a subclass of ``int``, it does not affect user code that expects integers, while on the other hand allowing printable representations for enumeration values:: >>> import socket >>> family = socket.AF_INET >>> family == 2 True >>> print(family) SocketFamily.AF_INET Internal constants are not seen by user code but are employed internally by stdlib modules. These can be implemented with ``Enum``. Some examples uncovered by a very partial skim through the stdlib: ``binhex``, ``imaplib``, ``http/client``, ``urllib/robotparser``, ``idlelib``, ``concurrent.futures``, ``turtledemo``. In addition, looking at the code of the Twisted library, there are many use cases for replacing internal state constants with enums. The same can be said about a lot of networking code (especially implementation of protocols) and can be seen in test protocols written with the Tulip library as well. Differences from PEP 354 ======================== Unlike PEP 354, enumeration values are not defined as a sequence of strings, but as attributes of a class. This design was chosen because it was felt that class syntax is more readable. Unlike PEP 354, enumeration values require an explicit integer value. This difference recognizes that enumerations often represent real-world values, or must interoperate with external real-world systems. For example, to store an enumeration in a database, it is better to convert it to an integer on the way in and back to an enumeration on the way out. Providing an integer value also provides an explicit ordering. However, there is no automatic conversion to and from the integer values, because explicit is better than implicit. Unlike PEP 354, this implementation does use a metaclass to define the enumeration's syntax, and allows for extended base-enumerations so that the common values in derived classes are identical (a singleton model). While PEP 354 dismisses this approach for its complexity, in practice any perceived complexity, though minimal, is hidden from users of the enumeration. Unlike PEP 354, enumeration values should only be tested by identity comparison. This is to emphasize the fact that enumeration values are singletons, much like ``None``. Acknowledgments =============== This PEP describes the ``flufl.enum`` package by Barry Warsaw. ``flufl.enum`` is based on an example by Jeremy Hylton. It has been modified and extended by Barry Warsaw for use in the GNU Mailman [5]_ project. Ben Finney is the author of the earlier enumeration PEP 354. References ========== .. [1] http://pythonhosted.org/flufl.enum/docs/using.html .. [2] http://www.python.org/dev/peps/pep-0354/ .. [3] http://mail.python.org/pipermail/python-ideas/2013-January/019003.html .. [4] http://mail.python.org/pipermail/python-ideas/2013-February/019373.html .. [5] http://www.list.org Copyright ========= This document has been placed in the public domain. Todo ==== * Mark PEP 354 "superseded by" this one, if accepted .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

29 223

Regression fix releases coming
by Georg Brandl 30 Apr '13

30 Apr '13

Hi all, since there turned out to be a "critical mass" of regressions in 2.7.4/ 3.2.4/3.3.1, Benjamin and I would like to release a releasing fixing these next weekend. Since we would like to cherry-pick only regression fixes into these releases, please make sure any regression issues you know of are set to "open" (even if fixed in the main repo) and status "release blocker" for the appropriate versions. If they aren't fixed yet, it would also be nice to have a patch ready by Saturday :) Thanks, Georg

4 4

Getting a list of registered codecs
by Paul Moore 30 Apr '13

30 Apr '13

Before I raise a bug for this, can someone confirm if I've simply missed something? I don't see any way, either in the docs or in the helpstrings from the codecs, of listing the codecs that have been registered. FWIW, I picked this up when I was looking at writing a simple encoding converter, and I wanted to add a flag to list what conversions were supported. Paul.

3 4

Enumeration item arguments?
by Ethan Furman 29 Apr '13

29 Apr '13

In the Planet example we saw the possibility of specifying arguments to enum item __init__: class Planet(Enum): MERCURY = (3.303e+23, 2.4397e6) VENUS = (4.869e+24, 6.0518e6) EARTH = (5.976e+24, 6.37814e6) MARS = (6.421e+23, 3.3972e6) JUPITER = (1.9e+27, 7.1492e7) SATURN = (5.688e+26, 6.0268e7) URANUS = (8.686e+25, 2.5559e7) NEPTUNE = (1.024e+26, 2.4746e7) def __init__(self, mass, radius): self.mass = mass # in kilograms self.radius = radius # in meters Do we want to support this? -- ~Ethan~

8 11

Purpose of files in $(DESTDIR)$(LIBPL)
by Bohuslav Kabrda 29 Apr '13

29 Apr '13

Hi, I'd like to ask about the purpose of files in $(DESTDIR)$(LIBPL) [1] - what is the reason to keep them/what are they useful for? I'm currently "taking over" Python packaging in Fedora and I'd like to know if these have some meaning for a distro-packaged Python (Dave Malcolm is not sure about them ;)). Thanks, Slavek. -- Regards, Bohuslav "Slavek" Kabrda. [1] http://hg.python.org/cpython/file/3.3/Makefile.pre.in#l1206

2 1

Re: [Python-Dev] Enumeration items: `type(EnumClass.item) is EnumClass` ?
by Adrian Sampson 29 Apr '13

29 Apr '13

On Apr 29, 2013, at 7:32 AM, Ethan Furman wrote: > On 04/28/2013 01:02 PM, Guido van Rossum wrote: >> On Sun, Apr 28, 2013 at 12:32 PM, Ethan Furman wrote: >> - should enum items be of the type of the Enum class? (i.e. type(SPRING) >> is Seasons) > > IMO Yes. This decision seems natural to me, so I wrote an enumeration library some time ago that uses a simple metaclass to achieve "type(Season.spring) is Season": https://github.com/sampsyo/beets/blob/master/beets/util/enumeration.py The module has other warts but perhaps it can be helpful anyway. Cheers, Adrian

1 0

Enums and data retrieval
by Ethan Furman 29 Apr '13

29 Apr '13

[starting new thread to not pollute the summary thread] On 04/28/2013 11:54 PM, Antoine Pitrou wrote:> On Sun, 28 Apr 2013 17:29:35 -0700 > Ethan Furman <ethan(a)stoneleaf.us> wrote: >> >> Not only is this inconsistent with the rest of Python*, but it's going to be a PITA for data storage/retrieval: >> >> datastore = dbf.Table('storage.dbf', 'event_name C(50); date D; season SEASON') >> >> def retrieve_record(...): >> result = [] >> for field_type, field_data in record: >> result.append(field_type(field_data)) > > I've never seen any kind of "data retrieval" which works like that. > Would you care to explain us the context? The more specific context would be my dbf package, which works with dBase III, Clipper, and Foxpro tables. When the fields of a record are requested they are transformed into Python data types, with code that looks pretty much like that retrieve_record snippet (w/o all the error checks, etc.). And no, it doesn't support enumerations (yet). A more general context would be anywhere that you need to convert the integer offset of an enum item back into the enum item itself; you may have gotten the integer offset from a postgres database, or an RPC call, and it seems to me the natural way get the enum item from that is with `EnumClass(offset)`. -- ~Ethan~

2 1

class name spaces inside an outer function
by Ethan Furman 28 Apr '13

28 Apr '13

I filed bug http://bugs.python.org/issue17853 last night. If somebody could point me in the right direction (mainly which files to look in), I'd be happy to attempt a patch. -- ~Ethan~

6 11

Re: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17357: Use more stern wording for
by Brett Cannon 27 Apr '13

27 Apr '13

This actually should have been for issue #17330, but I had the wrong bug open when I copy-and-pasted the number. On Sat, Apr 27, 2013 at 11:21 PM, brett.cannon <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/75e32a0bfd74 > changeset: 83525:75e32a0bfd74 > branch: 3.3 > parent: 83517:4b4ed1e11fd0 > user: Brett Cannon <brett(a)python.org> > date: Sat Apr 27 23:20:32 2013 -0400 > summary: > Issue #17357: Use more stern wording for > importlib.invalidate_caches(). > > files: > Doc/library/importlib.rst | 10 +++++----- > 1 files changed, 5 insertions(+), 5 deletions(-) > > > diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst > --- a/Doc/library/importlib.rst > +++ b/Doc/library/importlib.rst > @@ -103,9 +103,9 @@ > > Invalidate the internal caches of finders stored at > :data:`sys.meta_path`. If a finder implements ``invalidate_caches()`` then it > - will be called to perform the invalidation. This function may be needed if > - some modules are installed while your program is running and you expect the > - program to notice the changes. > + will be called to perform the invalidation. This function should be called > + if any modules are created/installed while your program is running to > + guarantee all finders will notice the new module's existence. > > .. versionadded:: 3.3 > > @@ -182,7 +182,7 @@ > > .. versionadded:: 3.3 > > - .. method:: find_loader(fullname): > + .. method:: find_loader(fullname) > > An abstract method for finding a :term:`loader` for the specified > module. Returns a 2-tuple of ``(loader, portion)`` where ``portion`` > @@ -194,7 +194,7 @@ > the empty list then no loader or location for a namespace package were > found (i.e. failure to find anything for the module). > > - .. method:: find_module(fullname): > + .. method:: find_module(fullname) > > A concrete implementation of :meth:`Finder.find_module` which is > equivalent to ``self.find_loader(fullname)[0]``. > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

1 0

Questions about Codon Alignment Proposal
by 阮铮 27 Apr '13

27 Apr '13

Hi Eric and Peter, I'm preparing the proposal for the codon alignment project. Two things I may want to hear your advice. 1) In the biopython wiki page, you mentioned "model selection" in the Approach & Goals. I'm not sure if there are any advantages to use codon alignment for model selection. Could you give me some references? Another thing is that model selection involves estimation of tree topology as well as branch lengthes and parameters across many substitution models. Will it be too computationally intensive for a python implementation? 2) You also mentioned the "validation (testing for frame shift)". Is there a test for frame shift? Or I can simply detect it by comparing amino acid sequences and nucleotide sequences. Best, Zheng Ruan

3 2