From steve at pearwood.info  Fri Feb  1 00:23:50 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 01 Feb 2013 10:23:50 +1100
Subject: [Python-ideas] Definition Symmetry
In-Reply-To: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
References: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
Message-ID: <510AFD06.1090305@pearwood.info>

On 01/02/13 03:35, Jason Keene wrote:
> Why do function definitions require parens?
>
>>>> class MyClass:
> ...     pass
> ...
>>>> def my_func:
>    File "<stdin>", line 1
>      def my_func:
>                 ^
> SyntaxError: invalid syntax
>
> This seems to me to break a symmetry with class definitions.

Why do you think that class definitions and function definitions
should be symmetrical?



-- 
Steven


From steve at pearwood.info  Fri Feb  1 00:26:33 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 01 Feb 2013 10:26:33 +1100
Subject: [Python-ideas] Extend module objects to support properties
In-Reply-To: <CAPTjJmrT_kvEN2L5g=nWNCdE+ktaxhK5SjYwE1OcWBQgdQVQiw@mail.gmail.com>
References: <51087225.3040801@hastings.org>
	<CADiSq7fRZhD8kWoHfpxg7HKriVr6qET6QRvCLGirXYdsUJHEfA@mail.gmail.com>
	<51094D8D.606@hastings.org> <5109BEC4.4050604@pearwood.info>
	<5109CE95.7060104@hastings.org>
	<CAPTjJmrT_kvEN2L5g=nWNCdE+ktaxhK5SjYwE1OcWBQgdQVQiw@mail.gmail.com>
Message-ID: <510AFDA9.9080507@pearwood.info>

On 01/02/13 09:00, Chris Angelico wrote:
> On Thu, Jan 31, 2013 at 12:53 PM, Larry Hastings<larry at hastings.org>  wrote:
>> But if the user assigns a different value to math.__dict__['pi'], math.pi
>> will diverge, which again could break code.  (Who might try to assign a
>> different value to pi?  The 1897 House Of Representatives of Indiana for
>> one!)
>>
>>
>> More generally, it's often useful to monkeypatch "constants" at runtime, for
>> testing purposes (and for less justifiable purposes).  Why prevent that?  I
>> cite the Consenting Adults rule.
>
> I've never actually been in the situation of doing it, but wouldn't it
> be reasonable to switch out math.pi to be (say) a decimal.Decimal
> rather than a float?


To what purpose? Monkey-patching math.pi doesn't affect any of the other math
functions. All it will do is break Python code that uses math.pi and expects
it to be a float.



-- 
Steven


From zuo at chopin.edu.pl  Fri Feb  1 00:27:07 2013
From: zuo at chopin.edu.pl (=?utf-8?B?SmFuIEthbGlzemV3c2tp?=)
Date: Fri, 01 Feb 2013 00:27:07 +0100
Subject: [Python-ideas] =?utf-8?q?Definition_Symmetry?=
Message-ID: <20130131232712.70B9BD60F5@filifionka.chopin.edu.pl>

First of all, the class definition concept and the function definition concept should *not* be considered 'symmetric'. They are fundamentally different in several aspects - especially the body of a class definition is executed once, at the *definition* time (and *before* the actual class is made) and the body of a function definition can be executed multiple times, at the *call* time (and obviously *after* the function is made).

Regards.
*j

-- 
[sent from phone]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/7272fcee/attachment.html>

From rosuav at gmail.com  Fri Feb  1 00:36:51 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 1 Feb 2013 10:36:51 +1100
Subject: [Python-ideas] Extend module objects to support properties
In-Reply-To: <510AFDA9.9080507@pearwood.info>
References: <51087225.3040801@hastings.org>
	<CADiSq7fRZhD8kWoHfpxg7HKriVr6qET6QRvCLGirXYdsUJHEfA@mail.gmail.com>
	<51094D8D.606@hastings.org> <5109BEC4.4050604@pearwood.info>
	<5109CE95.7060104@hastings.org>
	<CAPTjJmrT_kvEN2L5g=nWNCdE+ktaxhK5SjYwE1OcWBQgdQVQiw@mail.gmail.com>
	<510AFDA9.9080507@pearwood.info>
Message-ID: <CAPTjJmqZrTNLEvdidy_t8H7aEfNjGJom0rTP2n7tTQGK2NytNQ@mail.gmail.com>

On Fri, Feb 1, 2013 at 10:26 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On 01/02/13 09:00, Chris Angelico wrote:
>> I've never actually been in the situation of doing it, but wouldn't it
>> be reasonable to switch out math.pi to be (say) a decimal.Decimal
>> rather than a float?
>
> To what purpose? Monkey-patching math.pi doesn't affect any of the other
> math
> functions. All it will do is break Python code that uses math.pi and expects
> it to be a float.

True. I guess you'd just have your own local pi for that. (Your own
local bakery maybe?) Consider the notion withdrawn.

ChrisA


From ncoghlan at gmail.com  Fri Feb  1 01:50:03 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 1 Feb 2013 10:50:03 +1000
Subject: [Python-ideas] Extend module objects to support properties
In-Reply-To: <CALFfu7CTxbGQwdr9+oaChcvUD_Pd0c5S2hxEcQZSfTNPqjawXg@mail.gmail.com>
References: <51087225.3040801@hastings.org>
	<CADiSq7fRZhD8kWoHfpxg7HKriVr6qET6QRvCLGirXYdsUJHEfA@mail.gmail.com>
	<51094D8D.606@hastings.org>
	<CADiSq7cj8xLPXa-YjE-H9DEzXRbgjhm_B40ELQW8QjfGT3+htA@mail.gmail.com>
	<CALFfu7CTxbGQwdr9+oaChcvUD_Pd0c5S2hxEcQZSfTNPqjawXg@mail.gmail.com>
Message-ID: <CADiSq7e4puwp9=JBG8f4ceVmH148Jkv5yWs2eDKKN1214_WMJg@mail.gmail.com>

On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> That said, Larry's original proposal relates to sys, a built-in module
> written in C (in CPython of course).  In that case the solution is not
> quite the same, since module initialization interacts with sys.modules
> differently. [1][2]  Accommodating the original request would require
> more work, whether to muck with the import C-API or making sys an
> instance of another type, as someone suggested.

sys is already special cased so heavily in the interpreter
initialization, making it a little more special really wouldn't bother
me much :)

As far as customising other extension modules goes, the same "replace
yourself in sys.modules" trick should still work, you'd just do it via
the C API from your module init function.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From tjreedy at udel.edu  Fri Feb  1 02:07:27 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 31 Jan 2013 20:07:27 -0500
Subject: [Python-ideas] Definition Symmetry
In-Reply-To: <20130131160657.4620f918@anarchist.wooz.org>
References: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
	<510AC6F1.1060503@ei-grad.ru>
	<20130131160657.4620f918@anarchist.wooz.org>
Message-ID: <kef4gp$koh$1@ger.gmane.org>

On 1/31/2013 4:06 PM, Barry Warsaw wrote:
> On Jan 31, 2013, at 11:33 PM, Andrew Grigorev wrote:
>
>> Other strange thing is that the `raise` statement doesn't require to
>> instantiate an Exception object, allowing to pass an Exception class to it.
>>
>> raise NotImplementedError
>> raise NotImplementedError()
>>
>> Is there any difference between this two lines of code?

Questions like this below on python-list, not here.

> The main difference (I *think* this is still true) is that in the first
> example, if the exception is caught in C it can avoid instantiation.

The the second form allows addition of a message, which is usually a 
good idea.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Fri Feb  1 02:34:20 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 31 Jan 2013 20:34:20 -0500
Subject: [Python-ideas] Extend module objects to support properties
In-Reply-To: <CAPTjJmrT_kvEN2L5g=nWNCdE+ktaxhK5SjYwE1OcWBQgdQVQiw@mail.gmail.com>
References: <51087225.3040801@hastings.org>
	<CADiSq7fRZhD8kWoHfpxg7HKriVr6qET6QRvCLGirXYdsUJHEfA@mail.gmail.com>
	<51094D8D.606@hastings.org> <5109BEC4.4050604@pearwood.info>
	<5109CE95.7060104@hastings.org>
	<CAPTjJmrT_kvEN2L5g=nWNCdE+ktaxhK5SjYwE1OcWBQgdQVQiw@mail.gmail.com>
Message-ID: <kef637$so$1@ger.gmane.org>

On 1/31/2013 5:00 PM, Chris Angelico wrote:

> I've never actually been in the situation of doing it, but wouldn't it
> be reasonable to switch out math.pi to be (say) a decimal.Decimal
> rather than a float?

Mixed float-decimal arithmetic does not work. If the float functions in 
math used 'math.pi', that would not work, but they do not. On the other 
hand, the stdlib and test suite do have a few expressions like 2.0 * 
math.pi and math.pi / 180.0 that would have to be changed to 2 * math.pi 
and math.pi / 180. (This might be a reasonable change anyway.) Non-float 
inputs to math functions like sin are apparently converted to float when 
possible (I tested int, Decimal, and Fraction with sin). So if you can 
otherwise avoid float +-*/ decimal, the answer might be yes.

-- 
Terry Jan Reedy



From guido at python.org  Fri Feb  1 02:58:32 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 31 Jan 2013 17:58:32 -0800
Subject: [Python-ideas] PEP 3156: getting the socket or peer name from
	the transport
In-Reply-To: <CANSw7KzG-ePHkuA4tv2LQ-+Fio2nWO0qv38WsH5QBY-JzDq3Eg@mail.gmail.com>
References: <CAP7+vJ+_QwQW13b1mE1NSg8Z89MyYwHkyYmLwhumGo0iO-V+wA@mail.gmail.com>
	<CANSw7Kyi7d-Q5O=MqZTx5a-bOA3CQ4Me5RVafxJ0e7Kj1w693w@mail.gmail.com>
	<CAP7+vJ+BxgVPv0jBP=9GAdnXGdLRqgjCcBhVzVnJ4K7NDAxrjQ@mail.gmail.com>
	<CANSw7KwafX+qGiMszK7LVMs=xacmyEUjGLDPdqGzfW95GxbXfg@mail.gmail.com>
	<20130127122121.6b779ada@pitrou.net>
	<CANSw7KyuPCg9Ot6tY3ML_WBquO0PfFVqpzWPRpa3o8gmTyVS_A@mail.gmail.com>
	<1359288997.3488.2.camel@localhost.localdomain>
	<CAP7+vJ+av5zRDDLThgo7CCn_xLJ7rY3u-myP6Hihf50j0z7pYQ@mail.gmail.com>
	<EDC83381-4C64-4215-A90B-C72F2327BCA7@umbrellacode.com>
	<CAP7+vJKn6hE1zWujnDi=5dUtRsdovM7741G9bK0e4vQJvmbDPA@mail.gmail.com>
	<CANSw7KzG-ePHkuA4tv2LQ-+Fio2nWO0qv38WsH5QBY-JzDq3Eg@mail.gmail.com>
Message-ID: <CAP7+vJKHqnGCcjMsGcc9A623qrkiJzG-tv1Q5VWBGsW25e8FUw@mail.gmail.com>

On Thu, Jan 31, 2013 at 12:51 AM, Yuval Greenfield
<ubershmekel at gmail.com> wrote:
> On Mon, Jan 28, 2013 at 5:45 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> Hm. I'm not keen on precomputing all of that, since most protocols
>> won't need it, and the cost add up. This is not WSGI. The protocol has
>> the transport object and can ask it specific questions -- if through a
>> general API, like get_extra_info(key, [default]).
>>
>
> I forgot to ask before, but why is get_extra_info better than normal
> attributes and methods?
>
> val = transport.get_extra_info(key, None)
> if val is None:
>     pass
>
> # vs
>
> if hasattr(transport, key):
>     val = transport.key
> else:
>     pass

hasattr() smells bad. It also has namespace issues (hasattr(transport,
"write") returns true) and if people forget to use it (perhaps because
the transport they normally use always has a certain attribute) their
code is brittle. Defining a new API with a string key signals clearly
that the value may or may not exist, and reminds people to test the
result for None. (Of course they can forget that too. But it still
feels different to me.)

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Fri Feb  1 03:00:13 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 31 Jan 2013 18:00:13 -0800
Subject: [Python-ideas] PEP 3156: getting the socket or peer name from
	the transport
In-Reply-To: <CANSw7KywHzL==_Z-gGG89O6a5i1ebpX1AGF5gL-X-iv6YxOCbg@mail.gmail.com>
References: <CAP7+vJ+_QwQW13b1mE1NSg8Z89MyYwHkyYmLwhumGo0iO-V+wA@mail.gmail.com>
	<CANSw7Kyi7d-Q5O=MqZTx5a-bOA3CQ4Me5RVafxJ0e7Kj1w693w@mail.gmail.com>
	<CAP7+vJ+BxgVPv0jBP=9GAdnXGdLRqgjCcBhVzVnJ4K7NDAxrjQ@mail.gmail.com>
	<CANSw7KwafX+qGiMszK7LVMs=xacmyEUjGLDPdqGzfW95GxbXfg@mail.gmail.com>
	<20130127122121.6b779ada@pitrou.net>
	<CANSw7KyuPCg9Ot6tY3ML_WBquO0PfFVqpzWPRpa3o8gmTyVS_A@mail.gmail.com>
	<1359288997.3488.2.camel@localhost.localdomain>
	<CAP7+vJ+av5zRDDLThgo7CCn_xLJ7rY3u-myP6Hihf50j0z7pYQ@mail.gmail.com>
	<EDC83381-4C64-4215-A90B-C72F2327BCA7@umbrellacode.com>
	<CAP7+vJKn6hE1zWujnDi=5dUtRsdovM7741G9bK0e4vQJvmbDPA@mail.gmail.com>
	<CANSw7KzG-ePHkuA4tv2LQ-+Fio2nWO0qv38WsH5QBY-JzDq3Eg@mail.gmail.com>
	<CANSw7KywHzL==_Z-gGG89O6a5i1ebpX1AGF5gL-X-iv6YxOCbg@mail.gmail.com>
Message-ID: <CAP7+vJJ7Jhekkx+EPW5LvS-HtUO4Gw9H3YQvURoWNu+q0qyAEg@mail.gmail.com>

On Thu, Jan 31, 2013 at 12:52 AM, Yuval Greenfield
<ubershmekel at gmail.com> wrote:
>>     val = getattr(transport, key)

You probably meant

  val = getattr(transport, "key", None)

Your original example also forgot the quotes around "key" in the
hasattr() call.) The number of attempts needed to get your example
right tells me this is not a good proposal. :-)

-- 
--Guido van Rossum (python.org/~guido)


From jasonkeene at gmail.com  Fri Feb  1 03:13:32 2013
From: jasonkeene at gmail.com (Jason Keene)
Date: Thu, 31 Jan 2013 21:13:32 -0500
Subject: [Python-ideas] Definition Symmetry
In-Reply-To: <510AFD06.1090305@pearwood.info>
References: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
	<510AFD06.1090305@pearwood.info>
Message-ID: <CAPAaX_ifjthen8JiLvKetmTDbmrzAUKydqXxKfvF+68feqthyg@mail.gmail.com>

In a way they do the same thing, they both create an object
(function/class) from a suite and assign it to the name given after the
keyword (def/class).  Sure they do totally different things with the suite
in creating the object, but in essence it's a name assignment.

On Thursday, January 31, 2013, Steven D'Aprano wrote:

> On 01/02/13 03:35, Jason Keene wrote:
>
>> Why do function definitions require parens?
>>
>>  class MyClass:
>>>>>
>>>> ...     pass
>> ...
>>
>>> def my_func:
>>>>>
>>>>    File "<stdin>", line 1
>>      def my_func:
>>                 ^
>> SyntaxError: invalid syntax
>>
>> This seems to me to break a symmetry with class definitions.
>>
>
> Why do you think that class definitions and function definitions
> should be symmetrical?
>
>
>
> --
> Steven
> ______________________________**_________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130131/73bb827b/attachment.html>

From ericsnowcurrently at gmail.com  Fri Feb  1 03:58:42 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 31 Jan 2013 19:58:42 -0700
Subject: [Python-ideas] Extend module objects to support properties
In-Reply-To: <CADiSq7e4puwp9=JBG8f4ceVmH148Jkv5yWs2eDKKN1214_WMJg@mail.gmail.com>
References: <51087225.3040801@hastings.org>
	<CADiSq7fRZhD8kWoHfpxg7HKriVr6qET6QRvCLGirXYdsUJHEfA@mail.gmail.com>
	<51094D8D.606@hastings.org>
	<CADiSq7cj8xLPXa-YjE-H9DEzXRbgjhm_B40ELQW8QjfGT3+htA@mail.gmail.com>
	<CALFfu7CTxbGQwdr9+oaChcvUD_Pd0c5S2hxEcQZSfTNPqjawXg@mail.gmail.com>
	<CADiSq7e4puwp9=JBG8f4ceVmH148Jkv5yWs2eDKKN1214_WMJg@mail.gmail.com>
Message-ID: <CALFfu7AfUs+nUtcu0dRHTZXSH+wWHJ3pqGCYkxcF5ODoPYLd7A@mail.gmail.com>

On Jan 31, 2013 5:50 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
> On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow <ericsnowcurrently at gmail.com>
wrote:
> > That said, Larry's original proposal relates to sys, a built-in module
> > written in C (in CPython of course).  In that case the solution is not
> > quite the same, since module initialization interacts with sys.modules
> > differently. [1][2]  Accommodating the original request would require
> > more work, whether to muck with the import C-API or making sys an
> > instance of another type, as someone suggested.
>
> sys is already special cased so heavily in the interpreter
> initialization, making it a little more special really wouldn't bother
> me much :)
>
> As far as customising other extension modules goes, the same "replace
> yourself in sys.modules" trick should still work, you'd just do it via
> the C API from your module init function.

Don't extension modules get added to sys.modules only after their init
function returns?

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130131/c1a345df/attachment.html>

From ncoghlan at gmail.com  Fri Feb  1 04:07:37 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 1 Feb 2013 13:07:37 +1000
Subject: [Python-ideas] Extend module objects to support properties
In-Reply-To: <CALFfu7AfUs+nUtcu0dRHTZXSH+wWHJ3pqGCYkxcF5ODoPYLd7A@mail.gmail.com>
References: <51087225.3040801@hastings.org>
	<CADiSq7fRZhD8kWoHfpxg7HKriVr6qET6QRvCLGirXYdsUJHEfA@mail.gmail.com>
	<51094D8D.606@hastings.org>
	<CADiSq7cj8xLPXa-YjE-H9DEzXRbgjhm_B40ELQW8QjfGT3+htA@mail.gmail.com>
	<CALFfu7CTxbGQwdr9+oaChcvUD_Pd0c5S2hxEcQZSfTNPqjawXg@mail.gmail.com>
	<CADiSq7e4puwp9=JBG8f4ceVmH148Jkv5yWs2eDKKN1214_WMJg@mail.gmail.com>
	<CALFfu7AfUs+nUtcu0dRHTZXSH+wWHJ3pqGCYkxcF5ODoPYLd7A@mail.gmail.com>
Message-ID: <CADiSq7d_2Y4m1fARV6fx5QzvttxM6oC51rP0BdtR6AvA9J4_QA@mail.gmail.com>

On Fri, Feb 1, 2013 at 12:58 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>
> On Jan 31, 2013 5:50 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>>
>> On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow <ericsnowcurrently at gmail.com>
>> wrote:
>> > That said, Larry's original proposal relates to sys, a built-in module
>> > written in C (in CPython of course).  In that case the solution is not
>> > quite the same, since module initialization interacts with sys.modules
>> > differently. [1][2]  Accommodating the original request would require
>> > more work, whether to muck with the import C-API or making sys an
>> > instance of another type, as someone suggested.
>>
>> sys is already special cased so heavily in the interpreter
>> initialization, making it a little more special really wouldn't bother
>> me much :)
>>
>> As far as customising other extension modules goes, the same "replace
>> yourself in sys.modules" trick should still work, you'd just do it via
>> the C API from your module init function.
>
> Don't extension modules get added to sys.modules only after their init
> function returns?

Oh, you're right, they're currently responsible for creating their own
module object (I was thinking of the way I *wished* extension module
imports worked). Returning the module object from the init function
still works though - we just need to make sure the extension loader
can cope with the init function returning a non-module object :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From timothy.c.delaney at gmail.com  Fri Feb  1 01:19:14 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Fri, 1 Feb 2013 11:19:14 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
Message-ID: <CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>

Wow - this ended up being more difficult that I'd anticipated. Ensuring
that decorators work in Michael Foord-inspired enums, etc mandated a fairly
big redesign. As it was you couldn't do something like:

class A():
    a = 1

class B(Enum):
    B = A.a

because when the name 'A' was resolved in the definition of class B, it was
returning an int (the next enum value).

The end result may well still have some holes, but it's looking pretty good
to me.

Enums are constructed like:

class Color(Enum):
    RED, GREEN, BLUE
    CYAN = 10
    MAGENTA
    YELLOW
    BLACK

where the assigned enum starts a new count i.e. the above is 0, 1, 2, 10,
11, 12, 13. Arbitrary attributes may be assigned and not contribute to the
enumeration so long as the object being assigned does not implement
__index__ (if it does, it creates a discontiguous enumeration).

#!/usr/bin/env python3

import builtins
import collections
import operator

class EnumValue(int):
    def __new__(cls, key, value):
        e = super().__new__(cls, value)
        super().__setattr__(e, 'key', key)
        super().__setattr__(e, 'owner', None)
        return e

    def __setattr__(self, key, value):
        raise TypeError("Cannot set attribute of type %r" % (type(self),))

    def __str__(self):
        return "%s.%s" % (self.owner.__name__, self.key)

    def __repr__(self):
        if self.owner is not None:
            return "<%s '%s.%s': %d>" % (self.__qualname__,
self.owner.__qualname__, self.key, int(self))

        return "<%s '%s': %d>" % (self.__qualname__, self.key, int(self))

class EnumProxy(object):
    def __init__(self, key, value=None):
        self.key = key
        self.values = [value]
        self.used = False

    def __repr__(self):
        return "<%s '%s': %s>" % (self.__qualname__, self.key, self.values)

    def _get(self, used=None):
        if used:
            self.used = True

        try:
            return locals()[self.key]
        except KeyError:
            try:
                return globals()[self.key]
            except KeyError:
                try:
                    return getattr(builtins, self.key)
                except KeyError:
                    raise NameError(self.key, self.values)

    def __call__(self, *p, **kw):
        return self._get(True)(*p, **kw)

    def __getattr__(self, name):
        return getattr(self._get(True), name)

class EnumValues(collections.OrderedDict):
    def __init__(self):
        super().__init__()
        self.sealed = False

    def __getitem__(self, key):
        try:
            obj = super().__getitem__(key)

            if not self.sealed and isinstance(obj, EnumProxy):
                obj.values.append(None)

            return obj

        except KeyError:
            # Don't do anything with __dunder__ attributes
            if key[:2] == '__' and key[-2:] == '__':
                raise

            proxy = EnumProxy(key, None)
            super().__setitem__(key, proxy)
            return proxy

    def __setitem__(self, key, value):
        if key[:2] == '__' and key[-2:] == '__':
            return super().__setitem__(key, value)

        try:
            if isinstance(value, EnumProxy):
                value = value._get(True)

            elif not isinstance(value, EnumValue):
                value = operator.index(value)

        except TypeError:
            return super().__setitem__(key, value)

        try:
            o = super().__getitem__(key)

            if isinstance(o, EnumProxy):
                o.values.append(value)

        except KeyError:
            if isinstance(value, EnumProxy):
                int.__setattr__(value, 'key', key)
            else:
                value = EnumProxy(key, value)

            super().__setitem__(value.key, value)

class EnumMeta(type):

    @classmethod
    def __prepare__(metacls, name, bases):
        return EnumValues()

    def __new__(cls, name, bases, classdict):
        classdict.sealed = True

        del_list = []

        for v in classdict.values():
            if isinstance(v, EnumProxy) and v.used:
                del_list.append(v)

        for v in del_list:
            del classdict[v.key]

        result = type.__new__(cls, name, bases, dict(classdict))
        value = 0
        keys = {}
        values = {}

        for v in classdict.values():
            if isinstance(v, EnumProxy) and not v.used:
                if len(v.values) > 1:
                    raise AttributeError("Duplicate enum key '%s.%s'" %
(result.__qualname__, v.key,))

                if v.values[0] is not None:
                    value = v.values[0]

                if isinstance(value, EnumValue):

                    if (value.key is not None) and (value.key != v.key):
                        raise AttributeError("Assigned enum value to
non-matching key '%s': %r" % (v.key, value))

                    if value.owner is not None:
                        raise AttributeError("Assigned owned enum value to
key '%s': %r" % (v.key, value))

                    int.__setattr__(value, 'key', v.key)
                    v = value

                else:
                    v = EnumValue(v.key, value)

                setattr(result, v.key, v)
                value += 1

            if isinstance(v, EnumValue):
                int.__setattr__(v, 'owner', result)

                if v in values:
                    raise AttributeError("Duplicate enum value %d for keys:
'%s' and '%s'" % (int(v), values[v].key, v.key))

                keys[v.key] = v
                values[v] = v

        enum = sorted(values)

        result._key_to_enum = collections.OrderedDict()
        result._value_to_enum = values

        for e in enum:
            result._key_to_enum[e.key] = e

        return result

    def __getitem__(self, key):
        try:
            key = operator.index(key)
        except TypeError:
            return self._key_to_enum[key]
        else:
            return self._value_to_enum[key]

    def _items(self):
        return self._key_to_enum.items()

    def _keys(self):
        return self._key_to_enum.keys()

    def _values(self):
        return self._key_to_enum.values()

    def items(self):
        return self._items()

    def keys(self):
        return self._keys()

    def values(self):
        return self._values()

    def __iter__(self):
        return iter(self.values())

    def __repr__(self):
        r = super().__repr__()
        r = ['<enum', r[6:-1], ' ']
        r.append(str(self))
        r.append('>')
        return ''.join(r)

    def __str__(self):
        s = ['{']

        for k, v in self.items():
            if s[-1][-1:] != '{':
                s.append(', ')

            s.extend([k, ':', str(int(v))])

        s.append('}')
        return ''.join(s)

class Enum(metaclass=EnumMeta):
    def __getitem__(self, key):
        cls = type(self)
        return type(cls).__getitem__(cls, key)

    @classmethod
    def items(cls):
        return cls._items()

    @classmethod
    def keys(cls):
        return cls._keys()

    @classmethod
    def values(cls):
        return cls._values()

    def __iter__(self):
        return iter(self.values())

    def __repr__(self):
        r = super().__repr__()
        r = r.replace('object at 0x', 'enum at 0x')
        r = [r[:-1], ' ']
        r.append(str(self))
        r.append('>')
        return ''.join(r)

    def __str__(self):
        return str(type(self))

if __name__ == '__main__':

    class Color(Enum):
        RED, GREEN, BLUE
        ORANGE = "orange"
        CYAN = 10
        MAGENTA
        YELLOW
        BLACK

    import unittest

    class TestEnum(unittest.TestCase):

        EXPECTED_KEY_ORDER = ('RED', 'GREEN', 'BLUE', 'CYAN', 'MAGENTA',
'YELLOW', 'BLACK')
        EXPECTED_INT_VALUE_ORDER = (0, 1, 2, 10, 11, 12, 13)
        EXPECTED_ENUM_VALUE_ORDER = (Color.RED, Color.GREEN, Color.BLUE,
Color.CYAN, Color.MAGENTA, Color.YELLOW, Color.BLACK)
        EXPECTED_ITEMS_ORDER = tuple(zip(EXPECTED_KEY_ORDER,
EXPECTED_ENUM_VALUE_ORDER))

        def test_type(self):
            self.assertIsInstance(Color.RED, int)

        def test_class_enum_values(self):
            self.assertEqual(0, Color.RED)
            self.assertEqual(1, Color.GREEN)
            self.assertEqual(2, Color.BLUE)
            self.assertEqual(10, Color.CYAN)
            self.assertEqual(11, Color.MAGENTA)
            self.assertEqual(12, Color.YELLOW)
            self.assertEqual(13, Color.BLACK)
            self.assertEqual("orange", Color.ORANGE)

        def test_instance_enum_values(self):
            e = Color()
            self.assertIs(Color.RED, e.RED)
            self.assertIs(Color.GREEN, e.GREEN)
            self.assertIs(Color.BLUE, e.BLUE)
            self.assertIs(Color.CYAN, e.CYAN)
            self.assertIs(Color.MAGENTA, e.MAGENTA)
            self.assertIs(Color.YELLOW, e.YELLOW)
            self.assertIs(Color.BLACK, e.BLACK)
            self.assertIs(Color.ORANGE, e.ORANGE)

        def test_class_indexing(self):
            self.assertIs(Color.CYAN, Color['CYAN'])
            self.assertIs(Color.CYAN, Color[10])

        def test_instance_indexing(self):
            e = Color()
            self.assertIs(Color.CYAN, e['CYAN'])
            self.assertIs(Color.CYAN, e[10])

        def test_class_keys(self):
            self.assertEqual(self.EXPECTED_KEY_ORDER, tuple(Color.keys()))

        def test_instance_keys(self):
            self.assertEqual(tuple(Color.keys()), tuple(Color().keys()))

        def test_class_values(self):
            self.assertEqual(self.EXPECTED_INT_VALUE_ORDER,
tuple(Color.values()))
            self.assertEqual(self.EXPECTED_ENUM_VALUE_ORDER,
tuple(Color.values()))

        def test_instance_values(self):
            self.assertEqual(tuple(Color.values()), tuple(Color().values()))

        def test_class_items(self):
            self.assertEqual(self.EXPECTED_ITEMS_ORDER,
tuple(Color.items()))

        def test_instance_items(self):
            self.assertEqual(tuple(Color.items()), tuple(Color().items()))

        def test_owner(self):
            for e in Color:
                self.assertIs(e.owner, Color)

        def test_class_str(self):
            s = str(Color)

            for e in Color:
                self.assertIn('%s:%d' % (e.key, int(e)), s)

        def test_instance_str(self):
            self.assertEqual(str(Color), str(Color()))

        def test_class_repr(self):
            r = repr(Color)
            self.assertIn(Color.__qualname__, r)
            self.assertIn(str(Color), r)

        def test_instance_repr(self):
            e = Color()
            r = repr(e)
            self.assertIn(Color.__qualname__, r)
            self.assertIn('at 0x', r)
            self.assertIn(str(Color()), r)

        def _create_duplicate_key(self):
            class DuplicateKey(Enum):
                KEY,
                KEY

        def test_duplicate_key(self):
            self.assertRaises(AttributeError, self._create_duplicate_key)

        def _create_duplicate_value(self):
            class DuplicateValue(Enum):
                KEY1,
                KEY2 = 0

        def test_duplicate_value(self):
            self.assertRaises(AttributeError, self._create_duplicate_value)

        def _assign_wrong_key(self):
            class WrongKey(Enum):
                KEY1 = EnumValue('KEY2', 0)

        def test_wrong_key(self):
            self.assertRaises(AttributeError, self._assign_wrong_key)

        def test_unnamed_key1(self):
            class UnnamedKey(Enum):
                KEY1 = EnumValue(None, 5)

            self.assertEqual(UnnamedKey.KEY1, 5)
            self.assertIs(UnnamedKey, UnnamedKey.KEY1.owner)

        def test_unnamed_key1(self):
            unnamed = EnumValue(None, 5)

            class UnnamedKey(Enum):
                KEY1 = unnamed

            self.assertEqual(UnnamedKey.KEY1, 5)
            self.assertIs(UnnamedKey.KEY1, unnamed)
            self.assertIs(UnnamedKey, UnnamedKey.KEY1.owner)

        def _assign_wrong_owner(self):
            class WrongOwner(Enum):
                KEY1 = Color.RED

        def test_wrong_owner(self):
            self.assertRaises(AttributeError, self._assign_wrong_owner)

    unittest.main()

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/4401f17e/attachment.html>

From timothy.c.delaney at gmail.com  Fri Feb  1 05:18:57 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Fri, 1 Feb 2013 15:18:57 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
Message-ID: <CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>

Last version (for now). I'm really interested in people's opinions on this.
For this version I've taken some inspiration from flufl.enum (but there
remains the major difference that these enums subclass int).

- Enums are now subclassable;

- Added an Enum.make() method - a bit different to flufl.enum.make since my
enums have different semantics - each element must either be a name or a
(name, value) pair, and you can have a mix;

- Instantiating an enum now returns the appropriate EnumValue

- Enums now compare not equal with any enum that is not the same object
(but continue to compare equal with ints);

- Changed EnumValue.key -> EnumValue.name and EnumValue.owner ->
EnumValue.enum.

I didn't add __members__ as that use case is covered by having the Enum be
iterable + the immutable mapping interface.

#!/usr/bin/env python3

import builtins
import collections
import operator

class EnumValue(int):
    def __new__(cls, key, value):
        e = super().__new__(cls, value)
        super().__setattr__(e, 'name', key)
        super().__setattr__(e, 'enum', None)
        return e

    def __setattr__(self, key, value):
        raise TypeError("can't set attribute")

    def __eq__(self, other):
        if isinstance(other, EnumValue):
            return self is other

        return int(self) == other

    def __ne__(self, other):
        return not (self == other)

    def __hash__(self):
        return super().__hash__()

    def __str__(self):
        if self.enum is not None:
            return "%s.%s" % (self.enum.__name__, self.name)

        return self.name

    def __repr__(self):
        if self.enum is not None:
            return "<%s '%s.%s': %d>" % (self.__qualname__,
self.enum.__qualname__, self.name, int(self))

        return "<%s '%s': %d>" % (self.__qualname__, self.name, int(self))

class _EnumProxy(object):
    def __init__(self, key, value=None):
        self.key = key
        self.values = [value]
        self.used = False

    def __repr__(self):
        return "<%s '%s': %s>" % (self.__qualname__, self.key, self.values)

    def _get(self, used=None):
        if used:
            self.used = True

        try:
            return locals()[self.key]
        except KeyError:
            try:
                return globals()[self.key]
            except KeyError:
                try:
                    return getattr(builtins, self.key)
                except KeyError:
                    raise NameError(self.key, self.values)

    def __call__(self, *p, **kw):
        return self._get(True)(*p, **kw)

    def __getattr__(self, name):
        return getattr(self._get(True), name)

class EnumValues(collections.OrderedDict):
    def __init__(self):
        super().__init__()
        self.sealed = False

    def __getitem__(self, key):
        try:
            obj = super().__getitem__(key)

            if not self.sealed and isinstance(obj, _EnumProxy):
                obj.values.append(None)

            return obj

        except KeyError:
            # Don't do anything with __dunder__ attributes
            if key[:2] == '__' and key[-2:] == '__':
                raise

            proxy = _EnumProxy(key, None)
            super().__setitem__(key, proxy)
            return proxy

    def __setitem__(self, key, value):
        if key[:2] == '__' and key[-2:] == '__':
            return super().__setitem__(key, value)

        try:
            if isinstance(value, _EnumProxy):
                value = value._get(True)

            elif not isinstance(value, EnumValue):
                value = operator.index(value)

        except TypeError:
            return super().__setitem__(key, value)

        try:
            o = super().__getitem__(key)

            if isinstance(o, _EnumProxy):
                o.values.append(value)

        except KeyError:
            if not isinstance(value, _EnumProxy):
                value = _EnumProxy(key, value)

            super().__setitem__(value.key, value)

class EnumMeta(type):

    @classmethod
    def __prepare__(metacls, name, bases):
        return EnumValues()

    def __new__(cls, name, bases, classdict):
        classdict.sealed = True

        del_list = []

        for v in classdict.values():
            if isinstance(v, _EnumProxy) and v.used:
                del_list.append(v)

        for v in del_list:
            del classdict[v.key]

        result = type.__new__(cls, name, bases, dict(classdict))
        keys = {}
        values = {}
        result._key_to_enum = collections.OrderedDict()
        result._value_to_enum = values

        value = 0

        for b in result.__bases__:
            if isinstance(b, EnumMeta):
                keys.update(b._key_to_enum)
                values.update(b._value_to_enum)

        if values:
            value = max(values) + 1

        for v in classdict.values():
            if isinstance(v, _EnumProxy) and not v.used:
                if len(v.values) > 1:
                    raise AttributeError("Duplicate enum key '%s.%s'" %
(name, v.key,))
                elif v.key in keys:
                    raise AttributeError("Duplicate enum key '%s.%s'
(overriding '%s')" % (result.__name__, v.key, keys[v.key]))

                if v.values[0] is not None:
                    value = v.values[0]

                if isinstance(value, EnumValue):

                    if (value.name is not None) and (value.name != v.key):
                        raise AttributeError("Assigned enum value to
non-matching key '%s': %r" % (v.key, value))

                    if value.enum is not None:
                        raise AttributeError("Assigned owned enum value to
key '%s': %r" % (v.key, value))

                    int.__setattr__(value, 'name', v.key)
                    v = value

                else:
                    v = EnumValue(v.key, value)

                setattr(result, v.name, v)
                value += 1

            if isinstance(v, EnumValue):
                int.__setattr__(v, 'enum', result)
                int_v = int(v)

                if int_v in values:
                    raise AttributeError("Duplicate enum value %d for keys:
'%s.%s' and '%s.%s'" % (
                        int_v, values[int_v].enum.__name__,
values[int_v].name, result.__name__, v.name))

                keys[v.name] = v
                values[v] = v

        enum = sorted(values)

        for e in enum:
            result._key_to_enum[e.name] = e

        return result

    def __getitem__(self, key):
        try:
            key = operator.index(key)
        except TypeError:
            return self._key_to_enum[key]
        else:
            return self._value_to_enum[key]

    def items(self):
        return self._key_to_enum.items()

    def keys(self):
        return self._key_to_enum.keys()

    def values(self):
        return self._key_to_enum.values()

    def __iter__(self):
        return iter(self.values())

    def __repr__(self):
        r = super().__repr__()
        r = ['<enum', r[6:-1], ' ']
        r.append(str(self))
        r.append('>')
        return ''.join(r)

    def __str__(self):
        s = ['{']

        for k, v in self.items():
            if s[-1][-1:] != '{':
                s.append(', ')

            s.extend([k, ':', str(int(v))])

        s.append('}')
        return ''.join(s)

class Enum(metaclass=EnumMeta):
    def __new__(cls, value):
        return type(cls).__getitem__(cls, value)

    @staticmethod
    def make(clsname, elements):
        classdict = collections.OrderedDict()

        for e in elements:
            if isinstance(e, tuple):
                k, v = e
            else:
                k, v = e, None

            try:
                e = classdict[k]
                e.values.append(v)
            except KeyError:
                classdict[k] = _EnumProxy(k, v)

        result = EnumMeta(clsname, (Enum,), classdict)
        # For some reason, this is set to 'Enum' ...
        result.__qualname__ = clsname
        return result

if __name__ == '__main__':

    import unittest

    class Color(Enum):
        RED, GREEN, BLUE
        ORANGE = "orange"
        CYAN = 10
        MAGENTA
        YELLOW
        BLACK

    class TestEnum(unittest.TestCase):

        EXPECTED_KEY_ORDER = ('RED', 'GREEN', 'BLUE', 'CYAN', 'MAGENTA',
'YELLOW', 'BLACK')
        EXPECTED_INT_VALUE_ORDER = (0, 1, 2, 10, 11, 12, 13)
        EXPECTED_ENUM_VALUE_ORDER = (Color.RED, Color.GREEN, Color.BLUE,
Color.CYAN, Color.MAGENTA, Color.YELLOW, Color.BLACK)
        EXPECTED_ITEMS_ORDER = tuple(zip(EXPECTED_KEY_ORDER,
EXPECTED_ENUM_VALUE_ORDER))

        def test_type(self):
            self.assertIsInstance(Color.RED, int)

        def test_enum_values(self):
            self.assertEqual(0, Color.RED)
            self.assertEqual(1, Color.GREEN)
            self.assertEqual(2, Color.BLUE)
            self.assertEqual(10, Color.CYAN)
            self.assertEqual(11, Color.MAGENTA)
            self.assertEqual(12, Color.YELLOW)
            self.assertEqual(13, Color.BLACK)
            self.assertEqual("orange", Color.ORANGE)

        def test_indexing(self):
            self.assertIs(Color.CYAN, Color['CYAN'])
            self.assertIs(Color.CYAN, Color[10])

        def test_keys(self):
            self.assertEqual(self.EXPECTED_KEY_ORDER, tuple(Color.keys()))

        def test_values(self):
            self.assertEqual(self.EXPECTED_INT_VALUE_ORDER,
tuple(Color.values()))
            self.assertEqual(self.EXPECTED_ENUM_VALUE_ORDER,
tuple(Color.values()))

        def test_items(self):
            self.assertEqual(self.EXPECTED_ITEMS_ORDER,
tuple(Color.items()))

        def test_owner(self):
            for e in Color:
                self.assertIs(e.enum, Color)

        def test_equality(self):
            self.assertEqual(0, Color.RED)
            self.assertEqual(Color.RED, 0)
            self.assertEqual(Color.RED, Color.RED)
            self.assertNotEqual(Color.RED, Color.GREEN)

            class Color2(Enum):
                RED

            self.assertEqual(0, Color2.RED)
            self.assertEqual(Color2.RED, 0)
            self.assertNotEqual(Color.RED, Color2.RED)

        def test_str(self):
            s = str(Color)

            for e in Color:
                self.assertIn('%s:%d' % (e.name, int(e)), s)

        def test_repr(self):
            r = repr(Color)
            self.assertIn(Color.__qualname__, r)
            self.assertIn(str(Color), r)

        def test_instances(self):
            for e in Color:
                self.assertIs(e, Color(e))
                self.assertIs(e, Color(int(e)))
                self.assertIs(e, Color(e.name))

        def _create_duplicate_key(self):
            class DuplicateKey(Enum):
                KEY,
                KEY

        def test_duplicate_key(self):
            self.assertRaises(AttributeError, self._create_duplicate_key)

        def _create_duplicate_value(self):
            class DuplicateValue(Enum):
                KEY1,
                KEY2 = 0

        def test_duplicate_value(self):
            self.assertRaises(AttributeError, self._create_duplicate_value)

        def _assign_wrong_key(self):
            class WrongKey(Enum):
                KEY1 = EnumValue('KEY2', 0)

        def test_wrong_key(self):
            self.assertRaises(AttributeError, self._assign_wrong_key)

        def test_unnamed_key1(self):
            class UnnamedKey(Enum):
                KEY1 = EnumValue(None, 5)

            self.assertEqual(UnnamedKey.KEY1, 5)
            self.assertIs(UnnamedKey, UnnamedKey.KEY1.enum)

        def test_unnamed_key1(self):
            unnamed = EnumValue(None, 5)

            class UnnamedKey(Enum):
                KEY1 = unnamed

            self.assertEqual(UnnamedKey.KEY1, 5)
            self.assertIs(UnnamedKey.KEY1, unnamed)
            self.assertIs(UnnamedKey, UnnamedKey.KEY1.enum)

        def _assign_wrong_owner(self):
            class WrongOwner(Enum):
                KEY1 = Color.RED

        def test_wrong_owner(self):
            self.assertRaises(AttributeError, self._assign_wrong_owner)

        def test_subclassing(self):
            class ExtendedColor1(Color):
                PINK,
                GREY,
                WHITE = 5

            self.assertIs(Color.RED, ExtendedColor1.RED)

            for e in Color:
                self.assertIs(e, ExtendedColor1[e])

            self.assertEqual(14, ExtendedColor1.PINK)
            self.assertEqual(15, ExtendedColor1.GREY)
            self.assertEqual(5, ExtendedColor1.WHITE)

            class ExtendedColor2(Color):
                PINK,
                GREY,
                WHITE = 5

            self.assertIsNot(ExtendedColor1.PINK, ExtendedColor2.PINK)
            self.assertIs(ExtendedColor1.RED, ExtendedColor2.RED)

        def _create_duplicate_key_subclass(self):
            class ExtendedColor1(Color):
                RED

        def test_duplicate_key_subclass(self):
            self.assertRaises(AttributeError,
self._create_duplicate_key_subclass)

        def _create_duplicate_value_subclass(self):
            class ExtendedColor1(Color):
                PINK = 0

        def test_duplicate_key_subclass(self):
            self.assertRaises(AttributeError,
self._create_duplicate_value_subclass)

        def test_make(self):
            e = Enum.make('e', ('a', 'b', ('c', 3), 'd'))
            self.assertEqual(0, e.a)
            self.assertEqual(1, e.b)
            self.assertEqual(3, e.c)
            self.assertEqual(4, e.d)

        def test_duplicate_key_make(self):
            self.assertRaises(AttributeError, Enum.make, 'e', ('a', 'b',
('b', 3), 'd'))
            self.assertRaises(AttributeError, Enum.make, 'e', ('a', 'b',
('c', 3), 'b'))

        def test_duplicate_value_make(self):
            self.assertRaises(AttributeError, Enum.make, 'e', ('a', 'b',
('c', 1), 'd'))

    unittest.main()

Tim Delaney


On 1 February 2013 11:19, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> Wow - this ended up being more difficult that I'd anticipated. Ensuring
> that decorators work in Michael Foord-inspired enums, etc mandated a fairly
> big redesign. As it was you couldn't do something like:
>
> class A():
>     a = 1
>
> class B(Enum):
>     B = A.a
>
> because when the name 'A' was resolved in the definition of class B, it
> was returning an int (the next enum value).
>
> The end result may well still have some holes, but it's looking pretty
> good to me.
>
> Enums are constructed like:
>
> class Color(Enum):
>     RED, GREEN, BLUE
>     CYAN = 10
>     MAGENTA
>     YELLOW
>     BLACK
>
> where the assigned enum starts a new count i.e. the above is 0, 1, 2, 10,
> 11, 12, 13. Arbitrary attributes may be assigned and not contribute to the
> enumeration so long as the object being assigned does not implement
> __index__ (if it does, it creates a discontiguous enumeration).
>
> #!/usr/bin/env python3
>
> import builtins
> import collections
> import operator
>
> class EnumValue(int):
>     def __new__(cls, key, value):
>         e = super().__new__(cls, value)
>         super().__setattr__(e, 'key', key)
>         super().__setattr__(e, 'owner', None)
>         return e
>
>     def __setattr__(self, key, value):
>         raise TypeError("Cannot set attribute of type %r" % (type(self),))
>
>     def __str__(self):
>         return "%s.%s" % (self.owner.__name__, self.key)
>
>     def __repr__(self):
>         if self.owner is not None:
>             return "<%s '%s.%s': %d>" % (self.__qualname__,
> self.owner.__qualname__, self.key, int(self))
>
>         return "<%s '%s': %d>" % (self.__qualname__, self.key, int(self))
>
> class EnumProxy(object):
>     def __init__(self, key, value=None):
>         self.key = key
>         self.values = [value]
>         self.used = False
>
>     def __repr__(self):
>         return "<%s '%s': %s>" % (self.__qualname__, self.key, self.values)
>
>     def _get(self, used=None):
>         if used:
>             self.used = True
>
>         try:
>             return locals()[self.key]
>         except KeyError:
>             try:
>                 return globals()[self.key]
>             except KeyError:
>                 try:
>                     return getattr(builtins, self.key)
>                 except KeyError:
>                     raise NameError(self.key, self.values)
>
>     def __call__(self, *p, **kw):
>         return self._get(True)(*p, **kw)
>
>     def __getattr__(self, name):
>         return getattr(self._get(True), name)
>
> class EnumValues(collections.OrderedDict):
>     def __init__(self):
>         super().__init__()
>         self.sealed = False
>
>     def __getitem__(self, key):
>         try:
>             obj = super().__getitem__(key)
>
>             if not self.sealed and isinstance(obj, EnumProxy):
>                 obj.values.append(None)
>
>             return obj
>
>         except KeyError:
>             # Don't do anything with __dunder__ attributes
>             if key[:2] == '__' and key[-2:] == '__':
>                 raise
>
>             proxy = EnumProxy(key, None)
>             super().__setitem__(key, proxy)
>             return proxy
>
>     def __setitem__(self, key, value):
>         if key[:2] == '__' and key[-2:] == '__':
>             return super().__setitem__(key, value)
>
>         try:
>             if isinstance(value, EnumProxy):
>                 value = value._get(True)
>
>             elif not isinstance(value, EnumValue):
>                 value = operator.index(value)
>
>         except TypeError:
>             return super().__setitem__(key, value)
>
>         try:
>             o = super().__getitem__(key)
>
>             if isinstance(o, EnumProxy):
>                 o.values.append(value)
>
>         except KeyError:
>             if isinstance(value, EnumProxy):
>                 int.__setattr__(value, 'key', key)
>             else:
>                 value = EnumProxy(key, value)
>
>             super().__setitem__(value.key, value)
>
> class EnumMeta(type):
>
>     @classmethod
>     def __prepare__(metacls, name, bases):
>         return EnumValues()
>
>     def __new__(cls, name, bases, classdict):
>         classdict.sealed = True
>
>         del_list = []
>
>         for v in classdict.values():
>             if isinstance(v, EnumProxy) and v.used:
>                 del_list.append(v)
>
>         for v in del_list:
>             del classdict[v.key]
>
>         result = type.__new__(cls, name, bases, dict(classdict))
>         value = 0
>         keys = {}
>         values = {}
>
>         for v in classdict.values():
>             if isinstance(v, EnumProxy) and not v.used:
>                 if len(v.values) > 1:
>                     raise AttributeError("Duplicate enum key '%s.%s'" %
> (result.__qualname__, v.key,))
>
>                 if v.values[0] is not None:
>                     value = v.values[0]
>
>                 if isinstance(value, EnumValue):
>
>                     if (value.key is not None) and (value.key != v.key):
>                         raise AttributeError("Assigned enum value to
> non-matching key '%s': %r" % (v.key, value))
>
>                     if value.owner is not None:
>                         raise AttributeError("Assigned owned enum value to
> key '%s': %r" % (v.key, value))
>
>                     int.__setattr__(value, 'key', v.key)
>                     v = value
>
>                 else:
>                     v = EnumValue(v.key, value)
>
>                 setattr(result, v.key, v)
>                 value += 1
>
>             if isinstance(v, EnumValue):
>                 int.__setattr__(v, 'owner', result)
>
>                 if v in values:
>                     raise AttributeError("Duplicate enum value %d for
> keys: '%s' and '%s'" % (int(v), values[v].key, v.key))
>
>                 keys[v.key] = v
>                 values[v] = v
>
>         enum = sorted(values)
>
>         result._key_to_enum = collections.OrderedDict()
>         result._value_to_enum = values
>
>         for e in enum:
>             result._key_to_enum[e.key] = e
>
>         return result
>
>     def __getitem__(self, key):
>         try:
>             key = operator.index(key)
>         except TypeError:
>             return self._key_to_enum[key]
>         else:
>             return self._value_to_enum[key]
>
>     def _items(self):
>         return self._key_to_enum.items()
>
>     def _keys(self):
>         return self._key_to_enum.keys()
>
>     def _values(self):
>         return self._key_to_enum.values()
>
>     def items(self):
>         return self._items()
>
>     def keys(self):
>         return self._keys()
>
>     def values(self):
>         return self._values()
>
>     def __iter__(self):
>         return iter(self.values())
>
>     def __repr__(self):
>         r = super().__repr__()
>         r = ['<enum', r[6:-1], ' ']
>         r.append(str(self))
>         r.append('>')
>         return ''.join(r)
>
>     def __str__(self):
>         s = ['{']
>
>         for k, v in self.items():
>             if s[-1][-1:] != '{':
>                 s.append(', ')
>
>             s.extend([k, ':', str(int(v))])
>
>         s.append('}')
>         return ''.join(s)
>
> class Enum(metaclass=EnumMeta):
>     def __getitem__(self, key):
>         cls = type(self)
>         return type(cls).__getitem__(cls, key)
>
>     @classmethod
>     def items(cls):
>         return cls._items()
>
>     @classmethod
>     def keys(cls):
>         return cls._keys()
>
>     @classmethod
>     def values(cls):
>         return cls._values()
>
>     def __iter__(self):
>         return iter(self.values())
>
>     def __repr__(self):
>         r = super().__repr__()
>         r = r.replace('object at 0x', 'enum at 0x')
>         r = [r[:-1], ' ']
>         r.append(str(self))
>         r.append('>')
>         return ''.join(r)
>
>     def __str__(self):
>         return str(type(self))
>
> if __name__ == '__main__':
>
>     class Color(Enum):
>         RED, GREEN, BLUE
>         ORANGE = "orange"
>         CYAN = 10
>         MAGENTA
>         YELLOW
>         BLACK
>
>     import unittest
>
>     class TestEnum(unittest.TestCase):
>
>         EXPECTED_KEY_ORDER = ('RED', 'GREEN', 'BLUE', 'CYAN', 'MAGENTA',
> 'YELLOW', 'BLACK')
>         EXPECTED_INT_VALUE_ORDER = (0, 1, 2, 10, 11, 12, 13)
>         EXPECTED_ENUM_VALUE_ORDER = (Color.RED, Color.GREEN, Color.BLUE,
> Color.CYAN, Color.MAGENTA, Color.YELLOW, Color.BLACK)
>         EXPECTED_ITEMS_ORDER = tuple(zip(EXPECTED_KEY_ORDER,
> EXPECTED_ENUM_VALUE_ORDER))
>
>         def test_type(self):
>             self.assertIsInstance(Color.RED, int)
>
>         def test_class_enum_values(self):
>             self.assertEqual(0, Color.RED)
>             self.assertEqual(1, Color.GREEN)
>             self.assertEqual(2, Color.BLUE)
>             self.assertEqual(10, Color.CYAN)
>             self.assertEqual(11, Color.MAGENTA)
>             self.assertEqual(12, Color.YELLOW)
>             self.assertEqual(13, Color.BLACK)
>             self.assertEqual("orange", Color.ORANGE)
>
>         def test_instance_enum_values(self):
>             e = Color()
>             self.assertIs(Color.RED, e.RED)
>             self.assertIs(Color.GREEN, e.GREEN)
>             self.assertIs(Color.BLUE, e.BLUE)
>             self.assertIs(Color.CYAN, e.CYAN)
>             self.assertIs(Color.MAGENTA, e.MAGENTA)
>             self.assertIs(Color.YELLOW, e.YELLOW)
>             self.assertIs(Color.BLACK, e.BLACK)
>             self.assertIs(Color.ORANGE, e.ORANGE)
>
>         def test_class_indexing(self):
>             self.assertIs(Color.CYAN, Color['CYAN'])
>             self.assertIs(Color.CYAN, Color[10])
>
>         def test_instance_indexing(self):
>             e = Color()
>             self.assertIs(Color.CYAN, e['CYAN'])
>             self.assertIs(Color.CYAN, e[10])
>
>         def test_class_keys(self):
>             self.assertEqual(self.EXPECTED_KEY_ORDER, tuple(Color.keys()))
>
>         def test_instance_keys(self):
>             self.assertEqual(tuple(Color.keys()), tuple(Color().keys()))
>
>         def test_class_values(self):
>             self.assertEqual(self.EXPECTED_INT_VALUE_ORDER,
> tuple(Color.values()))
>             self.assertEqual(self.EXPECTED_ENUM_VALUE_ORDER,
> tuple(Color.values()))
>
>         def test_instance_values(self):
>             self.assertEqual(tuple(Color.values()),
> tuple(Color().values()))
>
>         def test_class_items(self):
>             self.assertEqual(self.EXPECTED_ITEMS_ORDER,
> tuple(Color.items()))
>
>         def test_instance_items(self):
>             self.assertEqual(tuple(Color.items()), tuple(Color().items()))
>
>         def test_owner(self):
>             for e in Color:
>                 self.assertIs(e.owner, Color)
>
>         def test_class_str(self):
>             s = str(Color)
>
>             for e in Color:
>                 self.assertIn('%s:%d' % (e.key, int(e)), s)
>
>         def test_instance_str(self):
>             self.assertEqual(str(Color), str(Color()))
>
>         def test_class_repr(self):
>             r = repr(Color)
>             self.assertIn(Color.__qualname__, r)
>             self.assertIn(str(Color), r)
>
>         def test_instance_repr(self):
>             e = Color()
>             r = repr(e)
>             self.assertIn(Color.__qualname__, r)
>             self.assertIn('at 0x', r)
>             self.assertIn(str(Color()), r)
>
>         def _create_duplicate_key(self):
>             class DuplicateKey(Enum):
>                 KEY,
>                 KEY
>
>         def test_duplicate_key(self):
>             self.assertRaises(AttributeError, self._create_duplicate_key)
>
>         def _create_duplicate_value(self):
>             class DuplicateValue(Enum):
>                 KEY1,
>                 KEY2 = 0
>
>         def test_duplicate_value(self):
>             self.assertRaises(AttributeError, self._create_duplicate_value)
>
>         def _assign_wrong_key(self):
>             class WrongKey(Enum):
>                 KEY1 = EnumValue('KEY2', 0)
>
>         def test_wrong_key(self):
>             self.assertRaises(AttributeError, self._assign_wrong_key)
>
>         def test_unnamed_key1(self):
>             class UnnamedKey(Enum):
>                 KEY1 = EnumValue(None, 5)
>
>             self.assertEqual(UnnamedKey.KEY1, 5)
>             self.assertIs(UnnamedKey, UnnamedKey.KEY1.owner)
>
>         def test_unnamed_key1(self):
>             unnamed = EnumValue(None, 5)
>
>             class UnnamedKey(Enum):
>                 KEY1 = unnamed
>
>             self.assertEqual(UnnamedKey.KEY1, 5)
>             self.assertIs(UnnamedKey.KEY1, unnamed)
>             self.assertIs(UnnamedKey, UnnamedKey.KEY1.owner)
>
>         def _assign_wrong_owner(self):
>             class WrongOwner(Enum):
>                 KEY1 = Color.RED
>
>         def test_wrong_owner(self):
>             self.assertRaises(AttributeError, self._assign_wrong_owner)
>
>     unittest.main()
>
> Tim Delaney
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/4bf9b66f/attachment.html>

From ben+python at benfinney.id.au  Fri Feb  1 05:28:33 2013
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 01 Feb 2013 15:28:33 +1100
Subject: [Python-ideas] Definition Symmetry
References: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
	<510AFD06.1090305@pearwood.info>
	<CAPAaX_ifjthen8JiLvKetmTDbmrzAUKydqXxKfvF+68feqthyg@mail.gmail.com>
Message-ID: <7wa9roejym.fsf@benfinney.id.au>

Jason Keene <jasonkeene at gmail.com> writes:

> In a way they do the same thing, they both create an object
> (function/class) from a suite and assign it to the name given after the
> keyword (def/class).  Sure they do totally different things with the suite
> in creating the object, but in essence it's a name assignment.

In a way, ?import? and ?break? do the same thing, they trigger the
compiler to compile a particular set of code bytes. Sure the code bytes
do totally different things, but in essence it's a statement.

Less facetiously: You can blur your vision as much as you like to make
?class? and ?def? look similar, but that doesn't diminish the importance
of the distinctions you're ignoring.

-- 
 \       ?Always do right. This will gratify some people, and astonish |
  `\                                            the rest.? ?Mark Twain |
_o__)                                                                  |
Ben Finney



From jasonkeene at gmail.com  Fri Feb  1 06:21:07 2013
From: jasonkeene at gmail.com (Jason Keene)
Date: Fri, 1 Feb 2013 00:21:07 -0500
Subject: [Python-ideas] Definition Symmetry
In-Reply-To: <7wa9roejym.fsf@benfinney.id.au>
References: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
	<510AFD06.1090305@pearwood.info>
	<CAPAaX_ifjthen8JiLvKetmTDbmrzAUKydqXxKfvF+68feqthyg@mail.gmail.com>
	<7wa9roejym.fsf@benfinney.id.au>
Message-ID: <CAPAaX_gJ32j1fz0SgvChv8feFnjjVvV1eqizFZtgmLMr5B4FgA@mail.gmail.com>

Both class and function definitions produce callable objects that are
assigned to a name with nearly identical syntax.  I don't think your
analogy with import/break statements is valid.  Additionally, both
definitions produce very similar byte codes:

              3 MAKE_FUNCTION            0
              6 STORE_FAST               0 (my_func)

              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_FAST               0 (MyClass)

The only argument I see for requiring empty parens in function definitions
is a historical one.

On Thursday, January 31, 2013, Ben Finney wrote:

> Jason Keene <jasonkeene at gmail.com <javascript:;>> writes:
>
> > In a way they do the same thing, they both create an object
> > (function/class) from a suite and assign it to the name given after the
> > keyword (def/class).  Sure they do totally different things with the
> suite
> > in creating the object, but in essence it's a name assignment.
>
> In a way, ?import? and ?break? do the same thing, they trigger the
> compiler to compile a particular set of code bytes. Sure the code bytes
> do totally different things, but in essence it's a statement.
>
> Less facetiously: You can blur your vision as much as you like to make
> ?class? and ?def? look similar, but that doesn't diminish the importance
> of the distinctions you're ignoring.
>
> --
>  \       ?Always do right. This will gratify some people, and astonish |
>   `\                                            the rest.? ?Mark Twain |
> _o__)                                                                  |
> Ben Finney
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <javascript:;>
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/fdcad2cf/attachment.html>

From jasonkeene at gmail.com  Fri Feb  1 06:41:32 2013
From: jasonkeene at gmail.com (Jason Keene)
Date: Fri, 1 Feb 2013 00:41:32 -0500
Subject: [Python-ideas] Definition Symmetry
In-Reply-To: <CAPAaX_gJ32j1fz0SgvChv8feFnjjVvV1eqizFZtgmLMr5B4FgA@mail.gmail.com>
References: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
	<510AFD06.1090305@pearwood.info>
	<CAPAaX_ifjthen8JiLvKetmTDbmrzAUKydqXxKfvF+68feqthyg@mail.gmail.com>
	<7wa9roejym.fsf@benfinney.id.au>
	<CAPAaX_gJ32j1fz0SgvChv8feFnjjVvV1eqizFZtgmLMr5B4FgA@mail.gmail.com>
Message-ID: <CAPAaX_jq+2PvSHWANCaRTay65iK5-dGbB4-h1w7fAdu0X+z=nw@mail.gmail.com>

>>> class MyClass(): pass
>>> class MyClass: pass
>>> def my_func(): pass
>>> def my_func: pass
  File "<stdin>", line 1
    def my_func:
               ^
SyntaxError: invalid syntax

http://www.youtube.com/watch?v=etuPF1yJRzg

If only more of us grew up watching sesame street.

On Friday, February 1, 2013, Jason Keene wrote:

> Both class and function definitions produce callable objects that are
> assigned to a name with nearly identical syntax.  I don't think your
> analogy with import/break statements is valid.  Additionally, both
> definitions produce very similar byte codes:
>
>               3 MAKE_FUNCTION            0
>               6 STORE_FAST               0 (my_func)
>
>               9 MAKE_FUNCTION            0
>              12 CALL_FUNCTION            0
>              15 BUILD_CLASS
>              16 STORE_FAST               0 (MyClass)
>
> The only argument I see for requiring empty parens in function
> definitions is a historical one.
>
> On Thursday, January 31, 2013, Ben Finney wrote:
>
>> Jason Keene <jasonkeene at gmail.com> writes:
>>
>> > In a way they do the same thing, they both create an object
>> > (function/class) from a suite and assign it to the name given after the
>> > keyword (def/class).  Sure they do totally different things with the
>> suite
>> > in creating the object, but in essence it's a name assignment.
>>
>> In a way, ?import? and ?break? do the same thing, they trigger the
>> compiler to compile a particular set of code bytes. Sure the code bytes
>> do totally different things, but in essence it's a statement.
>>
>> Less facetiously: You can blur your vision as much as you like to make
>> ?class? and ?def? look similar, but that doesn't diminish the importance
>> of the distinctions you're ignoring.
>>
>> --
>>  \       ?Always do right. This will gratify some people, and astonish |
>>   `\                                            the rest.? ?Mark Twain |
>> _o__)                                                                  |
>> Ben Finney
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/8bef1708/attachment.html>

From g.brandl at gmx.net  Fri Feb  1 10:12:24 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 01 Feb 2013 10:12:24 +0100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <20130130172627.32f64e71@pitrou.net>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<20130130172627.32f64e71@pitrou.net>
Message-ID: <keg0r3$ic9$1@ger.gmane.org>

Am 30.01.2013 17:26, schrieb Antoine Pitrou:

>> With a Python 3 metaclass that provides default values for *looked up*
>> entries you could have this:
>> 
>> class Color(Enum):
>>     RED, WHITE, BLUE
> 
> This relies on tuple evaluation order, and would also evaluate any
> other symbol looked up from inside the class body (which means I
> cannot add anything else than enum symbols to the class).
> 
> In other words, I'm afraid it would be somewhat fragile ;)

And it breaks static code checkers like pyflakes.

Georg



From storchaka at gmail.com  Fri Feb  1 10:27:16 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 01 Feb 2013 11:27:16 +0200
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
Message-ID: <keg1pk$ud8$1@ger.gmane.org>

I think that instead of the invention of implementation which covers all 
possible hypothetic scenarios, much more useful for the enums promotion 
would be if someone were to transform the existing constants in stdlib 
into enums with a minimum of the necessary capabilities.  This will show 
which features are the most important, which interface is more 
convenient from user's point of view, and what difficulties have to be 
faced.



From timothy.c.delaney at gmail.com  Fri Feb  1 12:41:58 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Fri, 1 Feb 2013 22:41:58 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <keg1pk$ud8$1@ger.gmane.org>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<keg1pk$ud8$1@ger.gmane.org>
Message-ID: <CAN8CLgmo1oX+OzdGj58ar5m13ejtaRdC-uf18OLkPdJMJ2VWVA@mail.gmail.com>

On 1 February 2013 20:27, Serhiy Storchaka <storchaka at gmail.com> wrote:

> I think that instead of the invention of implementation which covers all
> possible hypothetic scenarios


The scenarios I'm really interested in here is DRY; automatic assignment
(i.e. not having to specify any values); manual assignment (which should
allow any expression that is legal within a class definition) and being
able to introspect the enum.

Everything else falls out of supporting those scenarios within the normal
restrictions of enums (in particular 1:1 mapping) and conforming to obvious
interfaces.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/7ac0eaf6/attachment.html>

From barry at python.org  Fri Feb  1 16:36:56 2013
From: barry at python.org (Barry Warsaw)
Date: Fri, 1 Feb 2013 10:36:56 -0500
Subject: [Python-ideas] constant/enum type in stdlib
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
Message-ID: <20130201103656.0b24c40d@anarchist.wooz.org>

On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote:

>Last version (for now). I'm really interested in people's opinions on this.
>For this version I've taken some inspiration from flufl.enum (but there
>remains the major difference that these enums subclass int).

Why not package it up and put it in PyPI?  Better there than sitting in an
email thread of some mailing list full of crazy people. :)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/39dfa4d2/attachment.pgp>

From ned at nedbatchelder.com  Fri Feb  1 17:43:55 2013
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Fri, 01 Feb 2013 11:43:55 -0500
Subject: [Python-ideas] Definition Symmetry
In-Reply-To: <CAPAaX_gJ32j1fz0SgvChv8feFnjjVvV1eqizFZtgmLMr5B4FgA@mail.gmail.com>
References: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
	<510AFD06.1090305@pearwood.info>
	<CAPAaX_ifjthen8JiLvKetmTDbmrzAUKydqXxKfvF+68feqthyg@mail.gmail.com>
	<7wa9roejym.fsf@benfinney.id.au>
	<CAPAaX_gJ32j1fz0SgvChv8feFnjjVvV1eqizFZtgmLMr5B4FgA@mail.gmail.com>
Message-ID: <510BF0CB.6020705@nedbatchelder.com>

On 2/1/2013 12:21 AM, Jason Keene wrote:
> Both class and function definitions produce callable objects that are 
> assigned to a name with nearly identical syntax. I don't think your 
> analogy with import/break statements is valid.  Additionally, both 
> definitions produce very similar byte codes:
>
>               3 MAKE_FUNCTION            0
>               6 STORE_FAST               0 (my_func)
>
>               9 MAKE_FUNCTION            0
>              12 CALL_FUNCTION            0
>              15 BUILD_CLASS
>              16 STORE_FAST               0 (MyClass)
>
> The only argument I see for requiring empty parens in function 
> definitions is a historical one.

That's because you're ignoring the main one:  function calls require 
parens, so function definitions also require them.

--Ned.

>
> On Thursday, January 31, 2013, Ben Finney wrote:
>
>     Jason Keene <jasonkeene at gmail.com <javascript:;>> writes:
>
>     > In a way they do the same thing, they both create an object
>     > (function/class) from a suite and assign it to the name given
>     after the
>     > keyword (def/class).  Sure they do totally different things with
>     the suite
>     > in creating the object, but in essence it's a name assignment.
>
>     In a way, 'import' and 'break' do the same thing, they trigger the
>     compiler to compile a particular set of code bytes. Sure the code
>     bytes
>     do totally different things, but in essence it's a statement.
>
>     Less facetiously: You can blur your vision as much as you like to make
>     'class' and 'def' look similar, but that doesn't diminish the
>     importance
>     of the distinctions you're ignoring.
>
>     --
>      \       "Always do right. This will gratify some people, and
>     astonish |
>       `\                                            the rest." ---Mark
>     Twain |
>     _o__)                |
>     Ben Finney
>
>     _______________________________________________
>     Python-ideas mailing list
>     Python-ideas at python.org <javascript:;>
>     http://mail.python.org/mailman/listinfo/python-ideas
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/7478cded/attachment.html>

From ericsnowcurrently at gmail.com  Fri Feb  1 20:13:41 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 1 Feb 2013 12:13:41 -0700
Subject: [Python-ideas] Definition Symmetry
In-Reply-To: <CAPAaX_gJ32j1fz0SgvChv8feFnjjVvV1eqizFZtgmLMr5B4FgA@mail.gmail.com>
References: <CAPAaX_g8uuU9jhnWD_G6Xc3Anw8Pjkeu7UO5R_hDbXQdArkpPA@mail.gmail.com>
	<510AFD06.1090305@pearwood.info>
	<CAPAaX_ifjthen8JiLvKetmTDbmrzAUKydqXxKfvF+68feqthyg@mail.gmail.com>
	<7wa9roejym.fsf@benfinney.id.au>
	<CAPAaX_gJ32j1fz0SgvChv8feFnjjVvV1eqizFZtgmLMr5B4FgA@mail.gmail.com>
Message-ID: <CALFfu7D3WJTW4DCNSJtQVqAKCTwMPidAqkEi7J3Z4rQz-JAj5Q@mail.gmail.com>

On Thu, Jan 31, 2013 at 10:21 PM, Jason Keene <jasonkeene at gmail.com> wrote:
> Additionally, both definitions
> produce very similar byte codes:
>
>               3 MAKE_FUNCTION            0
>               6 STORE_FAST               0 (my_func)
>
>               9 MAKE_FUNCTION            0
>              12 CALL_FUNCTION            0
>              15 BUILD_CLASS
>              16 STORE_FAST               0 (MyClass)

That's just an implementation detail.

-eric


From eliben at gmail.com  Fri Feb  1 20:30:23 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Fri, 1 Feb 2013 11:30:23 -0800
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <20130201103656.0b24c40d@anarchist.wooz.org>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
Message-ID: <CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>

On Fri, Feb 1, 2013 at 7:36 AM, Barry Warsaw <barry at python.org> wrote:

> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote:
>
> >Last version (for now). I'm really interested in people's opinions on
> this.
> >For this version I've taken some inspiration from flufl.enum (but there
> >remains the major difference that these enums subclass int).
>
> Why not package it up and put it in PyPI?  Better there than sitting in an
> email thread of some mailing list full of crazy people. :)
>
>
I would actually prefer a place where it's easy to see the code, comment on
it and fork it like Bitbucket or Github. PyPI is good for other purposes...

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/38fe4c89/attachment.html>

From jsbueno at python.org.br  Fri Feb  1 20:47:14 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 1 Feb 2013 17:47:14 -0200
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
Message-ID: <CAH0mxTRsRvBqZAB_=+fRvdDcibEX=sq+Uo3NfLPfD-FzGE9mdQ@mail.gmail.com>

On 1 February 2013 17:30, Eli Bendersky <eliben at gmail.com> wrote:

>
>
>
> On Fri, Feb 1, 2013 at 7:36 AM, Barry Warsaw <barry at python.org> wrote:
>
>> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote:
>>
>> >Last version (for now). I'm really interested in people's opinions on
>> this.
>> >For this version I've taken some inspiration from flufl.enum (but there
>> >remains the major difference that these enums subclass int).
>>
>> Why not package it up and put it in PyPI?  Better there than sitting in an
>> email thread of some mailing list full of crazy people. :)
>>
>>
> I would actually prefer a place where it's easy to see the code, comment
> on it and fork it like Bitbucket or Github. PyPI is good for other
> purposes...
>
Indeed -
there are plenty of nice enums on pypi already - my call for getting then
to the stdlib
is that we can get a reliable enum/constant behavior  to be always there -
not only
for the constants in the stdlib, but for projects for which adding external
dependencies
would be expensive (as they currently have none)

  js
 -><-

>
>
> Eli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/49088fc3/attachment.html>

From ericsnowcurrently at gmail.com  Fri Feb  1 21:09:48 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 1 Feb 2013 13:09:48 -0700
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
Message-ID: <CALFfu7Cs1pEUqaKU2XrxqdJ6T-jZar0XJY5mdcjrwSP-+v573A@mail.gmail.com>

On Fri, Feb 1, 2013 at 12:30 PM, Eli Bendersky <eliben at gmail.com> wrote:
> On Fri, Feb 1, 2013 at 7:36 AM, Barry Warsaw <barry at python.org> wrote:
>> Why not package it up and put it in PyPI?  Better there than sitting in an
>> email thread of some mailing list full of crazy people. :)
>
> I would actually prefer a place where it's easy to see the code, comment on
> it and fork it like Bitbucket or Github. PyPI is good for other purposes...

+1

And put it on PyPI to make it more accessible.

-eric


From ericsnowcurrently at gmail.com  Fri Feb  1 21:11:39 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 1 Feb 2013 13:11:39 -0700
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <keg1pk$ud8$1@ger.gmane.org>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<keg1pk$ud8$1@ger.gmane.org>
Message-ID: <CALFfu7CAkyafD_8-3CNEe95S7eXjtHXsobG-vZ-4JQBY4DrpqA@mail.gmail.com>

On Fri, Feb 1, 2013 at 2:27 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> I think that instead of the invention of implementation which covers all
> possible hypothetic scenarios, much more useful for the enums promotion
> would be if someone were to transform the existing constants in stdlib into
> enums with a minimum of the necessary capabilities.  This will show which
> features are the most important, which interface is more convenient from
> user's point of view, and what difficulties have to be faced.

+1

There's a lot of precedent (and good reasons) for starting with a
minimal/basic implementation in the stdlib.

-eric


From jsbueno at python.org.br  Fri Feb  1 21:19:49 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 1 Feb 2013 18:19:49 -0200
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CALFfu7Cs1pEUqaKU2XrxqdJ6T-jZar0XJY5mdcjrwSP-+v573A@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
	<CALFfu7Cs1pEUqaKU2XrxqdJ6T-jZar0XJY5mdcjrwSP-+v573A@mail.gmail.com>
Message-ID: <CAH0mxTQ-d+R7ayMKry1tfZK-1zq1uSbLuN0BU9eTJHVue4X=+g@mail.gmail.com>

On 1 February 2013 18:09, Eric Snow <ericsnowcurrently at gmail.com> wrote:

> On Fri, Feb 1, 2013 at 12:30 PM, Eli Bendersky <eliben at gmail.com> wrote:
> > On Fri, Feb 1, 2013 at 7:36 AM, Barry Warsaw <barry at python.org> wrote:
> >> Why not package it up and put it in PyPI?  Better there than sitting in
> an
> >> email thread of some mailing list full of crazy people. :)
> >
> > I would actually prefer a place where it's easy to see the code, comment
> on
> > it and fork it like Bitbucket or Github. PyPI is good for other
> purposes...
>
> +1
>
> And put it on PyPI to make it more accessible.
>

As soon as there are some more goodies in there beyond the enumerator and
constants themselves.

I can see that a namedtuple - like utility function could be interesting to
create the enums -
and a way to import the generated constatnts to the current global
namespace that does not
violate good pratices.


I think that something like:

class MyEnum(Enum):
     RED, GREEN, BLUE

load_constants(MyEnum, globals() )

is a good solution that could respect both DRY and "explicit is better than
implicit"
and "Special cases aren't special enough to break the rules."

   js
  -><-

>
> -eric
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/785aed13/attachment.html>

From timothy.c.delaney at gmail.com  Fri Feb  1 21:28:57 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Sat, 2 Feb 2013 07:28:57 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <20130201103656.0b24c40d@anarchist.wooz.org>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
Message-ID: <CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>

On 2 February 2013 02:36, Barry Warsaw <barry at python.org> wrote:

> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote:
>
> >Last version (for now). I'm really interested in people's opinions on
> this.
> >For this version I've taken some inspiration from flufl.enum (but there
> >remains the major difference that these enums subclass int).
>
> Why not package it up and put it in PyPI?  Better there than sitting in an
> email thread of some mailing list full of crazy people. :)
>

Yep - I intend to. Was just hacking away initially to see if what I wanted
to achieve was feasible using Michael's metaclass as a base. It's in a
Mercurial repo so I'll put it up on BitBucket in the next day or two and
look at cleaning it up (documentation!) to put on PyPI.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/fe0f41d3/attachment.html>

From jsbueno at python.org.br  Fri Feb  1 21:39:28 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 1 Feb 2013 18:39:28 -0200
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
Message-ID: <CAH0mxTTKNSBDebjRTKGQjLAJcS3Rm+84rD4uz+sMa8vNZrb1aw@mail.gmail.com>

On 1 February 2013 18:28, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> On 2 February 2013 02:36, Barry Warsaw <barry at python.org> wrote:
>
>> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote:
>>
>> >Last version (for now). I'm really interested in people's opinions on
>> this.
>> >For this version I've taken some inspiration from flufl.enum (but there
>> >remains the major difference that these enums subclass int).
>>
>> Why not package it up and put it in PyPI?  Better there than sitting in an
>> email thread of some mailing list full of crazy people. :)
>>
>
> Yep - I intend to. Was just hacking away initially to see if what I wanted
> to achieve was feasible using Michael's metaclass as a base. It's in a
> Mercurial repo so I'll put it up on BitBucket in the next day or two and
> look at cleaning it up (documentation!) to put on PyPI.
>

Indeed . .as I said, we'd better put together a pre-PEP and making this
proof of concept impement most of it before going to Pypy -

There are other requisites that I think aren't taken care of that come form
the 2010 thread -
for example, pickle-ability.

  js
 -><-

>
> Tim Delaney
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/922a79f2/attachment.html>

From barry at python.org  Fri Feb  1 22:04:46 2013
From: barry at python.org (Barry Warsaw)
Date: Fri, 1 Feb 2013 16:04:46 -0500
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
Message-ID: <20130201160446.081bb6d3@anarchist.wooz.org>

On Feb 01, 2013, at 11:30 AM, Eli Bendersky wrote:

>I would actually prefer a place where it's easy to see the code, comment on
>it and fork it like Bitbucket or Github. PyPI is good for other purposes...

Well, sure, having the code in a publicly available vcs is always a good
idea.  My point really was that stuff ordinarily doesn't land in the stdlib
until it's lived on PyPI for a while.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/890be14c/attachment.pgp>

From ubershmekel at gmail.com  Fri Feb  1 22:53:14 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Fri, 1 Feb 2013 23:53:14 +0200
Subject: [Python-ideas] PEP 3156: getting the socket or peer name from
	the transport
In-Reply-To: <CAP7+vJKHqnGCcjMsGcc9A623qrkiJzG-tv1Q5VWBGsW25e8FUw@mail.gmail.com>
References: <CAP7+vJ+_QwQW13b1mE1NSg8Z89MyYwHkyYmLwhumGo0iO-V+wA@mail.gmail.com>
	<CANSw7Kyi7d-Q5O=MqZTx5a-bOA3CQ4Me5RVafxJ0e7Kj1w693w@mail.gmail.com>
	<CAP7+vJ+BxgVPv0jBP=9GAdnXGdLRqgjCcBhVzVnJ4K7NDAxrjQ@mail.gmail.com>
	<CANSw7KwafX+qGiMszK7LVMs=xacmyEUjGLDPdqGzfW95GxbXfg@mail.gmail.com>
	<20130127122121.6b779ada@pitrou.net>
	<CANSw7KyuPCg9Ot6tY3ML_WBquO0PfFVqpzWPRpa3o8gmTyVS_A@mail.gmail.com>
	<1359288997.3488.2.camel@localhost.localdomain>
	<CAP7+vJ+av5zRDDLThgo7CCn_xLJ7rY3u-myP6Hihf50j0z7pYQ@mail.gmail.com>
	<EDC83381-4C64-4215-A90B-C72F2327BCA7@umbrellacode.com>
	<CAP7+vJKn6hE1zWujnDi=5dUtRsdovM7741G9bK0e4vQJvmbDPA@mail.gmail.com>
	<CANSw7KzG-ePHkuA4tv2LQ-+Fio2nWO0qv38WsH5QBY-JzDq3Eg@mail.gmail.com>
	<CAP7+vJKHqnGCcjMsGcc9A623qrkiJzG-tv1Q5VWBGsW25e8FUw@mail.gmail.com>
Message-ID: <CANSw7Kz8OrPibY8YDKGqB-iBYsvcvTy-+wz9w5NVnxKuoAAiiA@mail.gmail.com>

On Fri, Feb 1, 2013 at 3:58 AM, Guido van Rossum <guido at python.org> wrote:

> hasattr() smells bad. It also has namespace issues (hasattr(transport,
>  "write") returns true) and if people forget to use it (perhaps because
> the transport they normally use always has a certain attribute) their
> code is brittle. Defining a new API with a string key signals clearly
> that the value may or may not exist, and reminds people to test the
> result for None. (Of course they can forget that too. But it still
> feels different to me.)


I understand. It's a good solution for clearly separating the standard
transport api from the extra-and-specific api. We want to make it obvious
when a protocol is breaking the transport abstraction - to promote playing
nicely with different transports.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130201/979aaf56/attachment.html>

From greg.ewing at canterbury.ac.nz  Sat Feb  2 01:53:03 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 02 Feb 2013 13:53:03 +1300
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAH0mxTQ-d+R7ayMKry1tfZK-1zq1uSbLuN0BU9eTJHVue4X=+g@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
	<CALFfu7Cs1pEUqaKU2XrxqdJ6T-jZar0XJY5mdcjrwSP-+v573A@mail.gmail.com>
	<CAH0mxTQ-d+R7ayMKry1tfZK-1zq1uSbLuN0BU9eTJHVue4X=+g@mail.gmail.com>
Message-ID: <510C636F.6060304@canterbury.ac.nz>

Joao S. O. Bueno wrote:
> class MyEnum(Enum):
>      RED, GREEN, BLUE
> 
> load_constants(MyEnum, globals() )

The "obvious" way to spell this would be

    from MyEnum import *

but it would be challenging to make that work, I suspect. :-(

-- 
Greg


From ethan at stoneleaf.us  Sat Feb  2 02:18:53 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 01 Feb 2013 17:18:53 -0800
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <510C636F.6060304@canterbury.ac.nz>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
	<CALFfu7Cs1pEUqaKU2XrxqdJ6T-jZar0XJY5mdcjrwSP-+v573A@mail.gmail.com>
	<CAH0mxTQ-d+R7ayMKry1tfZK-1zq1uSbLuN0BU9eTJHVue4X=+g@mail.gmail.com>
	<510C636F.6060304@canterbury.ac.nz>
Message-ID: <510C697D.6020400@stoneleaf.us>

On 02/01/2013 04:53 PM, Greg Ewing wrote:
> Joao S. O. Bueno wrote:
>> class MyEnum(Enum):
>>      RED, GREEN, BLUE
>>
>> load_constants(MyEnum, globals() )
>
> The "obvious" way to spell this would be
>
>     from MyEnum import *
>
> but it would be challenging to make that work, I suspect. :-(

It's not too tough:

8<---- constants.py --------------------------------------------------
class Colors(object):
     BLACK = 0
     RED = 1
     GREEN = 2
     BLUE = 3
     __all__ = ('BLACK','RED','GREEN','BLUE')
     @classmethod
     def register(cls):
         import sys
         sys.modules['%s.%s' % (__name__, cls.__name__)] = cls()
Colors.register()
8<---- constants.py --------------------------------------------------

--> from constants.Colors import *
--> RED
1
--> BLUE
3

~Ethan~


From greg.ewing at canterbury.ac.nz  Sat Feb  2 02:41:01 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 02 Feb 2013 14:41:01 +1300
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <510C697D.6020400@stoneleaf.us>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAF-Rda-OSew7zY0wjV2bhSWFwF5w1Ub48f5DaVBzpUqw_s-uDw@mail.gmail.com>
	<CALFfu7Cs1pEUqaKU2XrxqdJ6T-jZar0XJY5mdcjrwSP-+v573A@mail.gmail.com>
	<CAH0mxTQ-d+R7ayMKry1tfZK-1zq1uSbLuN0BU9eTJHVue4X=+g@mail.gmail.com>
	<510C636F.6060304@canterbury.ac.nz> <510C697D.6020400@stoneleaf.us>
Message-ID: <510C6EAD.80009@canterbury.ac.nz>

Ethan Furman wrote:
> On 02/01/2013 04:53 PM, Greg Ewing wrote:
> 
>> The "obvious" way to spell this would be
>>
>>     from MyEnum import *
>>
>> but it would be challenging to make that work, I suspect. :-(
> 
> 
> It's not too tough:

Yeah, I just took up my own challenge and came up with something
similar (apologies for the Python 2):

   class MetaEnum(type):

     def __init__(self, name, bases, dict):
       type.__init__(self, name, bases, dict)
       import sys
       sys.modules[name] = self

   class Enum(object):
     __metaclass__ = MetaEnum		

   class MyEnum(Enum):
   RED = 0
   GREEN = 1
   BLUE = 2

   from MyEnum import *

   print RED
   print GREEN
   print BLUE

I left off the module name so that you don't have to qualify the
import. A more general version would qualify it with all but the
last component of the module name, so you can import it relative
to the containing module.

-- 
Greg


From gotoalanlu at gmail.com  Sat Feb  2 07:24:34 2013
From: gotoalanlu at gmail.com (Hua Lu)
Date: Sat, 2 Feb 2013 00:24:34 -0600
Subject: [Python-ideas] frozenset literals
Message-ID: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>

This is similar to another proposal:
http://mail.python.org/pipermail/python-3000/2008-January/011798.html

Anyway, I was using ast.literal_eval and attempted to use frozenset({...})
as a key in a dictionary, which failed, because frozenset isn't a literal
(though putting frozenset in the environment would be a security risk). I
am currently working around this with tuples, but I'd like a literal for
representing frozensets as well. I also use frozensets elsewhere in the
code in ways similar to Raymond's original suggestion.

Perhaps something like f{...} for declaring frozenset( comprehension)?
literals?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/7ac1273f/attachment.html>

From stefan_ml at behnel.de  Sat Feb  2 08:34:27 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 02 Feb 2013 08:34:27 +0100
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
Message-ID: <keifi0$ej4$1@ger.gmane.org>

Hua Lu, 02.02.2013 07:24:
> This is similar to another proposal:
> http://mail.python.org/pipermail/python-3000/2008-January/011798.html
> 
> Anyway, I was using ast.literal_eval and attempted to use frozenset({...})
> as a key in a dictionary, which failed, because frozenset isn't a literal

This has nothing to do with being a literal or not. The way you created
your frozenset doesn't impact its behaviour.

Could you give an example of what's not working for you? Frozensets as dict
keys work just fine for me:

  Python 3.2.3 (default, Oct 19 2012, 19:53:16)
  [GCC 4.7.2] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> frozenset({1,2,3})
  frozenset({1, 2, 3})
  >>> hash(frozenset({1,2,3}))
  -7699079583225461316
  >>> { frozenset({1,2,3}) : 1 }
  {frozenset({1, 2, 3}): 1}

Stefan




From gotoalanlu at gmail.com  Sat Feb  2 08:38:06 2013
From: gotoalanlu at gmail.com (Hua Lu)
Date: Sat, 2 Feb 2013 01:38:06 -0600
Subject: [Python-ideas] frozenset literals
In-Reply-To: <keifi0$ej4$1@ger.gmane.org>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
Message-ID: <CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>

>>> import ast
>>> ast.literal_eval("{ 'key': 'val' }")
{'key': 'val'}
>>> ast.literal_eval("{ ('key',): 'val' }")
{('key',): 'val'}
>>> ast.literal_eval("{ frozenset({'key'}): 'val' }")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.3/ast.py", line 86, in literal_eval
    return _convert(node_or_string)
  File "/usr/lib/python3.3/ast.py", line 63, in _convert
    in zip(node.keys, node.values))
  File "/usr/lib/python3.3/ast.py", line 62, in <genexpr>
    return dict((_convert(k), _convert(v)) for k, v
  File "/usr/lib/python3.3/ast.py", line 85, in _convert
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Call object at 0x7f865a8c1450>



On Sat, Feb 2, 2013 at 1:34 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Hua Lu, 02.02.2013 07:24:
> > This is similar to another proposal:
> > http://mail.python.org/pipermail/python-3000/2008-January/011798.html
> >
> > Anyway, I was using ast.literal_eval and attempted to use
> frozenset({...})
> > as a key in a dictionary, which failed, because frozenset isn't a literal
>
> This has nothing to do with being a literal or not. The way you created
> your frozenset doesn't impact its behaviour.
>
> Could you give an example of what's not working for you? Frozensets as dict
> keys work just fine for me:
>
>   Python 3.2.3 (default, Oct 19 2012, 19:53:16)
>   [GCC 4.7.2] on linux2
>   Type "help", "copyright", "credits" or "license" for more information.
>   >>> frozenset({1,2,3})
>   frozenset({1, 2, 3})
>   >>> hash(frozenset({1,2,3}))
>   -7699079583225461316
>   >>> { frozenset({1,2,3}) : 1 }
>   {frozenset({1, 2, 3}): 1}
>
> Stefan
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/d8f06893/attachment.html>

From shane at umbrellacode.com  Sat Feb  2 12:46:28 2013
From: shane at umbrellacode.com (Shane Green)
Date: Sat, 02 Feb 2013 03:46:28 -0800
Subject: [Python-ideas] for line in input with open(path) as input...
Message-ID: <510CFC94.5000502@umbrellacode.com>

with open(path) as input:
     for line in input:
         do(line)

Using with to create reference to opened file returned by open() so it 
could temporarily be assigned to input for the sole purpose of iterating 
its contents never sat very well with me.

  * The context manager returned by open() exists only to create the
    context and return reference "input";
  * the context and code block created by the "with" only exists for
    inner "for" loop's code block to execute in.

Now, given a generator function:

def iterwith(cm):
     with cm as context:
         if context is None:
             context = cm
         for item in context:
             yield item

The previous code can be turned into:

for line in iterwith(open(path)):
     do(line)

So, questions:
     - Is there anything inherently wrong with the idea, or does it exist?
     - Is it a generally useful tool, or are the examples limited to files?
     - Is it possible a more general mechanism could have value, such as:

for line in file with open(path) as file:
     do(line)


The preceding could be leveraged to different effect:

for line in file with locked(path):
write(path + ".out", line)

Or,
     for line in input with nested(open(path),lock,open(opath)) as 
input,locked,output:
         output.write(line)

To revisit the original purpose of "with", this seems to cleanly address 
a very common scenario wherein:

resource = create_resource()
try:
     for item in resource:
         do_something(resource, item)
except:
     raise
finally:
     cleanup()

# Standard with approach...
with create_resource() as resource:
     for item in resource:
         do_something(resource, item)

# With for loop as context...
for item in resource with create_resource() as resource:
     do_something(resource, item)

And, given the translation into statements, maybe even crazy stuff...

[line for line in file with open(path) as file]

J/K, I think.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/47a071cc/attachment.html>

From steve at pearwood.info  Sat Feb  2 13:46:26 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 02 Feb 2013 23:46:26 +1100
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <510CFC94.5000502@umbrellacode.com>
References: <510CFC94.5000502@umbrellacode.com>
Message-ID: <510D0AA2.2060805@pearwood.info>

On 02/02/13 22:46, Shane Green wrote:
> with open(path) as input:
> for line in input:
> do(line)
>
> Using with to create reference to opened file returned by open() so it could temporarily be assigned to input for the sole purpose of iterating its contents never sat very well with me.

It's not the *sole* purpose. If all you want it to iterate over the file, you can do this:

for line in open(path):
     ...


and no context manager is created. The context manager is also responsible for closing the file immediately you exit the block, without waiting for the caller to manually close it, or the garbage collector to (eventually) close it. So it is not *solely* for iteration.

File context managers can also be used for more than just iteration:

with open(path) as input:
     text = input.read()


with open(path, 'r+') as output:
     output.write('ZZ')

and so forth.


> * The context manager returned by open() exists only to create the
> context and return reference "input";
> * the context and code block created by the "with" only exists for
> inner "for" loop's code block to execute in.


I don't understand that objection. As I see it, that's a bit like saying "the len function exists only to get the length of objects". What did you expect the context manager to exist for if not to do the things you say?

What am I missing?




-- 
Steven


From rosuav at gmail.com  Sat Feb  2 14:01:00 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 3 Feb 2013 00:01:00 +1100
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <510D0AA2.2060805@pearwood.info>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
Message-ID: <CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>

On Sat, Feb 2, 2013 at 11:46 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> It's not the *sole* purpose. If all you want it to iterate over the file,
> you can do this:
>
> for line in open(path):
>     ...

If I understand the OP, the issue is that the 'with' creates a name
binding and an indentation level for no purpose; it's like doing this:

f = open(path)
for line in f:
    ...

In that instance, it's possible to inline the function call and use
its result directly; it would be nice to be able to do the same with a
context manager. However, since 'with' isn't an expression, it's not
possible to directly inline the two.

I think the utility function iterwith() is a good - and probably the
best - method; it demands nothing special from the language, and works
quite happily.

ChrisA


From stefan_ml at behnel.de  Sat Feb  2 14:06:31 2013
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 02 Feb 2013 14:06:31 +0100
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
Message-ID: <kej30k$tpi$1@ger.gmane.org>

Hua Lu, 02.02.2013 08:38:
> >>> import ast
> >>> ast.literal_eval("{ 'key': 'val' }")
> {'key': 'val'}
> >>> ast.literal_eval("{ ('key',): 'val' }")
> {('key',): 'val'}
> >>> ast.literal_eval("{ frozenset({'key'}): 'val' }")
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python3.3/ast.py", line 86, in literal_eval
>     return _convert(node_or_string)
>   File "/usr/lib/python3.3/ast.py", line 63, in _convert
>     in zip(node.keys, node.values))
>   File "/usr/lib/python3.3/ast.py", line 62, in <genexpr>
>     return dict((_convert(k), _convert(v)) for k, v
>   File "/usr/lib/python3.3/ast.py", line 85, in _convert
>     raise ValueError('malformed node or string: ' + repr(node))
> ValueError: malformed node or string: <_ast.Call object at 0x7f865a8c1450>

So, why exactly are you using ast.literal_eval() to evaluate a non-literal
expression?

Stefan

PS: please reply without top-posting.




From shane at umbrellacode.com  Sat Feb  2 14:21:38 2013
From: shane at umbrellacode.com (Shane Green)
Date: Sat, 02 Feb 2013 05:21:38 -0800
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <510D0AA2.2060805@pearwood.info>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
Message-ID: <510D12E2.3040000@umbrellacode.com>

Sorry, I was definitely unclear: I understand that isn't their sole 
purpose, and what role they play, in both the iteration examples and 
others, and I don't mean to "object" to anything, nor was I suggesting 
context-manager current implementation was uneeded, etc.

My point is that, in the examples I listed, the sole purpose of the 
context-manager is to provide a context the for loop will execute in.

with manager as context:
     # No statements here...
     for items in context:
        # blah
     # no statements here...

I was suggesting this scenario, wherein the body of a for loop is, in 
fact, the block of code that acts upon some resource or set of resource 
being managed by a context manager, might be common enough to warrant a 
closer look, as the more "pythonic" approach might accept the for loop's 
body as the with statement's code block context.

with open(file) as input:
     # why have input exist here, if it's not used, other than...
     for line in input:
        # do something...
        # why have input--the context--exist outside of me?
     # why have input exist here?

And finally, why define the outer with block, for the sole pupose of 
containing the inner "for in.." loop's block?






> Steven D'Aprano <mailto:steve at pearwood.info>
> February 2, 2013 4:46 AM
> On 02/02/13 22:46, Shane Green wrote:
>> with open(path) as input:
>> for line in input:
>> do(line)
>>
>> Using with to create reference to opened file returned by open() so 
>> it could temporarily be assigned to input for the sole purpose of 
>> iterating its contents never sat very well with me.
>
> It's not the *sole* purpose. If all you want it to iterate over the 
> file, you can do this:
>
> for line in open(path):
>     ...
>
>
> and no context manager is created. The context manager is also 
> responsible for closing the file immediately you exit the block, 
> without waiting for the caller to manually close it, or the garbage 
> collector to (eventually) close it. So it is not *solely* for iteration.
>
> File context managers can also be used for more than just iteration:
>
> with open(path) as input:
>     text = input.read()
>
>
> with open(path, 'r+') as output:
>     output.write('ZZ')
>
> and so forth.
>
>
>> * The context manager returned by open() exists only to create the
>> context and return reference "input";
>> * the context and code block created by the "with" only exists for
>> inner "for" loop's code block to execute in.
>
>
> I don't understand that objection. As I see it, that's a bit like 
> saying "the len function exists only to get the length of objects". 
> What did you expect the context manager to exist for if not to do the 
> things you say?
>
> What am I missing?
>
>
>
>
> Shane Green <mailto:shane at umbrellacode.com>
> February 2, 2013 3:46 AM
> with open(path) as input:
>     for line in input:
>         do(line)
>
> Using with to create reference to opened file returned by open() so it 
> could temporarily be assigned to input for the sole purpose of 
> iterating its contents never sat very well with me.
>
>   * The context manager returned by open() exists only to create the
>     context and return reference "input";
>   * the context and code block created by the "with" only exists for
>     inner "for" loop's code block to execute in.
>
> Now, given a generator function:
>
> def iterwith(cm):
>     with cm as context:
>         if context is None:
>             context = cm
>         for item in context:
>             yield item
>
> The previous code can be turned into:
>
> for line in iterwith(open(path)):
>     do(line)
>
> So, questions:
>     - Is there anything inherently wrong with the idea, or does it exist?
>     - Is it a generally useful tool, or are the examples limited to files?
>     - Is it possible a more general mechanism could have value, such as:
>
> for line in file with open(path) as file:
>     do(line)
>
>
> The preceding could be leveraged to different effect:
>
> for line in file with locked(path):
> write(path + ".out", line)
>
> Or,
>     for line in input with nested(open(path),lock,open(opath)) as 
> input,locked,output:
>         output.write(line)
>
> To revisit the original purpose of "with", this seems to cleanly 
> address a very common scenario wherein:
>
> resource = create_resource()
> try:
>     for item in resource:
>         do_something(resource, item)
> except:
>     raise
> finally:
>     cleanup()
>
> # Standard with approach...
> with create_resource() as resource:
>     for item in resource:
>         do_something(resource, item)
>
> # With for loop as context...
> for item in resource with create_resource() as resource:
>     do_something(resource, item)
>
> And, given the translation into statements, maybe even crazy stuff...
>
> [line for line in file with open(path) as file]
>
> J/K, I think.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/3dab7dd7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/3dab7dd7/attachment.jpg>

From gotoalanlu at gmail.com  Sat Feb  2 14:33:17 2013
From: gotoalanlu at gmail.com (Hua Lu)
Date: Sat, 2 Feb 2013 07:33:17 -0600
Subject: [Python-ideas] frozenset literals
In-Reply-To: <kej30k$tpi$1@ger.gmane.org>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org>
Message-ID: <CAOOU=zigz+4SBM8rKLhKp1y-XNa=Z1ACnWLrOPEx1bO24QsWtQ@mail.gmail.com>

> So, why exactly are you using ast.literal_eval() to evaluate a non-literal
> expression?
>
> Stefan
>
> PS: please reply without top-posting.

Sorry, I'm quite new to mailing lists.

I want to be able to use frozensets in ast.literal_eval portions
(especially as semantically correct keys for certain domains where
tuples have too much structure). Writing an ast filter just to
accommodate a builtin type feels way too hacky to me, and there are
legitimate reasons for having frozenset literals exist, some of which
are discussed in the link to that 2008 discussion. Also, there are
serious hacks for eval, and after seeing some of those... ay caramba.

Thanks for being patient with my newbishness.


From storchaka at gmail.com  Sat Feb  2 14:50:27 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 02 Feb 2013 15:50:27 +0200
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
Message-ID: <kej5j3$lpa$1@ger.gmane.org>

On 02.02.13 15:01, Chris Angelico wrote:
> I think the utility function iterwith() is a good - and probably the
> best - method; it demands nothing special from the language, and works
> quite happily.

When should iterwith() call __exit__()?




From rosuav at gmail.com  Sat Feb  2 14:52:50 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 3 Feb 2013 00:52:50 +1100
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <kej5j3$lpa$1@ger.gmane.org>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
Message-ID: <CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>

On Sun, Feb 3, 2013 at 12:50 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 02.02.13 15:01, Chris Angelico wrote:
>>
>> I think the utility function iterwith() is a good - and probably the
>> best - method; it demands nothing special from the language, and works
>> quite happily.
>
>
> When should iterwith() call __exit__()?

According to the OP's posted code, as soon as the iterable runs out.
Not sure what happens if you don't exhaust it but I'm sure generator
functions have already solved that, too.

ChrisA


From storchaka at gmail.com  Sat Feb  2 14:53:15 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 02 Feb 2013 15:53:15 +0200
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
Message-ID: <kej5oa$lpa$2@ger.gmane.org>

On 02.02.13 08:24, Hua Lu wrote:
> Perhaps something like f{...} for declaring frozenset( comprehension)?
> literals?

You need also a syntax to specify an empty set and an empty frozenset.




From shane at umbrellacode.com  Sat Feb  2 14:55:14 2013
From: shane at umbrellacode.com (Shane Green)
Date: Sat, 02 Feb 2013 05:55:14 -0800
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <kej5j3$lpa$1@ger.gmane.org>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
Message-ID: <510D1AC2.1000304@umbrellacode.com>

iterwith() doesn't need to: __exit__() is invoked automatically when 
iteration completes and the loop exits.

> Serhiy Storchaka <mailto:storchaka at gmail.com>
> February 2, 2013 5:50 AM
>
>
> When should iterwith() call __exit__()?
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
> Chris Angelico <mailto:rosuav at gmail.com>
> February 2, 2013 5:01 AM
>
> If I understand the OP, the issue is that the 'with' creates a name
> binding and an indentation level for no purpose; it's like doing this:
>
> f = open(path)
> for line in f:
> ...
>
> In that instance, it's possible to inline the function call and use
> its result directly; it would be nice to be able to do the same with a
> context manager. However, since 'with' isn't an expression, it's not
> possible to directly inline the two.
>
> I think the utility function iterwith() is a good - and probably the
> best - method; it demands nothing special from the language, and works
> quite happily.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
> Steven D'Aprano <mailto:steve at pearwood.info>
> February 2, 2013 4:46 AM
> On 02/02/13 22:46, Shane Green wrote:
>> with open(path) as input:
>> for line in input:
>> do(line)
>>
>> Using with to create reference to opened file returned by open() so 
>> it could temporarily be assigned to input for the sole purpose of 
>> iterating its contents never sat very well with me.
>
> It's not the *sole* purpose. If all you want it to iterate over the 
> file, you can do this:
>
> for line in open(path):
>     ...
>
>
> and no context manager is created. The context manager is also 
> responsible for closing the file immediately you exit the block, 
> without waiting for the caller to manually close it, or the garbage 
> collector to (eventually) close it. So it is not *solely* for iteration.
>
> File context managers can also be used for more than just iteration:
>
> with open(path) as input:
>     text = input.read()
>
>
> with open(path, 'r+') as output:
>     output.write('ZZ')
>
> and so forth.
>
>
>> * The context manager returned by open() exists only to create the
>> context and return reference "input";
>> * the context and code block created by the "with" only exists for
>> inner "for" loop's code block to execute in.
>
>
> I don't understand that objection. As I see it, that's a bit like 
> saying "the len function exists only to get the length of objects". 
> What did you expect the context manager to exist for if not to do the 
> things you say?
>
> What am I missing?
>
>
>
>
> Shane Green <mailto:shane at umbrellacode.com>
> February 2, 2013 3:46 AM
> with open(path) as input:
>     for line in input:
>         do(line)
>
> Using with to create reference to opened file returned by open() so it 
> could temporarily be assigned to input for the sole purpose of 
> iterating its contents never sat very well with me.
>
>   * The context manager returned by open() exists only to create the
>     context and return reference "input";
>   * the context and code block created by the "with" only exists for
>     inner "for" loop's code block to execute in.
>
> Now, given a generator function:
>
> def iterwith(cm):
>     with cm as context:
>         if context is None:
>             context = cm
>         for item in context:
>             yield item
>
> The previous code can be turned into:
>
> for line in iterwith(open(path)):
>     do(line)
>
> So, questions:
>     - Is there anything inherently wrong with the idea, or does it exist?
>     - Is it a generally useful tool, or are the examples limited to files?
>     - Is it possible a more general mechanism could have value, such as:
>
> for line in file with open(path) as file:
>     do(line)
>
>
> The preceding could be leveraged to different effect:
>
> for line in file with locked(path):
> write(path + ".out", line)
>
> Or,
>     for line in input with nested(open(path),lock,open(opath)) as 
> input,locked,output:
>         output.write(line)
>
> To revisit the original purpose of "with", this seems to cleanly 
> address a very common scenario wherein:
>
> resource = create_resource()
> try:
>     for item in resource:
>         do_something(resource, item)
> except:
>     raise
> finally:
>     cleanup()
>
> # Standard with approach...
> with create_resource() as resource:
>     for item in resource:
>         do_something(resource, item)
>
> # With for loop as context...
> for item in resource with create_resource() as resource:
>     do_something(resource, item)
>
> And, given the translation into statements, maybe even crazy stuff...
>
> [line for line in file with open(path) as file]
>
> J/K, I think.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/6387b306/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/6387b306/attachment.jpg>

From shane at umbrellacode.com  Sat Feb  2 15:05:19 2013
From: shane at umbrellacode.com (Shane Green)
Date: Sat, 02 Feb 2013 06:05:19 -0800
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <510D1C4B.1020806@umbrellacode.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org> <510D1AC2.1000304@umbrellacode.com>
	<510D1C4B.1020806@umbrellacode.com>
Message-ID: <510D1D1F.4080206@umbrellacode.com>

*From:* Serhiy Storchaka <storchaka at gmail.com>
*Date:* February 2, 2013 5:50 AM
*To:* python-ideas at python.org
*Subject:* [Python-ideas] for line in input with open(path) as input...


When should iterwith() call __exit__()?

*From:* Shane Green <shane at umbrellacode.com>
*Date:* February 2, 2013 5:55 AM
*To:* Serhiy Storchaka <storchaka at gmail.com>
*CC:* python-ideas at python.org
*Subject:* [Python-ideas] for line in input with open(path) as input...
iterwith() doesn't need to: __exit__() is invoked automatically when 
iteration completes and the loop exits.

> I hadn't thought about the fact the context was created inside the 
> generator...  Nonetheless, cleanup would be automatic, always; and the 
> generator should exit eventually on close() if not exhausted, I 
> believe (need to review the details on gen cleanup).
>
>
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/292c4602/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/292c4602/attachment.jpg>

From ncoghlan at gmail.com  Sat Feb  2 15:18:23 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 00:18:23 +1000
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
	<CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
Message-ID: <CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>

The with statement block is needed to define *when* cleanup happens
(unconditionally at the end of the block).

The "iterwith" generator is currently pointless, as it results in
nondeterministic cleanup of the context manager, so you may as well not
bother and just rely on the underlying iterable's nondeterministic cleanup.

We're never going to add cleanup semantics directly to for loops because:
- separation of concerns is a good design principle
- Indentation levels are not a limited resource (anyone that thinks they
are may be forgetting that factoring out context managers, iterators and
subfunctions gives you more of them, and that judicious use of early
returns and continue statements can avoid wasting them)
- we already considered it when initially designing the with statement and
decided it was a bad idea.

I forget where that last part is written up. If it's not in PEP 343, 342,
346 or 340 (the full set of PEPs that led to the current with statement and
contextlib.contextmanager designs), it should be in one of the threads they
reference.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/2b60090b/attachment.html>

From steve at pearwood.info  Sat Feb  2 15:20:38 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 03 Feb 2013 01:20:38 +1100
Subject: [Python-ideas] frozenset literals
In-Reply-To: <kej30k$tpi$1@ger.gmane.org>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org>
Message-ID: <510D20B6.5040705@pearwood.info>

On 03/02/13 00:06, Stefan Behnel wrote:
> Hua Lu, 02.02.2013 08:38:
>>>>> import ast
>>>>> ast.literal_eval("{ 'key': 'val' }")
>> {'key': 'val'}
>>>>> ast.literal_eval("{ ('key',): 'val' }")
>> {('key',): 'val'}
>>>>> ast.literal_eval("{ frozenset({'key'}): 'val' }")
>> Traceback (most recent call last):
>>    File "<stdin>", line 1, in<module>
>>    File "/usr/lib/python3.3/ast.py", line 86, in literal_eval
>>      return _convert(node_or_string)
>>    File "/usr/lib/python3.3/ast.py", line 63, in _convert
>>      in zip(node.keys, node.values))
>>    File "/usr/lib/python3.3/ast.py", line 62, in<genexpr>
>>      return dict((_convert(k), _convert(v)) for k, v
>>    File "/usr/lib/python3.3/ast.py", line 85, in _convert
>>      raise ValueError('malformed node or string: ' + repr(node))
>> ValueError: malformed node or string:<_ast.Call object at 0x7f865a8c1450>
>
> So, why exactly are you using ast.literal_eval() to evaluate a non-literal
> expression?


Stefan, you did ask Hua Lu to show an example of what isn't working. It's just a demonstration of what doesn't work -- you can't create a frozenset using ast.literal_eval.

I think Hua Lu's original post made it quite clear. He wishes there to be a frozenset literal, because currently there is no way to have ast.literal_eval evaluate something containing a frozenset. Because there is no frozenset literal.

I think Raymond Hettinger's proposal back in 2008:

http://mail.python.org/pipermail/python-3000/2008-January/011798.html

and the following thread is worth reading. Guido even pronounced his agreement:

http://mail.python.org/pipermail/python-3000/2008-January/011814.html

but then changed his mind (as did Raymond). So the status quo remains.

Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:

http://mail.python.org/pipermail/python-3000/2008-January/011838.html

Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.



-- 
Steven


From ncoghlan at gmail.com  Sat Feb  2 15:28:55 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 00:28:55 +1000
Subject: [Python-ideas] frozenset literals
In-Reply-To: <kej5oa$lpa$2@ger.gmane.org>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<kej5oa$lpa$2@ger.gmane.org>
Message-ID: <CADiSq7dui1F5Koyk6hBVL=jTPydaFQS0J8sxtQbVjASnkhC5dg@mail.gmail.com>

FWIW, I could personally tolerate the introduction of s{} and fs{}
literals. We'd just declare the "s" prefix optional for non-empty sets to
match the current rules.

Encouraging the use of ast.literal_eval() over the security nightmare that
is eval() would be more than enough justification for me. (As a syntax
change, the idea would still need a PEP, though)

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/ed5de645/attachment.html>

From solipsis at pitrou.net  Sat Feb  2 15:28:56 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 2 Feb 2013 15:28:56 +0100
Subject: [Python-ideas] frozenset literals
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
Message-ID: <20130202152856.0ab823b9@pitrou.net>

On Sun, 03 Feb 2013 01:20:38 +1100
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:
> 
> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
> 
> Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.

I'm not sure what the point is. frozensets are too rarely used to
deserve a literal notation.

Regards

Antoine.




From solipsis at pitrou.net  Sat Feb  2 15:35:37 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 2 Feb 2013 15:35:37 +0100
Subject: [Python-ideas] frozenset literals
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<kej5oa$lpa$2@ger.gmane.org>
	<CADiSq7dui1F5Koyk6hBVL=jTPydaFQS0J8sxtQbVjASnkhC5dg@mail.gmail.com>
Message-ID: <20130202153537.1b5fca12@pitrou.net>

On Sun, 3 Feb 2013 00:28:55 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> FWIW, I could personally tolerate the introduction of s{} and fs{}
> literals. We'd just declare the "s" prefix optional for non-empty sets to
> match the current rules.
> 
> Encouraging the use of ast.literal_eval() over the security nightmare that
> is eval() would be more than enough justification for me. (As a syntax
> change, the idea would still need a PEP, though)

If it is enough of a justification, then why not literal support for
more useful datatypes (decimal, datetime...) rather than frozenset?

Regards

Antoine.




From ncoghlan at gmail.com  Sat Feb  2 15:40:47 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 00:40:47 +1000
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510D20B6.5040705@pearwood.info>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
Message-ID: <CADiSq7cNvX4afZZukkDrS3aN9TxtDfk=Y1RHm=OSbNDYfb6fjA@mail.gmail.com>

On 3 Feb 2013 00:21, "Steven D'Aprano" <steve at pearwood.info> wrote:
>
> On 03/02/13 00:06, Stefan Behnel wrote:
>>
>> Hua Lu, 02.02.2013 08:38:
>>>>>>
>>>>>> import ast
>>>>>> ast.literal_eval("{ 'key': 'val' }")
>>>
>>> {'key': 'val'}
>>>>>>
>>>>>> ast.literal_eval("{ ('key',): 'val' }")
>>>
>>> {('key',): 'val'}
>>>>>>
>>>>>> ast.literal_eval("{ frozenset({'key'}): 'val' }")
>>>
>>> Traceback (most recent call last):
>>>    File "<stdin>", line 1, in<module>
>>>    File "/usr/lib/python3.3/ast.py", line 86, in literal_eval
>>>      return _convert(node_or_string)
>>>    File "/usr/lib/python3.3/ast.py", line 63, in _convert
>>>      in zip(node.keys, node.values))
>>>    File "/usr/lib/python3.3/ast.py", line 62, in<genexpr>
>>>      return dict((_convert(k), _convert(v)) for k, v
>>>    File "/usr/lib/python3.3/ast.py", line 85, in _convert
>>>      raise ValueError('malformed node or string: ' + repr(node))
>>> ValueError: malformed node or string:<_ast.Call object at
0x7f865a8c1450>
>>
>>
>> So, why exactly are you using ast.literal_eval() to evaluate a
non-literal
>> expression?
>
>
>
> Stefan, you did ask Hua Lu to show an example of what isn't working. It's
just a demonstration of what doesn't work -- you can't create a frozenset
using ast.literal_eval.
>
> I think Hua Lu's original post made it quite clear. He wishes there to be
a frozenset literal, because currently there is no way to have
ast.literal_eval evaluate something containing a frozenset. Because there
is no frozenset literal.
>
> I think Raymond Hettinger's proposal back in 2008:
>
> http://mail.python.org/pipermail/python-3000/2008-January/011798.html
>
> and the following thread is worth reading. Guido even pronounced his
agreement:
>
> http://mail.python.org/pipermail/python-3000/2008-January/011814.html
>
> but then changed his mind (as did Raymond). So the status quo remains.
>
> Unfortunately the proposal to use f{ ... } for frozen sets cannot work
within the constraints of Python's lexer:
>
> http://mail.python.org/pipermail/python-3000/2008-January/011838.html

To clarify Guido's comment in that post, I'm fairly sure it *can* be made
to work, it just won' t be the same way that string prefixes work (because
the contents of dict and set displays are not opaque to the compiler the
way string contents are).

The hypothetical "What if we want to allow expr{} as a general construct?"
objection needs to be balanced against the immediate value of a more
expressive language subset for use in ast.literal_eval().

Cheers,
Nick.

>
> Unfortunately we're running out of useful, easy to enter symbols for
literals. Until such time (Python4000 perhaps, or more likely Python5000)
as we can use a rich set of Unicode literals, I don't think there is any
clear way to have a frozenset literal.
>
>
>
> --
> Steven
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/7b84a404/attachment.html>

From solipsis at pitrou.net  Sat Feb  2 15:46:21 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 2 Feb 2013 15:46:21 +0100
Subject: [Python-ideas] frozenset literals
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<CADiSq7cNvX4afZZukkDrS3aN9TxtDfk=Y1RHm=OSbNDYfb6fjA@mail.gmail.com>
Message-ID: <20130202154621.1133530a@pitrou.net>

On Sun, 3 Feb 2013 00:40:47 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 3 Feb 2013 00:21, "Steven D'Aprano" <steve at pearwood.info> wrote:
> >
> > On 03/02/13 00:06, Stefan Behnel wrote:
> >>
> >> Hua Lu, 02.02.2013 08:38:
> >>>>>>
> >>>>>> import ast
> >>>>>> ast.literal_eval("{ 'key': 'val' }")
> >>>
> >>> {'key': 'val'}
> >>>>>>
> >>>>>> ast.literal_eval("{ ('key',): 'val' }")
> >>>
> >>> {('key',): 'val'}
> >>>>>>
> >>>>>> ast.literal_eval("{ frozenset({'key'}): 'val' }")
> >>>
> >>> Traceback (most recent call last):
> >>>    File "<stdin>", line 1, in<module>
> >>>    File "/usr/lib/python3.3/ast.py", line 86, in literal_eval
> >>>      return _convert(node_or_string)
> >>>    File "/usr/lib/python3.3/ast.py", line 63, in _convert
> >>>      in zip(node.keys, node.values))
> >>>    File "/usr/lib/python3.3/ast.py", line 62, in<genexpr>
> >>>      return dict((_convert(k), _convert(v)) for k, v
> >>>    File "/usr/lib/python3.3/ast.py", line 85, in _convert
> >>>      raise ValueError('malformed node or string: ' + repr(node))
> >>> ValueError: malformed node or string:<_ast.Call object at
> 0x7f865a8c1450>
> >>
> >>
> >> So, why exactly are you using ast.literal_eval() to evaluate a
> non-literal
> >> expression?
> >
> >
> >
> > Stefan, you did ask Hua Lu to show an example of what isn't working. It's
> just a demonstration of what doesn't work -- you can't create a frozenset
> using ast.literal_eval.
> >
> > I think Hua Lu's original post made it quite clear. He wishes there to be
> a frozenset literal, because currently there is no way to have
> ast.literal_eval evaluate something containing a frozenset. Because there
> is no frozenset literal.
> >
> > I think Raymond Hettinger's proposal back in 2008:
> >
> > http://mail.python.org/pipermail/python-3000/2008-January/011798.html
> >
> > and the following thread is worth reading. Guido even pronounced his
> agreement:
> >
> > http://mail.python.org/pipermail/python-3000/2008-January/011814.html
> >
> > but then changed his mind (as did Raymond). So the status quo remains.
> >
> > Unfortunately the proposal to use f{ ... } for frozen sets cannot work
> within the constraints of Python's lexer:
> >
> > http://mail.python.org/pipermail/python-3000/2008-January/011838.html
> 
> To clarify Guido's comment in that post, I'm fairly sure it *can* be made
> to work, it just won' t be the same way that string prefixes work (because
> the contents of dict and set displays are not opaque to the compiler the
> way string contents are).
> 
> The hypothetical "What if we want to allow expr{} as a general construct?"
> objection needs to be balanced against the immediate value of a more
> expressive language subset for use in ast.literal_eval().

I've just tried: ast.literal_eval() works just fine on a normal set:

>>> ast.literal_eval("{1,2,3}")
{1, 2, 3}

So I'd like to know why people find it important to build a frozenset
rather than a normal set. The situation of wanting to use sets as hash
keys is very rare in my experience.

Regards

Antoine.




From ncoghlan at gmail.com  Sat Feb  2 15:56:57 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 00:56:57 +1000
Subject: [Python-ideas] frozenset literals
In-Reply-To: <20130202153537.1b5fca12@pitrou.net>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<kej5oa$lpa$2@ger.gmane.org>
	<CADiSq7dui1F5Koyk6hBVL=jTPydaFQS0J8sxtQbVjASnkhC5dg@mail.gmail.com>
	<20130202153537.1b5fca12@pitrou.net>
Message-ID: <CADiSq7cowSFabC9MvT1PiBzs7-5htGzLrmrO_4JDNr+zmpd+6Q@mail.gmail.com>

On 3 Feb 2013 00:39, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> On Sun, 3 Feb 2013 00:28:55 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> > FWIW, I could personally tolerate the introduction of s{} and fs{}
> > literals. We'd just declare the "s" prefix optional for non-empty sets
to
> > match the current rules.
> >
> > Encouraging the use of ast.literal_eval() over the security nightmare
that
> > is eval() would be more than enough justification for me. (As a syntax
> > change, the idea would still need a PEP, though)
>
> If it is enough of a justification, then why not literal support for
> more useful datatypes (decimal, datetime...) rather than frozenset?

The difference is that decimal and datetime already have safe "from string"
conversion operations. The empty set and frozen sets do not.

That said, a decimal literal proposal would be a reasonable follow-up to
the incorporation of cdecimal.

Datetime is a more difficult prospect, since there are good reasons
strptime is as flexible as it is.

Regardless, I'm not saying a PEP to support all the builtin container types
in ast.literal_eval would necessarily be accepted. I'm merely saying it is
*worth writing*.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/6ec770f6/attachment.html>

From solipsis at pitrou.net  Sat Feb  2 15:58:11 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 2 Feb 2013 15:58:11 +0100
Subject: [Python-ideas] frozenset literals
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<kej5oa$lpa$2@ger.gmane.org>
	<CADiSq7dui1F5Koyk6hBVL=jTPydaFQS0J8sxtQbVjASnkhC5dg@mail.gmail.com>
	<20130202153537.1b5fca12@pitrou.net>
	<CADiSq7cowSFabC9MvT1PiBzs7-5htGzLrmrO_4JDNr+zmpd+6Q@mail.gmail.com>
Message-ID: <20130202155811.0ba94896@pitrou.net>

On Sun, 3 Feb 2013 00:56:57 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 3 Feb 2013 00:39, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
> >
> > On Sun, 3 Feb 2013 00:28:55 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
> > > FWIW, I could personally tolerate the introduction of s{} and fs{}
> > > literals. We'd just declare the "s" prefix optional for non-empty sets
> to
> > > match the current rules.
> > >
> > > Encouraging the use of ast.literal_eval() over the security nightmare
> that
> > > is eval() would be more than enough justification for me. (As a syntax
> > > change, the idea would still need a PEP, though)
> >
> > If it is enough of a justification, then why not literal support for
> > more useful datatypes (decimal, datetime...) rather than frozenset?
> 
> The difference is that decimal and datetime already have safe "from string"
> conversion operations. The empty set and frozen sets do not.

You can call set() or frozenset() on a tuple literal, if you know you
want to end up with a set of frozenset.

Regards

Antoine.




From shane at umbrellacode.com  Sat Feb  2 16:16:41 2013
From: shane at umbrellacode.com (Shane Green)
Date: Sat, 02 Feb 2013 07:16:41 -0800
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
	<CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
	<CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>
Message-ID: <510D2DD9.7020109@umbrellacode.com>

Thanks Nick.  I definitely see your point about iterwith(); have been 
thinking about that since someone asked where __exit__() would be invoked.

I meant the following as a more compact way of expressing

for line in file with open(path) as file:
     process(line)

As a more compact way of expressing

with open(path) as file:
     for line in file:
         process(line)

Not a change to the semantics of for-loops; a point my iterwith() 
function has confuses greatly, I realize now. I'm not seeing a loss of 
separation of concerns there.

Indentation levels aren't limited, but flatter is better ;-)

I saw a bunch of back and forth regarding iteration and context 
management in the PEP, but didn't notice anything along these lines in 
particular . I'll have to go back and take a closer look.



Nick Coghlan wrote:
>
> The with statement block is needed to define *when* cleanup happens 
> (unconditionally at the end of the block).
>
> The "iterwith" generator is currently pointless, as it results in 
> nondeterministic cleanup of the context manager, so you may as well 
> not bother and just rely on the underlying iterable's nondeterministic 
> cleanup.
>
> We're never going to add cleanup semantics directly to for loops because:
> - separation of concerns is a good design principle
> - Indentation levels are not a limited resource (anyone that thinks 
> they are may be forgetting that factoring out context managers, 
> iterators and subfunctions gives you more of them, and that judicious 
> use of early returns and continue statements can avoid wasting them)
> - we already considered it when initially designing the with statement 
> and decided it was a bad idea.
>
> I forget where that last part is written up. If it's not in PEP 343, 
> 342, 346 or 340 (the full set of PEPs that led to the current with 
> statement and contextlib.contextmanager designs), it should be in one 
> of the threads they reference.
>
> Cheers,
> Nick.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/57c88fad/attachment.html>

From ubershmekel at gmail.com  Sat Feb  2 16:47:32 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Sat, 2 Feb 2013 17:47:32 +0200
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <510D2DD9.7020109@umbrellacode.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
	<CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
	<CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>
	<510D2DD9.7020109@umbrellacode.com>
Message-ID: <CANSw7KxAuY56Rn_3ZqmWeMU2Yf4Wy5kLGwWoQo=c3dwWZsWoiw@mail.gmail.com>

On Sat, Feb 2, 2013 at 5:16 PM, Shane Green <shane at umbrellacode.com> wrote:

> Thanks Nick.  I definitely see your point about iterwith(); have been
> thinking about that since someone asked where __exit__() would be invoked.
>
> I meant the following as a more compact way of expressing
>
> for line in file with open(path) as file:
>     process(line)
>
>

This is an interesting idea, though a bit too dense for my taste.


> Indentation levels aren't limited, but flatter is better ;-)
>
>
I really like Golang's solution (defer) which Nick sort of emulates with
ExitStack.

http://docs.python.org/3/library/contextlib.html#contextlib.ExitStack

If ExitStack ever became a language feature, we could write stuff like:


    def f():
        fhand = local open(path)
        process(fhand)
        ghand = local open(path2)
        process(ghand)

# which would be sort of equivalent to

    def g():
        try:
            fhand = None
            ghand = None
            fhand = local open(path)
            process(fhand)
            ghand = local open(path2)
            process(ghand)
        finally:
            if fhand is not None:
                fhand.close()
            if ghand is not None:
                ghand.close()



Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/8e707016/attachment.html>

From python at mrabarnett.plus.com  Sat Feb  2 19:30:30 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 02 Feb 2013 18:30:30 +0000
Subject: [Python-ideas] frozenset literals
In-Reply-To: <20130202152856.0ab823b9@pitrou.net>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
Message-ID: <510D5B46.8080409@mrabarnett.plus.com>

On 2013-02-02 14:28, Antoine Pitrou wrote:
> On Sun, 03 Feb 2013 01:20:38 +1100
> Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:
>>
>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
>>
>> Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.
>
> I'm not sure what the point is. frozensets are too rarely used to
> deserve a literal notation.
>
Does it need a special notation as such?

Wouldn't an alternative be to support "frozenset(...)" where "..." is
parseable by ast.literal_eval()?

The same thing could be done with some other built-in classes.


From gotoalanlu at gmail.com  Sat Feb  2 19:46:17 2013
From: gotoalanlu at gmail.com (Hua Lu)
Date: Sat, 2 Feb 2013 12:46:17 -0600
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510D5B46.8080409@mrabarnett.plus.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
Message-ID: <CAOOU=zh-9Ha1NNCKuoF5ReFh0WWT1eKWpLw7PNwV-izyxeGB-A@mail.gmail.com>

Perhaps, but we'd have to be careful with how we introduce those symbols:

http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html

You could possibly not allow attribute access but permit those
symbols... is there an exploit possible with that much filtering?

On Sat, Feb 2, 2013 at 12:30 PM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2013-02-02 14:28, Antoine Pitrou wrote:
>>
>> On Sun, 03 Feb 2013 01:20:38 +1100
>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>
>>>
>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work
>>> within the constraints of Python's lexer:
>>>
>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
>>>
>>> Unfortunately we're running out of useful, easy to enter symbols for
>>> literals. Until such time (Python4000 perhaps, or more likely Python5000) as
>>> we can use a rich set of Unicode literals, I don't think there is any clear
>>> way to have a frozenset literal.
>>
>>
>> I'm not sure what the point is. frozensets are too rarely used to
>> deserve a literal notation.
>>
> Does it need a special notation as such?
>
> Wouldn't an alternative be to support "frozenset(...)" where "..." is
> parseable by ast.literal_eval()?
>
> The same thing could be done with some other built-in classes.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From python at mrabarnett.plus.com  Sat Feb  2 20:07:09 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 02 Feb 2013 19:07:09 +0000
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CAOOU=zh-9Ha1NNCKuoF5ReFh0WWT1eKWpLw7PNwV-izyxeGB-A@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
	<CAOOU=zh-9Ha1NNCKuoF5ReFh0WWT1eKWpLw7PNwV-izyxeGB-A@mail.gmail.com>
Message-ID: <510D63DD.6020405@mrabarnett.plus.com>

On 2013-02-02 18:46, Hua Lu wrote:
> On Sat, Feb 2, 2013 at 12:30 PM, MRAB <python at mrabarnett.plus.com> wrote:
>> On 2013-02-02 14:28, Antoine Pitrou wrote:
>>> On Sun, 03 Feb 2013 01:20:38 +1100
>>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>>
>>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work
>>>> within the constraints of Python's lexer:
>>>>
>>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
>>>>
>>>> Unfortunately we're running out of useful, easy to enter symbols for
>>>> literals. Until such time (Python4000 perhaps, or more likely Python5000) as
>>>> we can use a rich set of Unicode literals, I don't think there is any clear
>>>> way to have a frozenset literal.
>>>
>>> I'm not sure what the point is. frozensets are too rarely used to
>>> deserve a literal notation.
>>>
>> Does it need a special notation as such?
>>
>> Wouldn't an alternative be to support "frozenset(...)" where "..." is
>> parseable by ast.literal_eval()?
>>
>> The same thing could be done with some other built-in classes.
>>
> Perhaps, but we'd have to be careful with how we introduce those symbols:
>
> http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html
>
> You could possibly not allow attribute access but permit those
> symbols... is there an exploit possible with that much filtering?
>
I'm talking about hard-coding "frozenset(...)" and a few other built-in
classes to return the appropriate object, so, for example, "set([0, 1,
2])" would be the same as "{0, 1, 2}".


From guido at python.org  Sat Feb  2 20:25:46 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 2 Feb 2013 11:25:46 -0800
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510D63DD.6020405@mrabarnett.plus.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
	<CAOOU=zh-9Ha1NNCKuoF5ReFh0WWT1eKWpLw7PNwV-izyxeGB-A@mail.gmail.com>
	<510D63DD.6020405@mrabarnett.plus.com>
Message-ID: <CAP7+vJLKtEvmmRk79b7SePwVJ6jZ-f0xKeY3_8UVVHMbc30-9g@mail.gmail.com>

On Sat, Feb 2, 2013 at 11:07 AM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2013-02-02 18:46, Hua Lu wrote:
>>
>> On Sat, Feb 2, 2013 at 12:30 PM, MRAB <python at mrabarnett.plus.com> wrote:
>>>
>>> On 2013-02-02 14:28, Antoine Pitrou wrote:
>>>>
>>>> On Sun, 03 Feb 2013 01:20:38 +1100
>>>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>>>
>>>>>
>>>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work
>>>>> within the constraints of Python's lexer:
>>>>>
>>>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
>>>>>
>>>>> Unfortunately we're running out of useful, easy to enter symbols for
>>>>> literals. Until such time (Python4000 perhaps, or more likely
>>>>> Python5000) as
>>>>> we can use a rich set of Unicode literals, I don't think there is any
>>>>> clear
>>>>> way to have a frozenset literal.
>>>>
>>>>
>>>> I'm not sure what the point is. frozensets are too rarely used to
>>>> deserve a literal notation.
>>>>
>>> Does it need a special notation as such?
>>>
>>> Wouldn't an alternative be to support "frozenset(...)" where "..." is
>>> parseable by ast.literal_eval()?
>>>
>>> The same thing could be done with some other built-in classes.
>>>
>> Perhaps, but we'd have to be careful with how we introduce those symbols:
>>
>> http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html
>>
>> You could possibly not allow attribute access but permit those
>> symbols... is there an exploit possible with that much filtering?
>>
> I'm talking about hard-coding "frozenset(...)" and a few other built-in
> classes to return the appropriate object, so, for example, "set([0, 1,
> 2])" would be the same as "{0, 1, 2}".

You're proposing to do this just for literal_eval(), right? But how
would you implement it? It seems it would require lots of special
cases. Where would you stop? dict(key1=..., key2=...)? list({...})?

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Sat Feb  2 20:30:46 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 2 Feb 2013 11:30:46 -0800
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <CANSw7KxAuY56Rn_3ZqmWeMU2Yf4Wy5kLGwWoQo=c3dwWZsWoiw@mail.gmail.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
	<CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
	<CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>
	<510D2DD9.7020109@umbrellacode.com>
	<CANSw7KxAuY56Rn_3ZqmWeMU2Yf4Wy5kLGwWoQo=c3dwWZsWoiw@mail.gmail.com>
Message-ID: <CAP7+vJK8znwp-2fw_p18ubJ+KT-dcLs-w0MSp3DuBbLvu+EbvA@mail.gmail.com>

None of these proposals have any merit. The last thing we need is more
ways to spell the same thing that can already be spelled in several
ways, all of which are just fine. Just because you save a line doesn't
make your code more readable.

-- 
--Guido van Rossum (python.org/~guido)


From raymond.hettinger at gmail.com  Sat Feb  2 20:54:45 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 2 Feb 2013 11:54:45 -0800
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510D20B6.5040705@pearwood.info>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
Message-ID: <F6D22965-6445-4186-A27D-72B7B23CBB3E@gmail.com>


On Feb 2, 2013, at 6:20 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> I think Raymond Hettinger's proposal back in 2008:
> 
> http://mail.python.org/pipermail/python-3000/2008-January/011798.html
> 
> and the following thread is worth reading. Guido even pronounced his agreement:
> 
> http://mail.python.org/pipermail/python-3000/2008-January/011814.html
> 
> but then changed his mind (as did Raymond). So the status quo remains.
> 
> Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:
> 
> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
> 
> Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset l

Wow, that was a nice recap of the historical discussions.
Thank you :-)

FWIW, if a workable frozenset literal were proposed, I would
have no objections.  

Their original use case was be useable 
as elements of other sets (for sets of sets) 
or as dictionary keys  (to implement graph structures).

Later, it became clear that it would be useful for code-optimization
to treat frozensets as constants that are built when a function
is defined, rather than later when it is invoked.

In the future, frozen sets may obtain other desirable properties.
For example, I'm still evaluating whether to store the 
key/hash entries in a dense array and storing only indicies
in a collision table.  For regular sets, the dense array needs
to over-allocate in order to grow efficiently.  In contrast,
a frozenset could free the excess memory and become
very compact (about a third of the current size). 

If that ever comes to fruition, then a frozenset literal would
be nice because of its compactness as compared to regular sets.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/9fe36ab2/attachment.html>

From ben at bendarnell.com  Sat Feb  2 21:10:37 2013
From: ben at bendarnell.com (Ben Darnell)
Date: Sat, 2 Feb 2013 15:10:37 -0500
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
Message-ID: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>

The event loop interface in PEP 3156 has an extensibility problem.  It
seems likely that it will have a method like listen_udp() by the time it's
done, but suppose that doesn't make it into the first official release.
 Third-party event loop implementations may want to provide UDP support as
an extension, but the most consistent way to provide that extension is by
adding new methods on the event loop object, where various extensions risk
conflicting with each other or with new methods that become standardized
later.

The PEP specifies the add_reader family of methods in part so that core
protocol implementations can be shared across all EventLoop implementations
that support these methods.  However, if those transports are implemented
in a common base class (like Twisted's PosixReactorBase), there is no way
for third-party transports to take advantage of a similar structure.  I'd
like to make it possible for transports to be developed independent of a
particular EventLoop implementation in a way that is consistent with the
way the core transports work.  (This is a bigger concern for tulip than it
is for twisted because twisted can update PosixReactorBase more frequently
than the stdlib can change)

I propose turning the interface around so that transport creation uses
static functions that take the event loop as an argument, with a
double-dispatch mechanism to allow the event loop to provide the actual
implementation when it can:

  def create_connection(protocol_factory, host, port, event_loop=None):
    if event_loop is None:
      event_loop = get_event_loop()
    # Note the use of a fully-qualified name in the registry
    impl = event_loop.get_implementation('tulip.create_connection')
    return impl(protocol_factory, host, port)

New third-party transports could provide fallbacks for event loops that
don't have their own implementations:

  if impl is None:
    # These supports_*() functions are placeholders for a to-be-determined
introspection interface.
    if supports_fd_interface(event_loop):
      return posix_udp_implementation(*args)
    elif supports_iocp_interface(event_loop):
      return iocp_udp_implementation(*args)
    else:
      raise Exception("This transport is not supported on this event loop")

Or they could plug into the event loop's implementation registry:

  LibUVEventLoop.register_implementation('mymodule.listen_udp',
libuv_udp_implementation)

This does introduce a little magic (is there any precedent for this kind of
multiple-dispatch in the standard library?), but I like the way it keeps
the event loop interface from getting too big and monolithic.  Third-party
transports can avoid naming conflicts without looking fundamentally
different from standard ones, and there's a clean path from doing something
that's platform-specific (e.g. with add_reader and friends) to supporting
multiple event loops to full standardization.

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/2cf5fcc7/attachment.html>

From markus at unterwaditzer.net  Sun Feb  3 00:52:17 2013
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Sun, 03 Feb 2013 00:52:17 +0100
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
Message-ID: <2bb9748e-9998-4800-9522-9c296de7623b@email.android.com>



Hua Lu <gotoalanlu at gmail.com> wrote:

>This is similar to another proposal:
>http://mail.python.org/pipermail/python-3000/2008-January/011798.html
>
>Anyway, I was using ast.literal_eval and attempted to use
>frozenset({...})
>as a key in a dictionary, which failed, because frozenset isn't a
>literal
>(though putting frozenset in the environment would be a security risk).
>I
>am currently working around this with tuples, but I'd like a literal
>for
>representing frozensets as well. I also use frozensets elsewhere in the
>code in ways similar to Raymond's original suggestion.
>
>Perhaps something like f{...} for declaring frozenset( comprehension)?
>literals?
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>http://mail.python.org/mailman/listinfo/python-ideas

What about {1, 2, 3}.frozen() ? While not exactly new syntax, i think it would be a good short way to create a frozenset.

-- Markus


From greg.ewing at canterbury.ac.nz  Sun Feb  3 02:27:39 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 03 Feb 2013 14:27:39 +1300
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CAP7+vJLKtEvmmRk79b7SePwVJ6jZ-f0xKeY3_8UVVHMbc30-9g@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
	<CAOOU=zh-9Ha1NNCKuoF5ReFh0WWT1eKWpLw7PNwV-izyxeGB-A@mail.gmail.com>
	<510D63DD.6020405@mrabarnett.plus.com>
	<CAP7+vJLKtEvmmRk79b7SePwVJ6jZ-f0xKeY3_8UVVHMbc30-9g@mail.gmail.com>
Message-ID: <510DBD0B.7040406@canterbury.ac.nz>

Guido van Rossum wrote:

> You're proposing to do this just for literal_eval(), right? But how
> would you implement it? It seems it would require lots of special
> cases. Where would you stop? dict(key1=..., key2=...)? list({...})?

I think you just do frozenset() and stop there until
someone asks for something else.

dict(key1=..., key2=...) and list({...}) are not needed,
because they can already be spelled as literals.

-- 
Greg



From steve at pearwood.info  Sun Feb  3 03:18:20 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 03 Feb 2013 13:18:20 +1100
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510D5B46.8080409@mrabarnett.plus.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
Message-ID: <510DC8EC.4010001@pearwood.info>

On 03/02/13 05:30, MRAB wrote:
> On 2013-02-02 14:28, Antoine Pitrou wrote:
>> On Sun, 03 Feb 2013 01:20:38 +1100
>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>
>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:
>>>
>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
>>>
>>> Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.
>>
>> I'm not sure what the point is. frozensets are too rarely used to
>> deserve a literal notation.
>>
> Does it need a special notation as such?
>
> Wouldn't an alternative be to support "frozenset(...)" where "..." is
> parseable by ast.literal_eval()?
>
> The same thing could be done with some other built-in classes.


I think that having literal_eval support non-literals is a bad, bad idea. Let's just not go there. It will surely end in tears.

However, I think that having something in between the strictness of literal_eval and the dangerous "anything goes" power of eval is a good idea. For a long time now I've toyed with an engine for building expression evaluators. Something that understands operators, function calls, etc, and you can tell it what names to accept. My main motivation is for evaluating mathematical expressions like:

5x^3 - 2x + log(1/y) + n!/??

I have some old, broken Pascal code that half does what I want, and I keep intending to revisit it some day in Python. In my copious spare time.

In any case, I think something like that belongs in a third-party module, at least at first.



-- 
Steven


From rosuav at gmail.com  Sun Feb  3 03:22:03 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 3 Feb 2013 13:22:03 +1100
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510DC8EC.4010001@pearwood.info>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
	<510DC8EC.4010001@pearwood.info>
Message-ID: <CAPTjJmpQ-9rkkVSTbyX3jPTZx0ty72gVgqcTFyeHpMpZHs+A6w@mail.gmail.com>

On Sun, Feb 3, 2013 at 1:18 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> I think that having literal_eval support non-literals is a bad, bad idea.
> Let's just not go there. It will surely end in tears.
>
> However, I think that having something in between the strictness of
> literal_eval and the dangerous "anything goes" power of eval is a good idea.
> For a long time now I've toyed with an engine for building expression
> evaluators. Something that understands operators, function calls, etc, and
> you can tell it what names to accept. My main motivation is for evaluating
> mathematical expressions like:
>
> 5x^3 - 2x + log(1/y) + n!/??

This sounds like a good idea, especially if there can be some way to
enforce that these names may ONLY be called - you can't piggyback on
log to get other functionality with log.__globals__ etc. That would
cover frozenset quite happily.

ChrisA


From python at mrabarnett.plus.com  Sun Feb  3 04:09:33 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 03 Feb 2013 03:09:33 +0000
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510DC8EC.4010001@pearwood.info>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
	<510DC8EC.4010001@pearwood.info>
Message-ID: <510DD4ED.9070300@mrabarnett.plus.com>

On 2013-02-03 02:18, Steven D'Aprano wrote:
> On 03/02/13 05:30, MRAB wrote:
>> On 2013-02-02 14:28, Antoine Pitrou wrote:
>>> On Sun, 03 Feb 2013 01:20:38 +1100
>>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>>
>>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer:
>>>>
>>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
>>>>
>>>> Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal.
>>>
>>> I'm not sure what the point is. frozensets are too rarely used to
>>> deserve a literal notation.
>>>
>> Does it need a special notation as such?
>>
>> Wouldn't an alternative be to support "frozenset(...)" where "..." is
>> parseable by ast.literal_eval()?
>>
>> The same thing could be done with some other built-in classes.
>
>
> I think that having literal_eval support non-literals is a bad, bad idea. Let's just not go there. It will surely end in tears.
>
It needn't go beyond a currently-supported literal passed as an
argument to frozenset, etc, for a very limited number of built-in
classes. A literal list or a literal set passed to frozenset would be
treated as a literal frozenset.

> However, I think that having something in between the strictness of literal_eval and the dangerous "anything goes" power of eval is a good idea. For a long time now I've toyed with an engine for building expression evaluators. Something that understands operators, function calls, etc, and you can tell it what names to accept. My main motivation is for evaluating mathematical expressions like:
>
> 5x^3 - 2x + log(1/y) + n!/??
>
> I have some old, broken Pascal code that half does what I want, and I keep intending to revisit it some day in Python. In my copious spare time.
>
> In any case, I think something like that belongs in a third-party module, at least at first.
>
>
>



From ncoghlan at gmail.com  Sun Feb  3 05:26:29 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 14:26:29 +1000
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <CANSw7KxAuY56Rn_3ZqmWeMU2Yf4Wy5kLGwWoQo=c3dwWZsWoiw@mail.gmail.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
	<CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
	<CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>
	<510D2DD9.7020109@umbrellacode.com>
	<CANSw7KxAuY56Rn_3ZqmWeMU2Yf4Wy5kLGwWoQo=c3dwWZsWoiw@mail.gmail.com>
Message-ID: <CADiSq7duqWT_AP3B-GmxPkox+6+tKJTRoNrNfbdZ6myZK_jrMw@mail.gmail.com>

On Sun, Feb 3, 2013 at 1:47 AM, Yuval Greenfield <ubershmekel at gmail.com> wrote:
> I really like Golang's solution (defer) which Nick sort of emulates with
> ExitStack.
>
> http://docs.python.org/3/library/contextlib.html#contextlib.ExitStack
>
> If ExitStack ever became a language feature

Why would it ever become a language feature? It works just fine as a
context manager, and the need for it isn't frequent enough to justify
special syntax.

> we could write stuff like:
>
>
>     def f():
>         fhand = local open(path)
>         process(fhand)
>         ghand = local open(path2)
>         process(ghand)

Why would you leave fhand open longer than necessary? The above would
be better written as:

    def f():
        with open(path) as fhand:
            process(fhand)
        with open(path2) as ghand:
            process(ghand)

If you need both files open at the same time, you can use a nested
context manager:

    def f():
        with open(path) as fhand:
            with open(path2) as ghand:
                process(fhand, ghand)

Or the nesting behaviour built into with statements themselves:

    def f():
        with open(path) as fhand, open(path2) as ghand:
            process(fhand, ghand)

It's only when the number of paths you need to open is dynamic that
ExitStack comes into play (this is actually very close to the example
in ExitStack's docstring, as handling a variable number of
simultaneously open files was the use case that highlighted the fatal
flaw in the way the old contextlib.nested design handled context
managers that acquired the resource in __init__ rather than
__enter__):

    def f(*paths):
        with contextlib.ExitStack() as stack:
            files = [stack.enter_context(open(path)) for path in paths]
            process(files)

Function and class definitions control name scope (amongst other
things), with statements control deterministic cleanup, loops control
iteration. That's what I mean by "separation of concerns" in relation
to these aspects of the language design and it's a *good* thing (and
one of the key reasons with statements behave like PEP 343, rather
than being closer to Guido's original looping idea that is described
in PEP 340).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From ncoghlan at gmail.com  Sun Feb  3 05:36:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 14:36:53 +1000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
Message-ID: <CADiSq7eEeDYJcA+JZJ0RTC4yCXYRsrOU7wTcmXz-c3LJtoxpZw@mail.gmail.com>

On Sun, Feb 3, 2013 at 6:10 AM, Ben Darnell <ben at bendarnell.com> wrote:
> The event loop interface in PEP 3156 has an extensibility problem.  It seems
> likely that it will have a method like listen_udp() by the time it's done,
> but suppose that doesn't make it into the first official release.
> Third-party event loop implementations may want to provide UDP support as an
> extension, but the most consistent way to provide that extension is by
> adding new methods on the event loop object, where various extensions risk
> conflicting with each other or with new methods that become standardized
> later.

The general idea of using factory functions for transport creation,
even for the "core transports" (like sockets, SSL and pipes) sounds
good to me. I've expressed some concerns previously about the breadth
of the event loop class API, and this would go a long way towards
alleviating them.

As far as precedents for the proposed dispatch/registration mechanism
goes, probably the closest we currently have is the codecs registry. A
"transport registry" (using Python dottedname identifiers as the
naming scheme) sounds promising to me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From ncoghlan at gmail.com  Sun Feb  3 05:50:45 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Feb 2013 14:50:45 +1000
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510DD4ED.9070300@mrabarnett.plus.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
	<510DC8EC.4010001@pearwood.info>
	<510DD4ED.9070300@mrabarnett.plus.com>
Message-ID: <CADiSq7fvVW_NBak4kRQrNHFscFM72m2wh=j7dMkEK6aAzyzmtQ@mail.gmail.com>

On Sun, Feb 3, 2013 at 1:09 PM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2013-02-03 02:18, Steven D'Aprano wrote:
>>
>> On 03/02/13 05:30, MRAB wrote:
>>>
>>> On 2013-02-02 14:28, Antoine Pitrou wrote:
>>>>
>>>> On Sun, 03 Feb 2013 01:20:38 +1100
>>>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>>>
>>>>>
>>>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work
>>>>> within the constraints of Python's lexer:
>>>>>
>>>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
>>>>>
>>>>> Unfortunately we're running out of useful, easy to enter symbols for
>>>>> literals. Until such time (Python4000 perhaps, or more likely Python5000) as
>>>>> we can use a rich set of Unicode literals, I don't think there is any clear
>>>>> way to have a frozenset literal.
>>>>
>>>>
>>>> I'm not sure what the point is. frozensets are too rarely used to
>>>> deserve a literal notation.
>>>>
>>> Does it need a special notation as such?
>>>
>>> Wouldn't an alternative be to support "frozenset(...)" where "..." is
>>> parseable by ast.literal_eval()?
>>>
>>> The same thing could be done with some other built-in classes.
>>
>>
>>
>> I think that having literal_eval support non-literals is a bad, bad idea.
>> Let's just not go there. It will surely end in tears.
>>
> It needn't go beyond a currently-supported literal passed as an
> argument to frozenset, etc, for a very limited number of built-in
> classes. A literal list or a literal set passed to frozenset would be
> treated as a literal frozenset.

This is why we need a PEP or a PyPI module. It's certainly not clear
to me that special casing in ast.literal_eval (or a new
"ast.limited_eval") is a superior solution to s{} and fs{} syntax for
creating the empty set and frozen sets. (And, as Raymond notes, there
are other compile-time benefits in terms of constant-caching when it
comes to dedicated syntax)

On the other hand, a "limited_eval" style solution might be easier to
extend to other builtins like range, reversed and enum, as well as to
container comprehensions and generator expressions. It also has the
virtue of being possible to write as a PyPI module, and made available
for *current* Python versions, rather than only being available in
Python 3.4+.

It's certainly a space worth exploring, even though the best way to
improve the status quo isn't immediately obvious.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From shane at umbrellacode.com  Sun Feb  3 07:17:56 2013
From: shane at umbrellacode.com (Shane Green)
Date: Sat, 02 Feb 2013 22:17:56 -0800
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <CADiSq7duqWT_AP3B-GmxPkox+6+tKJTRoNrNfbdZ6myZK_jrMw@mail.gmail.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
	<CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
	<CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>
	<510D2DD9.7020109@umbrellacode.com>
	<CANSw7KxAuY56Rn_3ZqmWeMU2Yf4Wy5kLGwWoQo=c3dwWZsWoiw@mail.gmail.com>
	<CADiSq7duqWT_AP3B-GmxPkox+6+tKJTRoNrNfbdZ6myZK_jrMw@mail.gmail.com>
Message-ID: <510E0114.1070809@umbrellacode.com>

Okay, Nick, thanks for iterating exactly what separation of concerns you 
meant: it clearly identifies where my thinking was going a bit awry.

My thinking was more along the lines of, "with" controls deterministic 
cleanup around a block of code, can't that block be an existing for loop?
The "with" statement still controls deterministic cleanup; its 
__enter__() still necessarily precedes evaluation of the for loop, and 
its __exit__() still immediately follows evaluation of the for loop.

But, there's not much to gain from the idea, at best, so it's a bit of a 
waste of time, I'm afraid...



02/02/2013 22:13:02





> Nick Coghlan <mailto:ncoghlan at gmail.com>
> February 2, 2013 8:26 PM
> On Sun, Feb 3, 2013 at 1:47 AM, Yuval Greenfield<ubershmekel at gmail.com>  wrote:
>> I really like Golang's solution (defer) which Nick sort of emulates with
>> ExitStack.
>>
>> http://docs.python.org/3/library/contextlib.html#contextlib.ExitStack
>>
>> If ExitStack ever became a language feature
>
> Why would it ever become a language feature? It works just fine as a
> context manager, and the need for it isn't frequent enough to justify
> special syntax.
>
>> we could write stuff like:
>>
>>
>>      def f():
>>          fhand = local open(path)
>>          process(fhand)
>>          ghand = local open(path2)
>>          process(ghand)
>
> Why would you leave fhand open longer than necessary? The above would
> be better written as:
>
>      def f():
>          with open(path) as fhand:
>              process(fhand)
>          with open(path2) as ghand:
>              process(ghand)
>
> If you need both files open at the same time, you can use a nested
> context manager:
>
>      def f():
>          with open(path) as fhand:
>              with open(path2) as ghand:
>                  process(fhand, ghand)
>
> Or the nesting behaviour built into with statements themselves:
>
>      def f():
>          with open(path) as fhand, open(path2) as ghand:
>              process(fhand, ghand)
>
> It's only when the number of paths you need to open is dynamic that
> ExitStack comes into play (this is actually very close to the example
> in ExitStack's docstring, as handling a variable number of
> simultaneously open files was the use case that highlighted the fatal
> flaw in the way the old contextlib.nested design handled context
> managers that acquired the resource in __init__ rather than
> __enter__):
>
>      def f(*paths):
>          with contextlib.ExitStack() as stack:
>              files = [stack.enter_context(open(path)) for path in paths]
>              process(files)
>
> Function and class definitions control name scope (amongst other
> things), with statements control deterministic cleanup, loops control
> iteration. That's what I mean by "separation of concerns" in relation
> to these aspects of the language design and it's a *good* thing (and
> one of the key reasons with statements behave like PEP 343, rather
> than being closer to Guido's original looping idea that is described
> in PEP 340).
>
> Cheers,
> Nick.
>
> Yuval Greenfield <mailto:ubershmekel at gmail.com>
> February 2, 2013 7:47 AM
> On Sat, Feb 2, 2013 at 5:16 PM, Shane Green <shane at umbrellacode.com 
> <mailto:shane at umbrellacode.com>> wrote:
>
>     Thanks Nick.  I definitely see your point about iterwith(); have
>     been thinking about that since someone asked where __exit__()
>     would be invoked.
>
>     I meant the following as a more compact way of expressing
>
>     for line in file with open(path) as file:
>         process(line)
>
>
>
> This is an interesting idea, though a bit too dense for my taste.
>
>
>     Indentation levels aren't limited, but flatter is better ;-)
>
>
> I really like Golang's solution (defer) which Nick sort of emulates 
> with ExitStack.
>
> http://docs.python.org/3/library/contextlib.html#contextlib.ExitStack
>
> If ExitStack ever became a language feature, we could write stuff like:
>
>
>     def f():
>         fhand = local open(path)
>         process(fhand)
>         ghand = local open(path2)
>         process(ghand)
>
> # which would be sort of equivalent to
>
>     def g():
>         try:
>             fhand = None
>             ghand = None
>             fhand = local open(path)
>             process(fhand)
>             ghand = local open(path2)
>             process(ghand)
>         finally:
>             if fhand is not None:
>                 fhand.close()
>             if ghand is not None:
>                 ghand.close()
>
>
>
> Yuval
> Shane Green <mailto:shane at umbrellacode.com>
> February 2, 2013 7:16 AM
> Thanks Nick.  I definitely see your point about iterwith(); have been 
> thinking about that since someone asked where __exit__() would be 
> invoked.
>
> I meant the following as a more compact way of expressing
>
> for line in file with open(path) as file:
>     process(line)
>
> As a more compact way of expressing
>
> with open(path) as file:
>     for line in file:
>         process(line)
>
> Not a change to the semantics of for-loops; a point my iterwith() 
> function has confuses greatly, I realize now. I'm not seeing a loss of 
> separation of concerns there.
>
> Indentation levels aren't limited, but flatter is better ;-)
>
> I saw a bunch of back and forth regarding iteration and context 
> management in the PEP, but didn't notice anything along these lines in 
> particular . I'll have to go back and take a closer look.
>
>
>
> Nick Coghlan <mailto:ncoghlan at gmail.com>
> February 2, 2013 6:18 AM
>
> The with statement block is needed to define *when* cleanup happens 
> (unconditionally at the end of the block).
>
> The "iterwith" generator is currently pointless, as it results in 
> nondeterministic cleanup of the context manager, so you may as well 
> not bother and just rely on the underlying iterable's nondeterministic 
> cleanup.
>
> We're never going to add cleanup semantics directly to for loops because:
> - separation of concerns is a good design principle
> - Indentation levels are not a limited resource (anyone that thinks 
> they are may be forgetting that factoring out context managers, 
> iterators and subfunctions gives you more of them, and that judicious 
> use of early returns and continue statements can avoid wasting them)
> - we already considered it when initially designing the with statement 
> and decided it was a bad idea.
>
> I forget where that last part is written up. If it's not in PEP 343, 
> 342, 346 or 340 (the full set of PEPs that led to the current with 
> statement and contextlib.contextmanager designs), it should be in one 
> of the threads they reference.
>
> Cheers,
> Nick.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
> Chris Angelico <mailto:rosuav at gmail.com>
> February 2, 2013 5:52 AM
>
> According to the OP's posted code, as soon as the iterable runs out.
> Not sure what happens if you don't exhaust it but I'm sure generator
> functions have already solved that, too.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/6453c48f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1215 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/6453c48f/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1203 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/6453c48f/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130202/6453c48f/attachment-0002.jpg>

From ubershmekel at gmail.com  Sun Feb  3 09:38:33 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Sun, 3 Feb 2013 10:38:33 +0200
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <CADiSq7duqWT_AP3B-GmxPkox+6+tKJTRoNrNfbdZ6myZK_jrMw@mail.gmail.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
	<CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
	<CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>
	<510D2DD9.7020109@umbrellacode.com>
	<CANSw7KxAuY56Rn_3ZqmWeMU2Yf4Wy5kLGwWoQo=c3dwWZsWoiw@mail.gmail.com>
	<CADiSq7duqWT_AP3B-GmxPkox+6+tKJTRoNrNfbdZ6myZK_jrMw@mail.gmail.com>
Message-ID: <CANSw7Kz0p6bDHWY_zf-GWgQyD8LqqnrEweauN_9pXFE9cRvoHA@mail.gmail.com>

On Sun, Feb 3, 2013 at 6:26 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> If you need both files open at the same time, you can use a nested
>  context manager:
>
>     def f():
>         with open(path) as fhand:
>             with open(path2) as ghand:
>                 process(fhand, ghand)
>
> Or the nesting behaviour built into with statements themselves:
>
>     def f():
>         with open(path) as fhand, open(path2) as ghand:
>             process(fhand, ghand)
>
>
This is indeed what I was looking for. The "with" statement does give a lot
of control, though I do dislike the double indentation and also dislike
opening 2 files on one line. Sucks to be me.


> Function and class definitions control name scope (amongst other
> things), with statements control deterministic cleanup, loops control
> iteration. That's what I mean by "separation of concerns" in relation
> to these aspects of the language design and it's a *good* thing (and
> one of the key reasons with statements behave like PEP 343, rather
> than being closer to Guido's original looping idea that is described
> in PEP 340).
>
>
Yes, separation of concerns is indeed a good thing and as Guido mentioned,
there are already too many ways to do this.

Thanks for the enlightenment,

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/cebad62e/attachment.html>

From solipsis at pitrou.net  Sun Feb  3 10:04:55 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 3 Feb 2013 10:04:55 +0100
Subject: [Python-ideas] frozenset literals
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<20130202152856.0ab823b9@pitrou.net>
	<510D5B46.8080409@mrabarnett.plus.com>
	<510DC8EC.4010001@pearwood.info>
	<510DD4ED.9070300@mrabarnett.plus.com>
	<CADiSq7fvVW_NBak4kRQrNHFscFM72m2wh=j7dMkEK6aAzyzmtQ@mail.gmail.com>
Message-ID: <20130203100455.5657a9d7@pitrou.net>

On Sun, 3 Feb 2013 14:50:45 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> This is why we need a PEP or a PyPI module. It's certainly not clear
> to me that special casing in ast.literal_eval (or a new
> "ast.limited_eval") is a superior solution to s{} and fs{} syntax for
> creating the empty set and frozen sets.

Well, it's superior because it doesn't need new syntax.

> (And, as Raymond notes, there
> are other compile-time benefits in terms of constant-caching when it
> comes to dedicated syntax)

Which is a rather dubious benefit when it comes to a little-used
datatype. If you want to avoid the cost of instantiating a frozenset in
every loop iteration, just hoist it manually out of the loop.

> On the other hand, a "limited_eval" style solution might be easier to
> extend to other builtins like range, reversed and enum, as well as to
> container comprehensions and generator expressions. It also has the
> virtue of being possible to write as a PyPI module, and made available
> for *current* Python versions, rather than only being available in
> Python 3.4+.

Agreed.

Regards

Antoine.




From yorik.sar at gmail.com  Sun Feb  3 10:23:03 2013
From: yorik.sar at gmail.com (Yuriy Taraday)
Date: Sun, 3 Feb 2013 13:23:03 +0400
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
Message-ID: <CABocrW4RHngjjeb0AbBUyF7F1w_aaxuJd8U6ytGSmgEDpUMkhg@mail.gmail.com>

On Sat, Feb 2, 2013 at 11:38 AM, Hua Lu <gotoalanlu at gmail.com> wrote:

> >>> import ast
> >>> ast.literal_eval("{ 'key': 'val' }")
> {'key': 'val'}
> >>> ast.literal_eval("{ ('key',): 'val' }")
> {('key',): 'val'}
> >>> ast.literal_eval("{ frozenset({'key'}): 'val' }")
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python3.3/ast.py", line 86, in literal_eval
>     return _convert(node_or_string)
>   File "/usr/lib/python3.3/ast.py", line 63, in _convert
>     in zip(node.keys, node.values))
>   File "/usr/lib/python3.3/ast.py", line 62, in <genexpr>
>     return dict((_convert(k), _convert(v)) for k, v
>   File "/usr/lib/python3.3/ast.py", line 85, in _convert
>     raise ValueError('malformed node or string: ' + repr(node))
> ValueError: malformed node or string: <_ast.Call object at 0x7f865a8c1450>
>

It looks like we can special-case usage of set literal as a key for a dict
or a member for another set and create a frozenset constant instead.

So that adict[{1, 2, 3}] should be interpreted as adict[frozenset([1, 2,
3])] and { {'b', 'a', 'r'}, 'foo' } as { frozenset('bar'), 'foo' }

This will provide a minimal change to the interpreter while making it
possible to use any literal-parsing with frozensets while keeping method
calls out of literal_eval.

-- 

Kind regards, Yuriy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/d2bf3e7f/attachment.html>

From eliben at gmail.com  Sun Feb  3 15:54:01 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 3 Feb 2013 06:54:01 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CADiSq7eEeDYJcA+JZJ0RTC4yCXYRsrOU7wTcmXz-c3LJtoxpZw@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CADiSq7eEeDYJcA+JZJ0RTC4yCXYRsrOU7wTcmXz-c3LJtoxpZw@mail.gmail.com>
Message-ID: <CAF-Rda9xghVySNLTfZnzZnEK9C-pcra+CMeGM-zSiq257Lm=aw@mail.gmail.com>

On Sat, Feb 2, 2013 at 8:36 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Sun, Feb 3, 2013 at 6:10 AM, Ben Darnell <ben at bendarnell.com> wrote:
> > The event loop interface in PEP 3156 has an extensibility problem.  It
> seems
> > likely that it will have a method like listen_udp() by the time it's
> done,
> > but suppose that doesn't make it into the first official release.
> > Third-party event loop implementations may want to provide UDP support
> as an
> > extension, but the most consistent way to provide that extension is by
> > adding new methods on the event loop object, where various extensions
> risk
> > conflicting with each other or with new methods that become standardized
> > later.
>
> The general idea of using factory functions for transport creation,
> even for the "core transports" (like sockets, SSL and pipes) sounds
> good to me. I've expressed some concerns previously about the breadth
> of the event loop class API, and this would go a long way towards
> alleviating them.
>
>
+1

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/368d3701/attachment.html>

From guido at python.org  Sun Feb  3 18:30:45 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 3 Feb 2013 09:30:45 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
Message-ID: <CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>

Ben: I wrote a long reply, it is inline below. However it's possible
that I am not seeing the use case right. Your proposal is written in
very general terms; perhaps you can come up with a more specific
example to defend it further? The UDP example somehow doesn't seem
very compelling to me.

On Sat, Feb 2, 2013 at 12:10 PM, Ben Darnell <ben at bendarnell.com> wrote:
> The event loop interface in PEP 3156 has an extensibility problem.  It seems
> likely that it will have a method like listen_udp() by the time it's done,
> but suppose that doesn't make it into the first official release.
> Third-party event loop implementations may want to provide UDP support as an
> extension, but the most consistent way to provide that extension is by
> adding new methods on the event loop object, where various extensions risk
> conflicting with each other or with new methods that become standardized
> later.

This may be based on a misunderstanding. If a specific event loop
implementation wants to offer a new transport, there is no reason to
add it as a method to the event loop. The app that wants to use that
transport has to import that event loop too, so the app might as well
call a module-level function that's specific to that event loop, in
order to instantiate the new transport.

We may have to point this out in the PEP, since it is likely that
implementers wanting to offer new features will think of adding new
methods to their event loop first. But that's a precious namespace
(since it's shared by all event loop implementations), whereas their
own implementation's module namespace is less precious.

> The PEP specifies the add_reader family of methods in part so that core
> protocol implementations can be shared across all EventLoop implementations
> that support these methods.  However, if those transports are implemented in
> a common base class (like Twisted's PosixReactorBase), there is no way for
> third-party transports to take advantage of a similar structure.

I'm not sure I understand this (the "there is no way" part). Is the
problem that that base class is private, or that the add_reader
methods may not be there, or what?

> I'd like
> to make it possible for transports to be developed independent of a
> particular EventLoop implementation in a way that is consistent with the way
> the core transports work.

Hm, now we're talking about something else. Transports implemented
*independently* from an event loop implementation should not assume
more than the standardized API. This feels rather limiting unless
we're talking about transports built on top of other protocols (e.g.
the mythical TCP-over-HTTP transport :-).

My expectation is that most transport implementations (as opposed to
protocols) are probably tied closely to an event loop implementation
(and note that the various Tulip event loop implementations using
select, poll, epoll, kqueue, are a single implementation for this
purpose -- OTOH the IocpEventLoop (in the iocp branch) is a different
implementation and, indeed, has different transport implementations!

> (This is a bigger concern for tulip than it is
> for twisted because twisted can update PosixReactorBase more frequently than
> the stdlib can change)

I think what you're really getting at here is that there is no 3rd
significant party cottage industry creating new transports. Transports
in practice are all part of the Twisted distribution, so development
is only gated by backwards compatibility requirements with user apps
(APIs, once offered, must remain supported and stable), not by
compatibilities with older versions of the rest of the framework (a
new transport introduced in Twisted 12.1 doesn't have to work with
Twisted 12.0).

> I propose turning the interface around so that transport creation uses
> static functions that take the event loop as an argument, with a
> double-dispatch mechanism to allow the event loop to provide the actual
> implementation when it can:
>
>   def create_connection(protocol_factory, host, port, event_loop=None):
>     if event_loop is None:
>       event_loop = get_event_loop()
>     # Note the use of a fully-qualified name in the registry
>     impl = event_loop.get_implementation('tulip.create_connection')
>     return impl(protocol_factory, host, port)

Hm. I don't see what this adds. It still always gets the protocol from
the event loop so moving this standardized method out of the event
loop class doesn't seem to buy anything. The implementation-dependent
work is just moved into get_implementation(). I also don't see why we
need a registry.

> New third-party transports could provide fallbacks for event loops that
> don't have their own implementations:
>
>   if impl is None:
>     # These supports_*() functions are placeholders for a to-be-determined
>     # introspection interface.
>     if supports_fd_interface(event_loop):
>       return posix_udp_implementation(*args)
>     elif supports_iocp_interface(event_loop):
>       return iocp_udp_implementation(*args)
>     else:
>       raise Exception("This transport is not supported on this event loop")

It seems you want each transport implementation to provide its own
create_connection() function, right? There's nothing wrong with that,
and I don't see that just because 3rd party transports will be
instantiated through a module-level function (in a specific module)
that means that the standard transports specified by the PEP (standard
in semantics, not in implementation!) can't be instantiated through
methods on the event loop.

Perhaps you are placing a higher value on consistency between standard
and non-standard transports? To me, it is actually positive to be
inconsistent here, so readers are made fully aware that a non-standard
transport is being used.

(The idea of introspection functions is fine, however.)

> Or they could plug into the event loop's implementation registry:
>
>   LibUVEventLoop.register_implementation('mymodule.listen_udp',
> libuv_udp_implementation)

Yeah, but wouldn't this likely be a private affair between UV's event
loop and UV's UDP transport, which are being distributed together?
Users who wish to use UDP (assuming PEP 3156 ends up not specifying
UDP support) are required to depend on a non-standard feature of their
event loop, you can't hide that with a registry. (Note that a UDP
transport has a different API than a TCP transport, and requires the
protocol to implement different methods as well.)

> This does introduce a little magic (is there any precedent for this kind of
> multiple-dispatch in the standard library?), but I like the way it keeps the
> event loop interface from getting too big and monolithic.  Third-party
> transports can avoid naming conflicts without looking fundamentally
> different from standard ones, and there's a clean path from doing something
> that's platform-specific (e.g. with add_reader and friends) to supporting
> multiple event loops to full standardization.

I actually consider it a good thing that when a concept is
standardized, the "name" of the API changes. When we adopt a 3rd party
module in the stdlib we typically give it a new name too, to avoid
confusion about which version is meant (since inevitably the 3rd party
has more variability than the version adopted into the stdlib).

With all that said, if a particular event loop implementation prefers
to add non-standard methods to their event loop object, and then
starts lobbying for its adoption in the standard, I can't stop them,
and they may even have a good shot at getting adopted in the next
Python version. But they should be aware of the risk they run, that
the next version of the stdlib might expose a different API under
their chose name. It will be easier for their users to transition if
they choose a way to spell their extension that is *not* likely to be
standardized, e.g. a function in their own module, or an event loop
method name with a custom prefix like uv_listen_udp().

I'm looking forward to explanations of why I am preventing the
developing of 3rd party transports with this response....

-- 
--Guido van Rossum (python.org/~guido)


From ben at bendarnell.com  Sun Feb  3 20:55:05 2013
From: ben at bendarnell.com (Ben Darnell)
Date: Sun, 3 Feb 2013 14:55:05 -0500
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
Message-ID: <CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>

On Sun, Feb 3, 2013 at 12:30 PM, Guido van Rossum <guido at python.org> wrote:

> Ben: I wrote a long reply, it is inline below. However it's possible
> that I am not seeing the use case right. Your proposal is written in
> very general terms; perhaps you can come up with a more specific
> example to defend it further? The UDP example somehow doesn't seem
> very compelling to me.
>

UDP is a real-life example from tornado  - we don't have any built-in
support for UDP, but people who need it have been able to build it without
touching tornado itself.  The same argument would apply to pipes or any
number of other (admittedly much more esoteric) network protocols.  I'll
elaborate on the UDP example below.


>
> On Sat, Feb 2, 2013 at 12:10 PM, Ben Darnell <ben at bendarnell.com> wrote:
> > The event loop interface in PEP 3156 has an extensibility problem.  It
> seems
> > likely that it will have a method like listen_udp() by the time it's
> done,
> > but suppose that doesn't make it into the first official release.
> > Third-party event loop implementations may want to provide UDP support
> as an
> > extension, but the most consistent way to provide that extension is by
> > adding new methods on the event loop object, where various extensions
> risk
> > conflicting with each other or with new methods that become standardized
> > later.
>
> This may be based on a misunderstanding. If a specific event loop
> implementation wants to offer a new transport, there is no reason to
> add it as a method to the event loop. The app that wants to use that
> transport has to import that event loop too, so the app might as well
> call a module-level function that's specific to that event loop, in
> order to instantiate the new transport.
>
> We may have to point this out in the PEP, since it is likely that
> implementers wanting to offer new features will think of adding new
> methods to their event loop first. But that's a precious namespace
> (since it's shared by all event loop implementations), whereas their
> own implementation's module namespace is less precious.
>

Right.  Third-party extensions to the event loop interface are inherently
problematic, so we'll have to provide them in some other way.  I'm
proposing a pattern for that "some other way" and then realizing that I
like it even for first-party interfaces.


>
> > The PEP specifies the add_reader family of methods in part so that core
> > protocol implementations can be shared across all EventLoop
> implementations
> > that support these methods.  However, if those transports are
> implemented in
> > a common base class (like Twisted's PosixReactorBase), there is no way
> for
> > third-party transports to take advantage of a similar structure.
>
> I'm not sure I understand this (the "there is no way" part). Is the
> problem that that base class is private, or that the add_reader
> methods may not be there, or what?
>

Suppose twisted did not have UDP support built in.  Most reactor
implementations subclass PosixReactorBase (with IOCPReactor as the notable
exception).  Twisted can add UDP support and implement listenUDP in
PosixReactorBase and IOCPReactor, and suddenly most reactors (even
third-party ones like TornadoReactor) support UDP for free.  Those that
don't (a hypothetical LibUVReactor?) can implement it themselves and
interoperate with everything else.

If a third party wanted to add UDP support separately from twisted's
release schedule, they can't do with an interface that is generically
usable across all reactors.  They could make a static function listenUDP()
that works with any IReactorFDSet, and maybe special-case IOCPReactor, but
then there'd be no way for a third-party LibUVReactor to participate.


>
> > I'd like
> > to make it possible for transports to be developed independent of a
> > particular EventLoop implementation in a way that is consistent with the
> way
> > the core transports work.
>
> Hm, now we're talking about something else. Transports implemented
> *independently* from an event loop implementation should not assume
> more than the standardized API. This feels rather limiting unless
> we're talking about transports built on top of other protocols (e.g.
> the mythical TCP-over-HTTP transport :-).



> My expectation is that most transport implementations (as opposed to
> protocols) are probably tied closely to an event loop implementation
> (and note that the various Tulip event loop implementations using
> select, poll, epoll, kqueue, are a single implementation for this
> purpose -- OTOH the IocpEventLoop (in the iocp branch) is a different
> implementation and, indeed, has different transport implementations!
>

add_reader is not very limiting except for its platform-specificity.  It's
possible to have a generic protocol across all posixy event loops and then
special-case the small number of interesting non-posixy ones (or maybe
there is some other class of methods that could be standardized for other
platforms?  Is there some set of methods analogous to add_reader that
multiple IOCP-based loops could share?)



>
> > (This is a bigger concern for tulip than it is
> > for twisted because twisted can update PosixReactorBase more frequently
> than
> > the stdlib can change)
>
> I think what you're really getting at here is that there is no 3rd
> significant party cottage industry creating new transports. Transports
> in practice are all part of the Twisted distribution, so development
> is only gated by backwards compatibility requirements with user apps
> (APIs, once offered, must remain supported and stable), not by
> compatibilities with older versions of the rest of the framework (a
> new transport introduced in Twisted 12.1 doesn't have to work with
> Twisted 12.0).
>

I was thinking more about release schedules.  Twisted has several releases
a year, but the standard library moves much more slowly.


>
> > I propose turning the interface around so that transport creation uses
> > static functions that take the event loop as an argument, with a
> > double-dispatch mechanism to allow the event loop to provide the actual
> > implementation when it can:
> >
> >   def create_connection(protocol_factory, host, port, event_loop=None):
> >     if event_loop is None:
> >       event_loop = get_event_loop()
> >     # Note the use of a fully-qualified name in the registry
> >     impl = event_loop.get_implementation('tulip.create_connection')
> >     return impl(protocol_factory, host, port)
>
> Hm. I don't see what this adds. It still always gets the protocol from
> the event loop so moving this standardized method out of the event
> loop class doesn't seem to buy anything. The implementation-dependent
> work is just moved into get_implementation(). I also don't see why we
> need a registry.
>

This version doesn't change much, it's mainly to set the stage for the
following variations.  However, it does have a few nice properties - it
keeps the (public) event loop interface small and manageable, and callers
don't need to touch actual event loop objects unless they want to have more
than one.  From a stylistic perspective I like this style of interface more
than using dozens of methods on the event loop object itself (even if those
dozens of methods are still there but hidden as an implementation detail).


>
> > New third-party transports could provide fallbacks for event loops that
> > don't have their own implementations:
> >
> >   if impl is None:
> >     # These supports_*() functions are placeholders for a
> to-be-determined
> >     # introspection interface.
> >     if supports_fd_interface(event_loop):
> >       return posix_udp_implementation(*args)
> >     elif supports_iocp_interface(event_loop):
> >       return iocp_udp_implementation(*args)
> >     else:
> >       raise Exception("This transport is not supported on this event
> loop")
>
> It seems you want each transport implementation to provide its own
> create_connection() function, right? There's nothing wrong with that,
> and I don't see that just because 3rd party transports will be
> instantiated through a module-level function (in a specific module)
> that means that the standard transports specified by the PEP (standard
> in semantics, not in implementation!) can't be instantiated through
> methods on the event loop.
>
> Perhaps you are placing a higher value on consistency between standard
> and non-standard transports? To me, it is actually positive to be
> inconsistent here, so readers are made fully aware that a non-standard
> transport is being used.
>

When third-party modules get absorbed into the standard library, it's often
possible to support both just by trying different imports until one works
(unittest.mock vs mock, json vs simplejson, etc).  Sometimes a module's
interface gets cleaned up and rearranged in the process, but that seems to
be less common.  It would be nice if a third-party transport could get
standardized and the only thing callers would need to change is their
imports.  However, this is a minor concern; as I wrote up this design I
realized I liked it for first-party work even if there were no third-party
modules to be consistent with.


>
> (The idea of introspection functions is fine, however.)
>
> > Or they could plug into the event loop's implementation registry:
> >
> >   LibUVEventLoop.register_implementation('mymodule.listen_udp',
> > libuv_udp_implementation)
>
> Yeah, but wouldn't this likely be a private affair between UV's event
> loop and UV's UDP transport, which are being distributed together?
>

But libuv doesn't necessarily contain the transport creation function.  The
idea is that someone can propose a transport interface in a third-party
module (mymodule.listen_udp in this example), implement it themselves for
some event loop implementations, and other event loops can declare
themselves compatible with it.

(And in an admittedly far-fetched scenario, if there were two third-party
UDP interfaces and LibUVEventLoop implemented one of them, yet another
party could build a bridge between the two, and then they'd plug it in with
register_implementation)



> Users who wish to use UDP (assuming PEP 3156 ends up not specifying
> UDP support) are required to depend on a non-standard feature of their
> event loop, you can't hide that with a registry. (Note that a UDP
> transport has a different API than a TCP transport, and requires the
> protocol to implement different methods as well.)
>

Yes, third-party transports will be non-standard, but in practice the
add_reader family makes it easy to get broad coverage in a quasi-standard
way, and the registry makes it possible to fill in the gaps.



>
> > This does introduce a little magic (is there any precedent for this kind
> of
> > multiple-dispatch in the standard library?), but I like the way it keeps
> the
> > event loop interface from getting too big and monolithic.  Third-party
> > transports can avoid naming conflicts without looking fundamentally
> > different from standard ones, and there's a clean path from doing
> something
> > that's platform-specific (e.g. with add_reader and friends) to supporting
> > multiple event loops to full standardization.
>
> I actually consider it a good thing that when a concept is
> standardized, the "name" of the API changes. When we adopt a 3rd party
> module in the stdlib we typically give it a new name too, to avoid
> confusion about which version is meant (since inevitably the 3rd party
> has more variability than the version adopted into the stdlib).
>

The *module* gets a new name (and that is indeed a good thing), but the
functions and classes within (usually) stay the same.


>
> With all that said, if a particular event loop implementation prefers
> to add non-standard methods to their event loop object, and then
> starts lobbying for its adoption in the standard, I can't stop them,
> and they may even have a good shot at getting adopted in the next
> Python version. But they should be aware of the risk they run, that
> the next version of the stdlib might expose a different API under
> their chose name. It will be easier for their users to transition if
> they choose a way to spell their extension that is *not* likely to be
> standardized, e.g. a function in their own module, or an event loop
> method name with a custom prefix like uv_listen_udp().
>

Agreed.


>
> I'm looking forward to explanations of why I am preventing the
> developing of 3rd party transports with this response....
>

I don't think the status quo prevents the development of third-party
transports, but it does implicitly encourage two bad habits:  A) adding
methods to the event loop, inviting name collisions, or B) just building on
add_reader and friends without thinking about non-posix platforms.  Of
course, no one expects a groundswell of third-party development at the
event loop and transport level, so this could just be so much
overengineering, but I like it from a stylistic perspective even without
the third-party benefits.

-Ben


>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/1212bcd4/attachment.html>

From gotoalanlu at gmail.com  Sun Feb  3 21:45:14 2013
From: gotoalanlu at gmail.com (Hua Lu)
Date: Sun, 3 Feb 2013 14:45:14 -0600
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CABocrW4RHngjjeb0AbBUyF7F1w_aaxuJd8U6ytGSmgEDpUMkhg@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<CABocrW4RHngjjeb0AbBUyF7F1w_aaxuJd8U6ytGSmgEDpUMkhg@mail.gmail.com>
Message-ID: <CAOOU=zjn_0A4Ukc6WLDnoxAZG6B-=Y8dOaXjegMKmBPa4cHKcA@mail.gmail.com>

> It looks like we can special-case usage of set literal as a key for a dict
> or a member for another set and create a frozenset constant instead.
>
> So that adict[{1, 2, 3}] should be interpreted as adict[frozenset([1, 2,
> 3])] and { {'b', 'a', 'r'}, 'foo' } as { frozenset('bar'), 'foo' }

This could interfere with the behavior of types overriding the index
operator. Also, it's a special case.

> This will provide a minimal change to the interpreter while making it
> possible to use any literal-parsing with frozensets while keeping method
> calls out of literal_eval.

Why shouldn't we change the interpreter when it could make the language better?

Honestly, I can't think of an element which will require special
syntax like symbol{...}. Furthermore, it could be really confusable
with current syntax. Example:

    for fn { bar(foo) for foo in baz }:

What if bar was a higher order function and that syntax meant
something like "make a generator applying each thing in the set to an
original fn before introducing fn in the scope" when you really meant
`for fn in {...}`?

Python 3 has already made some calls more explicitly a function (e.g.
print) so having that as syntactic sugar for that would also be weird.
There doesn't seem to be any _use_ for syntax like symbol{...}, so I
feel for example s{...}, f{...}, and d{...} would be a natural
extension to Python insofar as b'...' and u'...' are in the language,
_even though_ sets and maps feel more general mathematically speaking
(so the need for special use syntax _may_ be inappropriate (but people
write math in languages ;)).


From abarnert at yahoo.com  Sun Feb  3 22:35:19 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 3 Feb 2013 13:35:19 -0800
Subject: [Python-ideas] for line in input with open(path) as input...
In-Reply-To: <510D2DD9.7020109@umbrellacode.com>
References: <510CFC94.5000502@umbrellacode.com>
	<510D0AA2.2060805@pearwood.info>
	<CAPTjJmra_g=OhHRexMB_6DYoQNdyZ7ggg6KYfYSXNE=LiWi+hQ@mail.gmail.com>
	<kej5j3$lpa$1@ger.gmane.org>
	<CAPTjJmreH+MFaTuSwYZg+sGxSErnnVQDRmywQb1Q2oBxixNXdw@mail.gmail.com>
	<CADiSq7eEUGRr2b5SnX0ZdAXyHsA+N-Y8cOCNF6tXrDnZCJ6JkA@mail.gmail.com>
	<510D2DD9.7020109@umbrellacode.com>
Message-ID: <8B2CB8DF-31AF-4290-AA4A-E2EC27D98F22@yahoo.com>

On Feb 2, 2013, at 7:16, Shane Green <shane at umbrellacode.com> wrote:

> Thanks Nick.  I definitely see your point about iterwith(); have been thinking about that since someone asked where __exit__() would be 

If iterwith calls close when the iterator is exhausted, it's useful. In that case, close on __exit__ is only a fallback in cases where you don't exhaust it. If it only has close on __exit__, it's useless, because its the same as not doing anything.

A few months ago I proposed and unproposed a similar extension to generator expression syntax. People suggested multiple versions of an iterwith style function, and I think everyone (including me) agreed that this was a better answer even if you ignore the obvious huge benefit that it doesn't require changing the language.

While your idea isn't identical to mine (in a generator expression, the with would have to come before the for, which doesn't feel as natural as your postfix, and it also breaks the lifting of the outermost iterable, which isn't an issue in the for statement), I think the same thing ends up being true in your case.

    for line in iterwith(open(path)):

    for line in f with open(path) as f:

I think the first one is more readable. It also doesn't make you repeat the name of an otherwise-unnecessary variable twice. And it doesn't expose that variable to the loop body. If you _want_ to use f, i think you want the explicit with statement for clarity.

The only real advantage to the second is that it immediately closes the file on break, instead of only doing so on normal exit or return. Sometimes that's important, but again, I think in most of those cases you'll want the explicit with scope for clarity.



From timothy.c.delaney at gmail.com  Sun Feb  3 22:39:57 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Mon, 4 Feb 2013 08:39:57 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
Message-ID: <CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>

On 2 February 2013 07:28, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> On 2 February 2013 02:36, Barry Warsaw <barry at python.org> wrote:
>
>> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote:
>>
>> >Last version (for now). I'm really interested in people's opinions on
>> this.
>> >For this version I've taken some inspiration from flufl.enum (but there
>> >remains the major difference that these enums subclass int).
>>
>> Why not package it up and put it in PyPI?  Better there than sitting in an
>> email thread of some mailing list full of crazy people. :)
>>
>
> Yep - I intend to. Was just hacking away initially to see if what I wanted
> to achieve was feasible using Michael's metaclass as a base. It's in a
> Mercurial repo so I'll put it up on BitBucket in the next day or two and
> look at cleaning it up (documentation!) to put on PyPI.
>

Public repository on BitBucket: https://bitbucket.org/magao/enum

Feel free to raise issues there, clone and make pull requests, etc.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130204/cdc67fda/attachment.html>

From greg.ewing at canterbury.ac.nz  Sun Feb  3 22:43:48 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 04 Feb 2013 10:43:48 +1300
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAF-Rda9xghVySNLTfZnzZnEK9C-pcra+CMeGM-zSiq257Lm=aw@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CADiSq7eEeDYJcA+JZJ0RTC4yCXYRsrOU7wTcmXz-c3LJtoxpZw@mail.gmail.com>
	<CAF-Rda9xghVySNLTfZnzZnEK9C-pcra+CMeGM-zSiq257Lm=aw@mail.gmail.com>
Message-ID: <510EDA14.1030905@canterbury.ac.nz>

> On Sat, Feb 2, 2013 at 8:36 PM, Nick Coghlan <ncoghlan at gmail.com 
> <mailto:ncoghlan at gmail.com>> wrote:
> 
>     The general idea of using factory functions for transport creation,
>     even for the "core transports" (like sockets, SSL and pipes) sounds
>     good to me. I've expressed some concerns previously about the breadth
>     of the event loop class API, and this would go a long way towards
>     alleviating them.

How would this work, exactly? The implementation of these
functions will depend on the platform and event loop being
used, so there still needs to be some kind of dispatching
mechanism based on the current event loop.

-- 
Greg


From timothy.c.delaney at gmail.com  Mon Feb  4 00:07:01 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Mon, 4 Feb 2013 10:07:01 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
Message-ID: <CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>

On 4 February 2013 08:39, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> Public repository on BitBucket: https://bitbucket.org/magao/enum
>
> Feel free to raise issues there, clone and make pull requests, etc.
>

As Eli has noted in the issues, no comments or anything except the unit
tests yet. Lots of magic. Needs significant cleanup.

And I'll note again in case it's not clear - this currently *only* works
with Python 3.3.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130204/ea8ebc92/attachment.html>

From ncoghlan at gmail.com  Mon Feb  4 00:19:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 4 Feb 2013 09:19:18 +1000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <510EDA14.1030905@canterbury.ac.nz>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CADiSq7eEeDYJcA+JZJ0RTC4yCXYRsrOU7wTcmXz-c3LJtoxpZw@mail.gmail.com>
	<CAF-Rda9xghVySNLTfZnzZnEK9C-pcra+CMeGM-zSiq257Lm=aw@mail.gmail.com>
	<510EDA14.1030905@canterbury.ac.nz>
Message-ID: <CADiSq7cDk6+JTv3Zxn2gMnTeoXHeWwmsi7xWXPyxwDm_QkbHQA@mail.gmail.com>

On 4 Feb 2013 07:45, "Greg Ewing" <greg.ewing at canterbury.ac.nz> wrote:
>>
>> On Sat, Feb 2, 2013 at 8:36 PM, Nick Coghlan <ncoghlan at gmail.com <mailto:
ncoghlan at gmail.com>> wrote:
>>
>>     The general idea of using factory functions for transport creation,
>>     even for the "core transports" (like sockets, SSL and pipes) sounds
>>     good to me. I've expressed some concerns previously about the breadth
>>     of the event loop class API, and this would go a long way towards
>>     alleviating them.
>
>
> How would this work, exactly? The implementation of these
> functions will depend on the platform and event loop being
> used, so there still needs to be some kind of dispatching
> mechanism based on the current event loop.

Ben covered that in his original post: the event loop has a registry of
transport factories, keyed by the factory name (e.g.
"mycoolmodule.myneattransport". The factory functions check the event loop
registry for an implementation and use it if they find it. Otherwise, they
fall back to a non-optimised implementation based on the standard event
loop API, or they throw an error indicating that the transport doesn't
support the current event loop.

It's a really elegant design that:
- provides a consistent user experience between "first party" and "third
party" transport implementations, rather than switching arbitrarily from
"event loop method" to "module level function" based on an implementation
detail
- allows an optimised transport implementation to be added to an event loop
without requiring application code to change in order to take advantage of
it and without even requiring cooperation from the event loop developers
- the transport creation API also extends cleanly to the creation of
protocol stacks

A per event loop implementation transport registry makes sense for the same
reason the codec registry makes sense. The only differences are that:
- the different transports may offer different APIs
- the registry isn't global, but specific to each event loop implementation.

Cheers,
Nick.

>
> --
> Greg
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130204/0945bd10/attachment.html>

From jbvsmo at gmail.com  Mon Feb  4 00:53:26 2013
From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=)
Date: Sun, 3 Feb 2013 21:53:26 -0200
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
Message-ID: <CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>

Hi, about this enum/const thing, The use case I like more is a class where
you know all the
instances and not just a sequence of names.
Particularly It would be nice to have custom attributes and methods besides
the value and the name.

I have my own implementation with a basic api somewhat borrowed from
flufl.enum (plus a lot of other stuff),
but with this kind of support: https://github.com/jbvsmo/makeobj

I couldn't find the best way to express enums with the current python
syntax, so I also wrote a simple
regex-parsed language to fit objects with an arbitrary level of complexity.
I think, enumeration per se
is not much more useful than just a bunch of integers... Having this kind
of control IMO is.

Although Java is not a good example of anything, they have a similar
feature. What do you people think?

Jo?o Bernardo


2013/2/3 Tim Delaney <timothy.c.delaney at gmail.com>

> On 4 February 2013 08:39, Tim Delaney <timothy.c.delaney at gmail.com> wrote:
>
>> Public repository on BitBucket: https://bitbucket.org/magao/enum
>>
>> Feel free to raise issues there, clone and make pull requests, etc.
>>
>
> As Eli has noted in the issues, no comments or anything except the unit
> tests yet. Lots of magic. Needs significant cleanup.
>
> And I'll note again in case it's not clear - this currently *only* works
> with Python 3.3.
>
> Tim Delaney
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/5e024275/attachment.html>

From timothy.c.delaney at gmail.com  Mon Feb  4 01:17:29 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Mon, 4 Feb 2013 11:17:29 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
Message-ID: <CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>

On 4 February 2013 10:53, Jo?o Bernardo <jbvsmo at gmail.com> wrote:

> Hi, about this enum/const thing, The use case I like more is a class where
> you know all the
> instances and not just a sequence of names.
> Particularly It would be nice to have custom attributes and methods
> besides the value and the name.
>
> I have my own implementation with a basic api somewhat borrowed from
> flufl.enum (plus a lot of other stuff),
> but with this kind of support: https://github.com/jbvsmo/makeobj
>

I considered it, and in fact you could almost do it with my implementation
by using a custom subclass of EnumValue (except trying it has just exposed
a design flaw with the whole _EnumProxy bit). Works if you create the enum
in the same module as EnumValues, fails otherwise. Going to have to have a
rethink.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130204/15f15a23/attachment.html>

From desnacked at riseup.net  Mon Feb  4 01:18:35 2013
From: desnacked at riseup.net (desnacked at riseup.net)
Date: Sun, 3 Feb 2013 16:18:35 -0800
Subject: [Python-ideas] Specification of procedures to store cryptographic
	secrets
Message-ID: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>

Some hours ago I sent an email to python-crypto asking how to securely
wipe cryptographic secrets from memory:
http://mail.python.org/pipermail/python-crypto/2013-February/001170.html

Antoine said that cryptographic secret wiping could be achieved if one
uses bytearrays carefully and then overwrites their contents after use. I
agree that this sounds reasonable, but I think it would be even better if
that was a documented property of bytearrays.

If that property of bytearrays was specified in the Python standards, it
would be easier for people who write cryptographic applications and
libraries to use bytearrays correctly, and it would also guarantee that
this property won't change in future versions of Python. Furthermore, it
would help authors of cryptographic libraries to design their APIs and
internal functions in a way that would allow the secure erasure of
sensitive data.

Would this make sense or am I asking too much from Python?



From jbvsmo at gmail.com  Mon Feb  4 01:25:55 2013
From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=)
Date: Sun, 3 Feb 2013 22:25:55 -0200
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
Message-ID: <CAOyAWgj_R=3LR221jfiJZkE4JNJcASDyTdXbs9u3Ax8-27SH6g@mail.gmail.com>

2013/2/3 Tim Delaney <timothy.c.delaney at gmail.com>

> On 4 February 2013 10:53, Jo?o Bernardo <jbvsmo at gmail.com> wrote:
>
>> Hi, about this enum/const thing, The use case I like more is a class
>> where you know all the
>> instances and not just a sequence of names.
>> Particularly It would be nice to have custom attributes and methods
>> besides the value and the name.
>>
>> I have my own implementation with a basic api somewhat borrowed from
>> flufl.enum (plus a lot of other stuff),
>> but with this kind of support: https://github.com/jbvsmo/makeobj
>>
>
> I considered it, and in fact you could almost do it with my implementation
> by using a custom subclass of EnumValue (except trying it has just exposed
> a design flaw with the whole _EnumProxy bit). Works if you create the enum
> in the same module as EnumValues, fails otherwise. Going to have to have a
> rethink.
>
>
For attributes, it would probably be easy to do, but for methods you will
probably need a new _EnumProxy subclass for each class.
I did this with a metaclass factory.


Jo?o Bernardo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/6c758bcb/attachment.html>

From dustin at v.igoro.us  Mon Feb  4 01:30:09 2013
From: dustin at v.igoro.us (Dustin J. Mitchell)
Date: Sun, 3 Feb 2013 19:30:09 -0500
Subject: [Python-ideas] Specification of procedures to store
	cryptographic secrets
In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
Message-ID: <CAJtE5vQLU=E1GLK9yFxMWUuPB6U-49oVVYxAkHAOoRfVs-8riQ@mail.gmail.com>

On Sun, Feb 3, 2013 at 7:18 PM,  <desnacked at riseup.net> wrote:
> If that property of bytearrays was specified in the Python standards, it
> would be easier for people who write cryptographic applications and
> libraries to use bytearrays correctly, and it would also guarantee that
> this property won't change in future versions of Python. Furthermore, it
> would help authors of cryptographic libraries to design their APIs and
> internal functions in a way that would allow the secure erasure of
> sensitive data.

It would similarly be helpful to add low-level support for "pinning"
such memory so that it is not written to backing store.  While that
can be done with the mmap module, the details are tricky.

I don't think that this belongs in the Python core, though.  Rather, I
think that this should be implemented in a module which can be used in
conjunction with bytearrays, mmap, and any other necessary pieces of
the core and stdlib.  In fact, such a thing might already exist - I
haven't looked (it's really not within my area of interest)

Putting such a thing in the stdlib might achieve the guarantee you
suggest, but it might not.  It really just shifts responsibility for
ensuring good cryptographic programming onto people who spend their
time implementing programming languages.

Dustin


From christian at python.org  Mon Feb  4 02:11:00 2013
From: christian at python.org (Christian Heimes)
Date: Mon, 04 Feb 2013 02:11:00 +0100
Subject: [Python-ideas] Specification of procedures to store
	cryptographic secrets
In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
Message-ID: <ken1r1$ih9$1@ger.gmane.org>

> Some hours ago I sent an email to python-crypto asking how to securely
> wipe cryptographic secrets from memory:
> http://mail.python.org/pipermail/python-crypto/2013-February/001170.html
> 
> Antoine said that cryptographic secret wiping could be achieved if one
> uses bytearrays carefully and then overwrites their contents after use. I
> agree that this sounds reasonable, but I think it would be even better if
> that was a documented property of bytearrays.

That might work if you never ever resize a bytearray during its life
cycle. A resize op calls realloc() which may copy the data to a new
memory region. The old region isn't zeroed.

The approach only takes care of the object itself on the heap. Some
function may store data on the stack or make a temporary copy to another
memory location on the heap. You have to compensate for that.
libtomcrypt has a function burn_stack() that allocates and overwrites
memory on the stack with a recursive function call.

Christian



From greg at krypto.org  Mon Feb  4 02:54:39 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 3 Feb 2013 17:54:39 -0800
Subject: [Python-ideas] Specification of procedures to store
	cryptographic secrets
In-Reply-To: <ken1r1$ih9$1@ger.gmane.org>
References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
	<ken1r1$ih9$1@ger.gmane.org>
Message-ID: <CAGE7PNLFoV8yGx+6PRG5BoR9y7krSerLPveFWPR0x7v-dBcHgA@mail.gmail.com>

Correct.  this isn't something that belongs in the core python language and
types.  something needing memory-pinning and secure wiping should be
implemented as a special type (c extension module) for use with the c
extension libraries that need those properties.  as soon as anything enters
python's own types or values ever make it into python code in any way, no
guarantees can ever be made as to how many copies were made and scattered
around the process's own address space.  assume "many".

Python doesn't implement any sort of chain of custody for data internally.



On Sun, Feb 3, 2013 at 5:11 PM, Christian Heimes <christian at python.org>wrote:

> > Some hours ago I sent an email to python-crypto asking how to securely
> > wipe cryptographic secrets from memory:
> > http://mail.python.org/pipermail/python-crypto/2013-February/001170.html
> >
> > Antoine said that cryptographic secret wiping could be achieved if one
> > uses bytearrays carefully and then overwrites their contents after use. I
> > agree that this sounds reasonable, but I think it would be even better if
> > that was a documented property of bytearrays.
>
> That might work if you never ever resize a bytearray during its life
> cycle. A resize op calls realloc() which may copy the data to a new
> memory region. The old region isn't zeroed.
>
> The approach only takes care of the object itself on the heap. Some
> function may store data on the stack or make a temporary copy to another
> memory location on the heap. You have to compensate for that.
> libtomcrypt has a function burn_stack() that allocates and overwrites
> memory on the stack with a recursive function call.
>
> Christian
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/3b2e47c4/attachment.html>

From christian at python.org  Mon Feb  4 03:12:38 2013
From: christian at python.org (Christian Heimes)
Date: Mon, 04 Feb 2013 03:12:38 +0100
Subject: [Python-ideas] Specification of procedures to store
 cryptographic secrets
In-Reply-To: <CAGE7PNLFoV8yGx+6PRG5BoR9y7krSerLPveFWPR0x7v-dBcHgA@mail.gmail.com>
References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
	<ken1r1$ih9$1@ger.gmane.org>
	<CAGE7PNLFoV8yGx+6PRG5BoR9y7krSerLPveFWPR0x7v-dBcHgA@mail.gmail.com>
Message-ID: <510F1916.4040307@python.org>

Am 04.02.2013 02:54, schrieb Gregory P. Smith:
> Correct.  this isn't something that belongs in the core python language
> and types.  something needing memory-pinning and secure wiping should be
> implemented as a special type (c extension module) for use with the c
> extension libraries that need those properties.  as soon as anything
> enters python's own types or values ever make it into python code in any
> way, no guarantees can ever be made as to how many copies were made and
> scattered around the process's own address space.  assume "many".
> 
> Python doesn't implement any sort of chain of custody for data internally.

I agree! A custom type came into my mind, too. Data wiping
is merely a small part of the general issue. A confident and secure
container for secrets must do more. For example it has to prevent the
memory page from getting swapped to disk with mlock(2). Lot's of bad
things can happen when you look at L1/L2/L3 CPU cache, hyper threading
and virtualization. All that stuff makes it hard to conceal secrets.

On the bright side attacks rarely crack cryptography. In most cases it's
easier, faster and less costly to do social engineering. Humans are
lazy, ignorant and bribable.

Christian


From tjreedy at udel.edu  Mon Feb  4 03:31:20 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 03 Feb 2013 21:31:20 -0500
Subject: [Python-ideas] Specification of procedures to store
	cryptographic secrets
In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
Message-ID: <ken6i6$kc2$1@ger.gmane.org>

On 2/3/2013 7:18 PM, desnacked at riseup.net 
wrote:
> Some hours ago I sent an email to python-crypto asking how to securely
> wipe cryptographic secrets from memory:
> http://mail.python.org/pipermail/python-crypto/2013-February/001170.html
>
> Antoine said that cryptographic secret wiping could be achieved if one
> uses bytearrays carefully and then overwrites their contents after use. I

I presume he meant with CPython with its non-compacting gc on current 
major OSes. Perhaps the system also needs to be unloaded enough that the 
memory is not written to disk. Or the secret is written and erased 
before that would happen.

> agree that this sounds reasonable, but I think it would be even better if
> that was a documented property of bytearrays.

I do not think such a low-level special-case property would be 
appropriate. Python is a high-level languages for manipulating fairly 
abstract objects defined by interface and behavior. The reference manual 
defining the language intentionally says almost nothing about the 
hardware and memory of an implementation. This is partly why Python is 
relatively easy to read and mentally execute in a human brain. One 
usually does not need to mentally simulate a linear byte memory.

> If that property of bytearrays was specified in the Python standards, it
> would be easier for people who write cryptographic applications and
> libraries to use bytearrays correctly, and it would also guarantee that
> this property won't change in future versions of Python.

This would mean that Python could not run on hardware that made the 
guarantee impossible. What if a future OS ran directly off an SSD, 
either dispensing with current DRAM, or using it as the outer cache 
layer? My understanding is that SSDs run independently with their own os 
and that external access is to logical rather than physical memory. What 
if a farther future system had a write-once, read-many, never-erase 
petabyte or exabyte 3d cube memory, with the SSD only serving as an 
index? Of course, it is possible that security concern will figure into 
future designs.

The statement 'del x' only means "break the association between the name 
'x' and the object currently associated with 'x'". If that is the last 
link to the object, it becomes inaccessible from Python and *eligible* 
to be physically deleted. What what happens in concrete hardware is 
explicitly not Python's concern. From 3.1. Objects, values and types:

"Objects are never explicitly destroyed; however, when they become 
unreachable they may be garbage-collected. An implementation is allowed 
to postpone garbage collection or omit it altogether ..."

 > Furthermore, it
> would help authors of cryptographic libraries to design their APIs and
> internal functions in a way that would allow the secure erasure of
> sensitive data.

I agree with Dustin that you need a 3rd-party crytobytes module. It 
could be specific to OS and hardware, keep up with changes, and refuse 
to run if the required guarantees cannot be met.

> Would this make sense or am I asking too much from Python?

To me, it makes perfect sense for you to want a cryptobytes class that 
does exactly what you want it to do. And, again to me, you are asking 
too much for such to be part of the stdlib. Whether you are asking too 
much of any particular OS is beyond my knowledge. If the OS can provide 
the guarantees, a 3rd party Python wrapping should be possible.

-- 
Terry Jan Reedy



From eliben at gmail.com  Mon Feb  4 05:51:35 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 3 Feb 2013 20:51:35 -0800
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
Message-ID: <CAF-Rda-DGq5qX_9VcLHg=cuKfOGV6BhavfcdpR6o-AGWKDDAfw@mail.gmail.com>

On Sun, Feb 3, 2013 at 3:53 PM, Jo?o Bernardo <jbvsmo at gmail.com> wrote:

> Hi, about this enum/const thing, The use case I like more is a class where
> you know all the
> instances and not just a sequence of names.
> Particularly It would be nice to have custom attributes and methods
> besides the value and the name.
>
> I have my own implementation with a basic api somewhat borrowed from
> flufl.enum (plus a lot of other stuff),
> but with this kind of support: https://github.com/jbvsmo/makeobj
>
> I couldn't find the best way to express enums with the current python
> syntax, so I also wrote a simple
> regex-parsed language to fit objects with an arbitrary level of
> complexity. I think, enumeration per se
> is not much more useful than just a bunch of integers... Having this kind
> of control IMO is.
>
> Although Java is not a good example of anything, they have a similar
> feature. What do you people think?
>
>

Personally, I disagree with the "more features is better" approach.
Features have a cost - they complicate the implementation which makes it
fragile, harder to maintain and harder to understand. Even more
importantly, they make *user* code harder to understand.

Therefore it's IMHO best to decide on a basic functionality that brings
most of the benefits, and then think about how to make the implementation
*simpler*. All I really want from an enum is what I have in C and lack in
Python - a nice, minimally type-safe way to name special constants. Having
this in hand, I want the simplest and cleanest syntax possible, not more
features.

Tim's implementation strikes a good balance - the syntax is as minimal as
can be in a library implementation, and it provides all the basic features
well. It adds some more, possibly at the cost of complexity, which may or
may not be good. This is why I think that a PEP weighing features
for/against inclusion is a logical next step.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130203/d5eda837/attachment.html>

From greg.ewing at canterbury.ac.nz  Mon Feb  4 06:25:45 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 04 Feb 2013 18:25:45 +1300
Subject: [Python-ideas] Specification of procedures to store
 cryptographic secrets
In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
Message-ID: <510F4659.4050607@canterbury.ac.nz>

desnacked at riseup.net wrote:
> Antoine said that cryptographic secret wiping could be achieved if one
> uses bytearrays carefully and then overwrites their contents after use. I
> agree that this sounds reasonable, but I think it would be even better if
> that was a documented property of bytearrays.

I think to fully guarantee that you would need a promise
from the OS that overwriting a particular piece of your
virtual address space removes all evidence of that data
from swap space, etc. I don't know whether any current
OSes provide that kind of guarantee.

-- 
Greg


From mal at egenix.com  Mon Feb  4 12:21:20 2013
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 04 Feb 2013 12:21:20 +0100
Subject: [Python-ideas] Specification of procedures to store
 cryptographic secrets
In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net>
Message-ID: <510F99B0.3000600@egenix.com>

On 04.02.2013 01:18, desnacked at riseup.net wrote:
> Some hours ago I sent an email to python-crypto asking how to securely
> wipe cryptographic secrets from memory:
> http://mail.python.org/pipermail/python-crypto/2013-February/001170.html
> 
> Antoine said that cryptographic secret wiping could be achieved if one
> uses bytearrays carefully and then overwrites their contents after use. I
> agree that this sounds reasonable, but I think it would be even better if
> that was a documented property of bytearrays.
> 
> If that property of bytearrays was specified in the Python standards, it
> would be easier for people who write cryptographic applications and
> libraries to use bytearrays correctly, and it would also guarantee that
> this property won't change in future versions of Python. Furthermore, it
> would help authors of cryptographic libraries to design their APIs and
> internal functions in a way that would allow the secure erasure of
> sensitive data.
> 
> Would this make sense or am I asking too much from Python?

I don't think there's any safe way to store crypto information in
memory. You'd have to use a dedicated hardware crypto device to avoid
leaking the keys (think memory reallocation, the OS swapping memory
to disk, your code running on a VM, etc.). See e.g.
http://c0decstuff.blogspot.de/2011/01/in-memory-extraction-of-ssl-private.html
for an example on how to do this intentionally.

Not even OpenSSL tries to address this, so I think it's asking a
bit much from Python ;-)

That said, adding a little more security to a custom blob type
would certainly not hurt :-)

Here's some inspiration for locking and cleaning memory:
http://c0decstuff.blogspot.de/2011/01/in-memory-extraction-of-ssl-private.html
(pages 36ff)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 04 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/


From guido at python.org  Mon Feb  4 20:02:57 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 4 Feb 2013 11:02:57 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
Message-ID: <CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>

I'm going to try and snip as much as I can to get the the heart of this...

On Sun, Feb 3, 2013 at 11:55 AM, Ben Darnell <ben at bendarnell.com> wrote:
> UDP is a real-life example from tornado  - we don't have any built-in
> support for UDP, but people who need it have been able to build it without
> touching tornado itself.  The same argument would apply to pipes or any
> number of other (admittedly much more esoteric) network protocols.  I'll
> elaborate on the UDP example below.

Hm. UDP is relatively easy because it uses sockets. Pipes are harder
-- testing for the presence of add_reader (etc. -- I will leave this
off from now on) isn't enough, because select on Windows does not
support pipes.

Thinking about what you could mean by "more esoteric protocols",
there's really not much at the level of TCP and UDP that comes to
mind. UNIX-domain sockets, and perhaps the (root-only) protocol for
sniffing packets (raw sockets?).

A new feature just landed in Tulip (I still have to update PEP 3156)
where you can pass a pre-constructed socket object to
create_connection() and start_serving(), which will make it a little
easier to support esoteric ways of setting up the socket; however,
create_connection() is still limited to sockets that implement a byte
stream, because of the way the transport/protocol API works.

> Right.  Third-party extensions to the event loop interface are inherently
> problematic, so we'll have to provide them in some other way.  I'm proposing
> a pattern for that "some other way" and then realizing that I like it even
> for first-party interfaces.

Glad that is out of the way. But I'm still skeptical -- first, as I
explained before, I am actually in favor of using different styles for
1st and 3rd party interfaces, so the status of the interface used is
obvious to the reader (and the coder, in case they are copy-pasting
recipes :-); second, I don't expect there will be too many
opportunities to put the pattern at work.

Note that I'm only talking about 3rd party *interfaces* -- if a 3rd
party module implements a 1st party interface (e.g. the stream
transport/protocol interface specified in PEP 3156) it can just use
the event loop create_connection method (assuming it is also
implementing a new event loop -- otherwise what would be the point of
the 3rd party code?).

And note that even UDP requires a different interface between
transport and protocol -- e.g. the protocol method called should be
packet_received() rather than data_received(), and the protocol should
probably not implement write/writelines but send and send_multiple.
And the signatures of these methods will be different because
(depending on how you use UDP) you have to have a parameter for the
peer address.

And yet, implementing UDP as pure 3rd party code using add_reader is
simple, as long as the event loop supports add_reader. You just can't
use create_connection or start_serving -- but those are really just
convenience methods that are easily reimplemented. (We could refactor
the standard implementations to have more reusable parts, but we'd run
into the same problem as with add_reader -- while most UNIXy event
loops will easily support such refactorings, that's not the case with
event loops based on IOCP, other other libraries that don't naturally
offer add_reader functionality. (Not sure if that's the case for
libuv.)

All this makes me skeptical that a single API should be used to
register "transports". At the very least you will need different
registries for each distinct transport/protocol interface; in
addition, custom transports (even if they implement the same
transport/protocol interface) may have different constructor arguments
(e.g. consider plain TCP vs. SSL in Tulip).

> Suppose twisted did not have UDP support built in.  Most reactor
> implementations subclass PosixReactorBase (with IOCPReactor as the notable
> exception).  Twisted can add UDP support and implement listenUDP in
> PosixReactorBase and IOCPReactor, and suddenly most reactors (even
> third-party ones like TornadoReactor) support UDP for free.

If you say so. I don't know enough about Twisted's internals to verify
this claim. Depending on how things were factored I could easily
imagining something in PosixReactorBase making the assumption of a
stream protocol somewhere. In a stream protocol like TCP, it is safe
to collapse two consecutive sends into one, and to split one send into
multiples. But not for datagram protocols like UDP. In an ideal world,
knowledge of all this is completely left out of the reactor. But, in a
hypothetical world where Twisted only supported streams, who knows
whether that is done?

> Those that
> don't (a hypothetical LibUVReactor?) can implement it themselves and
> interoperate with everything else.

In practice I suspect that the number of 3rd party event loop
implementations that support add_reader and let a different 3rd
party's UDP implementation succeed will be vanishingly small. Even
smaller if you don't count the ones that are essentially clones or
subclasses of Tulip's UNIX support.

> If a third party wanted to add UDP support separately from twisted's release
> schedule, they can't do with an interface that is generically usable across
> all reactors.  They could make a static function listenUDP() that works with
> any IReactorFDSet, and maybe special-case IOCPReactor, but then there'd be
> no way for a third-party LibUVReactor to participate.

This sounds unavoidable no matter how you refactor the interface.
There are potentially event loop implementations that don't use socket
objects at all. (And yes, those will have to reject the 'sock'
argument to create_connection and start_serving; and I have to change
start_serving's return type to be something other than a socket.) When
you implement a new 3rd party transport, you are pretty much
inevitably limiting yourself to a subset of event loops. That subset
won't be empty, and it will be sufficient for your purpose, but the
ideal of portability across all (or even most) event loops, including
ones that haven't been written yet, is unattainable. I certainly
haven't seen an indication that your proposed registry will address
this.

> add_reader is not very limiting except for its platform-specificity.  It's
> possible to have a generic protocol across all posixy event loops and then
> special-case the small number of interesting non-posixy ones (or maybe there
> is some other class of methods that could be standardized for other
> platforms?  Is there some set of methods analogous to add_reader that
> multiple IOCP-based loops could share?)

Not exactly analogous -- the whole point of IOCP is that it is not
"ready-based" but "completion-based". The sock_recv (etc.) methods on
the event loop are my attempt to suggest a way for other
completion-based event loops to open themselves up for new transport
implementations, but this is much more limiting than add_reader --
e.g. I suspect that IOCP will let you read from a named pipe, but you
must use a different library call than for receiving from a socket;
even receiving a packet from UDP will require a different method. This
issue doesn't exist in the same way for add_reader, because the system
call to do the read is not made by the event loop, it is made by the
transport.

> This version doesn't change much, it's mainly to set the stage for the
> following variations.  However, it does have a few nice properties - it
> keeps the (public) event loop interface small and manageable, and callers
> don't need to touch actual event loop objects unless they want to have more
> than one.

Not quite -- the call_soon(), call_later() etc. functionality is also
exposed as event loop methods.

> From a stylistic perspective I like this style of interface more
> than using dozens of methods on the event loop object itself (even if those
> dozens of methods are still there but hidden as an implementation detail).

You can't argue about style. :-)

> When third-party modules get absorbed into the standard library, it's often
> possible to support both just by trying different imports until one works
> (unittest.mock vs mock, json vs simplejson, etc).  Sometimes a module's
> interface gets cleaned up and rearranged in the process, but that seems to
> be less common.  It would be nice if a third-party transport could get
> standardized and the only thing callers would need to change is their
> imports.  However, this is a minor concern; as I wrote up this design I
> realized I liked it for first-party work even if there were no third-party
> modules to be consistent with.

You can't argue about style. :-)

> But libuv doesn't necessarily contain the transport creation function.

But a libuv-based PEP 3156-compliant event loop implementation must.

> The
> idea is that someone can propose a transport interface in a third-party
> module (mymodule.listen_udp in this example), implement it themselves for
> some event loop implementations, and other event loops can declare
> themselves compatible with it.

Aha! This is the executive summary of your proposal, or at least your
goal for it.

This is hypothesizing rather a lot of goodwill and coordination
between different 3rd party developers. And the registry offered by
the event loop comes down to not much more than a dictionary with keys
that follow a certain convention (e.g. fully-qualified package+module
name plus some identifier for the feature) and nothing can be said
about what the items stored in the registry are (since a packet
transport and a stream transport are not interchangeable, and even two
stream transports may not be).

Given that for each 3rd party transport the details of how to
implement a compatible version of it will vary hugely, both depending
on what the transport is trying to do and how the event loop works, I
expect that the market for this registry will be rather small. And
when a particular 3rd party transport wants to enable other 3rd party
events to support them, they can implement their own registry, which
the other 3rd party could then plug into. (But see below.)

> (And in an admittedly far-fetched scenario, if there were two third-party
> UDP interfaces and LibUVEventLoop implemented one of them, yet another party
> could build a bridge between the two, and then they'd plug it in with
> register_implementation)

I do see one argument in favor of having a standard registry on the
event loop, even if it's just a dict with register/lookup APIs and a
naming convention, and no semantics assigned to the items registered.
That argument is to make 3rd party transport implementers aware of the
possibility that some other 3rd party might want to offer a compatible
implementation aimed at an event loop that's not supported natively by
the (former) 3rd party transport. And I could even be convinced that
the standard protocols should use this registry so that the source
code serves as an example of best practices.

Still, it's a pretty weak argument IMO -- I don't expect there to be a
significant cottage industry cranking out 3rd party protocol
implementations, assuming we add UDP to PEP 3156, which is my plan.
And I don't think that create_connection() should be used to create
UDP connections -- the signature of the protocol factory passed in
would be quite different, for starters, and the set of options needed
to configure the transport is also different (assuming we want to
support both connected and connection-less UDP).

> I don't think the status quo prevents the development of third-party
> transports, but it does implicitly encourage two bad habits:  A) adding
> methods to the event loop, inviting name collisions, or B) just building on
> add_reader and friends without thinking about non-posix platforms.  Of
> course, no one expects a groundswell of third-party development at the event
> loop and transport level, so this could just be so much overengineering, but
> I like it from a stylistic perspective even without the third-party
> benefits.

Yeah, so that's the rub: I'm not so keen on adding extra machinery to
the PEP that I don't expect to be used much. I have more important
fish to fry (such as adding UDP :-). And adding a registry one whole
Python release cycle later (e.g. in 3.5, assuming PEP 3156 is
standardized and included in 3.4) doesn't strike me as such a bad
thing -- I don't think we're painting ourselves into much of a corner
by not having a registry right from the start.

-- 
--Guido van Rossum (python.org/~guido)


From ericsnowcurrently at gmail.com  Tue Feb  5 00:57:05 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Mon, 4 Feb 2013 16:57:05 -0700
Subject: [Python-ideas] Extend module objects to support properties
In-Reply-To: <CADiSq7e4puwp9=JBG8f4ceVmH148Jkv5yWs2eDKKN1214_WMJg@mail.gmail.com>
References: <51087225.3040801@hastings.org>
	<CADiSq7fRZhD8kWoHfpxg7HKriVr6qET6QRvCLGirXYdsUJHEfA@mail.gmail.com>
	<51094D8D.606@hastings.org>
	<CADiSq7cj8xLPXa-YjE-H9DEzXRbgjhm_B40ELQW8QjfGT3+htA@mail.gmail.com>
	<CALFfu7CTxbGQwdr9+oaChcvUD_Pd0c5S2hxEcQZSfTNPqjawXg@mail.gmail.com>
	<CADiSq7e4puwp9=JBG8f4ceVmH148Jkv5yWs2eDKKN1214_WMJg@mail.gmail.com>
Message-ID: <CALFfu7CneFVSpQTmEjeenMHMGy3CWqYB8A3FqiYyR9U0mn7Nqw@mail.gmail.com>

On Thu, Jan 31, 2013 at 5:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> ... or making sys an
>> instance of another type, as someone suggested.
>
> sys is already special cased so heavily in the interpreter
> initialization, making it a little more special really wouldn't bother
> me much :)

One benefit of this is that sys.modules could be made a [carefully
done] property that wraps interp->modules, making sys.modules truly
replaceable, which would simplify a number of different places where
we've had to work around it.

-eric


From ben at bendarnell.com  Tue Feb  5 05:50:20 2013
From: ben at bendarnell.com (Ben Darnell)
Date: Mon, 4 Feb 2013 23:50:20 -0500
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
Message-ID: <CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>

On Mon, Feb 4, 2013 at 2:02 PM, Guido van Rossum <guido at python.org> wrote:

> I'm going to try and snip as much as I can to get the the heart of this...
>
>
Me too...


>
> Thinking about what you could mean by "more esoteric protocols",
> there's really not much at the level of TCP and UDP that comes to
> mind. UNIX-domain sockets, and perhaps the (root-only) protocol for
> sniffing packets (raw sockets?).
>

Looking at twisted as an example, the things that are supported across both
PosixReactorBase and IOCPReactor are TCP, UDP, SSL, multicast, and
subprocesses.  Unix domain sockets aren't that interesting here since any
system that supports them will (presumably?) support add_reader and things
will just work.


>
> A new feature just landed in Tulip (I still have to update PEP 3156)
> where you can pass a pre-constructed socket object to
> create_connection() and start_serving(), which will make it a little
> easier to support esoteric ways of setting up the socket; however,
> create_connection() is still limited to sockets that implement a byte
> stream, because of the way the transport/protocol API works.
>

Cool.  That's definitely useful (especially on the server side), but we'll
still need a separate interface for datagrams.


>
> > Right.  Third-party extensions to the event loop interface are inherently
> > problematic, so we'll have to provide them in some other way.  I'm
> proposing
> > a pattern for that "some other way" and then realizing that I like it
> even
> > for first-party interfaces.
>
> Glad that is out of the way. But I'm still skeptical -- first, as I
> explained before, I am actually in favor of using different styles for
> 1st and 3rd party interfaces, so the status of the interface used is
> obvious to the reader (and the coder, in case they are copy-pasting
> recipes :-); second, I don't expect there will be too many
> opportunities to put the pattern at work.
>

You can't argue about style. :-)



> > The
> > idea is that someone can propose a transport interface in a third-party
> > module (mymodule.listen_udp in this example), implement it themselves for
> > some event loop implementations, and other event loops can declare
> > themselves compatible with it.
>
> Aha! This is the executive summary of your proposal, or at least your
> goal for it.
>
> This is hypothesizing rather a lot of goodwill and coordination
> between different 3rd party developers.


Yeah, history is unfortunately not very supportive of the idea that
developers of asynchronous frameworks will coordinate on this kind of thing
:)


> And the registry offered by
> the event loop comes down to not much more than a dictionary with keys
> that follow a certain convention (e.g. fully-qualified package+module
> name plus some identifier for the feature) and nothing can be said
> about what the items stored in the registry are (since a packet
> transport and a stream transport are not interchangeable, and even two
> stream transports may not be).
>
> Given that for each 3rd party transport the details of how to
> implement a compatible version of it will vary hugely, both depending
> on what the transport is trying to do and how the event loop works, I
> expect that the market for this registry will be rather small. And
> when a particular 3rd party transport wants to enable other 3rd party
> events to support them, they can implement their own registry, which
> the other 3rd party could then plug into. (But see below.)
>

True.


>
> > (And in an admittedly far-fetched scenario, if there were two third-party
> > UDP interfaces and LibUVEventLoop implemented one of them, yet another
> party
> > could build a bridge between the two, and then they'd plug it in with
> > register_implementation)
>
> I do see one argument in favor of having a standard registry on the
> event loop, even if it's just a dict with register/lookup APIs and a
> naming convention, and no semantics assigned to the items registered.
> That argument is to make 3rd party transport implementers aware of the
> possibility that some other 3rd party might want to offer a compatible
> implementation aimed at an event loop that's not supported natively by
> the (former) 3rd party transport. And I could even be convinced that
> the standard protocols should use this registry so that the source
> code serves as an example of best practices.
>
> Still, it's a pretty weak argument IMO -- I don't expect there to be a
> significant cottage industry cranking out 3rd party protocol
> implementations, assuming we add UDP to PEP 3156, which is my plan.
>

Yeah, as long as we get TCP, UDP, SSL, and pipes (at least for
subprocesses), I'm hard pressed to imagine anything else that would be in
so much demand it would need to be supported by all event loops.


> And I don't think that create_connection() should be used to create
> UDP connections -- the signature of the protocol factory passed in
> would be quite different, for starters, and the set of options needed
> to configure the transport is also different (assuming we want to
> support both connected and connection-less UDP).
>

Of course.  (I may have been unclear somewhere along the way, but it was
never my intention for create_connection to work for both TCP and UDP)


>
> > I don't think the status quo prevents the development of third-party
> > transports, but it does implicitly encourage two bad habits:  A) adding
> > methods to the event loop, inviting name collisions, or B) just building
> on
> > add_reader and friends without thinking about non-posix platforms.  Of
> > course, no one expects a groundswell of third-party development at the
> event
> > loop and transport level, so this could just be so much overengineering,
> but
> > I like it from a stylistic perspective even without the third-party
> > benefits.
>
> Yeah, so that's the rub: I'm not so keen on adding extra machinery to
> the PEP that I don't expect to be used much. I have more important
> fish to fry (such as adding UDP :-). And adding a registry one whole
> Python release cycle later (e.g. in 3.5, assuming PEP 3156 is
> standardized and included in 3.4) doesn't strike me as such a bad
> thing -- I don't think we're painting ourselves into much of a corner
> by not having a registry right from the start.
>

Fair enough.  I may use this pattern when/if I retrofit Tornado to be
IOCP-friendly since we currently create transports with static functions,
but for Tulip let's wait and see if the problem this proposal is trying to
solve ever materializes.

-Ben


>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130204/c2ee1ddd/attachment.html>

From saghul at gmail.com  Tue Feb  5 15:48:43 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Tue, 05 Feb 2013 15:48:43 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: Remove 'when' from Handler
Message-ID: <51111BCB.6070609@gmail.com>

Hi,

While working on some changes for rose (an eventloop for tulip based on 
libuv) I realized the Handler class is a bit too coupled to the event 
loop implementation currently present in Tulip.

pyuv (and pyev, and probably others) provide timers already, so there is 
no need to implement them "manually" and keep a heap with them. This 
basically means that Handler objects will always have the 'when' value 
set to None, and the comparison functions don't really apply, moreover 
__eq__ would even be incorrect in its current form.

IMHO it would be good to have a base Handler class which doesn't take 
time into account and then a Timer class which does. Thoughts?

Also, I see that the PEP does mention the possibility of providing a 
__call__ method on the Handler class itself, which would basically call 
the callback and suppress exceptions (I guess by logging them). Was this 
ever discussed? Sounds like a good idea to me :-)


Regards,

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From guido at python.org  Tue Feb  5 16:49:43 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 5 Feb 2013 07:49:43 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: Remove 'when' from Handler
In-Reply-To: <51111BCB.6070609@gmail.com>
References: <51111BCB.6070609@gmail.com>
Message-ID: <CAP7+vJLwYRy9GLw_aEzSWVpxmXje=rs4utTCW0_0U5NapgweVw@mail.gmail.com>

I think both suggestions are good ones. The when field has bothered me
for a while, although it did not come to the surface. Adding the
call/catch/log logic on the Handler seems nice too, but I think I
would rather not use handler() -- I would rather use handler.call().
I.e. make the method name 'call', not '__call__'. (The latter almost
always makes APIs more confusing IMO.)

Can you contribute code?

--Guido

On Tue, Feb 5, 2013 at 6:48 AM, Sa?l Ibarra Corretg? <saghul at gmail.com> wrote:
> Hi,
>
> While working on some changes for rose (an eventloop for tulip based on
> libuv) I realized the Handler class is a bit too coupled to the event loop
> implementation currently present in Tulip.
>
> pyuv (and pyev, and probably others) provide timers already, so there is no
> need to implement them "manually" and keep a heap with them. This basically
> means that Handler objects will always have the 'when' value set to None,
> and the comparison functions don't really apply, moreover __eq__ would even
> be incorrect in its current form.
>
> IMHO it would be good to have a base Handler class which doesn't take time
> into account and then a Timer class which does. Thoughts?
>
> Also, I see that the PEP does mention the possibility of providing a
> __call__ method on the Handler class itself, which would basically call the
> callback and suppress exceptions (I guess by logging them). Was this ever
> discussed? Sounds like a good idea to me :-)
>
>
> Regards,
>
> --
> Sa?l Ibarra Corretg?
> http://saghul.net/blog | http://about.me/saghul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
--Guido van Rossum (python.org/~guido)


From saghul at gmail.com  Tue Feb  5 17:20:15 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Tue, 05 Feb 2013 17:20:15 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: Remove 'when' from Handler
In-Reply-To: <CAP7+vJLwYRy9GLw_aEzSWVpxmXje=rs4utTCW0_0U5NapgweVw@mail.gmail.com>
References: <51111BCB.6070609@gmail.com>
	<CAP7+vJLwYRy9GLw_aEzSWVpxmXje=rs4utTCW0_0U5NapgweVw@mail.gmail.com>
Message-ID: <5111313F.6010708@gmail.com>

Guido van Rossum wrote:
> I think both suggestions are good ones. The when field has bothered me
> for a while, although it did not come to the surface. Adding the
> call/catch/log logic on the Handler seems nice too, but I think I
> would rather not use handler() -- I would rather use handler.call().
> I.e. make the method name 'call', not '__call__'. (The latter almost
> always makes APIs more confusing IMO.)
>
> Can you contribute code?
>

Sure, I've never used the codereview thing nor mercurial , but I guess 
this a good time to start :-)

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From guido at python.org  Tue Feb  5 17:53:05 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 5 Feb 2013 08:53:05 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
Message-ID: <CAP7+vJKuteMTVbT49PNvRBBF1+EvHJ6-MW84DBpoqYXe5MxwrQ@mail.gmail.com>

On Mon, Feb 4, 2013 at 8:50 PM, Ben Darnell <ben at bendarnell.com> wrote:
> Fair enough.  I may use this pattern when/if I retrofit Tornado to be
> IOCP-friendly since we currently create transports with static functions,
> but for Tulip let's wait and see if the problem this proposal is trying to
> solve ever materializes.

Great!

-- 
--Guido van Rossum (python.org/~guido)


From p.f.moore at gmail.com  Tue Feb  5 21:32:24 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 5 Feb 2013 20:32:24 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
Message-ID: <CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>

On 5 February 2013 04:50, Ben Darnell <ben at bendarnell.com> wrote:
>> Still, it's a pretty weak argument IMO -- I don't expect there to be a
>> significant cottage industry cranking out 3rd party protocol
>> implementations, assuming we add UDP to PEP 3156, which is my plan.
>
> Yeah, as long as we get TCP, UDP, SSL, and pipes (at least for
> subprocesses), I'm hard pressed to imagine anything else that would be in so
> much demand it would need to be supported by all event loops.

There are two thing that come to *my* mind whenever this sort of
debate comes up. I'll freely admit that they are 100% theoretical in
terms of my actual requirements, but neither is particular
implausible.

- Synchronisation primitlives like Windows event objects - wanting to
integrate code to be run when an event is set into an event loop seems
relatively reasonable.
- GUI input events - I don't know about Unix, but Windows GUI events
are a separate notification stream from network or pipe data, and it's
very plausible that someone would want to integrate GUI events into an
async app. Twisted, for instance, has GUI event loop integration
facilities, I believe.

(I know that "write your own event loop" is always a solution. But I'm
not sure that doesn't set the bar a bit too high - it depends on how
easy it is to reuse and extend the implementation of the standard
event loops, which is something I'm not really clear on yet).

Paul


From guido at python.org  Tue Feb  5 22:28:11 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 5 Feb 2013 13:28:11 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
Message-ID: <CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>

On Tue, Feb 5, 2013 at 12:32 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 5 February 2013 04:50, Ben Darnell <ben at bendarnell.com> wrote:
>>> Still, it's a pretty weak argument IMO -- I don't expect there to be a
>>> significant cottage industry cranking out 3rd party protocol
>>> implementations, assuming we add UDP to PEP 3156, which is my plan.
>>
>> Yeah, as long as we get TCP, UDP, SSL, and pipes (at least for
>> subprocesses), I'm hard pressed to imagine anything else that would be in so
>> much demand it would need to be supported by all event loops.
>
> There are two thing that come to *my* mind whenever this sort of
> debate comes up. I'll freely admit that they are 100% theoretical in
> terms of my actual requirements, but neither is particular
> implausible.
>
> - Synchronisation primitlives like Windows event objects - wanting to
> integrate code to be run when an event is set into an event loop seems
> relatively reasonable.

Since this is about interacting with the threading world, you can
always wrap that in a concurrent.futures.Future, and then wrap that in
eventloop.wrap_future().

> - GUI input events - I don't know about Unix, but Windows GUI events
> are a separate notification stream from network or pipe data, and it's
> very plausible that someone would want to integrate GUI events into an
> async app. Twisted, for instance, has GUI event loop integration
> facilities, I believe.

That's way too large a topic to try to anticipate without thorough
research. And it's not very likely that you can do this in a portable
way either -- the best you can probably hope for is have a PEP
3156-compliant event loop that lets you use portable async networking
APIs (transports and protocols) while also letting you write
platform-*specific* GUI code. The best route here will probably be the
PEP 3156 bridge that Twisted is going to develop once the PEP and
Twisted's Python 3 port stabilize. (Unfortunately, wxPython is not
ported to Python 3.)

> (I know that "write your own event loop" is always a solution. But I'm
> not sure that doesn't set the bar a bit too high - it depends on how
> easy it is to reuse and extend the implementation of the standard
> event loops, which is something I'm not really clear on yet).

Check the most recent changes that landed in Tulip. The EventLoop
class has been refactored to bits and pieces, with exactly that
purpose in mind. (The PEP is a bit behind ATM, and I just broke the
ProactorEventLoop's ability to act as a server, but I'm on that.)

-- 
--Guido van Rossum (python.org/~guido)


From p.f.moore at gmail.com  Tue Feb  5 23:08:02 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 5 Feb 2013 22:08:02 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
Message-ID: <CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>

Thanks for the responses. I think you're right on both counts - in
particular, I'd forgotten the ability to wrap futures for the event
case.

On 5 February 2013 21:28, Guido van Rossum <guido at python.org> wrote:
> Check the most recent changes that landed in Tulip. The EventLoop
> class has been refactored to bits and pieces, with exactly that
> purpose in mind. (The PEP is a bit behind ATM, and I just broke the
> ProactorEventLoop's ability to act as a server, but I'm on that.)

I'll try to get a chance to look. I haven't really checked the code
for a while, as I've been busy. Did you ever look at my subprocess
code? It was in a bitbucket repo rather than reitveld, my apologies if
that was a problem, I can investigate how to use reitveld if needed.

Paul


From saghul at gmail.com  Tue Feb  5 23:35:19 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Tue, 05 Feb 2013 23:35:19 +0100
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <51070056.8020006@gmail.com>
References: <51070056.8020006@gmail.com>
Message-ID: <51118927.3060506@gmail.com>

Hi again,

I just updated rose [0] to match latest changes in Tulip API and remove 
Tulip itself from the code, now a proper EventLoopPolicy is defined, 
which will in turn use the pyuv-based event loop.


Regards,

[0]: https://github.com/saghul/rose

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From guido at python.org  Tue Feb  5 23:39:30 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 5 Feb 2013 14:39:30 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
Message-ID: <CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>

On Tue, Feb 5, 2013 at 2:08 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> Did you ever look at my subprocess
> code? It was in a bitbucket repo rather than reitveld, my apologies if
> that was a problem, I can investigate how to use reitveld if needed.

I glanced at it, but didn't really review it in depth. Do you think
it's ready for integration? I do like the idea of using as much from
the subprocess module as possible. We also need to look at supporting
the same API on Windows (since subprocess supports it). I have access
to a Windows box now.

If you think you have something that's ready to integrate (even if
just UNIX), please do use Rietveld (codereview.appspot.com). The best
strategy is to
leave your changes uncommitted in a current checkout of Tulip, and run
the upload.py script that you
can download here: https://codereview.appspot.com/static/upload.py
and which is documented here:
http://code.google.com/p/rietveld/wiki/UploadPyUsage

(You are already a PSF contributed, right?)

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Tue Feb  5 23:58:02 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 5 Feb 2013 14:58:02 -0800
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <51118927.3060506@gmail.com>
References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com>
Message-ID: <CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>

On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg? <saghul at gmail.com> wrote:
> I just updated rose [0] to match latest changes in Tulip API and remove
> Tulip itself from the code, now a proper EventLoopPolicy is defined, which
> will in turn use the pyuv-based event loop.

Neat!

I wonder if you could see how many of Tulip's own tests pass with your
event loop implementation?

> [0]: https://github.com/saghul/rose
>
>
> --
> Sa?l Ibarra Corretg?
> http://saghul.net/blog | http://about.me/saghul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas

-- 
--Guido van Rossum (python.org/~guido)


From fafhrd91 at gmail.com  Wed Feb  6 00:10:58 2013
From: fafhrd91 at gmail.com (Nikolay Kim)
Date: Tue, 5 Feb 2013 15:10:58 -0800
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com>
	<CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
Message-ID: <E541592F-4923-49E3-AE31-C25176686ABC@gmail.com>

I'll try to run gunicorn worker as well

On Feb 5, 2013, at 2:58 PM, Guido van Rossum <guido at python.org> wrote:

> On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg? <saghul at gmail.com> wrote:
>> I just updated rose [0] to match latest changes in Tulip API and remove
>> Tulip itself from the code, now a proper EventLoopPolicy is defined, which
>> will in turn use the pyuv-based event loop.
> 
> Neat!
> 
> I wonder if you could see how many of Tulip's own tests pass with your
> event loop implementation?
> 
>> [0]: https://github.com/saghul/rose
>> 
>> 
>> --
>> Sa?l Ibarra Corretg?
>> http://saghul.net/blog | http://about.me/saghul
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



From fafhrd91 at gmail.com  Wed Feb  6 00:32:02 2013
From: fafhrd91 at gmail.com (Nikolay Kim)
Date: Tue, 5 Feb 2013 15:32:02 -0800
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com>
	<CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
Message-ID: <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com>


On Feb 5, 2013, at 2:58 PM, Guido van Rossum <guido at python.org> wrote:

> On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg? <saghul at gmail.com> wrote:
>> I just updated rose [0] to match latest changes in Tulip API and remove
>> Tulip itself from the code, now a proper EventLoopPolicy is defined, which
>> will in turn use the pyuv-based event loop.
> 
> Neat!
> 
> I wonder if you could see how many of Tulip's own tests pass with your
> event loop implementation?
> 

two tests are failing. i think thats very good result.

======================================================================
FAIL: test_start_serving_cant_bind (tulip.events_test.UVEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/unittest/mock.py", line 1088, in patched
    return func(*args, **keywargs)
  File "/Users/nikolay/dev/tulip/src/tulip/tulip/events_test.py", line 497, in test_start_serving_cant_bind
    self.assertRaises(Err, self.event_loop.run_until_complete, fut)
AssertionError: Err not raised by run_until_complete

======================================================================
FAIL: test_baseexception_during_cancel (tulip.tasks_test.TaskTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/nikolay/dev/tulip/src/tulip/tulip/tasks_test.py", line 451, in test_baseexception_during_cancel
    self.assertRaises(BaseException, self.event_loop.run_once)
AssertionError: BaseException not raised by run_once



>> [0]: https://github.com/saghul/rose
>> 
>> 
>> --
>> Sa?l Ibarra Corretg?
>> http://saghul.net/blog | http://about.me/saghul
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



From p.f.moore at gmail.com  Wed Feb  6 00:56:18 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 5 Feb 2013 23:56:18 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
Message-ID: <CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>

On 5 February 2013 22:39, Guido van Rossum <guido at python.org> wrote:
> On Tue, Feb 5, 2013 at 2:08 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> Did you ever look at my subprocess
>> code? It was in a bitbucket repo rather than reitveld, my apologies if
>> that was a problem, I can investigate how to use reitveld if needed.
>
> I glanced at it, but didn't really review it in depth. Do you think
> it's ready for integration? I do like the idea of using as much from
> the subprocess module as possible. We also need to look at supporting
> the same API on Windows (since subprocess supports it). I have access
> to a Windows box now.

Functionally, I'm happy with the patch (on Unix, it needs something
for Windows but I don't know IOCP very well, so I'm not sure how much
I can do there, ironically). I'm a bit concerned with the usability of
the API in the coroutine style (it's fine using callbacks, which I'm
comfortable with, but I'm struggling to get my head round coroutines,
so I've no intuition as to whether it feels natural in that style).

> If you think you have something that's ready to integrate (even if
> just UNIX), please do use Rietveld (codereview.appspot.com). The best
> strategy is to
> leave your changes uncommitted in a current checkout of Tulip, and run
> the upload.py script that you
> can download here: https://codereview.appspot.com/static/upload.py
> and which is documented here:
> http://code.google.com/p/rietveld/wiki/UploadPyUsage

OK, I'll give it a try - probably not for a week or two, as I'm away a
lot at the moment...

> (You are already a PSF contributed, right?)

TBH, I'm not entirely sure. I *should* be, as there's code in Python
from me, but I don't recall ever sending in a form. I'll send a new
form in just to be certain.

Paul.


From saghul at gmail.com  Wed Feb  6 00:56:51 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Wed, 06 Feb 2013 00:56:51 +0100
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com>
References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com>
	<CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
	<381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com>
Message-ID: <51119C43.9060604@gmail.com>

Nikolay Kim wrote:
> On Feb 5, 2013, at 2:58 PM, Guido van Rossum<guido at python.org>  wrote:
>
>> On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg?<saghul at gmail.com>  wrote:
>>> I just updated rose [0] to match latest changes in Tulip API and remove
>>> Tulip itself from the code, now a proper EventLoopPolicy is defined, which
>>> will in turn use the pyuv-based event loop.
>> Neat!
>>
>> I wonder if you could see how many of Tulip's own tests pass with your
>> event loop implementation?
>>

Hum, they all pass for me on OSX. Did you use the runtests.py script 
from rose?

>
> two tests are failing. i think thats very good result.
>
> ======================================================================
> FAIL: test_start_serving_cant_bind (tulip.events_test.UVEventLoopTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/unittest/mock.py", line 1088, in patched
>      return func(*args, **keywargs)
>    File "/Users/nikolay/dev/tulip/src/tulip/tulip/events_test.py", line 497, in test_start_serving_cant_bind
>      self.assertRaises(Err, self.event_loop.run_until_complete, fut)
> AssertionError: Err not raised by run_until_complete
>

I overwrote this test in rose/events_test.py

> ======================================================================
> FAIL: test_baseexception_during_cancel (tulip.tasks_test.TaskTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "/Users/nikolay/dev/tulip/src/tulip/tulip/tasks_test.py", line 451, in test_baseexception_during_cancel
>      self.assertRaises(BaseException, self.event_loop.run_once)
> AssertionError: BaseException not raised by run_once
>

Hum, I have not seen this one.


-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From fafhrd91 at gmail.com  Wed Feb  6 01:30:37 2013
From: fafhrd91 at gmail.com (Nikolay Kim)
Date: Tue, 5 Feb 2013 16:30:37 -0800
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <51119C43.9060604@gmail.com>
References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com>
	<CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
	<381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com>
	<51119C43.9060604@gmail.com>
Message-ID: <051B29A4-9023-4778-9B0B-976E1633676C@gmail.com>

i used uv_events as default event loop and ran all tulip tests with it. but those failures are not related to uv_events anyway
I'd inherit uv_events.EventLoop from base_events.BaseEventLoop.


On Feb 5, 2013, at 3:56 PM, Sa?l Ibarra Corretg? <saghul at gmail.com> wrote:

> Nikolay Kim wrote:
>> On Feb 5, 2013, at 2:58 PM, Guido van Rossum<guido at python.org>  wrote:
>> 
>>> On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg?<saghul at gmail.com>  wrote:
>>>> I just updated rose [0] to match latest changes in Tulip API and remove
>>>> Tulip itself from the code, now a proper EventLoopPolicy is defined, which
>>>> will in turn use the pyuv-based event loop.
>>> Neat!
>>> 
>>> I wonder if you could see how many of Tulip's own tests pass with your
>>> event loop implementation?
>>> 
> 
> Hum, they all pass for me on OSX. Did you use the runtests.py script from rose?
> 
>> 
>> two tests are failing. i think thats very good result.
>> 
>> ======================================================================
>> FAIL: test_start_serving_cant_bind (tulip.events_test.UVEventLoopTests)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/unittest/mock.py", line 1088, in patched
>>     return func(*args, **keywargs)
>>   File "/Users/nikolay/dev/tulip/src/tulip/tulip/events_test.py", line 497, in test_start_serving_cant_bind
>>     self.assertRaises(Err, self.event_loop.run_until_complete, fut)
>> AssertionError: Err not raised by run_until_complete
>> 
> 
> I overwrote this test in rose/events_test.py
> 
>> ======================================================================
>> FAIL: test_baseexception_during_cancel (tulip.tasks_test.TaskTests)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "/Users/nikolay/dev/tulip/src/tulip/tulip/tasks_test.py", line 451, in test_baseexception_during_cancel
>>     self.assertRaises(BaseException, self.event_loop.run_once)
>> AssertionError: BaseException not raised by run_once
>> 
> 
> Hum, I have not seen this one.
> 
> 
> -- 
> Sa?l Ibarra Corretg?
> http://saghul.net/blog | http://about.me/saghul



From tjreedy at udel.edu  Wed Feb  6 01:30:16 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 05 Feb 2013 19:30:16 -0500
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
Message-ID: <kes87a$p5u$1@ger.gmane.org>

On 2/5/2013 6:56 PM, Paul Moore wrote:
> On 5 February 2013 22:39, Guido van Rossum <guido at python.org> wrote:

>> (You are already a PSF contributed, right?)
>
> TBH, I'm not entirely sure. I *should* be, as there's code in Python
> from me, but I don't recall ever sending in a form. I'll send a new
> form in just to be certain.

This is now recorded in user records on the tracker.
As for you, Paul, http://bugs.python.org/user301
indicates that there is no current record. If you ever did send a paper 
form, it got lost a few years ago like mine and many others. You can now 
send a scan by email.

http://www.python.org/psf/contrib/

People with contributor agreements are also indicated by '*' after their 
name in issue messages. Other past or prospective contributors who are 
not sure of their recorded contributor status can check either way and 
send a form if needed.

-- 
Terry Jan Reedy



From saghul at gmail.com  Wed Feb  6 01:45:39 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Wed, 06 Feb 2013 01:45:39 +0100
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <051B29A4-9023-4778-9B0B-976E1633676C@gmail.com>
References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com>
	<CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
	<381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com>
	<51119C43.9060604@gmail.com>
	<051B29A4-9023-4778-9B0B-976E1633676C@gmail.com>
Message-ID: <5111A7B3.7050504@gmail.com>

Nikolay Kim wrote:
> i used uv_events as default event loop and ran all tulip tests with it. but those failures are not related to uv_events anyway
> I'd inherit uv_events.EventLoop from base_events.BaseEventLoop.
>

Why? There is not much code that can be reused, since timers are not 
needed and the _run_once function is also not applicable.

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From guido at python.org  Wed Feb  6 02:04:16 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 5 Feb 2013 17:04:16 -0800
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <5111A7B3.7050504@gmail.com>
References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com>
	<CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
	<381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com>
	<51119C43.9060604@gmail.com>
	<051B29A4-9023-4778-9B0B-976E1633676C@gmail.com>
	<5111A7B3.7050504@gmail.com>
Message-ID: <CAP7+vJJd--1t=eAG+HZrmeMSdGS1bCGkJtk8qg4FEZU+RBhCbA@mail.gmail.com>

On Tue, Feb 5, 2013 at 4:45 PM, Sa?l Ibarra Corretg? <saghul at gmail.com> wrote:
> Nikolay Kim wrote:
>>
>> i used uv_events as default event loop and ran all tulip tests with it.
>> but those failures are not related to uv_events anyway
>> I'd inherit uv_events.EventLoop from base_events.BaseEventLoop.
>>
>
> Why? There is not much code that can be reused, since timers are not needed
> and the _run_once function is also not applicable.

Because the test mocks base_events.socket. :-) That's not a great
reason, I agree. Perhaps you and Nikolay can work on a better way to
do the mocking so it doesn't rely on base_events?

-- 
--Guido van Rossum (python.org/~guido)


From ubershmekel at gmail.com  Wed Feb  6 20:19:10 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 6 Feb 2013 21:19:10 +0200
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <kes87a$p5u$1@ger.gmane.org>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
Message-ID: <CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>

Any others wanting to toy with tulip on windows can have the
_overlapped.pyd I painfully made for python3.3-32bit.

https://www.dropbox.com/s/20ljdyafpe25ekh/_overlapped.pyd

BTW I needed to require windows vista for it to compile...

#define _WIN32_WINNT 0x0600
#define NTDDI_VERSION 0x06000000


Yuval Greenfield
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130206/027e31b8/attachment.html>

From guido at python.org  Wed Feb  6 20:29:56 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 6 Feb 2013 11:29:56 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
Message-ID: <CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>

Great! Are you keeping that up to date? Richard is checking changes at
a furious pace. :-)

On Wed, Feb 6, 2013 at 11:19 AM, Yuval Greenfield <ubershmekel at gmail.com> wrote:
> Any others wanting to toy with tulip on windows can have the _overlapped.pyd
> I painfully made for python3.3-32bit.
>
> https://www.dropbox.com/s/20ljdyafpe25ekh/_overlapped.pyd
>
> BTW I needed to require windows vista for it to compile...
>
> #define _WIN32_WINNT 0x0600
> #define NTDDI_VERSION 0x06000000
>
>
> Yuval Greenfield
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
--Guido van Rossum (python.org/~guido)


From ubershmekel at gmail.com  Wed Feb  6 22:23:13 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 6 Feb 2013 23:23:13 +0200
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
Message-ID: <CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>

On Wed, Feb 6, 2013 at 9:29 PM, Guido van Rossum <guido at python.org> wrote:

> Great! Are you keeping that up to date? Richard is checking changes at
> a furious pace. :-)
>
>
I updated the file so the same link has the new pyd that is up to date as
of
http://code.google.com/p/tulip/source/detail?r=65c456e2c20ece3adabc6d5f37d78957e9aeaf18-
Fix return type of SetFromWindowsErr().

E:\Dropbox\dev\python\tulip>c:\python33\python runtests.py
....s......s............s.............sss......ss.....sss...s..........sss......
....................................................
----------------------------------------------------------------------
Ran 132 tests in 8.513s

OK (skipped=15)

Strangely, the first time I run the tests I get a big pile of output
exceptions though the tests do pass. E.g.

.........sss......ss.ERROR:root:Exception in task
Traceback (most recent call last):
  File "E:\Dropbox\dev\python\tulip\tulip\tasks.py", line 96, in _step
    result = coro.send(value)
  File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 235, in
create_c
onnection
    raise exceptions[0]
  File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 226, in
create_c
onnection
    yield self.sock_connect(sock, address)
  File "c:\python33\lib\unittest\mock.py", line 846, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "c:\python33\lib\unittest\mock.py", line 901, in _mock_call
    raise effect
OSError
.ERROR:root:Exception in task
[...]



Indeed, it's hard to keep up.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130206/085947b0/attachment.html>

From shibturn at gmail.com  Wed Feb  6 22:48:37 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Wed, 06 Feb 2013 21:48:37 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
Message-ID: <keuj3n$bl0$1@ger.gmane.org>

On 06/02/2013 9:23pm, Yuval Greenfield wrote:
> Strangely, the first time I run the tests I get a big pile of output
> exceptions though the tests do pass. E.g.
>
> .........sss......ss.ERROR:root:Exception in task
> Traceback (most recent call last):
>    File "E:\Dropbox\dev\python\tulip\tulip\tasks.py", line 96, in _step
>      result = coro.send(value)
>    File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 235, in
> create_c
> onnection
>      raise exceptions[0]
>    File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 226, in
> create_c
> onnection
>      yield self.sock_connect(sock, address)
>    File "c:\python33\lib\unittest\mock.py", line 846, in __call__
>      return _mock_self._mock_call(*args, **kwargs)
>    File "c:\python33\lib\unittest\mock.py", line 901, in _mock_call
>      raise effect
> OSError
> .ERROR:root:Exception in task
> [...]

I see this sometimes too.

It seems that these are expected errors caused by using 
self.assertRaises(...).  Why these errors are logged only sometimes I 
don't understand.

-- 
Richard



From p.f.moore at gmail.com  Wed Feb  6 23:20:17 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 6 Feb 2013 22:20:17 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <keuj3n$bl0$1@ger.gmane.org>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
Message-ID: <CACac1F-QvvZzq6U2X4jek+W=7seQeNy+42Xeyg=qVORzNevOuw@mail.gmail.com>

On 6 February 2013 21:48, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 06/02/2013 9:23pm, Yuval Greenfield wrote:
>>
>> Strangely, the first time I run the tests I get a big pile of output
>> exceptions though the tests do pass. E.g.

I just tried a build on Windows 7 64-bit with Python 3.3 (i.e., not a
3.4 checkout). I got a lot of these types of error as well. I also got
a couple of genuine ones. One was a build problem - PY_ULONG_MAX
doesn't exist. I changed it to ULONG_MAX which got the compile to work
- but it may not be correct, I guess. The other was:

ERROR: test_sock_accept (tulip.events_test.ProactorEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Work\Scratch\tulip\tulip\events_test.py", line 309, in
test_sock_accept
    conn, addr = self.event_loop.run_until_complete(f)
  File "C:\Work\Scratch\tulip\tulip\base_events.py", line 104, in
run_until_complete
    return future.result()  # May raise future.exception().
  File "C:\Work\Scratch\tulip\tulip\futures.py", line 148, in result
    raise self._exception
  File "C:\Work\Scratch\tulip\tulip\windows_events.py", line 132, in _poll
    value = callback()
  File "C:\Work\Scratch\tulip\tulip\windows_events.py", line 83, in
finish_accept
    listener.fileno())
OSError: [WinError 10014] The system detected an invalid pointer
address in attempting to use a pointer argument in a call

Interestingly, I saw the same OSError occurring in some of the
"ERROR:root:Accept failed" logging stuff.

I don't know if that's of any use...

Paul


From amauryfa at gmail.com  Wed Feb  6 23:37:06 2013
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 6 Feb 2013 23:37:06 +0100
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
Message-ID: <CAGmFidYhU_Qvo1jcgTx1gApuxOvzvuooQnPnr79s36k0hDcNKA@mail.gmail.com>

2013/2/5 Guido van Rossum <guido at python.org>

> > - GUI input events - I don't know about Unix, but Windows GUI events
> > are a separate notification stream from network or pipe data, and it's
> > very plausible that someone would want to integrate GUI events into an
> > async app. Twisted, for instance, has GUI event loop integration
> > facilities, I believe.
>
> That's way too large a topic to try to anticipate without thorough
> research. And it's not very likely that you can do this in a portable
> way either -- the best you can probably hope for is have a PEP
> 3156-compliant event loop that lets you use portable async networking
> APIs (transports and protocols) while also letting you write
> platform-*specific* GUI code. The best route here will probably be the
> PEP 3156 bridge that Twisted is going to develop once the PEP and
> Twisted's Python 3 port stabilize. (Unfortunately, wxPython is not
> ported to Python 3.)


That's not exactly true: http://wiki.wxpython.org/ProjectPhoenix#Links
It's still a work in progress, but the basic interface is unlikely to
change.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130206/a36dde36/attachment.html>

From guido at python.org  Wed Feb  6 23:43:54 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 6 Feb 2013 14:43:54 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <keuj3n$bl0$1@ger.gmane.org>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
Message-ID: <CAP7+vJL4uX=gpziqhUaY2WMTffvCKhacDnfy72uP+UJjQEfz9w@mail.gmail.com>

On Wed, Feb 6, 2013 at 1:48 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 06/02/2013 9:23pm, Yuval Greenfield wrote:
>>
>> Strangely, the first time I run the tests I get a big pile of output
>> exceptions though the tests do pass. E.g.
>>
>> .........sss......ss.ERROR:root:Exception in task
>> Traceback (most recent call last):
>>    File "E:\Dropbox\dev\python\tulip\tulip\tasks.py", line 96, in _step
>>      result = coro.send(value)
>>    File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 235, in
>> create_c
>> onnection
>>      raise exceptions[0]
>>    File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 226, in
>> create_c
>> onnection
>>      yield self.sock_connect(sock, address)
>>    File "c:\python33\lib\unittest\mock.py", line 846, in __call__
>>      return _mock_self._mock_call(*args, **kwargs)
>>    File "c:\python33\lib\unittest\mock.py", line 901, in _mock_call
>>      raise effect
>> OSError
>> .ERROR:root:Exception in task
>> [...]
>
>
> I see this sometimes too.
>
> It seems that these are expected errors caused by using
> self.assertRaises(...).  Why these errors are logged only sometimes I don't
> understand.

Me neither. :-(

The message "Exception in task" means that it is a task that raises an
exception. I used to ignore these; now I log them always, but ideally
they should only be logged when whoever waits for the Task doesn't
catch them (or, better, when nobody waits for the task). I tried to
implement that part but I couldn't get it to work (yet) -- I will have
to get back to this at some point, because accurate exception logging
(never silent, but not too spammy either) is very important for a good
user experience.

But it remains a mystery why they sometimes show and not other times.
It suggests there's some indeterminate timing in some tests. If it
happens only the first time when the tests are run this usually points
to a timing behavior that's different when the source code is parsed
as opposed to read from a .pyc file.

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Wed Feb  6 23:44:41 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 6 Feb 2013 14:44:41 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAGmFidYhU_Qvo1jcgTx1gApuxOvzvuooQnPnr79s36k0hDcNKA@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAP7+vJ+gMhXHs+2zmr+H=sMLwJ5+60OGd38YL6LRYqDkpRgjJw@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CAGmFidYhU_Qvo1jcgTx1gApuxOvzvuooQnPnr79s36k0hDcNKA@mail.gmail.com>
Message-ID: <CAP7+vJLj0zon=-_7Fwc1-an5cdvbBFoxs1jYCTJS42Jw6Oc4WA@mail.gmail.com>

On Wed, Feb 6, 2013 at 2:37 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> 2013/2/5 Guido van Rossum <guido at python.org>
>> (Unfortunately, wxPython is not
>> ported to Python 3.)

> That's not exactly true: http://wiki.wxpython.org/ProjectPhoenix#Links
> It's still a work in progress, but the basic interface is unlikely to
> change.

Glad to hear it!

-- 
--Guido van Rossum (python.org/~guido)


From shibturn at gmail.com  Thu Feb  7 00:58:03 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Wed, 06 Feb 2013 23:58:03 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJL4uX=gpziqhUaY2WMTffvCKhacDnfy72uP+UJjQEfz9w@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
	<CAP7+vJL4uX=gpziqhUaY2WMTffvCKhacDnfy72uP+UJjQEfz9w@mail.gmail.com>
Message-ID: <keuqmd$glm$1@ger.gmane.org>

On 06/02/2013 10:43pm, Guido van Rossum wrote:
> Me neither.:-(
>
> The message "Exception in task" means that it is a task that raises an
> exception. I used to ignore these; now I log them always, but ideally
> they should only be logged when whoever waits for the Task doesn't
> catch them (or, better, when nobody waits for the task). I tried to
> implement that part but I couldn't get it to work (yet) -- I will have
> to get back to this at some point, because accurate exception logging
> (never silent, but not too spammy either) is very important for a good
> user experience.
>
> But it remains a mystery why they sometimes show and not other times.
> It suggests there's some indeterminate timing in some tests. If it
> happens only the first time when the tests are run this usually points
> to a timing behavior that's different when the source code is parsed
> as opposed to read from a .pyc file.

Commenting out the one use of suppress_log_errors() makes the "expected 
errors" appear on Linux too.  But I would have thought that that would 
only effect the test which uses suppress_log_errors().

diff -r 65c456e2c20e tulip/events_test.py
--- a/tulip/events_test.py      Wed Feb 06 19:08:14 2013 +0000
+++ b/tulip/events_test.py      Wed Feb 06 23:51:32 2013 +0000
@@ -508,7 +508,7 @@
          self.assertFalse(sock.close.called)

      def test_accept_connection_exception(self):
-        self.suppress_log_errors()
+        #self.suppress_log_errors()

          sock = unittest.mock.Mock()
          sock.accept.side_effect = socket.error

-- 
Richard



From guido at python.org  Thu Feb  7 01:11:37 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 6 Feb 2013 16:11:37 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <keuqmd$glm$1@ger.gmane.org>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
	<CAP7+vJL4uX=gpziqhUaY2WMTffvCKhacDnfy72uP+UJjQEfz9w@mail.gmail.com>
	<keuqmd$glm$1@ger.gmane.org>
Message-ID: <CAP7+vJJE6fsrq78=R==LhOkR7T+juGVSkgWR1p1vebbowH==pQ@mail.gmail.com>

On Wed, Feb 6, 2013 at 3:58 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 06/02/2013 10:43pm, Guido van Rossum wrote:
>>
>> Me neither.:-(
>>
>> The message "Exception in task" means that it is a task that raises an
>> exception. I used to ignore these; now I log them always, but ideally
>> they should only be logged when whoever waits for the Task doesn't
>> catch them (or, better, when nobody waits for the task). I tried to
>> implement that part but I couldn't get it to work (yet) -- I will have
>> to get back to this at some point, because accurate exception logging
>> (never silent, but not too spammy either) is very important for a good
>> user experience.
>>
>> But it remains a mystery why they sometimes show and not other times.
>> It suggests there's some indeterminate timing in some tests. If it
>> happens only the first time when the tests are run this usually points
>> to a timing behavior that's different when the source code is parsed
>> as opposed to read from a .pyc file.
>
>
> Commenting out the one use of suppress_log_errors() makes the "expected
> errors" appear on Linux too.  But I would have thought that that would only
> effect the test which uses suppress_log_errors().
>
> diff -r 65c456e2c20e tulip/events_test.py
> --- a/tulip/events_test.py      Wed Feb 06 19:08:14 2013 +0000
> +++ b/tulip/events_test.py      Wed Feb 06 23:51:32 2013 +0000
> @@ -508,7 +508,7 @@
>          self.assertFalse(sock.close.called)
>
>      def test_accept_connection_exception(self):
> -        self.suppress_log_errors()
> +        #self.suppress_log_errors()
>
>          sock = unittest.mock.Mock()
>          sock.accept.side_effect = socket.error

Good catch! What's going on is that the super().tearDown() call was
missing from EventLoopTestsMixin. I'll fix that -- then we'll have to
separately add suppress_log_errors() calls to various tests (I'll do
that at a slower pace).

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Thu Feb  7 01:21:25 2013
From: guido at python.org (Guido van Rossum)
Date: Wed, 6 Feb 2013 16:21:25 -0800
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJJE6fsrq78=R==LhOkR7T+juGVSkgWR1p1vebbowH==pQ@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
	<CAP7+vJL4uX=gpziqhUaY2WMTffvCKhacDnfy72uP+UJjQEfz9w@mail.gmail.com>
	<keuqmd$glm$1@ger.gmane.org>
	<CAP7+vJJE6fsrq78=R==LhOkR7T+juGVSkgWR1p1vebbowH==pQ@mail.gmail.com>
Message-ID: <CAP7+vJKzZEqCoHYPPiErWVA6GQ2btkn=ELNYEqVHGP4uCwiVYQ@mail.gmail.com>

On Wed, Feb 6, 2013 at 4:11 PM, Guido van Rossum <guido at python.org> wrote:
> On Wed, Feb 6, 2013 at 3:58 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
>> On 06/02/2013 10:43pm, Guido van Rossum wrote:
>>>
>>> Me neither.:-(
>>>
>>> The message "Exception in task" means that it is a task that raises an
>>> exception. I used to ignore these; now I log them always, but ideally
>>> they should only be logged when whoever waits for the Task doesn't
>>> catch them (or, better, when nobody waits for the task). I tried to
>>> implement that part but I couldn't get it to work (yet) -- I will have
>>> to get back to this at some point, because accurate exception logging
>>> (never silent, but not too spammy either) is very important for a good
>>> user experience.
>>>
>>> But it remains a mystery why they sometimes show and not other times.
>>> It suggests there's some indeterminate timing in some tests. If it
>>> happens only the first time when the tests are run this usually points
>>> to a timing behavior that's different when the source code is parsed
>>> as opposed to read from a .pyc file.
>>
>>
>> Commenting out the one use of suppress_log_errors() makes the "expected
>> errors" appear on Linux too.  But I would have thought that that would only
>> effect the test which uses suppress_log_errors().
>>
>> diff -r 65c456e2c20e tulip/events_test.py
>> --- a/tulip/events_test.py      Wed Feb 06 19:08:14 2013 +0000
>> +++ b/tulip/events_test.py      Wed Feb 06 23:51:32 2013 +0000
>> @@ -508,7 +508,7 @@
>>          self.assertFalse(sock.close.called)
>>
>>      def test_accept_connection_exception(self):
>> -        self.suppress_log_errors()
>> +        #self.suppress_log_errors()
>>
>>          sock = unittest.mock.Mock()
>>          sock.accept.side_effect = socket.error
>
> Good catch! What's going on is that the super().tearDown() call was
> missing from EventLoopTestsMixin. I'll fix that -- then we'll have to
> separately add suppress_log_errors() calls to various tests (I'll do
> that at a slower pace).

Should all be fixed now. (Please check on Windows, I can't check it right now.)

-- 
--Guido van Rossum (python.org/~guido)


From ubershmekel at gmail.com  Thu Feb  7 01:32:00 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Thu, 7 Feb 2013 02:32:00 +0200
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CAP7+vJKzZEqCoHYPPiErWVA6GQ2btkn=ELNYEqVHGP4uCwiVYQ@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
	<CAP7+vJL4uX=gpziqhUaY2WMTffvCKhacDnfy72uP+UJjQEfz9w@mail.gmail.com>
	<keuqmd$glm$1@ger.gmane.org>
	<CAP7+vJJE6fsrq78=R==LhOkR7T+juGVSkgWR1p1vebbowH==pQ@mail.gmail.com>
	<CAP7+vJKzZEqCoHYPPiErWVA6GQ2btkn=ELNYEqVHGP4uCwiVYQ@mail.gmail.com>
Message-ID: <CANSw7Kx=7p-u_DHDFGYzSq30VmksVwAfXjh4QJQhJO26b4gdCQ@mail.gmail.com>

On Thu, Feb 7, 2013 at 2:21 AM, Guido van Rossum <guido at python.org> wrote:

> Should all be fixed now. (Please check on Windows, I can't check it right
> now.)
>
>
Removed the pyc's and ran on windows a few times. There was no strange
exception output. Seems to be fixed.

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130207/6a09bc9e/attachment.html>

From shibturn at gmail.com  Thu Feb  7 01:33:38 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Thu, 07 Feb 2013 00:33:38 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CACac1F-QvvZzq6U2X4jek+W=7seQeNy+42Xeyg=qVORzNevOuw@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
	<CACac1F-QvvZzq6U2X4jek+W=7seQeNy+42Xeyg=qVORzNevOuw@mail.gmail.com>
Message-ID: <keusp3$vhr$1@ger.gmane.org>

On 06/02/2013 10:20pm, Paul Moore wrote:
> I just tried a build on Windows 7 64-bit with Python 3.3 (i.e., not a
> 3.4 checkout). I got a lot of these types of error as well. I also got
> a couple of genuine ones. One was a build problem - PY_ULONG_MAX
> doesn't exist. I changed it to ULONG_MAX which got the compile to work
> - but it may not be correct, I guess. The other was:
>
> ERROR: test_sock_accept (tulip.events_test.ProactorEventLoopTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "C:\Work\Scratch\tulip\tulip\events_test.py", line 309, in
> test_sock_accept
>      conn, addr = self.event_loop.run_until_complete(f)
>    File "C:\Work\Scratch\tulip\tulip\base_events.py", line 104, in
> run_until_complete
>      return future.result()  # May raise future.exception().
>    File "C:\Work\Scratch\tulip\tulip\futures.py", line 148, in result
>      raise self._exception
>    File "C:\Work\Scratch\tulip\tulip\windows_events.py", line 132, in _poll
>      value = callback()
>    File "C:\Work\Scratch\tulip\tulip\windows_events.py", line 83, in
> finish_accept
>      listener.fileno())
> OSError: [WinError 10014] The system detected an invalid pointer
> address in attempting to use a pointer argument in a call
>
> Interestingly, I saw the same OSError occurring in some of the
> "ERROR:root:Accept failed" logging stuff.
>
> I don't know if that's of any use...

Thanks.

It should be fixed now, but I have not tested it on a 64 bit build.  (My 
64 bit setup seems to be all screwed up.)

-- 
Richard



From p.f.moore at gmail.com  Thu Feb  7 11:13:39 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 7 Feb 2013 10:13:39 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <keusp3$vhr$1@ger.gmane.org>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
	<CACac1F-QvvZzq6U2X4jek+W=7seQeNy+42Xeyg=qVORzNevOuw@mail.gmail.com>
	<keusp3$vhr$1@ger.gmane.org>
Message-ID: <CACac1F_k_5Ech9WoCMGbNVzvmXWK2jJKAiU7J3Qzkb0gT8G07Q@mail.gmail.com>

I can test it when I get home...

On Thursday, 7 February 2013, Richard Oudkerk wrote:

> On 06/02/2013 10:20pm, Paul Moore wrote:
>
>> I just tried a build on Windows 7 64-bit with Python 3.3 (i.e., not a
>> 3.4 checkout). I got a lot of these types of error as well. I also got
>> a couple of genuine ones. One was a build problem - PY_ULONG_MAX
>> doesn't exist. I changed it to ULONG_MAX which got the compile to work
>> - but it may not be correct, I guess. The other was:
>>
>> ERROR: test_sock_accept (tulip.events_test.**ProactorEventLoopTests)
>> ------------------------------**------------------------------**
>> ----------
>> Traceback (most recent call last):
>>    File "C:\Work\Scratch\tulip\tulip\**events_test.py", line 309, in
>> test_sock_accept
>>      conn, addr = self.event_loop.run_until_**complete(f)
>>    File "C:\Work\Scratch\tulip\tulip\**base_events.py", line 104, in
>> run_until_complete
>>      return future.result()  # May raise future.exception().
>>    File "C:\Work\Scratch\tulip\tulip\**futures.py", line 148, in result
>>      raise self._exception
>>    File "C:\Work\Scratch\tulip\tulip\**windows_events.py", line 132, in
>> _poll
>>      value = callback()
>>    File "C:\Work\Scratch\tulip\tulip\**windows_events.py", line 83, in
>> finish_accept
>>      listener.fileno())
>> OSError: [WinError 10014] The system detected an invalid pointer
>> address in attempting to use a pointer argument in a call
>>
>> Interestingly, I saw the same OSError occurring in some of the
>> "ERROR:root:Accept failed" logging stuff.
>>
>> I don't know if that's of any use...
>>
>
> Thanks.
>
> It should be fixed now, but I have not tested it on a 64 bit build.  (My
> 64 bit setup seems to be all screwed up.)
>
> --
> Richard
>
> ______________________________**_________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130207/f4037d36/attachment.html>

From p.f.moore at gmail.com  Thu Feb  7 18:44:54 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 7 Feb 2013 17:44:54 +0000
Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface
In-Reply-To: <CACac1F_k_5Ech9WoCMGbNVzvmXWK2jJKAiU7J3Qzkb0gT8G07Q@mail.gmail.com>
References: <CAFkYKJ6CZ7x6rJX6c8bNuyLUASKGfL1ZLkZRh6JG7Ycz4FZB6g@mail.gmail.com>
	<CAFkYKJ4_Cm-26pzyJQYRX7n9kbAkUkdXFXqk30zD6do1apa8pg@mail.gmail.com>
	<CAP7+vJLrZ+VGxTby7R=mDoqAxce-T_FVFPj+=JtW7uDEoVXegg@mail.gmail.com>
	<CAFkYKJ5oO0c9yYgqptvCVvjfWhhioDkqG-3gj5roT-rLMHrJXA@mail.gmail.com>
	<CACac1F_XQkLEJ4P9a0uN3UMxBsuNPg67Rdm7jRV2Whq2QgJDwQ@mail.gmail.com>
	<CAP7+vJLSwGBqCYj30kd97oh-ik7x4wH8+5=xao1An3BchEZuoQ@mail.gmail.com>
	<CACac1F8ofC+9DjyQap8AQY5-zCQgevYdkekTkrhNXfP_oxLyBA@mail.gmail.com>
	<CAP7+vJLxw42hET=qhqzzd38f_Ntniv806doe5Xfyagwq5yTjoA@mail.gmail.com>
	<CACac1F8+7XGkMv5FccrT9MrkPc0EZ7eo-E7FB5mOHK=kkehqYA@mail.gmail.com>
	<kes87a$p5u$1@ger.gmane.org>
	<CANSw7Ky6ijV6-3WKGL17_F6ZZ38VHpdjHaR09ihKvsM-pEFj4A@mail.gmail.com>
	<CAP7+vJJ8z_ReeB2pKdq7=s+EgjAQNJreO8ixwXJ2c42MDmBzuQ@mail.gmail.com>
	<CANSw7Kzmc_i3egUH1meoiiOwr5i5RpS9=K84GpAZ__aXyqULig@mail.gmail.com>
	<keuj3n$bl0$1@ger.gmane.org>
	<CACac1F-QvvZzq6U2X4jek+W=7seQeNy+42Xeyg=qVORzNevOuw@mail.gmail.com>
	<keusp3$vhr$1@ger.gmane.org>
	<CACac1F_k_5Ech9WoCMGbNVzvmXWK2jJKAiU7J3Qzkb0gT8G07Q@mail.gmail.com>
Message-ID: <CACac1F_wQnVYsxgY6qhxpRwk9p301x4uet4K-9pQokA+nW=ZZQ@mail.gmail.com>

On 7 February 2013 10:13, Paul Moore <p.f.moore at gmail.com> wrote:
> On Thursday, 7 February 2013, Richard Oudkerk wrote:
>>
>> It should be fixed now, but I have not tested it on a 64 bit build.  (My
>> 64 bit setup seems to be all screwed up.)
>
> I can test it when I get home...

Yep, works fine now on 64-bit Windows 7.

Ran 132 tests in 9.048s

OK (skipped=15)

No errors, no spurious logging of exceptions.

Thanks!
Paul


From fafhrd91 at gmail.com  Fri Feb  8 01:18:14 2013
From: fafhrd91 at gmail.com (Nikolay Kim)
Date: Thu, 7 Feb 2013 16:18:14 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: Synchronization primitives
Message-ID: <A38BF10C-F6C9-49D2-B31E-1448FFC450CD@gmail.com>

I think Tulip should have synchronization primitives by default. 
Here is my motivation:

  1. it is more convenient to use locking primitive with existing semantics rather than tulip magic.
crawl.py example could use semaphore instead of tasks.wait with timeout 0.
  2. while it seems easy to implement semaphore with tulip, it still requires deep understanding of 
tulips control flow. i had 2 not very obvious bugs in my semaphore implementation:

     a) release() could be called from different co-routines during same scheduling step.
     b) bug with acquire and release during same scheduling step.
         example:
      1. task tries to acquire locked semaphore
      2. other task releases semaphore

      in result - semaphore gets acquired 2 times at the end of scheduling
step, because actual semaphore acquisition happens in "call_soon" and at this stage
release() call is already released semaphore but first waiter does not acquire it yet.

Thoughts?

my implementation: https://codereview.appspot.com/download/issue7230045_15001.diff



From saghul at gmail.com  Fri Feb  8 01:24:35 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Fri, 08 Feb 2013 01:24:35 +0100
Subject: [Python-ideas] libuv based eventloop for tulip experiment
In-Reply-To: <CAP7+vJJd--1t=eAG+HZrmeMSdGS1bCGkJtk8qg4FEZU+RBhCbA@mail.gmail.com>
References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com>
	<CAP7+vJKF2bUUXY0ZHz=UCPL4H00uac5RovokZFQQaezF3mBF4Q@mail.gmail.com>
	<381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com>
	<51119C43.9060604@gmail.com>
	<051B29A4-9023-4778-9B0B-976E1633676C@gmail.com>
	<5111A7B3.7050504@gmail.com>
	<CAP7+vJJd--1t=eAG+HZrmeMSdGS1bCGkJtk8qg4FEZU+RBhCbA@mail.gmail.com>
Message-ID: <511445C3.8080601@gmail.com>

Guido van Rossum wrote:
> On Tue, Feb 5, 2013 at 4:45 PM, Sa?l Ibarra Corretg?<saghul at gmail.com>  wrote:
>> Nikolay Kim wrote:
>>> i used uv_events as default event loop and ran all tulip tests with it.
>>> but those failures are not related to uv_events anyway
>>> I'd inherit uv_events.EventLoop from base_events.BaseEventLoop.
>>>
>> Why? There is not much code that can be reused, since timers are not needed
>> and the _run_once function is also not applicable.
>
> Because the test mocks base_events.socket. :-) That's not a great
> reason, I agree. Perhaps you and Nikolay can work on a better way to
> do the mocking so it doesn't rely on base_events?
>

I see. I modified it to inherit from BaseEventLoop now. I found a bug in 
the way rose executes handler callbacks 
(https://github.com/saghul/rose/commit/94e984cf756c3d0730acf804c18cab682cffd6d6) 
I need to think how to implement this, probably saving the first base 
exception, stopping the processing and reraising it...

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From gotoalanlu at gmail.com  Fri Feb  8 03:34:30 2013
From: gotoalanlu at gmail.com (Hua Lu)
Date: Thu, 7 Feb 2013 20:34:30 -0600
Subject: [Python-ideas] frozenset literals
In-Reply-To: <CAOOU=zjn_0A4Ukc6WLDnoxAZG6B-=Y8dOaXjegMKmBPa4cHKcA@mail.gmail.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<CABocrW4RHngjjeb0AbBUyF7F1w_aaxuJd8U6ytGSmgEDpUMkhg@mail.gmail.com>
	<CAOOU=zjn_0A4Ukc6WLDnoxAZG6B-=Y8dOaXjegMKmBPa4cHKcA@mail.gmail.com>
Message-ID: <CAOOU=zgtRjAmzf-PXcDganut_-kb3T9Z8dy7qFUAH1F8NUXndA@mail.gmail.com>

Hey all, I have a simple hack around this problem for the time being.
It involves adding a parameter to ast.literal_eval. See this diff:

$ diff ast.py.bak ast.py
38c38
< def literal_eval(node_or_string):
---
> def literal_eval(node_or_string, use_frozensets=False):
49a50
>     set_t = frozenset if use_frozensets else set
60c61
<             return set(map(_convert, node.elts))
---
>             return set_t(map(_convert, node.elts))

Use is as follows:
>>> import ast
>>> ast.literal_eval("{1,2,3}")
{1, 2, 3}
>>> ast.literal_eval("{1,2,3}", use_frozensets=True)
frozenset({1, 2, 3})
>>> ast.literal_eval("{{1,2,3}: 'foo'}", use_frozensets=True)
{frozenset({1, 2, 3}): 'foo'}

Regards,
Alan


From solipsis at pitrou.net  Fri Feb  8 10:06:11 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 8 Feb 2013 10:06:11 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: Synchronization primitives
References: <A38BF10C-F6C9-49D2-B31E-1448FFC450CD@gmail.com>
Message-ID: <20130208100611.1d7c53fd@pitrou.net>

Le Thu, 7 Feb 2013 16:18:14 -0800,
Nikolay Kim <fafhrd91 at gmail.com> a ?crit :
> I think Tulip should have synchronization primitives by default. 
> Here is my motivation:
> 
>   1. it is more convenient to use locking primitive with existing
> semantics rather than tulip magic. crawl.py example could use
> semaphore instead of tasks.wait with timeout 0. 2. while it seems
> easy to implement semaphore with tulip, it still requires deep
> understanding of tulips control flow. i had 2 not very obvious bugs
> in my semaphore implementation:
> 
>      a) release() could be called from different co-routines during
> same scheduling step. b) bug with acquire and release during same
> scheduling step. example:
>       1. task tries to acquire locked semaphore
>       2. other task releases semaphore
> 
>       in result - semaphore gets acquired 2 times at the end of
> scheduling step, because actual semaphore acquisition happens in
> "call_soon" and at this stage release() call is already released
> semaphore but first waiter does not acquire it yet.

I think Twisted has DeferredSemaphore for the same reasons.

Regards

Antoine.




From guido at python.org  Fri Feb  8 18:44:33 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 8 Feb 2013 09:44:33 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: Synchronization primitives
In-Reply-To: <20130208100611.1d7c53fd@pitrou.net>
References: <A38BF10C-F6C9-49D2-B31E-1448FFC450CD@gmail.com>
	<20130208100611.1d7c53fd@pitrou.net>
Message-ID: <CAP7+vJLD4ia=LVtUJj9iefZ7VkN6FKOE_UEMzydFpzQ4Oj0J+w@mail.gmail.com>

On Fri, Feb 8, 2013 at 1:06 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Thu, 7 Feb 2013 16:18:14 -0800,
> Nikolay Kim <fafhrd91 at gmail.com> a ?crit :
>> I think Tulip should have synchronization primitives by default.
>> Here is my motivation:
>>
>>   1. it is more convenient to use locking primitive with existing
>> semantics rather than tulip magic. crawl.py example could use
>> semaphore instead of tasks.wait with timeout 0. 2. while it seems
>> easy to implement semaphore with tulip, it still requires deep
>> understanding of tulips control flow. i had 2 not very obvious bugs
>> in my semaphore implementation:
>>
>>      a) release() could be called from different co-routines during
>> same scheduling step. b) bug with acquire and release during same
>> scheduling step. example:
>>       1. task tries to acquire locked semaphore
>>       2. other task releases semaphore
>>
>>       in result - semaphore gets acquired 2 times at the end of
>> scheduling step, because actual semaphore acquisition happens in
>> "call_soon" and at this stage release() call is already released
>> semaphore but first waiter does not acquire it yet.
>
> I think Twisted has DeferredSemaphore for the same reasons.

So, is anyone against adding Nikolay's locks implementation to PEP
3156? It's a pretty simple API, and looks similar to the locks in
threading.py except for a sprinkling of "yield from". It does not
depend on Tulip internals.

-- 
--Guido van Rossum (python.org/~guido)


From ncoghlan at gmail.com  Sat Feb  9 04:26:56 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 9 Feb 2013 13:26:56 +1000
Subject: [Python-ideas] PEP 3156 / Tulip: Synchronization primitives
In-Reply-To: <CAP7+vJLD4ia=LVtUJj9iefZ7VkN6FKOE_UEMzydFpzQ4Oj0J+w@mail.gmail.com>
References: <A38BF10C-F6C9-49D2-B31E-1448FFC450CD@gmail.com>
	<20130208100611.1d7c53fd@pitrou.net>
	<CAP7+vJLD4ia=LVtUJj9iefZ7VkN6FKOE_UEMzydFpzQ4Oj0J+w@mail.gmail.com>
Message-ID: <CADiSq7fzj5j-MCw_azMcRTibNCggneSw=7BGUZf=mSUM+Z7dJw@mail.gmail.com>

On Sat, Feb 9, 2013 at 3:44 AM, Guido van Rossum <guido at python.org> wrote:
> On Fri, Feb 8, 2013 at 1:06 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> I think Twisted has DeferredSemaphore for the same reasons.
>
> So, is anyone against adding Nikolay's locks implementation to PEP
> 3156? It's a pretty simple API, and looks similar to the locks in
> threading.py except for a sprinkling of "yield from". It does not
> depend on Tulip internals.

Common need + easy to get the implementation details wrong = sounds
like a good candidate for inclusion to me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From timothy.c.delaney at gmail.com  Sun Feb 10 23:43:45 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Mon, 11 Feb 2013 09:43:45 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
Message-ID: <CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>

On 4 February 2013 11:17, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> On 4 February 2013 10:53, Jo?o Bernardo <jbvsmo at gmail.com> wrote:
>
>> Hi, about this enum/const thing, The use case I like more is a class
>> where you know all the
>> instances and not just a sequence of names.
>> Particularly It would be nice to have custom attributes and methods
>> besides the value and the name.
>>
>> I have my own implementation with a basic api somewhat borrowed from
>> flufl.enum (plus a lot of other stuff),
>> but with this kind of support: https://github.com/jbvsmo/makeobj
>>
>
> I considered it, and in fact you could almost do it with my implementation
> by using a custom subclass of EnumValue (except trying it has just exposed
> a design flaw with the whole _EnumProxy bit). Works if you create the enum
> in the same module as EnumValues, fails otherwise. Going to have to have a
> rethink.
>

Fixed the _EnumProxy issue (but it's a kludge - I've used sys._getframe() -
there's probably a better way). I've also made it so that you can override
the metaclass to return a subclass of EnumValue.

Now you can do something like:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum, EnumValue, EnumMeta
>>>
>>> class MyEnumValue1(EnumValue):
...     pass
...
>>> class MyEnumMeta1(EnumMeta):
...     @classmethod
...     def _create_value(cls, key, value):
...         return MyEnumValue1(key, value)
...
>>> class MyEnum1(Enum, metaclass=MyEnumMeta1):
...     VALUE1,
...     VALUE2
...
>>> class MyEnumValue2(EnumValue):
...     pass
...
>>> class MyEnumMeta2(MyEnumMeta1):
...     @classmethod
...     def _create_value(cls, key, value):
...         return MyEnumValue2(key, value)
...
>>> class MyEnum2(MyEnum1, metaclass=MyEnumMeta2):
...     VALUE3,
...     VALUE4
...
>>> print(repr(MyEnum1))
<enum '__main__.MyEnum1' {<MyEnumValue1 'MyEnum1.VALUE1': 0>, <MyEnumValue1
'MyEnum1.VALUE2': 1>}>
>>> print(repr(MyEnum2))
<enum '__main__.MyEnum2' {<MyEnumValue1 'MyEnum1.VALUE1': 0>, <MyEnumValue1
'MyEnum1.VALUE2': 1>, <MyEnumValue2 'MyEnum2.VALUE3': 2>, <MyEnumValue2
'MyEnum2.VALUE4': 3>}>
>>>

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130211/2e83ee00/attachment.html>

From timothy.c.delaney at gmail.com  Mon Feb 11 07:59:55 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Mon, 11 Feb 2013 17:59:55 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
Message-ID: <CAN8CLgnHv1SCT=yj2qpbG0bAKpNYCG=OjDMJ13XxtVgX4QksFQ@mail.gmail.com>

On 11 February 2013 09:43, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> I've also made it so that you can override the metaclass to return a
> subclass of EnumValue.
>

Expanded on this a bit to simplify things. Now to use a different type you
can just go:

 Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum, EnumValue
>>>
>>> class MyEnumValue1(EnumValue):
...     pass
...
>>> class MyEnum1(Enum, metaclass=Enum.subtype(MyEnumValue1)):
...     VALUE1,
...     VALUE2
...
>>> class MyEnumValue2(EnumValue):
...     pass
...
>>> class MyEnum2(MyEnum1, metaclass=MyEnum1.subtype(MyEnumValue2)):
...     VALUE3,
...     VALUE4
...
>>> print(repr(MyEnum1))
<enum __main__.MyEnum1 {<MyEnumValue1 'VALUE1': 0>, <MyEnumValue1 'VALUE2':
1>}>
>>> print(repr(MyEnum2))
<enum __main__.MyEnum2 {<MyEnumValue1 'VALUE1': 0>, <MyEnumValue1 'VALUE2':
1>, <MyEnumValue2 'VALUE3': 2>, <MyEnumValue2 'VALUE4': 3>}>

The parameter passed to subtype() can be any callable that takes 2
parameters (key, value) and returns an instance of EnumValue.
Unfortunately, I can't see any way to avoid specifying the base class twice
(once as the base class, once for the metaclass=).

Similarly, passing the 'value_type' parameter to Enum.make() creates an
appropriate metaclass from the base class:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum, EnumValue
>>> class MyEnumValue1(EnumValue):
...     pass
...
>>> MyEnum1 = Enum.make('MyEnum1', ('VALUE1', 'VALUE2'),
value_type=MyEnumValue1)
>>> print(repr(MyEnum1))
<enum __main__.MyEnum1 {<MyEnumValue1 'VALUE1': 0>, <MyEnumValue1 'VALUE2':
1>}>

Whilst the original "use a different metaclass to produce values of a
different type" was possibly feature-creep, these modifications at least
make it pretty painless to use.

Overall, I'm pretty happy with where this is now, except for the use of
sys._getframe().

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130211/1e062b06/attachment.html>

From saghul at gmail.com  Mon Feb 11 13:21:44 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Mon, 11 Feb 2013 13:21:44 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
Message-ID: <5118E258.90705@gmail.com>

Hi,

While analyzing some code I came across the following situation:

- User calls run_until_complete for some Task
- The task itself calls loop.stop() at some point
- run_until_complete raises TimeoutError

Here is a very simple example: https://gist.github.com/saghul/4754117

Something seems a bit off here. While one could argue that stopping the 
loop while run_until_complete is ongoing, getting a TimeoutError doesn't 
feel right. I think we should detect that the loop was stopped but the 
future is not done and raise something like NotCompleted or 
FutureNotCompleted.

Thoughts?

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From eliben at gmail.com  Mon Feb 11 14:28:45 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 11 Feb 2013 05:28:45 -0800
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
Message-ID: <CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>

On Sun, Feb 10, 2013 at 2:43 PM, Tim Delaney <timothy.c.delaney at gmail.com>wrote:

> On 4 February 2013 11:17, Tim Delaney <timothy.c.delaney at gmail.com> wrote:
>
>> On 4 February 2013 10:53, Jo?o Bernardo <jbvsmo at gmail.com> wrote:
>>
>>> Hi, about this enum/const thing, The use case I like more is a class
>>> where you know all the
>>> instances and not just a sequence of names.
>>> Particularly It would be nice to have custom attributes and methods
>>> besides the value and the name.
>>>
>>> I have my own implementation with a basic api somewhat borrowed from
>>> flufl.enum (plus a lot of other stuff),
>>> but with this kind of support: https://github.com/jbvsmo/makeobj
>>>
>>
>> I considered it, and in fact you could almost do it with my
>> implementation by using a custom subclass of EnumValue (except trying it
>> has just exposed a design flaw with the whole _EnumProxy bit). Works if you
>> create the enum in the same module as EnumValues, fails otherwise. Going to
>> have to have a rethink.
>>
>
> Fixed the _EnumProxy issue (but it's a kludge - I've used sys._getframe()
> - there's probably a better way). I've also made it so that you can
> override the metaclass to return a subclass of EnumValue.
>
> Now you can do something like:
>
> Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
> bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from enum import Enum, EnumValue, EnumMeta
> >>>
> >>> class MyEnumValue1(EnumValue):
> ...     pass
> ...
> >>> class MyEnumMeta1(EnumMeta):
> ...     @classmethod
> ...     def _create_value(cls, key, value):
> ...         return MyEnumValue1(key, value)
> ...
> >>> class MyEnum1(Enum, metaclass=MyEnumMeta1):
>  ...     VALUE1,
> ...     VALUE2
> ...
> >>> class MyEnumValue2(EnumValue):
> ...     pass
> ...
> >>> class MyEnumMeta2(MyEnumMeta1):
> ...     @classmethod
> ...     def _create_value(cls, key, value):
> ...         return MyEnumValue2(key, value)
> ...
> >>> class MyEnum2(MyEnum1, metaclass=MyEnumMeta2):
> ...     VALUE3,
> ...     VALUE4
> ...
> >>> print(repr(MyEnum1))
> <enum '__main__.MyEnum1' {<MyEnumValue1 'MyEnum1.VALUE1': 0>,
> <MyEnumValue1 'MyEnum1.VALUE2': 1>}>
> >>> print(repr(MyEnum2))
> <enum '__main__.MyEnum2' {<MyEnumValue1 'MyEnum1.VALUE1': 0>,
> <MyEnumValue1 'MyEnum1.VALUE2': 1>, <MyEnumValue2 'MyEnum2.VALUE3': 2>,
> <MyEnumValue2 'MyEnum2.VALUE4': 3>}>
> >>>
>

Can you elaborate on the utility of this feature? What realistic use cases
do you see for it? I think that at this point it's important to weigh all
benefits of features vs. implementation complexity, and there's absolutely
no need to support every feature every other enum implementation has. I
want to stress again that the most important characteristic of your
implementation is the clean syntax which means that enums are so easy to
define they don't really need special Python syntax and a library feature
can do. However, there's a big leap from this to defining custom
metaclasses for enums.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130211/27f5899c/attachment.html>

From guido at python.org  Mon Feb 11 17:06:16 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 08:06:16 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <5118E258.90705@gmail.com>
References: <5118E258.90705@gmail.com>
Message-ID: <CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>

Yeah, I think the base implementation of run_until_complete is wrong.
I think I didn't have stop() when I coded that.

Could you change it to something that just registers a lambda calling
stop() on the Future, and see if that makes your test case behave
better? (If so, please add a unit test and submit for code review.)

On Mon, Feb 11, 2013 at 4:21 AM, Sa?l Ibarra Corretg? <saghul at gmail.com> wrote:
> Hi,
>
> While analyzing some code I came across the following situation:
>
> - User calls run_until_complete for some Task
> - The task itself calls loop.stop() at some point
> - run_until_complete raises TimeoutError
>
> Here is a very simple example: https://gist.github.com/saghul/4754117
>
> Something seems a bit off here. While one could argue that stopping the loop
> while run_until_complete is ongoing, getting a TimeoutError doesn't feel
> right. I think we should detect that the loop was stopped but the future is
> not done and raise something like NotCompleted or FutureNotCompleted.
>
> Thoughts?
>
> --
> Sa?l Ibarra Corretg?
> http://saghul.net/blog | http://about.me/saghul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



-- 
--Guido van Rossum (python.org/~guido)


From saghul at gmail.com  Mon Feb 11 19:18:18 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Mon, 11 Feb 2013 19:18:18 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
Message-ID: <511935EA.3040201@gmail.com>

Guido van Rossum wrote:
> Yeah, I think the base implementation of run_until_complete is wrong.
> I think I didn't have stop() when I coded that.
>
> Could you change it to something that just registers a lambda calling
> stop() on the Future, and see if that makes your test case behave
> better? (If so, please add a unit test and submit for code review.)
>

Hum, I guess I'm missing something, but futures don't have a stop() 
method. The code already calls loop.stop() when the future is done.

What I had in mind was to add a _called to Handler and expose it with a 
readonly property, so then we could check if handler.called and raise 
TimeoutError or not. As for raising another exception, maybe it's not 
such a good idea, since caller can always check future.done() later anyway.

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From guido at python.org  Mon Feb 11 19:26:58 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 10:26:58 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <511935EA.3040201@gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
Message-ID: <CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>

On Mon, Feb 11, 2013 at 10:18 AM, Sa?l Ibarra Corretg? <saghul at gmail.com>wrote:

> Guido van Rossum wrote:
>
>> Yeah, I think the base implementation of run_until_complete is wrong.
>> I think I didn't have stop() when I coded that.
>>
>> Could you change it to something that just registers a lambda calling
>> stop() on the Future, and see if that makes your test case behave
>> better? (If so, please add a unit test and submit for code review.)
>>
>>
> Hum, I guess I'm missing something, but futures don't have a stop()
> method. The code already calls loop.stop() when the future is done.
>

Oops, I read your post (and the code) too fast.


> What I had in mind was to add a _called to Handler and expose it with a
> readonly property, so then we could check if handler.called and raise
> TimeoutError or not. As for raising another exception, maybe it's not such
> a good idea, since caller can always check future.done() later anyway.
>

I'm not so keen on that.

But you were probably calling run_until_complete() without an explicit
timeout. In that case, it should not call run() but run_forever(), and it
should never raise TimeoutError.

You could still get a TimeoutError if a timeout was given; in that case, I
think you can fix this case by passing call_later() a helper function that
sets a nonlocal variable which you then inspect.

You could also use a different mechanism, e.g. call cancel() on a Future
when the timeout occurs. (But I think that might be less reliable, since I
think Tasks can catch cancellations.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130211/fdf7dced/attachment.html>

From saghul at gmail.com  Mon Feb 11 20:29:28 2013
From: saghul at gmail.com (=?UTF-8?Q?Sa=C3=BAl_Ibarra_Corretg=C3=A9?=)
Date: Mon, 11 Feb 2013 20:29:28 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
	<CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
Message-ID: <CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>

> But you were probably calling run_until_complete() without an explicit
> timeout. In that case, it should not call run() but run_forever(), and it
> should never raise TimeoutError.
>

Right, this was actually the case. I'll submit a code review addressing this.

> You could still get a TimeoutError if a timeout was given; in that case, I
> think you can fix this case by passing call_later() a helper function that
> sets a nonlocal variable which you then inspect.
>

Sure, but then what should we do: return like nothing has happened or
raise an exception so that the caller knows that the future is not
complete?

> You could also use a different mechanism, e.g. call cancel() on a Future
> when the timeout occurs. (But I think that might be less reliable, since I
> think Tasks can catch cancellations.)
>

Yeah, that would not work, in my example the task is already started,
I don't think cancel() will work midway.

-- 
/Sa?l
http://saghul.net | http://sipdoc.net


From guido at python.org  Mon Feb 11 20:37:39 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 11:37:39 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
	<CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
	<CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>
Message-ID: <CAP7+vJKb-iJuR4XC_VZtbqzWvM+B4WgN5pc1_-MxDCGeN7XPdQ@mail.gmail.com>

On Mon, Feb 11, 2013 at 11:29 AM, Sa?l Ibarra Corretg? <saghul at gmail.com>wrote:

> > But you were probably calling run_until_complete() without an explicit
> > timeout. In that case, it should not call run() but run_forever(), and it
> > should never raise TimeoutError.
> >
>
> Right, this was actually the case. I'll submit a code review addressing
> this.
>

Cool.


>  > You could still get a TimeoutError if a timeout was given; in that
> case, I
> > think you can fix this case by passing call_later() a helper function
> that
> > sets a nonlocal variable which you then inspect.
>
> Sure, but then what should we do: return like nothing has happened or
> raise an exception so that the caller knows that the future is not
> complete?
>

The same thing as when no timeout is specified. I presume this means
returning None. The alternative would be to call future.result() in all
cases -- it would raise InvalidStateError, but that doesn't feel very
useful. Perhaps this is actually the correct response? What is the reason
you have something else that calls stop()?


> > You could also use a different mechanism, e.g. call cancel() on a Future
> > when the timeout occurs. (But I think that might be less reliable, since
> I
> > think Tasks can catch cancellations.)
>
> Yeah, that would not work, in my example the task is already started,
> I don't think cancel() will work midway.
>

Actually, it may. There is code to throw CancelledError into the generator,
in _step(). But I don't think it will necessarily break out of all blocking
I/O. (Though it really should -- perhaps this is something useful to add
tests for.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130211/d526eaf8/attachment.html>

From timothy.c.delaney at gmail.com  Mon Feb 11 21:09:41 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Tue, 12 Feb 2013 07:09:41 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
Message-ID: <CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>

On 12 February 2013 00:28, Eli Bendersky <eliben at gmail.com> wrote:

> On Sun, Feb 10, 2013 at 2:43 PM, Tim Delaney <timothy.c.delaney at gmail.com>wrote:
>
>> On 4 February 2013 11:17, Tim Delaney <timothy.c.delaney at gmail.com>wrote:
>>
>>> On 4 February 2013 10:53, Jo?o Bernardo <jbvsmo at gmail.com> wrote:
>>>
>>>> Particularly It would be nice to have custom attributes and methods
>>>> besides the value and the name.
>>>>
>>>
>> I've also made it so that you can override the metaclass to return a
>> subclass of EnumValue.
>>
>> Can you elaborate on the utility of this feature? What realistic use
> cases do you see for it? I think that at this point it's important to weigh
> all benefits of features vs. implementation complexity, and there's
> absolutely no need to support every feature every other enum implementation
> has. I want to stress again that the most important characteristic of your
> implementation is the clean syntax which means that enums are so easy to
> define they don't really need special Python syntax and a library feature
> can do. However, there's a big leap from this to defining custom
> metaclasses for enums.
>

The custom metaclass is purely a mechanism to make it easy to use a custom
class for the enum value. It wold be possible to manually assign a custom
enum type to each enum, but then you wold lose the ability to just define
the names. By using a custom metaclass, you can have it automatically
assign the enum value type that you want.

My next email specifies a simplified syntax for specifying the custom
metaclass (you don't need to create one at all).

Supporting this functionality was actually very simple. However, I am
wondering though how useful this is without being able to specify
additional parameters for the enums. I've often used enums with quite
complex behaviour (e.g. in java) - the enum part is purely to have a unique
value assigned to each instance of the class and not repeating myself for
no good reason. I couldn't quite do the same with this as it currently is -
whilst I can have whatever behaviour I want, there's nothing to key it off
except the name and value which probably isn't enough. I've been trying to
think of a syntax which would work to pass additional parameters and I
can't think of anything cleaner than having a specialised class to pass the
additional parameters - but I need to somehow be able to specify the
optional integer value of the enum. Maybe have the first parameter be None?

class EnumParams():
    def __init__(self, value=None, *p, **kw):
        self.value = value
        ....

    def _resolve_proxies(self):
        # names would have been affected by EnumValues.__getattr__ - need
to resolve them from _EnumProxy to the actual values

...

class MyEnumValue(EnumValue):
    def __new__(cls, key, value, *p):
        e = super().__new__(cls, key, value)
        e.p = p

    def dump(self):
        print(self.p)

class MyEnum(Enum, metaclass=Enum.subtype(MyEnumValue)):
    A = EnumParams(None, 'extra', 'params')
    B = EnumParams(3, 'more', 'params')

Thoughts? Is the extra complexity worth it? The thing is, this doesn't take
away from the ability to specify the very simple clean enums - but it would
give the enums pretty much the full capabilities of java enums.

I'd like to have all these features available so that any PEP could
reference them and discuss the pros and cons (including how well they work
in practice).

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/19d42e31/attachment.html>

From timothy.c.delaney at gmail.com  Mon Feb 11 21:11:35 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Tue, 12 Feb 2013 07:11:35 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
Message-ID: <CAN8CLgkzbCE91Bvib3Ad0Q0R_k+1v54UG7m=FoNku5gHG54Veg@mail.gmail.com>

And I've just realised that my enums will be broken when assigning a
literal containing a name lookup e.g.

v = 1

class MyEnum(Enum):
    A, B
    other_attr = (v,)

other_attr will have an _EnumProxy instance. But I think I can do away with
the _EnumProxy entirely now that I'm using sys._getframe(). Let me try
something.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/676fb278/attachment.html>

From timothy.c.delaney at gmail.com  Mon Feb 11 21:22:03 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Tue, 12 Feb 2013 07:22:03 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgkzbCE91Bvib3Ad0Q0R_k+1v54UG7m=FoNku5gHG54Veg@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
	<CAN8CLgkzbCE91Bvib3Ad0Q0R_k+1v54UG7m=FoNku5gHG54Veg@mail.gmail.com>
Message-ID: <CAN8CLgmBKRkNxXZ3mw3OUTg10_L1R5Sm8Yv=TEyysOPm3B9N7g@mail.gmail.com>

On 12 February 2013 07:11, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> And I've just realised that my enums will be broken when assigning a
> literal containing a name lookup e.g.
>
> v = 1
>
> class MyEnum(Enum):
>     A, B
>     other_attr = (v,)
>
> other_attr will have an _EnumProxy instance. But I think I can do away
> with the _EnumProxy entirely now that I'm using sys._getframe(). Let me try
> something.
>

Too early in the morning. The above would avoid the whole _EnumProxy
because 'v' would be successfully looked up in the global namespace. But I
may still get rid of some instances of _EnumProxy.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/3b85ca11/attachment.html>

From saghul at gmail.com  Mon Feb 11 21:53:45 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Mon, 11 Feb 2013 21:53:45 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <CAP7+vJKb-iJuR4XC_VZtbqzWvM+B4WgN5pc1_-MxDCGeN7XPdQ@mail.gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
	<CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
	<CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>
	<CAP7+vJKb-iJuR4XC_VZtbqzWvM+B4WgN5pc1_-MxDCGeN7XPdQ@mail.gmail.com>
Message-ID: <51195A59.1020500@gmail.com>


>
> The same thing as when no timeout is specified. I presume this means
> returning None. The alternative would be to call future.result() in all
> cases -- it would raise InvalidStateError, but that doesn't feel very
> useful. Perhaps this is actually the correct response? What is the
> reason you have something else that calls stop()?
>

I didn't run into this in practice, I just thought of it :-) I guess 
that some exception handler or some third party lib could do it without 
you knowing, but I don't have a good example on why someone would want 
to do it.

>
>     Yeah, that would not work, in my example the task is already started,
>     I don't think cancel() will work midway.
>
> Actually, it may. There is code to throw CancelledError into the
> generator, in _step(). But I don't think it will necessarily break out
> of all blocking I/O. (Though it really should -- perhaps this is
> something useful to add tests for.)
>

Well, it may work for tasks, but since run_until_complete applies to 
futures in general I'm not sure if that would still work.

I added the following review which fixes the problem and adds a test 
case for it: https://codereview.appspot.com/7301076/ A few days ago I 
also submitted this one: https://codereview.appspot.com/7312046/ which 
adds a few more tests for run_until_complete.


Regards,

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From guido at python.org  Mon Feb 11 22:42:13 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 13:42:13 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <51195A59.1020500@gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
	<CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
	<CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>
	<CAP7+vJKb-iJuR4XC_VZtbqzWvM+B4WgN5pc1_-MxDCGeN7XPdQ@mail.gmail.com>
	<51195A59.1020500@gmail.com>
Message-ID: <CAP7+vJLY4g4HUhwKaDvh8qXNNoO47rj_9j=6Ko0GwsRER+GUEA@mail.gmail.com>

On Mon, Feb 11, 2013 at 12:53 PM, Sa?l Ibarra Corretg? <saghul at gmail.com>wrote:

>
>
>> The same thing as when no timeout is specified. I presume this means
>> returning None. The alternative would be to call future.result() in all
>> cases -- it would raise InvalidStateError, but that doesn't feel very
>> useful. Perhaps this is actually the correct response? What is the
>> reason you have something else that calls stop()?
>>
>>
> I didn't run into this in practice, I just thought of it :-) I guess that
> some exception handler or some third party lib could do it without you
> knowing, but I don't have a good example on why someone would want to do it.
>

I think there are actually two cases: where stop() is called from the task
given to run_until_complete(), or from another task. In the former case,
because the exception bubbles out of the task's coroutine, the task will be
marked as done (and its exception set to _StopError) -- at least I think
so. In the latter case, the task will still be runnable, and it will
continue to run when the event loop is started again. (Hm... maybe
run_forever() should be called start(), to match stop()?)

I think your unittest only tests the former case.

I'm not sure what run_until_complete() should return in the latter case. In
fact it makes me doubt the behavior of run_until_complete() in general.
Maybe it should return the Future (although that's kind of redundant)? Or
an enum indicating what happened? (Cases would be result, exception,
timeout, running -- and maybe cancelled?)


>
>
>>     Yeah, that would not work, in my example the task is already started,
>>     I don't think cancel() will work midway.
>>
>> Actually, it may. There is code to throw CancelledError into the
>> generator, in _step(). But I don't think it will necessarily break out
>> of all blocking I/O. (Though it really should -- perhaps this is
>> something useful to add tests for.)
>>
>>
> Well, it may work for tasks, but since run_until_complete applies to
> futures in general I'm not sure if that would still work.
>
> I added the following review which fixes the problem and adds a test case
> for it: https://codereview.appspot.com/7301076/ A few days ago I also
> submitted this one: https://codereview.appspot.com/7312046/ which adds a
> few more tests for run_until_complete.
>

Approved the latter. Gave some detailed feedback on the former; but see my
higher-order mumblings above...

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130211/baff6f83/attachment.html>

From saghul at gmail.com  Mon Feb 11 23:23:35 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Mon, 11 Feb 2013 23:23:35 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <CAP7+vJLY4g4HUhwKaDvh8qXNNoO47rj_9j=6Ko0GwsRER+GUEA@mail.gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
	<CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
	<CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>
	<CAP7+vJKb-iJuR4XC_VZtbqzWvM+B4WgN5pc1_-MxDCGeN7XPdQ@mail.gmail.com>
	<51195A59.1020500@gmail.com>
	<CAP7+vJLY4g4HUhwKaDvh8qXNNoO47rj_9j=6Ko0GwsRER+GUEA@mail.gmail.com>
Message-ID: <51196F67.6080609@gmail.com>


>
> I think there are actually two cases: where stop() is called from the
> task given to run_until_complete(), or from another task. In the former
> case, because the exception bubbles out of the task's coroutine, the
> task will be marked as done (and its exception set to _StopError) -- at
> least I think so. In the latter case, the task will still be runnable,
> and it will continue to run when the event loop is started again. (Hm...
> maybe run_forever() should be called start(), to match stop()?)
>
> I think your unittest only tests the former case.
>

Well, I think they are actually the same. In the gist I posted earlier I 
called loop.stop(), I didn't raise the exception inside the task, and 
loop.stop() will call call_soon so the actual raising will happen in the 
context on _run_once unless I'm mistaken.

Here is an updated simple test with it's output: 
https://gist.github.com/saghul/4758151

> I'm not sure what run_until_complete() should return in the latter case.
> In fact it makes me doubt the behavior of run_until_complete() in
> general. Maybe it should return the Future (although that's kind of
> redundant)? Or an enum indicating what happened? (Cases would be result,
> exception, timeout, running -- and maybe cancelled?)
>

I think the current behavior is ok, it the user really wants to check if 
the future was actually completed she can check with done(), if she 
saved the future in a variable, that is.

My other preferred approach would be to raise an exception, since I 
consider this an exceptional case: I told the loop to run until this 
future is completed so I expect that to happen. Since it didn't, 
NotCompletedError (or something alike) sounds good to me.


-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From guido at python.org  Mon Feb 11 23:31:34 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 14:31:34 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <51196F67.6080609@gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
	<CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
	<CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>
	<CAP7+vJKb-iJuR4XC_VZtbqzWvM+B4WgN5pc1_-MxDCGeN7XPdQ@mail.gmail.com>
	<51195A59.1020500@gmail.com>
	<CAP7+vJLY4g4HUhwKaDvh8qXNNoO47rj_9j=6Ko0GwsRER+GUEA@mail.gmail.com>
	<51196F67.6080609@gmail.com>
Message-ID: <CAP7+vJLNB2V=h2dCbv2dQKv4uonsiNrTABL_edN_bKq6PrdKOw@mail.gmail.com>

On Mon, Feb 11, 2013 at 2:23 PM, Sa?l Ibarra Corretg? <saghul at gmail.com>wrote:

>
>
>> I think there are actually two cases: where stop() is called from the
>> task given to run_until_complete(), or from another task. In the former
>> case, because the exception bubbles out of the task's coroutine, the
>> task will be marked as done (and its exception set to _StopError) -- at
>> least I think so. In the latter case, the task will still be runnable,
>> and it will continue to run when the event loop is started again. (Hm...
>> maybe run_forever() should be called start(), to match stop()?)
>>
>> I think your unittest only tests the former case.
>>
>>
> Well, I think they are actually the same. In the gist I posted earlier I
> called loop.stop(), I didn't raise the exception inside the task, and
> loop.stop() will call call_soon so the actual raising will happen in the
> context on _run_once unless I'm mistaken.
>

Oh, you're right! Not my day for reading carefully. :-)


> Here is an updated simple test with it's output:
> https://gist.github.com/saghul/4758151
>
>
>  I'm not sure what run_until_complete() should return in the latter case.
>> In fact it makes me doubt the behavior of run_until_complete() in
>> general. Maybe it should return the Future (although that's kind of
>> redundant)? Or an enum indicating what happened? (Cases would be result,
>> exception, timeout, running -- and maybe cancelled?)
>>
>>
> I think the current behavior is ok, it the user really wants to check if
> the future was actually completed she can check with done(), if she saved
> the future in a variable, that is.
>
> My other preferred approach would be to raise an exception, since I
> consider this an exceptional case: I told the loop to run until this future
> is completed so I expect that to happen. Since it didn't, NotCompletedError
> (or something alike) sounds good to me.
>

Good. How about calling future.result(), and letting it raise
InvalidStateError ? (Except if the timeout handler was called.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130211/df72edb6/attachment.html>

From saghul at gmail.com  Mon Feb 11 23:35:39 2013
From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=)
Date: Mon, 11 Feb 2013 23:35:39 +0100
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <CAP7+vJLNB2V=h2dCbv2dQKv4uonsiNrTABL_edN_bKq6PrdKOw@mail.gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
	<CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
	<CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>
	<CAP7+vJKb-iJuR4XC_VZtbqzWvM+B4WgN5pc1_-MxDCGeN7XPdQ@mail.gmail.com>
	<51195A59.1020500@gmail.com>
	<CAP7+vJLY4g4HUhwKaDvh8qXNNoO47rj_9j=6Ko0GwsRER+GUEA@mail.gmail.com>
	<51196F67.6080609@gmail.com>
	<CAP7+vJLNB2V=h2dCbv2dQKv4uonsiNrTABL_edN_bKq6PrdKOw@mail.gmail.com>
Message-ID: <5119723B.8060703@gmail.com>


>
> Good. How about calling future.result(), and letting it raise
> InvalidStateError ? (Except if the timeout handler was called.)
>

I did a quick test to see how it feels and I do like it. I'll update the 
review with this change plus the other suggested ones.

-- 
Sa?l Ibarra Corretg?
http://saghul.net/blog | http://about.me/saghul


From guido at python.org  Mon Feb 11 23:36:52 2013
From: guido at python.org (Guido van Rossum)
Date: Mon, 11 Feb 2013 14:36:52 -0800
Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in
	run_until_complete
In-Reply-To: <5119723B.8060703@gmail.com>
References: <5118E258.90705@gmail.com>
	<CAP7+vJK4wihRnzxSvzF9HKX+EmT_B_mv8Tzppq7H0SpZtz5vEg@mail.gmail.com>
	<511935EA.3040201@gmail.com>
	<CAP7+vJKqxF=SVfWao1_63SKG=ju2y4L596PK7WdBgcJM89qTQA@mail.gmail.com>
	<CANSmkSGWx4KxbsAAhyLsD1CXuS0ak5Z=vfgdOkKVbDyNug86zA@mail.gmail.com>
	<CAP7+vJKb-iJuR4XC_VZtbqzWvM+B4WgN5pc1_-MxDCGeN7XPdQ@mail.gmail.com>
	<51195A59.1020500@gmail.com>
	<CAP7+vJLY4g4HUhwKaDvh8qXNNoO47rj_9j=6Ko0GwsRER+GUEA@mail.gmail.com>
	<51196F67.6080609@gmail.com>
	<CAP7+vJLNB2V=h2dCbv2dQKv4uonsiNrTABL_edN_bKq6PrdKOw@mail.gmail.com>
	<5119723B.8060703@gmail.com>
Message-ID: <CAP7+vJ+X=N0TkV61Q06PbJp6P_FD+dTzGk9f1equqOewb06kmw@mail.gmail.com>

Great!


On Mon, Feb 11, 2013 at 2:35 PM, Sa?l Ibarra Corretg? <saghul at gmail.com>wrote:

>
>
>> Good. How about calling future.result(), and letting it raise
>> InvalidStateError ? (Except if the timeout handler was called.)
>>
>>
> I did a quick test to see how it feels and I do like it. I'll update the
> review with this change plus the other suggested ones.
>
>
> --
> Sa?l Ibarra Corretg?
> http://saghul.net/blog | http://about.me/saghul
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130211/6a278a78/attachment.html>

From timothy.c.delaney at gmail.com  Tue Feb 12 07:49:23 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Tue, 12 Feb 2013 17:49:23 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
Message-ID: <CAN8CLgmn3ikWQR57G54z_QMDpB5dFMcpVCJvsThBBqhE6BN5GA@mail.gmail.com>

On 12 February 2013 07:09, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

>
> Supporting this functionality was actually very simple. However, I am
> wondering though how useful this is without being able to specify
> additional parameters for the enums. I've often used enums with quite
> complex behaviour (e.g. in java) - the enum part is purely to have a unique
> value assigned to each instance of the class and not repeating myself for
> no good reason. I couldn't quite do the same with this as it currently is -
> whilst I can have whatever behaviour I want, there's nothing to key it off
> except the name and value which probably isn't enough. I've been trying to
> think of a syntax which would work to pass additional parameters and I
> can't think of anything cleaner than having a specialised class to pass the
> additional parameters - but I need to somehow be able to specify the
> optional integer value of the enum. Maybe have the first parameter be None?
>

OK - I've implemented this. EnumValue has grown 'args' and 'kwargs'
attributes.

Enums can be constructed like:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum, EnumParams, _
>>>
>>> class MyEnum(Enum):
...     A, B
...     C = EnumParams(None, 'i', 'j', k='k', l='l')
...     D = EnumParams(10, 'm', n='n')
...     E
...     F = EnumParams(None, 'o')
...     G = 20
...     H, I
...     J = EnumParams(None, p='p')
...
>>> print(repr(MyEnum))
<enum __main__.MyEnum {<EnumValue 'A': 0>, <EnumValue 'B': 1>, <EnumValue
'C': 2>, <EnumValue 'D': 10>, <EnumValue 'E': 11>, <EnumValue 'F': 12>,
<EnumValue 'G': 20>, <EnumValue 'H': 21>, <EnumValue 'I': 22>, <EnumValue
'J': 23>}>

_ is aliased to EnumParams so you can do:

>>> class MyEnum(Enum):...     A, B
...     C = _(None, 'i', 'j', k='k', l='l')
...     D = _(10, 'm', n='n')
...     E
...     F = _(None, 'o')
...     G = 20
...     H, I
...     J = _(None, p='p')
...
>>> print(repr(MyEnum))
<enum __main__.MyEnum {<EnumValue 'A': 0>, <EnumValue 'B': 1>, <EnumValue
'C': 2>, <EnumValue 'D': 10>, <EnumValue 'E': 11>, <EnumValue 'F': 12>,
<EnumValue 'G': 20>, <EnumValue 'H': 21>, <EnumValue 'I': 22>, <EnumValue
'J': 23>}>

which looks a bit cleaner to me, but might be a little confusing. Thoughts?

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/32a923cb/attachment.html>

From timothy.c.delaney at gmail.com  Tue Feb 12 09:06:38 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Tue, 12 Feb 2013 19:06:38 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgmn3ikWQR57G54z_QMDpB5dFMcpVCJvsThBBqhE6BN5GA@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
	<CAN8CLgmn3ikWQR57G54z_QMDpB5dFMcpVCJvsThBBqhE6BN5GA@mail.gmail.com>
Message-ID: <CAN8CLg=V7PCc4pedH=+evOf5BDKryakLH1VO24geVNx=rVmqQg@mail.gmail.com>

I'm evil. I just had a very bad thought, and I think I kinda like it.

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum, _
>>>
>>> class Color(Enum):
...     RED
...     GREEN = _(None, 'green')
...     BLUE = _(None, 'blue', hex='0000FF')
...     def __init__(self):
...         print('__init__', repr(self), self.args, self.kwargs)
...     def dump(self):
...         print(self, self.args, self.kwargs)
...
__init__ <EnumValue 'Color.RED': 0> () {}
__init__ <EnumValue 'Color.GREEN': 1> ('green',) {}
__init__ <EnumValue 'Color.BLUE': 2> ('blue',) {'hex': '0000FF'}

>>> print(repr(Color))
<enum __main__.Color {<EnumValue 'RED': 0>, <EnumValue 'GREEN': 1>,
<EnumValue 'BLUE': 2>}>
>>>
>>> for e in Color:
...     e.dump()
...     Color(e).dump()
...
Color.RED () {}
Color.RED () {}
Color.GREEN ('green',) {}
Color.GREEN ('green',) {}
Color.BLUE ('blue',) {'hex': '0000FF'}
Color.BLUE ('blue',) {'hex': '0000FF'}

When you request an attribute on the EnumValue that doesn't exist, it gets
it from the owning Enum class. If the attribute is an instance method (or
static method - they have the same type) then it gets turned into an
instance method of the EnumValue.

Also, when the EnumValue becomes owned by Enum, Enum.__init__ is called as
an EnumValue instance method (with no parameters).

This actually makes a kind of perverse sense. Conceptually the actual
EnumValues are the instances of the Enum (and in fact, I've made them
pretend to actually be so). I think it also entirely does away with the
need for subclasses of EnumValue and EnumMeta - you can do everything by
specifying attributes and behaviour in the Enum instance methods.

I've pushed this to https://bitbucket.org/magao/enum - have a play and see
how evil you think this is.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/347be251/attachment.html>

From eliben at gmail.com  Tue Feb 12 16:21:04 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 12 Feb 2013 07:21:04 -0800
Subject: [Python-ideas] PEP for enum library type?
Message-ID: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>

Hi all,

The ideas thrown around and Tim's prototype in the thread "constant/enum
type in stdlib" show that it's possible to use some metaclass magic to
implement very convenient syntax for enums, while only including them in
the stdlib (no language changes):

from enum import Enum

class Color(Enum):
  RED, BLUE, GREEN

I'm, for one, convinced that with this in hand we should *not* add special
syntax for enums, since the gains would be minimal over the above proposal.

While the parallel thread (and Tim's Bitbucket issue tracker) discusses his
proposed implementation, I think it may be worthwhile to start from the
other direction by writing a PEP that aims to include this in 3.4. The way
I see it, the PEP should discuss and help us settle upon a minimal set of
features deemed important in an enum.

If that sounds OK to people, then someone should write that PEP :-) This
could be Tim, or Barry who's been maintaining flufl.enum for a long time.
If no one steps up, I will gladly do it.

Thoughts?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/38674503/attachment.html>

From jsbueno at python.org.br  Tue Feb 12 16:30:20 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Tue, 12 Feb 2013 13:30:20 -0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
Message-ID: <CAH0mxTQKA6MA6kYc61MebAC+rP73z9aWh8Uo6fwd57NUgb72Mg@mail.gmail.com>

On 12 February 2013 13:21, Eli Bendersky <eliben at gmail.com> wrote:
> Hi all,
>
> The ideas thrown around and Tim's prototype in the thread "constant/enum
> type in stdlib" show that it's possible to use some metaclass magic to
> implement very convenient syntax for enums, while only including them in the
> stdlib (no language changes):
>
> from enum import Enum
>
> class Color(Enum):
>   RED, BLUE, GREEN
>
> I'm, for one, convinced that with this in hand we should *not* add special
> syntax for enums, since the gains would be minimal over the above proposal.
>
> While the parallel thread (and Tim's Bitbucket issue tracker) discusses his
> proposed implementation, I think it may be worthwhile to start from the
> other direction by writing a PEP that aims to include this in 3.4. The way I
> see it, the PEP should discuss and help us settle upon a minimal set of
> features deemed important in an enum.
>
> If that sounds OK to people, then someone should write that PEP :-) This
> could be Tim, or Barry who's been maintaining flufl.enum for a long time. If
> no one steps up, I will gladly do it.

I'd be fine with writing the PEP as well - maybe we could do it together?


>
> Thoughts?
>
> Eli
>
>


From solipsis at pitrou.net  Tue Feb 12 16:31:02 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 12 Feb 2013 16:31:02 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
Message-ID: <20130212163102.50dbf570@pitrou.net>

Le Tue, 12 Feb 2013 07:21:04 -0800,
Eli Bendersky <eliben at gmail.com> a ?crit :
> 
> from enum import Enum
> 
> class Color(Enum):
>   RED, BLUE, GREEN

I think I'd favour a namedtuple-style approach, e.g.:

class Color(Enum):
    fields = ('RED', 'BLUE', 'GREEN')

(which allows you to build enums programmatically too)

Also, I think you have to provide several classes, at least
SequenceEnum and StringEnum, and perhaps also BitmaskEnum.
It makes sense for them to be separate types, 1) because it makes it
explicit how the given enum behaves, 2) because combining integer and
str constants in an enum would be crazy.

Regards

Antoine.




From barry at python.org  Tue Feb 12 16:42:02 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 12 Feb 2013 10:42:02 -0500
Subject: [Python-ideas] constant/enum type in stdlib
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
	<CAN8CLgmn3ikWQR57G54z_QMDpB5dFMcpVCJvsThBBqhE6BN5GA@mail.gmail.com>
Message-ID: <20130212104202.550f5ae7@anarchist.wooz.org>

On Feb 12, 2013, at 05:49 PM, Tim Delaney wrote:

>_ is aliased to EnumParams so you can do:

_() is very commonly used for gettext translations, by long established
convention.  This is convention is inherited by Python from GNU gettext.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/5e4a93a3/attachment.pgp>

From ethan at stoneleaf.us  Tue Feb 12 16:31:42 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 07:31:42 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
Message-ID: <511A605E.3090104@stoneleaf.us>

On 02/12/2013 07:21 AM, Eli Bendersky wrote:
> While the parallel thread (and Tim's Bitbucket issue tracker) discusses
> his proposed implementation, I think it may be worthwhile to start from
> the other direction by writing a PEP that aims to include this in 3.4.
> The way I see it, the PEP should discuss and help us settle upon a
> minimal set of features deemed important in an enum.
>
> If that sounds OK to people, then someone should write that PEP :-) This
> could be Tim, or Barry who's been maintaining flufl.enum for a long
> time. If no one steps up, I will gladly do it.

A PEP would be excellent.  I tried searching for one a couple days ago 
to help my understanding with Tim's methods, but sadly didn't find one. :(

--
~Ethan~


From ethan at stoneleaf.us  Tue Feb 12 18:07:12 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 09:07:12 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212163102.50dbf570@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
Message-ID: <511A76C0.4070005@stoneleaf.us>

On 02/12/2013 07:31 AM, Antoine Pitrou wrote:
> Le Tue, 12 Feb 2013 07:21:04 -0800,
> Eli Bendersky <eliben at gmail.com> a ?crit :
>>
>> from enum import Enum
>>
>> class Color(Enum):
>>    RED, BLUE, GREEN
>
> I think I'd favour a namedtuple-style approach, e.g.:
>
> class Color(Enum):
>      fields = ('RED', 'BLUE', 'GREEN')
>
> (which allows you to build enums programmatically too)
>
> Also, I think you have to provide several classes, at least
> SequenceEnum and StringEnum, and perhaps also BitmaskEnum.
> It makes sense for them to be separate types, 1) because it makes it
> explicit how the given enum behaves, 2) because combining integer and
> str constants in an enum would be crazy.

I partially agree with Antoine; there should be three types of Enum 
available:

    - sequence (most common: 0, 1, 2, 3, etc.)
    - flag (1, 2, 4, 8, etc.)
    - unique ('on', 'off'; 'north', 'south', 'east', 'west')

and they shouldn't be mixed together higgledy-piggledy in the same class.

However, I like the simplicity of Tim's approach.

The result of the implementation I've been playing with looks something 
like:

class Color(Enum):
     type = 'sequence'
     RED, GREEN, BLUE

class Geometry(Enum):
     type = 'unique'
     LEFT, RIGHT, TOP, BOTTOM
     WEST, EAST, NORTH, SOUTH

class FieldType(Enum):
     type = 'flag'
     BINARY, AUTOINC, NULLABLE


The metaclass takes care of assigning the next value and selecting the 
proper subclass; 'unique' values are lower-cased (LEFT -> 'left'); 
subclassing is possible; and each Enum value is properly an instance of 
it's Enum class.

One important reason for having the different types of Enum is for 
prototyping.

--
~Ethan~


From jsbueno at python.org.br  Tue Feb 12 19:05:44 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Tue, 12 Feb 2013 16:05:44 -0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212163102.50dbf570@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
Message-ID: <CAH0mxTRH3nKn+=nhOqNZHaH+jX4SuwdLTfL8g+V-oMqCA9Cing@mail.gmail.com>

On 12 February 2013 13:31, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Tue, 12 Feb 2013 07:21:04 -0800,
> Eli Bendersky <eliben at gmail.com> a ?crit :
>>
>> from enum import Enum
>>
>> class Color(Enum):
>>   RED, BLUE, GREEN
>
> I think I'd favour a namedtuple-style approach, e.g.:
>
> class Color(Enum):
>     fields = ('RED', 'BLUE', 'GREEN')
>
> (which allows you to build enums programmatically too)

That is too much ' ' typing  - I think it would be ok, to have
it like that, but a helper function that would work just like
the namedtuple call:
Color = Enum("Color", "RED GREEN BLUE", int)

The magic of
class Color(Enum):
    RED, GREEN, BLUE

is tempting, but basically, what is required so that it does not break

class Color(Enum):
    RED = other_module.value

(without making "other_module" a new enum value :-)  ) if not worth it,
IMHO.

such magic is implemented in Tim's proof of concept.

A midle term could be to allow declarations like

class Color(IntEnum):
    RED, BLUE, GREEN

but with no other logic or expressions in the class body -- all names
showing up are strictly part of the sequence, and them have a constructor call,
with keyword arguments for all the other cases.







>
> Also, I think you have to provide several classes, at least
> SequenceEnum and StringEnum, and perhaps also BitmaskEnum.
> It makes sense for them to be separate types, 1) because it makes it
> explicit how the given enum behaves, 2) because combining integer and
> str constants in an enum would be crazy.

+1

One other important point, that is missed somewhere along Tim's implementation:
the  repr and str of the enum values should be the value's name - no need
to the fancy representation. Actually, the fancy representation defeats most
of the purpose of having the constants/enums

They could have a property like Enum._qualified_name if one needs it.



>
> Regards
>
> Antoine.
>
>


From guido at python.org  Tue Feb 12 19:08:44 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 10:08:44 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511A76C0.4070005@stoneleaf.us>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
Message-ID: <CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>

I'm torn. I like the clean look of Tim's:

class Flag(Enum):
   RED, WHITE, BLUE

However, I don't like the magic needed for its implementation -- and
anybody who thinks they know Python pretty well and thinks about this
syntax for more than two seconds has to wonder how on earth it's done
(and the answer is pretty horrible). It is also pretty brittle to
depend on the lookup order -- and I suspect it will be prone to
masking other bugs (any typo in a name used in class-level code will
essentially generate a bogus new enum value).

OTOH I don't like having the string quotes of Antoine's counter-proposal:

class Flag(Enum):
    fields = ('RED', 'WHITE', 'BLUE')

Whenever I see quoted names that are also used unquoted, I cringe a
little (and this includes calls to hasattr() or getattr() with a
string literal for the name).

The compromise that I would personally lean towards is more like this:

class Flag(Enum):
   RED, WHITE, BLUE = <something>

Even if the <something> requires you to know how many values you are
defining, like range(3). If we have to have a somewhat more verbose
syntax that doesn't bother me too much.

FWIW I do like being able to define enums that are ints and strings
(but not mixed); masks/flags I see more as a special case of ints (if
you use this you are likely trying to match some API defined in C or
C++). I also agree that it must be possible to customize the enum
class and the behavior of the values by defining methods on the class.

--
--Guido van Rossum (python.org/~guido)


From ethan at stoneleaf.us  Tue Feb 12 19:31:30 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 10:31:30 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
Message-ID: <511A8A82.4010602@stoneleaf.us>

On 02/12/2013 10:08 AM, Guido van Rossum wrote:
> I'm torn. I like the clean look of Tim's:
>
> class Flag(Enum):
>     RED, WHITE, BLUE
>
> However, I don't like the magic needed for its implementation -- and
> anybody who thinks they know Python pretty well and thinks about this
> syntax for more than two seconds has to wonder how on earth it's done
> (and the answer is pretty horrible). It is also pretty brittle to
> depend on the lookup order -- and I suspect it will be prone to
> masking other bugs (any typo in a name used in class-level code will
> essentially generate a bogus new enum value).

It seems to me the point of an enum is to give names to an order, so why 
would the lookup order be a problem?

> OTOH I don't like having the string quotes of Antoine's counter-proposal:
>
> class Flag(Enum):
>      fields = ('RED', 'WHITE', 'BLUE')
>
> The compromise that I would personally lean towards is more like this:
>
> class Flag(Enum):
>     RED, WHITE, BLUE = <something>
>
> Even if the <something> requires you to know how many values you are
> defining, like range(3). If we have to have a somewhat more verbose
> syntax that doesn't bother me too much.


Having just dealt with using a poor-person's Enum (THIS, THAT, THE_OTHER 
= range(3)) and then having to update it a few times (including all the 
places in the code that depended on the length), I can safely say that 
the range(n) construct is a PITA.

If you don't mind verbose, we could always go for something like:

class Color(Enum):
     start_defs()
     BLACK
     RED, GREEN, BLUE
     CYAN, YELLOW, MAGENTA
     end_defs()

and if we mandate that the enum names must come first we can even ditch 
the 'start_defs' call.

As far as typos go, I don't think that's a new problem (it certainly 
isn't for me, anyway ;) and my best defense is plenty of tests.

--
~Ethan~


From guido at python.org  Tue Feb 12 20:13:37 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 11:13:37 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511A8A82.4010602@stoneleaf.us>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
Message-ID: <CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>

On Tue, Feb 12, 2013 at 10:31 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 02/12/2013 10:08 AM, Guido van Rossum wrote:
>>
>> I'm torn. I like the clean look of Tim's:
>>
>> class Flag(Enum):
>>     RED, WHITE, BLUE
>>
>> However, I don't like the magic needed for its implementation -- and
>> anybody who thinks they know Python pretty well and thinks about this
>> syntax for more than two seconds has to wonder how on earth it's done
>> (and the answer is pretty horrible). It is also pretty brittle to
>> depend on the lookup order -- and I suspect it will be prone to
>> masking other bugs (any typo in a name used in class-level code will
>> essentially generate a bogus new enum value).
>
>
> It seems to me the point of an enum is to give names to an order, so why
> would the lookup order be a problem?

Because the language requirement that the expressions in a tuple are
evaluated left-to-right is fairly weak. (There are other, similar
contexts where the order is not l2r.)

>> OTOH I don't like having the string quotes of Antoine's counter-proposal:
>>
>> class Flag(Enum):
>>      fields = ('RED', 'WHITE', 'BLUE')
>>
>> The compromise that I would personally lean towards is more like this:
>>
>> class Flag(Enum):
>>     RED, WHITE, BLUE = <something>
>>
>> Even if the <something> requires you to know how many values you are
>> defining, like range(3). If we have to have a somewhat more verbose
>> syntax that doesn't bother me too much.
>
>
>
> Having just dealt with using a poor-person's Enum (THIS, THAT, THE_OTHER =
> range(3)) and then having to update it a few times (including all the places
> in the code that depended on the length), I can safely say that the range(n)
> construct is a PITA.
>
> If you don't mind verbose, we could always go for something like:
>
> class Color(Enum):
>     start_defs()
>     BLACK
>     RED, GREEN, BLUE
>     CYAN, YELLOW, MAGENTA
>     end_defs()
>
> and if we mandate that the enum names must come first we can even ditch the
> 'start_defs' call.

If that's where you're leaning I would much rather do this:

class Color(Enum):
  BLACK = val()
  RED = val()
  # etc.

EIBTI.

> As far as typos go, I don't think that's a new problem (it certainly isn't
> for me, anyway ;) and my best defense is plenty of tests.

So with Tim's implementation, what happens here:

class Color(Enum):
  RED, GREEN, BLUE
  if sys.platform == 'win32':
    MAGENTA

and you happen to have no "import sys" in your module? The sys lookup
will succeed, create a new enum, and then you will get an error
something like this:

AttributeError: 'Enum' object has no attribute 'platform'

--
--Guido van Rossum (python.org/~guido)


From barry at python.org  Tue Feb 12 20:37:48 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 12 Feb 2013 14:37:48 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
Message-ID: <20130212143748.23112336@anarchist.wooz.org>

On Feb 12, 2013, at 04:31 PM, Antoine Pitrou wrote:

>(which allows you to build enums programmatically too)

I prefer an alternative API for creating enums programmatically, e.g.:

http://pythonhosted.org/flufl.enum/docs/using.html#alternative-api

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/7dedbd82/attachment.pgp>

From barry at python.org  Tue Feb 12 20:45:18 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 12 Feb 2013 14:45:18 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
Message-ID: <20130212144518.3be491c5@anarchist.wooz.org>

On Feb 12, 2013, at 09:07 AM, Ethan Furman wrote:

>The result of the implementation I've been playing with looks something like:
>
>class Color(Enum):
>     type = 'sequence'
>     RED, GREEN, BLUE
>
>class Geometry(Enum):
>     type = 'unique'
>     LEFT, RIGHT, TOP, BOTTOM
>     WEST, EAST, NORTH, SOUTH
>
>class FieldType(Enum):
>     type = 'flag'
>     BINARY, AUTOINC, NULLABLE

IMHO, none of these approach the simplicity and explicitness of this API:

class Color(Enum):
    RED = 1
    GREEN = 2
    BLUE = 3

class FieldType(Enum):
    BINARY = 1
    AUTOINC = 2
    NULLABLE = 4

The savings in a few characters, does not (again IMHO) make up for all the
magic and implicitness you have to guess at with the top versions.

I also don't find much value in the 'unique' style, which seem like they're
just another way to name string constants.  Here though, you could argue that
the DRY principle at least makes them interesting, since IIUC, the explicit
alternative would be

class Geometry:
    LEFT = 'left'
    RIGHT = 'right'
    TOP = 'top'
    # ... and so on ...

I just wouldn't call these "enums" though. ;)  Again, being a little more
explicit seems like a win, or at least not so bad.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/40520683/attachment.pgp>

From barry at python.org  Tue Feb 12 20:50:36 2013
From: barry at python.org (Barry Warsaw)
Date: Tue, 12 Feb 2013 14:50:36 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<CAH0mxTRH3nKn+=nhOqNZHaH+jX4SuwdLTfL8g+V-oMqCA9Cing@mail.gmail.com>
Message-ID: <20130212145036.32a98928@anarchist.wooz.org>

On Feb 12, 2013, at 04:05 PM, Joao S. O. Bueno wrote:

>That is too much ' ' typing - I think it would be ok, to have it like that,
>but a helper function that would work just like the namedtuple call: Color =
>Enum("Color", "RED GREEN BLUE", int)

How about:

    >>> from flufl.enum import make
    >>> make('Colors', 'RED GREEN BLUE'.split())
    <Colors {RED: 1, GREEN: 2, BLUE: 3}>

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/ea444847/attachment.pgp>

From guido at python.org  Tue Feb 12 20:59:54 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 11:59:54 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
Message-ID: <CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>

After all the defenses I still don't like Tim's proposed syntax. Color me Barry.

On Tue, Feb 12, 2013 at 11:54 AM, Tim Delaney <tim.delaney at aptare.com> wrote:
> On 13 February 2013 06:13, Guido van Rossum <guido at python.org> wrote:
>>
>> So with Tim's implementation, what happens here:
>>
>> class Color(Enum):
>>   RED, GREEN, BLUE
>>   if sys.platform == 'win32':
>>     MAGENTA
>>
>> and you happen to have no "import sys" in your module? The sys lookup
>> will succeed, create a new enum, and then you will get an error
>> something like this:
>>
>> AttributeError: 'Enum' object has no attribute 'platform'
>
>
> Well, that particular case would work (you'd get a NameError: sys) due to
> having done an attribute lookup on sys, but the following:
>
> class Color(Enum):
>     RED, GREEN, BLUE
>     if platfor == 'win32':
>         MAGENTA
>
> would create an enum value 'platfor'.
>
> Personally, I don't think it's possible to have an enum with a simple
> interface and powerful semantics with just python code without it being
> fragile. I think I've got it fairly close, but there is horrible magic in
> there (multiple kinds) and there are definitely still edge cases. Any
> complete enum implementation is going to need some special handling by the
> parser I think.
>
> I'm actually thinking that to simplify things, I need a sentinel object to
> mark the end of the enum list (which allows other names after it). But that
> still wouldn't handle the case above (the if statement).
>
> BTW, for anyone who hasn't seen the magic code (largely uncommented, no
> explanations yet of how it's doing it - I probably won't get to that until
> the weekend) it's here: https://bitbucket.org/magao/enum
>
> Tim Delaney



--
--Guido van Rossum (python.org/~guido)


From ethan at stoneleaf.us  Tue Feb 12 20:26:28 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 11:26:28 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
Message-ID: <511A9764.2060902@stoneleaf.us>

On 02/12/2013 11:13 AM, Guido van Rossum wrote:
> On Tue, Feb 12, 2013 at 10:31 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 02/12/2013 10:08 AM, Guido van Rossum wrote:
>>>
>>> I'm torn. I like the clean look of Tim's:
>>>
>>> class Flag(Enum):
>>>      RED, WHITE, BLUE
>>>
>>> However, I don't like the magic needed for its implementation -- and
>>> anybody who thinks they know Python pretty well and thinks about this
>>> syntax for more than two seconds has to wonder how on earth it's done
>>> (and the answer is pretty horrible). It is also pretty brittle to
>>> depend on the lookup order -- and I suspect it will be prone to
>>> masking other bugs (any typo in a name used in class-level code will
>>> essentially generate a bogus new enum value).
>>
>>
>> It seems to me the point of an enum is to give names to an order, so why
>> would the lookup order be a problem?
>
> Because the language requirement that the expressions in a tuple are
> evaluated left-to-right is fairly weak. (There are other, similar
> contexts where the order is not l2r.)

Ah.


>> If you don't mind verbose, we could always go for something like:
>>
>> class Color(Enum):
>>      start_defs()
>>      BLACK
>>      RED, GREEN, BLUE
>>      CYAN, YELLOW, MAGENTA
>>      end_defs()
>>
>> and if we mandate that the enum names must come first we can even ditch the
>> 'start_defs' call.
>
> If that's where you're leaning I would much rather do this:
>
> class Color(Enum):
>    BLACK = val()
>    RED = val()
>    # etc.
>
> EIBTI.

Personally, I'd be fine with:

class Color(Enum):
    BLACK
    RED
    BLUE
    PURPLE

which avoids the l2r issue, and the repetitive use of val() (or whatever 
it's called).


>> As far as typos go, I don't think that's a new problem (it certainly isn't
>> for me, anyway ;) and my best defense is plenty of tests.
>
> So with Tim's implementation, what happens here:
>
> class Color(Enum):
>    RED, GREEN, BLUE
>    if sys.platform == 'win32':
>      MAGENTA
>
> and you happen to have no "import sys" in your module? The sys lookup
> will succeed, create a new enum, and then you will get an error
> something like this:
>
> AttributeError: 'Enum' object has no attribute 'platform'

We could minimize that issue by requiring that enum names be ALLCAPS -- 
if something comes through that isn't, don't supply a value and let the 
AttributeError perculate up.

--
~Ethan~


From timothy.c.delaney at gmail.com  Tue Feb 12 21:13:21 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 07:13:21 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
Message-ID: <CAN8CLgmNUHRPj0piWRz+2zCUX-NNT2AZgOiLm_0FF0qbXJo2Wg@mail.gmail.com>

Sorry to anyone who gets this twice - sent it from my work address which is
not subscribed to Python-Ideas.

On 13 February 2013 06:13, Guido van Rossum <guido at python.org> wrote:

> > As far as typos go, I don't think that's a new problem (it certainly
> isn't
> > for me, anyway ;) and my best defense is plenty of tests.
>
> So with Tim's implementation, what happens here:
>
> class Color(Enum):
>   RED, GREEN, BLUE
>   if sys.platform == 'win32':
>     MAGENTA
>
> and you happen to have no "import sys" in your module? The sys lookup
> will succeed, create a new enum, and then you will get an error
> something like this:
>
> AttributeError: 'Enum' object has no attribute 'platform'


Well, that particular case would work (you'd get a NameError: sys) due to
having done an attribute lookup on sys, but the following:

class Color(Enum):
    RED, GREEN, BLUE
    if platfor == 'win32':
        MAGENTA

would create an enum value 'platfor'.

Personally, I don't think it's possible to have an enum with a simple
interface and powerful semantics with just python code without it being
fragile. I think I've got it fairly close, but there is horrible magic in
there (multiple kinds) and there are definitely still edge cases. Any
complete enum implementation is going to need some special handling by the
parser I think.

I'm actually thinking that to simplify things, I need a sentinel object to
mark the end of the enum list (which allows other names after it). But that
still wouldn't handle the case above (the if statement).

BTW, for anyone who hasn't seen the magic code (largely uncommented, no
explanations yet of how it's doing it - I probably won't get to that until
the weekend) it's here: https://bitbucket.org/magao/enum

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/4d8d4e6f/attachment.html>

From timothy.c.delaney at gmail.com  Tue Feb 12 21:14:59 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 07:14:59 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <20130212104202.550f5ae7@anarchist.wooz.org>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
	<CAN8CLgmn3ikWQR57G54z_QMDpB5dFMcpVCJvsThBBqhE6BN5GA@mail.gmail.com>
	<20130212104202.550f5ae7@anarchist.wooz.org>
Message-ID: <CAN8CLgm-GbjVPPLGLvUwV4r06jjrXwu6ZO6kUe9SVyj7f4GsDw@mail.gmail.com>

On 13 February 2013 02:42, Barry Warsaw <barry at python.org> wrote:

> On Feb 12, 2013, at 05:49 PM, Tim Delaney wrote:
>
> >_ is aliased to EnumParams so you can do:
>
> _() is very commonly used for gettext translations, by long established
> convention.  This is convention is inherited by Python from GNU gettext.
>

Good point - it's actually what inspired me to use it, but a bad idea for
me to do so. I've taken that out.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/19f41122/attachment.html>

From ethan at stoneleaf.us  Tue Feb 12 20:58:30 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 11:58:30 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212144518.3be491c5@anarchist.wooz.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
Message-ID: <511A9EE6.5070803@stoneleaf.us>

On 02/12/2013 11:45 AM, Barry Warsaw wrote:
> On Feb 12, 2013, at 09:07 AM, Ethan Furman wrote:
>
>> The result of the implementation I've been playing with looks something like:
>>
>> class Color(Enum):
>>      type = 'sequence'
>>      RED, GREEN, BLUE
>>
>> class Geometry(Enum):
>>      type = 'unique'
>>      LEFT, RIGHT, TOP, BOTTOM
>>      WEST, EAST, NORTH, SOUTH
>>
>> class FieldType(Enum):
>>      type = 'flag'
>>      BINARY, AUTOINC, NULLABLE
>
> IMHO, none of these approach the simplicity and explicitness of this API:
>
> class Color(Enum):
>      RED = 1
>      GREEN = 2
>      BLUE = 3
>
> class FieldType(Enum):
>      BINARY = 1
>      AUTOINC = 2
>      NULLABLE = 4
>
> The savings in a few characters, does not (again IMHO) make up for all the
> magic and implicitness you have to guess at with the top versions.

0)  Once you read the docs you don't have to guess  ;)
1)  Magic is fun.  :)
2)  I hate repetition  (= 1  = 2  = 3 is repetitive)
3)  I make mistakes when repeating stuff (... = 11   = 11  = 13)


> I also don't find much value in the 'unique' style, which seem like they're
> just another way to name string constants.  Here though, you could argue that
> the DRY principle at least makes them interesting, since IIUC, the explicit
> alternative would be
>
> class Geometry:
>      LEFT = 'left'
>      RIGHT = 'right'
>      TOP = 'top'
>      # ... and so on ...

Class membership can also be an advantage.

--
~Ethan~


From jsbueno at python.org.br  Tue Feb 12 21:33:13 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Tue, 12 Feb 2013 18:33:13 -0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212145036.32a98928@anarchist.wooz.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<CAH0mxTRH3nKn+=nhOqNZHaH+jX4SuwdLTfL8g+V-oMqCA9Cing@mail.gmail.com>
	<20130212145036.32a98928@anarchist.wooz.org>
Message-ID: <CAH0mxTTwPRPY5MTTYMZZR-uSXBp0vY=MNAqV2WQ_6zCpgHMeMQ@mail.gmail.com>

On 12 February 2013 17:50, Barry Warsaw <barry at python.org> wrote:
>>That is too much ' ' typing - I think it would be ok, to have it like that,
>>but a helper function that would work just like the namedtuple call: Color =
>>Enum("Color", "RED GREEN BLUE", int)
>
> How about:
>
>     >>> from flufl.enum import make
>     >>> make('Colors', 'RED GREEN BLUE'.split())
>     <Colors {RED: 1, GREEN: 2, BLUE: 3}>


Not bad - I think that wahtever is agreeded in a nice api that takes
advantage of the
"class" statement we should have a callabale constructor that takes
names as strings.

And moreover, it is nice to have a way to load the created
enums/constants to the current namespace -
and it can be done, without "magic" to the global namespace having a
call that takes a "namespace" argument
to which one would normally pass in blobals()

So:
from enum import load, make

would allow for:

>>> Colors = make("Colors", "RED GREEN BLUE".split())
>>> load(Colors, globals())
>>> RED
RED


(I think that allowing a call without the "split()" would be better - )
That would make some *linters complain - but those should update
themselves to the fact in time.

I consider a way to load the created constants  into the module namespace
very important to the concept of enums/constant itself.  Agreeing on a call
that can get "globals()" and update it would preclude the idea of being able
to do "from MyEnumClass import *" that showed up on the previous thread.

  js
 -><-


From ubershmekel at gmail.com  Tue Feb 12 21:38:21 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Tue, 12 Feb 2013 22:38:21 +0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511A9EE6.5070803@stoneleaf.us>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511A9EE6.5070803@stoneleaf.us>
Message-ID: <CANSw7Ky44ykS+ywrYvS6ynR3t0GWbfxjAqu-9caSP2+o4F8esw@mail.gmail.com>

On Tue, Feb 12, 2013 at 9:58 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> 1)  Magic is fun.  :)
>

I heartily agree. Check this out:

>>> RED, GREEN, BLUE = enum()
>>> print(RED, GREEN, BLUE)
0 1 2
>>> LEFT, TOP, RIGHT, DOWN = enum()
>>> print(LEFT, TOP, RIGHT, DOWN)
0 1 2 3


And here's the dirty trick....


import inspect
def enum():
    """If you use this  your computer will most likely burst ablaze and
your teeth will fall off"""
    frame = inspect.stack()[1][0]
    # 4 = 3 for the function call "enum()", 1 for the opcode
"UNPACK_SEQUENCE",
    # the next two bytes represent how many things we need to unpack.
    a, b = frame.f_code.co_code[frame.f_lasti + 4:frame.f_lasti + 6]
    enum_len = a + b * 256
    return range(enum_len)




Cheers,


Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/a78cbdaf/attachment.html>

From joshua.landau.ws at gmail.com  Tue Feb 12 21:43:50 2013
From: joshua.landau.ws at gmail.com (Joshua Landau)
Date: Tue, 12 Feb 2013 20:43:50 +0000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511A9764.2060902@stoneleaf.us>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<511A9764.2060902@stoneleaf.us>
Message-ID: <CAN1F8qVS-whJDUJNWz4BrmkciNu4sqh1T=B8iTuV7dDHoASWLQ@mail.gmail.com>

On 12 February 2013 19:26, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 02/12/2013 11:13 AM, Guido van Rossum wrote:
>
>> On Tue, Feb 12, 2013 at 10:31 AM, Ethan Furman <ethan at stoneleaf.us>
>> wrote:
>>
>>> On 02/12/2013 10:08 AM, Guido van Rossum wrote:
>>>
>>>>
>>>> I'm torn. I like the clean look of Tim's:
>>>>
>>>> class Flag(Enum):
>>>>      RED, WHITE, BLUE
>>>>
>>>> However, I don't like the magic needed for its implementation -- and
>>>> anybody who thinks they know Python pretty well and thinks about this
>>>> syntax for more than two seconds has to wonder how on earth it's done
>>>> (and the answer is pretty horrible). It is also pretty brittle to
>>>> depend on the lookup order -- and I suspect it will be prone to
>>>> masking other bugs (any typo in a name used in class-level code will
>>>> essentially generate a bogus new enum value).
>>>>
>>>
>>>
>>> It seems to me the point of an enum is to give names to an order, so why
>>> would the lookup order be a problem?
>>>
>>
>> Because the language requirement that the expressions in a tuple are
>> evaluated left-to-right is fairly weak. (There are other, similar
>> contexts where the order is not l2r.)
>
> <snip>

> Personally, I'd be fine with:
>
> class Color(Enum):
>    BLACK
>    RED
>    BLUE
>    PURPLE
>
> which avoids the l2r issue, and the repetitive use of val() (or whatever
> it's called).


Can you force people not to use:

class Color(Enum):
   BLACK, RED, BLUE, PURPLE

?
If I'm understanding the implementation, if you support one then you
support the other. Hence the problem will still stand. That said, I'm not
personally too worried about it.

 As far as typos go, I don't think that's a new problem (it certainly isn't
>>> for me, anyway ;) and my best defense is plenty of tests.
>>>
>>
>> So with Tim's implementation, what happens here:
>>
>> class Color(Enum):
>>    RED, GREEN, BLUE
>>    if sys.platform == 'win32':
>>      MAGENTA
>>
>> and you happen to have no "import sys" in your module? The sys lookup
>> will succeed, create a new enum, and then you will get an error
>> something like this:
>>
>> AttributeError: 'Enum' object has no attribute 'platform'
>>
>
> We could minimize that issue by requiring that enum names be ALLCAPS -- if
> something comes through that isn't, don't supply a value and let the
> AttributeError perculate up.
>

I think it's worse than that. I'm using Tim Delaney's code and I'm not
really sure why it's doing what it's doing.

import sys

class Color(Enum):
   RED, GREEN, BLUE
   if sys.platform:
     MAGENTA

works and doesn't create Color.sys.

import sys

class Color(Enum):
   RED, GREEN, BLUE
   if sys:
     MAGENTA

works and _does_ create Color.sys.

So somehow there's enough magic to realise that if you are accessing an
attribute, it's not going to be an enum. In other words, you don't need the
ALLCAPS anyway. You might think "oh, good!". I think "oh noes!". It's way
too magic for me.

The other idea:

class Color(Enum):
    values = "RED", "GREEN", "BLUE", "MAGENTA", "OTHER", "COLORS", "HERE",
"TOO"

could be written

class Color(Enum):
    values = "RED GREEN BLUE MAGENTA OTHER COLORS HERE TOO".split()

so I'm OK with that, but then you're in the territory when you might as
well go with the method Barry suggested.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/155b2fd7/attachment.html>

From eliben at gmail.com  Tue Feb 12 21:58:44 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 12 Feb 2013 12:58:44 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
Message-ID: <CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>

On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum <guido at python.org> wrote:

> After all the defenses I still don't like Tim's proposed syntax. Color me
> Barry.
>

In this case, I see no reason not to use Barry's flufl.enum - it's pretty
good and has been around for a while. The main advantage of Tim's
implementation syntax (wasn't it Michael Foord who originally proposed it?)
is that it lets us do less typing which is great. IOW, sure with flufl.enum
I can do this and be reasonably happy:

class Color(Enum):
  RED = 1
  BLUE = 2
  GREEN = 3

But in 99% of the cases in Python code I absolutely don't care about the
values of these enumerations (*), so I'd much, *much* rather write:

class Color(Enum):
  RED, BLUE, GREEN

Yes, the implementation is messy, but it can be documented and tested well,
and it's fairly small and confined. But this is a matter of preference, and
I can see your point (Guido) about preferring the more explicit syntax in
order to avoid too much magic.

Eli

(*) And where I do care, I'd still enjoy them being generated automatically.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/5049f7d3/attachment.html>

From solipsis at pitrou.net  Tue Feb 12 22:04:12 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 12 Feb 2013 22:04:12 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
Message-ID: <20130212220412.4bd3e604@pitrou.net>

On Tue, 12 Feb 2013 12:58:44 -0800
Eli Bendersky <eliben at gmail.com> wrote:

> On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum <guido at python.org> wrote:
> 
> > After all the defenses I still don't like Tim's proposed syntax. Color me
> > Barry.
> >
> 
> In this case, I see no reason not to use Barry's flufl.enum - it's pretty
> good and has been around for a while. The main advantage of Tim's
> implementation syntax (wasn't it Michael Foord who originally proposed it?)
> is that it lets us do less typing which is great. IOW, sure with flufl.enum
> I can do this and be reasonably happy:
> 
> class Color(Enum):
>   RED = 1
>   BLUE = 2
>   GREEN = 3

I still hope enum values are strings by default - or, if not, that they
get nice reprs. Integer enums are only useful for interoperability with
stubborn low-level libraries :-)

Regards

Antoine.




From timothy.c.delaney at gmail.com  Tue Feb 12 22:14:51 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 08:14:51 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
Message-ID: <CAN8CLgkbfTUEvY23RwnxX5NU9pNXQux7dqbxaMdsNRxoWLTh2w@mail.gmail.com>

On 13 February 2013 07:58, Eli Bendersky <eliben at gmail.com> wrote:

> On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum <guido at python.org>wrote:
>
>> After all the defenses I still don't like Tim's proposed syntax. Color me
>> Barry.
>>
>
>  The main advantage of Tim's implementation syntax (wasn't it Michael
> Foord who originally proposed it?)
>

Indeed it was and I don't want to take any of the "credit" away from
Michael. I simply took his original inspired idea and added several extra
layers of magic on top ;) I've updated my README to make that clear.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/42f1c5bc/attachment.html>

From drekin at gmail.com  Tue Feb 12 22:19:13 2013
From: drekin at gmail.com (drekin at gmail.com)
Date: Tue, 12 Feb 2013 13:19:13 -0800 (PST)
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
Message-ID: <511ab1d1.83b80e0a.01e6.057d@mx.google.com>

Hello.

> masks/flags I see more as a special case of ints (if you use this you are likely trying to match some API defined in C or C++).

Comfortably encoding flags as int is just a half of FlagsEnum functionality. It has also abstract set semantics, see http://mail.python.org/pipermail/python-ideas/2013-January/019150.html .

Regards, Drekin



From timothy.c.delaney at gmail.com  Tue Feb 12 22:19:41 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 08:19:41 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
Message-ID: <CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>

On 13 February 2013 07:58, Eli Bendersky <eliben at gmail.com> wrote:

>
> But in 99% of the cases in Python code I absolutely don't care about the
> values of these enumerations (*), so I'd much, *much* rather write:
>
> class Color(Enum):
>   RED, BLUE, GREEN
>
> Yes, the implementation is messy, but it can be documented and tested
> well, and it's fairly small and confined. But this is a matter of
> preference, and I can see your point (Guido) about preferring the more
> explicit syntax in order to avoid too much magic.
>
> Eli
>
> (*) And where I do care, I'd still enjoy them being generated
> automatically.
>

I'm of the opinion that the most fundamental property of an enum is that it
can generate its values automatically. Everything else is a nice-to-have,
but without that property it's not much use as an enum.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/23a58b82/attachment.html>

From guido at python.org  Tue Feb 12 22:29:17 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 13:29:17 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
Message-ID: <CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>

Please. Anything *except*

class Color(Enum):
  RED, BLUE, GREEN

And not this eiter:

class Color(Enum):
  RED
  BLUE
  GREEN

On Tue, Feb 12, 2013 at 12:58 PM, Eli Bendersky <eliben at gmail.com> wrote:
> On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum <guido at python.org> wrote:
>>
>> After all the defenses I still don't like Tim's proposed syntax. Color me
>> Barry.
>
>
> In this case, I see no reason not to use Barry's flufl.enum - it's pretty
> good and has been around for a while. The main advantage of Tim's
> implementation syntax (wasn't it Michael Foord who originally proposed it?)
> is that it lets us do less typing which is great. IOW, sure with flufl.enum
> I can do this and be reasonably happy:
>
>
> class Color(Enum):
>   RED = 1
>   BLUE = 2
>   GREEN = 3
>
> But in 99% of the cases in Python code I absolutely don't care about the
> values of these enumerations (*), so I'd much, *much* rather write:
>
> class Color(Enum):
>   RED, BLUE, GREEN
>
> Yes, the implementation is messy, but it can be documented and tested well,
> and it's fairly small and confined. But this is a matter of preference, and
> I can see your point (Guido) about preferring the more explicit syntax in
> order to avoid too much magic.
>
> Eli
>
> (*) And where I do care, I'd still enjoy them being generated automatically.
>
>



--
--Guido van Rossum (python.org/~guido)


From rosuav at gmail.com  Tue Feb 12 22:36:11 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 13 Feb 2013 08:36:11 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLg=V7PCc4pedH=+evOf5BDKryakLH1VO24geVNx=rVmqQg@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
	<CAN8CLgmn3ikWQR57G54z_QMDpB5dFMcpVCJvsThBBqhE6BN5GA@mail.gmail.com>
	<CAN8CLg=V7PCc4pedH=+evOf5BDKryakLH1VO24geVNx=rVmqQg@mail.gmail.com>
Message-ID: <CAPTjJmozAB99EO0SBtqG0QzBevex4RknFGTtQ8VeZ-=RMdSrvQ@mail.gmail.com>

On Tue, Feb 12, 2013 at 7:06 PM, Tim Delaney
<timothy.c.delaney at gmail.com> wrote:
> This actually makes a kind of perverse sense. Conceptually the actual
> EnumValues are the instances of the Enum (and in fact, I've made them
> pretend to actually be so).

Not so perverse. I think that makes very good sense (but then, I know
C++ enums, so maybe I'm tainted). Looks good, though I've not actually
played with the code.

ChrisA


From stefan at bytereef.org  Tue Feb 12 22:46:55 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 12 Feb 2013 22:46:55 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
Message-ID: <20130212214655.GA20475@sleipnir.bytereef.org>

Guido van Rossum <guido at python.org> wrote:
> Please. Anything *except*
> 
> class Color(Enum):
>   RED, BLUE, GREEN
> 
> And not this eiter:
> 
> class Color(Enum):
>   RED
>   BLUE
>   GREEN

Is a new enum keyword still a possibility? To me that seems to be the
cleanest way.


Stefan Krah




From ryan at ryanhiebert.com  Tue Feb 12 22:55:06 2013
From: ryan at ryanhiebert.com (ryan at ryanhiebert.com)
Date: Tue, 12 Feb 2013 13:55:06 -0800
Subject: [Python-ideas] enums vs symbols
In-Reply-To: <20130212214655.GA20475@sleipnir.bytereef.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
	<20130212214655.GA20475@sleipnir.bytereef.org>
Message-ID: <28F639F9-CF62-4AF7-BB80-9CFEEA2F385C@ryanhiebert.com>

Forgive me, a lurker, but I wonder: how does an enum differ in usage from something like a Symbol from ruby, other than that an enum is required to be declared beforehand?

I'm definitely a python guy, but the idea of identifiers that are their own value interests me, and seems like it could have some overlap with the concept and usage of enums.

If there's not much difference in usage between enums and symbols apart from declaration, perhaps we could make the declaration optional and combine the concepts? This would certainly require parser changes.

Ryan



From guido at python.org  Tue Feb 12 22:58:04 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 13:58:04 -0800
Subject: [Python-ideas] enums vs symbols
In-Reply-To: <28F639F9-CF62-4AF7-BB80-9CFEEA2F385C@ryanhiebert.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
	<20130212214655.GA20475@sleipnir.bytereef.org>
	<28F639F9-CF62-4AF7-BB80-9CFEEA2F385C@ryanhiebert.com>
Message-ID: <CAP7+vJ+fVEQxqDixwKfjkz1TcdFP2YOwgjtVu+yf4ut21pNkww@mail.gmail.com>

I don't know Ruby symbols, but if they are anything like Lisp symbols,
they are completely different. Lisp symbols are interned strings --
Python just uses strings (and if they happen to be interned you get a
modest speedup). This is for things like:

if hasattr(x, 'foo'):
    x.foo()

On Tue, Feb 12, 2013 at 1:55 PM,  <ryan at ryanhiebert.com> wrote:
> Forgive me, a lurker, but I wonder: how does an enum differ in usage from something like a Symbol from ruby, other than that an enum is required to be declared beforehand?
>
> I'm definitely a python guy, but the idea of identifiers that are their own value interests me, and seems like it could have some overlap with the concept and usage of enums.
>
> If there's not much difference in usage between enums and symbols apart from declaration, perhaps we could make the declaration optional and combine the concepts? This would certainly require parser changes.
>
> Ryan
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



--
--Guido van Rossum (python.org/~guido)


From guido at python.org  Tue Feb 12 22:58:49 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 13:58:49 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212220412.4bd3e604@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<20130212220412.4bd3e604@pitrou.net>
Message-ID: <CAP7+vJ+dp2YSSCRdTCvk-NUmWHOrFfqmWthU9VGeTKRK5u=sKw@mail.gmail.com>

Definitely they need nice reprs.

On Tue, Feb 12, 2013 at 1:04 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 12 Feb 2013 12:58:44 -0800
> Eli Bendersky <eliben at gmail.com> wrote:
>
>> On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum <guido at python.org> wrote:
>>
>> > After all the defenses I still don't like Tim's proposed syntax. Color me
>> > Barry.
>> >
>>
>> In this case, I see no reason not to use Barry's flufl.enum - it's pretty
>> good and has been around for a while. The main advantage of Tim's
>> implementation syntax (wasn't it Michael Foord who originally proposed it?)
>> is that it lets us do less typing which is great. IOW, sure with flufl.enum
>> I can do this and be reasonably happy:
>>
>> class Color(Enum):
>>   RED = 1
>>   BLUE = 2
>>   GREEN = 3
>
> I still hope enum values are strings by default - or, if not, that they
> get nice reprs. Integer enums are only useful for interoperability with
> stubborn low-level libraries :-)
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



--
--Guido van Rossum (python.org/~guido)


From guido at python.org  Tue Feb 12 22:59:44 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 13:59:44 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
Message-ID: <CAP7+vJJ9JMA38Gye8i_aeFuiYtjpbdG9DRaQZaxqKA=axY0bfQ@mail.gmail.com>

And I'm of the opinion that the most fundamental property of an enum
is that it doesn't print as an int.

On Tue, Feb 12, 2013 at 1:19 PM, Tim Delaney
<timothy.c.delaney at gmail.com> wrote:
> On 13 February 2013 07:58, Eli Bendersky <eliben at gmail.com> wrote:
>>
>>
>> But in 99% of the cases in Python code I absolutely don't care about the
>> values of these enumerations (*), so I'd much, *much* rather write:
>>
>> class Color(Enum):
>>   RED, BLUE, GREEN
>>
>> Yes, the implementation is messy, but it can be documented and tested
>> well, and it's fairly small and confined. But this is a matter of
>> preference, and I can see your point (Guido) about preferring the more
>> explicit syntax in order to avoid too much magic.
>>
>> Eli
>>
>> (*) And where I do care, I'd still enjoy them being generated
>> automatically.
>
>
> I'm of the opinion that the most fundamental property of an enum is that it
> can generate its values automatically. Everything else is a nice-to-have,
> but without that property it's not much use as an enum.
>
> Tim Delaney



--
--Guido van Rossum (python.org/~guido)


From timothy.c.delaney at gmail.com  Tue Feb 12 23:09:04 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 09:09:04 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJJ9JMA38Gye8i_aeFuiYtjpbdG9DRaQZaxqKA=axY0bfQ@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
	<CAP7+vJJ9JMA38Gye8i_aeFuiYtjpbdG9DRaQZaxqKA=axY0bfQ@mail.gmail.com>
Message-ID: <CAN8CLgnzPfuiEFe-BpV3n2Ar7P6Et98fGkPdWA52WTRwqKzPUw@mail.gmail.com>

On 13 February 2013 08:59, Guido van Rossum <guido at python.org> wrote:

> And I'm of the opinion that the most fundamental property of an enum
> is that it doesn't print as an int.


That part is easy.

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum
>>> class Color(Enum):
...     RED
...
>>> str(Color.RED)
'Color.RED'
>>> repr(Color.RED)
"<EnumValue 'Color.RED': 0>"
>>> str(Color)
'{RED:0}'
>>> repr(Color)
"<enum __main__.Color {<EnumValue 'RED': 0>}>"

Making an appropriate str/repr just requires deciding what it should be. I
chose to make the reprs give more information, but there's no reason they
couldn't be simplified.

I'll see if I can come up with a syntax for declaration that we're both
happy with, but I doubt I'll be as happy with it as what I've currently got
(despite its edge cases).

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/76c44a48/attachment.html>

From ryan at ryanhiebert.com  Tue Feb 12 23:13:53 2013
From: ryan at ryanhiebert.com (ryan at ryanhiebert.com)
Date: Tue, 12 Feb 2013 14:13:53 -0800
Subject: [Python-ideas] enums vs symbols
In-Reply-To: <CAP7+vJ+fVEQxqDixwKfjkz1TcdFP2YOwgjtVu+yf4ut21pNkww@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
	<20130212214655.GA20475@sleipnir.bytereef.org>
	<28F639F9-CF62-4AF7-BB80-9CFEEA2F385C@ryanhiebert.com>
	<CAP7+vJ+fVEQxqDixwKfjkz1TcdFP2YOwgjtVu+yf4ut21pNkww@mail.gmail.com>
Message-ID: <0233EDDB-8034-42D6-9052-505E16D878E4@ryanhiebert.com>

Thanks for that. I think that when I consider the cases that I don't like the idea of using strings, using a proper enum makes better sense anyway.

On Feb 12, 2013, at 1:58 PM, Guido van Rossum <guido at python.org> wrote:

> I don't know Ruby symbols, but if they are anything like Lisp symbols,
> they are completely different. Lisp symbols are interned strings --
> Python just uses strings (and if they happen to be interned you get a
> modest speedup). This is for things like:
> 
> if hasattr(x, 'foo'):
>    x.foo()
> 
> On Tue, Feb 12, 2013 at 1:55 PM,  <ryan at ryanhiebert.com> wrote:
>> Forgive me, a lurker, but I wonder: how does an enum differ in usage from something like a Symbol from ruby, other than that an enum is required to be declared beforehand?
>> 
>> I'm definitely a python guy, but the idea of identifiers that are their own value interests me, and seems like it could have some overlap with the concept and usage of enums.
>> 
>> If there's not much difference in usage between enums and symbols apart from declaration, perhaps we could make the declaration optional and combine the concepts? This would certainly require parser changes.
>> 
>> Ryan
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> 
> 
> 
> --
> --Guido van Rossum (python.org/~guido)



From eliben at gmail.com  Tue Feb 12 23:13:57 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 12 Feb 2013 14:13:57 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212220412.4bd3e604@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<20130212220412.4bd3e604@pitrou.net>
Message-ID: <CAF-Rda-AiL0KQhSF3RoBAKdb8jDeAu4FuFxvyODO7h6S2wY93g@mail.gmail.com>

On Tue, Feb 12, 2013 at 1:04 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Tue, 12 Feb 2013 12:58:44 -0800
> Eli Bendersky <eliben at gmail.com> wrote:
>
> > On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum <guido at python.org>
> wrote:
> >
> > > After all the defenses I still don't like Tim's proposed syntax. Color
> me
> > > Barry.
> > >
> >
> > In this case, I see no reason not to use Barry's flufl.enum - it's pretty
> > good and has been around for a while. The main advantage of Tim's
> > implementation syntax (wasn't it Michael Foord who originally proposed
> it?)
> > is that it lets us do less typing which is great. IOW, sure with
> flufl.enum
> > I can do this and be reasonably happy:
> >
> > class Color(Enum):
> >   RED = 1
> >   BLUE = 2
> >   GREEN = 3
>
> I still hope enum values are strings by default - or, if not, that they
> get nice reprs. Integer enums are only useful for interoperability with
> stubborn low-level libraries :-)
>

I agree, and Tim's implementation provides a very nice string
representation, no matter what the value underneath is. I don't really care
what the value underneath is, actually.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/d9c0cfaa/attachment.html>

From ethan at stoneleaf.us  Tue Feb 12 23:14:22 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 14:14:22 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CANSw7Ky44ykS+ywrYvS6ynR3t0GWbfxjAqu-9caSP2+o4F8esw@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511A9EE6.5070803@stoneleaf.us>
	<CANSw7Ky44ykS+ywrYvS6ynR3t0GWbfxjAqu-9caSP2+o4F8esw@mail.gmail.com>
Message-ID: <511ABEBE.90201@stoneleaf.us>

On 02/12/2013 12:38 PM, Yuval Greenfield wrote:
> On Tue, Feb 12, 2013 at 9:58 PM, Ethan Furman <ethan at stoneleaf.us
> <mailto:ethan at stoneleaf.us>> wrote:
>
>     1)  Magic is fun.  :)
>
>
> I heartily agree. Check this out:
>
>  >>> RED, GREEN, BLUE = enum()
>  >>> print(RED, GREEN, BLUE)
> 0 1 2
>  >>> LEFT, TOP, RIGHT, DOWN = enum()
>  >>> print(LEFT, TOP, RIGHT, DOWN)
> 0 1 2 3
>
>
> And here's the dirty trick....
>
> import inspect
> def enum():
>      """If you use this  your computer will most likely burst ablaze and
> your teeth will fall off"""
>      frame = inspect.stack()[1][0]
>      # 4 = 3 for the function call "enum()", 1 for the opcode
> "UNPACK_SEQUENCE",
>      # the next two bytes represent how many things we need to unpack.
>      a, b = frame.f_code.co_code[frame.f_lasti + 4:frame.f_lasti + 6]
>      enum_len = a + b * 256
>      return range(enum_len)

That is pretty cool.

Now all it needs is a nice repr(), and appropriate comparisons with only 
int and same enum group.

Is it cross-Python?

--
~Ethan~



From jonathan at slenders.be  Tue Feb 12 23:29:38 2013
From: jonathan at slenders.be (Jonathan Slenders)
Date: Tue, 12 Feb 2013 23:29:38 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212214655.GA20475@sleipnir.bytereef.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
	<20130212214655.GA20475@sleipnir.bytereef.org>
Message-ID: <CAKfyG3yJS454zAq7KPLN023J1WxbLi5uqPsVgKRiaS4cYCapPQ@mail.gmail.com>

2013/2/12 Stefan Krah <stefan at bytereef.org>

> Guido van Rossum <guido at python.org> wrote:
> > Please. Anything *except*
> >
> > class Color(Enum):
> >   RED, BLUE, GREEN
> >
> > And not this eiter:
> >
> > class Color(Enum):
> >   RED
> >   BLUE
> >   GREEN
>
> Is a new enum keyword still a possibility? To me that seems to be the
> cleanest way.


I was just thinking the same. We don't like to have more syntax than
necessary, but we should also not introduce too much magic...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/9079cf18/attachment.html>

From timothy.c.delaney at gmail.com  Tue Feb 12 23:32:25 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 09:32:25 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAKfyG3yJS454zAq7KPLN023J1WxbLi5uqPsVgKRiaS4cYCapPQ@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
	<20130212214655.GA20475@sleipnir.bytereef.org>
	<CAKfyG3yJS454zAq7KPLN023J1WxbLi5uqPsVgKRiaS4cYCapPQ@mail.gmail.com>
Message-ID: <CAN8CLgmb=Ebwky0BAKLv50pr=+LvkoV29d6ZQ7BTK-ZZb63SqQ@mail.gmail.com>

On 13 February 2013 09:29, Jonathan Slenders <jonathan at slenders.be> wrote:

> Is a new enum keyword still a possibility? To me that seems to be the
>
>> cleanest way.
>
>
> I was just thinking the same. We don't like to have more syntax than
> necessary, but we should also not introduce too much magic...
>

That's my conclusion as well. To get a nice* enum with guaranteed semantics
and no edge cases, I think it needs parser support.

* for my definition of "nice".

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/beb58f56/attachment.html>

From steve at pearwood.info  Tue Feb 12 23:51:50 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Feb 2013 09:51:50 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212144518.3be491c5@anarchist.wooz.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
Message-ID: <511AC786.80807@pearwood.info>

On 13/02/13 06:45, Barry Warsaw wrote:
[...]
> IMHO, none of these approach the simplicity and explicitness of this API:
>
> class Color(Enum):
>      RED = 1
>      GREEN = 2
>      BLUE = 3
>
> class FieldType(Enum):
>      BINARY = 1
>      AUTOINC = 2
>      NULLABLE = 4
>
> The savings in a few characters, does not (again IMHO) make up for all the
> magic and implicitness you have to guess at with the top versions.
>
> I also don't find much value in the 'unique' style, which seem like they're
> just another way to name string constants.  Here though, you could argue that
> the DRY principle at least makes them interesting, since IIUC, the explicit
> alternative would be
>
> class Geometry:
>      LEFT = 'left'
>      RIGHT = 'right'
>      TOP = 'top'
>      # ... and so on ...
>
> I just wouldn't call these "enums" though. ;)  Again, being a little more
> explicit seems like a win, or at least not so bad.

My perspective from the peanut gallery is, not so. That "little more"
explicitness actually raises the barrier to using enums enough that they
are underused. In my experience, good practice or not, too people avoid
defining enum-like constants:

LEFT = 'left'
RIGHT = 'right'
...

and just embed the string literals in their code instead.

def move(distance, direction):
     if direction == 'left':
         ...


It's hard to avoid the "magic constant" anti-pattern, and too easy to fall
into the bad habit.

Speaking as an ex-Pascal programmer, I really miss the simplicity of creating
enumerated types.

type
   units = (inches, feet, furlongs, miles);
   relationship = (parent, sibling, child);


which would correspond to Tim's suggestion:


class units(Enum):
     INCHES, FEET, FURLONGS, MILES

class relationship(Enum):
     PARENT, SIBLING, CHILD


I would use that, since I don't really care what the values of the enums are.
All I care is:

- they're unique within a type/class;
- they print like their name (e.g. FEET not 1);
- (optional) they can combine as bitmasks.


Not having to set their value explicitly is a *good thing*. If I don't know
the underlying value, I'm not tempted to do this:

some_func(42, unit=1)  # should be unit=FEET


But if I had to use something like this:

class units(Enum):
     INCHES = val()
     FEET = val()
     FURLONGS = val()
     MILES = val()


or this:

class relationship(Enum):
     PARENT, SIBLING, CHILD = range(3)


I'd probably wouldn't bother. I'd stick to "poor man's enum", like this:

PARENT, SIBLING, CHILD = range(3)

or (to my shame, but I have to be realistic) magic constants.



-- 
Steven


From markus at unterwaditzer.net  Tue Feb 12 23:50:01 2013
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Tue, 12 Feb 2013 23:50:01 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
Message-ID: <d6bb7f2f-dda7-451f-9b11-2739d81aac8c@email.android.com>

Guido van Rossum <guido at python.org> wrote:

>I'm torn. I like the clean look of Tim's:
>
>class Flag(Enum):
>   RED, WHITE, BLUE
>
>However, I don't like the magic needed for its implementation -- and
>anybody who thinks they know Python pretty well and thinks about this
>syntax for more than two seconds has to wonder how on earth it's done
>(and the answer is pretty horrible). It is also pretty brittle to
>depend on the lookup order -- and I suspect it will be prone to
>masking other bugs (any typo in a name used in class-level code will
>essentially generate a bogus new enum value).
>
>OTOH I don't like having the string quotes of Antoine's
>counter-proposal:
>
>class Flag(Enum):
>    fields = ('RED', 'WHITE', 'BLUE')
>
>Whenever I see quoted names that are also used unquoted, I cringe a
>little (and this includes calls to hasattr() or getattr() with a
>string literal for the name).
>
>The compromise that I would personally lean towards is more like this:
>
>class Flag(Enum):
>   RED, WHITE, BLUE = <something>
>
>Even if the <something> requires you to know how many values you are
>defining, like range(3). If we have to have a somewhat more verbose
>syntax that doesn't bother me too much.
>
>FWIW I do like being able to define enums that are ints and strings
>(but not mixed); masks/flags I see more as a special case of ints (if
>you use this you are likely trying to match some API defined in C or
>C++). I also agree that it must be possible to customize the enum
>class and the behavior of the values by defining methods on the class.
>
>--
>--Guido van Rossum (python.org/~guido)
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>http://mail.python.org/mailman/listinfo/python-ideas

Forgive my naiveness, but why do we need a new type for enums? Wouldn't a new builtin function that returns a dict suffice? Something like:

def enum(*keys):
    return dict((k, i) for i, k in enumerate(keys))

loglevels = enum('debug', 'warning', 'error')

-- Markus


From guido at python.org  Tue Feb 12 23:56:45 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 14:56:45 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511AC786.80807@pearwood.info>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
Message-ID: <CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>

Frankly, enums are not that useful in small programs. For large
programs or libraries, and especially for public APIs, the extra cost
of defining the enum shouldn't count against them.

Let's just import Barry's enums into the stdlib.

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Tue Feb 12 23:58:15 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 14:58:15 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <d6bb7f2f-dda7-451f-9b11-2739d81aac8c@email.android.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<d6bb7f2f-dda7-451f-9b11-2739d81aac8c@email.android.com>
Message-ID: <CAP7+vJKa8CCyKYfvdh-BnTE=1uiHTcBpUmA94W3x1AzkzJkSQw@mail.gmail.com>

On Tue, Feb 12, 2013 at 2:50 PM, Markus Unterwaditzer
<markus at unterwaditzer.net> wrote:
> Forgive my naiveness, but why do we need a new type for enums? Wouldn't a new builtin function that returns a dict suffice? Something like:
>
> def enum(*keys):
>     return dict((k, i) for i, k in enumerate(keys))
>
> loglevels = enum('debug', 'warning', 'error')

Because you shouldn't have to put the enum names in quotes.

-- 
--Guido van Rossum (python.org/~guido)


From steve at pearwood.info  Wed Feb 13 00:00:38 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Feb 2013 10:00:38 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212220412.4bd3e604@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<20130212220412.4bd3e604@pitrou.net>
Message-ID: <511AC996.3080008@pearwood.info>

On 13/02/13 08:04, Antoine Pitrou wrote:

> I still hope enum values are strings by default - or, if not, that they
> get nice reprs.

99.9% of the time I don't care what value enums have, in fact it is an
advantage if I don't know. Pascal, as I remember it, doesn't give you any
way to find out. But I consider enums displaying as their name to be
essential to an enum.



-- 
Steven


From jonathan at slenders.be  Wed Feb 13 00:01:41 2013
From: jonathan at slenders.be (Jonathan Slenders)
Date: Wed, 13 Feb 2013 00:01:41 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAN8CLgmb=Ebwky0BAKLv50pr=+LvkoV29d6ZQ7BTK-ZZb63SqQ@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
	<20130212214655.GA20475@sleipnir.bytereef.org>
	<CAKfyG3yJS454zAq7KPLN023J1WxbLi5uqPsVgKRiaS4cYCapPQ@mail.gmail.com>
	<CAN8CLgmb=Ebwky0BAKLv50pr=+LvkoV29d6ZQ7BTK-ZZb63SqQ@mail.gmail.com>
Message-ID: <CAKfyG3x1+6WDWtTNTjGwnjN2fk34vuG-4Amqz5Jy6gX9=jr=mA@mail.gmail.com>

Just quickly thinking a little out of the box.

Suppose, that we added grammar for scheme-like symbols. (You know, these
atomic identifiers, prefixed with a quote. I don't know much about scheme,
but as I understand, they are just constants.)

In Python, the syntax could be similar, we could simply do in a global
module scope:

sym symbol

which would translate to:

locals()['symbol'] = Symbol('symbol')

Where Symbol is a built-in that cannot change, which doesn't have a
content, except for its name, and never equals anything but itself. From
then on, we can access it through the name-binding 'symbol'

Then we could probably create enums, using the classes as we have:

class Colors:
    sym red
    sym green
    sym blue

There are probably some gaps here in my reasoning, but this could also work.

Cheers,
Jonathan



2013/2/12 Tim Delaney <timothy.c.delaney at gmail.com>

> On 13 February 2013 09:29, Jonathan Slenders <jonathan at slenders.be> wrote:
>
>> Is a new enum keyword still a possibility? To me that seems to be the
>>
>>> cleanest way.
>>
>>
>> I was just thinking the same. We don't like to have more syntax than
>> necessary, but we should also not introduce too much magic...
>>
>
> That's my conclusion as well. To get a nice* enum with guaranteed
> semantics and no edge cases, I think it needs parser support.
>
> * for my definition of "nice".
>
> Tim Delaney
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/35607490/attachment.html>

From eliben at gmail.com  Wed Feb 13 00:06:36 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 12 Feb 2013 15:06:36 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
Message-ID: <CAF-Rda_ArRXUoXbuWHAmsxmWNtFZZALEiwZ-CuxXC7DR6qYSFQ@mail.gmail.com>

On Tue, Feb 12, 2013 at 2:56 PM, Guido van Rossum <guido at python.org> wrote:

> Frankly, enums are not that useful in small programs. For large
> programs or libraries, and especially for public APIs, the extra cost
> of defining the enum shouldn't count against them.
>
> Let's just import Barry's enums into the stdlib.
>

SGTM. So no need to write a PEP?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/d699242b/attachment.html>

From steve at pearwood.info  Wed Feb 13 00:08:19 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Feb 2013 10:08:19 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212145036.32a98928@anarchist.wooz.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<CAH0mxTRH3nKn+=nhOqNZHaH+jX4SuwdLTfL8g+V-oMqCA9Cing@mail.gmail.com>
	<20130212145036.32a98928@anarchist.wooz.org>
Message-ID: <511ACB63.7030901@pearwood.info>

On 13/02/13 06:50, Barry Warsaw wrote:
> On Feb 12, 2013, at 04:05 PM, Joao S. O. Bueno wrote:
>
>> That is too much ' ' typing - I think it would be ok, to have it like that,
>> but a helper function that would work just like the namedtuple call: Color =
>> Enum("Color", "RED GREEN BLUE", int)
>
> How about:
>
>      >>>  from flufl.enum import make
>      >>>  make('Colors', 'RED GREEN BLUE'.split())
>      <Colors {RED: 1, GREEN: 2, BLUE: 3}>


That reminds me of namedtuple. I could live with something close to that.

Like namedtuple, it should do the split for you.



-- 
Steven


From timothy.c.delaney at gmail.com  Wed Feb 13 00:09:05 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 10:09:05 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
Message-ID: <CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>

On 13 February 2013 09:56, Guido van Rossum <guido at python.org> wrote:

> Frankly, enums are not that useful in small programs. For large
> programs or libraries, and especially for public APIs, the extra cost
> of defining the enum shouldn't count against them.
>
> Let's just import Barry's enums into the stdlib.


That's entirely your call. FWIW I probably won't use them, as they fail to
meet my needs for an enum, #1 being not having to specify their values in
any way if I don't want to. Once you get past 3 or 4 values, it's too easy
to miss or reuse a value.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/394e0c83/attachment.html>

From steve at pearwood.info  Wed Feb 13 00:09:39 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 13 Feb 2013 10:09:39 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CANSw7Ky44ykS+ywrYvS6ynR3t0GWbfxjAqu-9caSP2+o4F8esw@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511A9EE6.5070803@stoneleaf.us>
	<CANSw7Ky44ykS+ywrYvS6ynR3t0GWbfxjAqu-9caSP2+o4F8esw@mail.gmail.com>
Message-ID: <511ACBB3.2050509@pearwood.info>

On 13/02/13 07:38, Yuval Greenfield wrote:
> On Tue, Feb 12, 2013 at 9:58 PM, Ethan Furman<ethan at stoneleaf.us>  wrote:
>
>> 1)  Magic is fun.  :)
>>
>
> I heartily agree. Check this out:
>
>>>> RED, GREEN, BLUE = enum()
>>>> print(RED, GREEN, BLUE)
> 0 1 2
>>>> LEFT, TOP, RIGHT, DOWN = enum()
>>>> print(LEFT, TOP, RIGHT, DOWN)
> 0 1 2 3
>
>
> And here's the dirty trick....

[snip code]

Awesome!

:-)




-- 
Steven


From markus at unterwaditzer.net  Wed Feb 13 00:09:49 2013
From: markus at unterwaditzer.net (Markus Unterwaditzer)
Date: Wed, 13 Feb 2013 00:09:49 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJKa8CCyKYfvdh-BnTE=1uiHTcBpUmA94W3x1AzkzJkSQw@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<d6bb7f2f-dda7-451f-9b11-2739d81aac8c@email.android.com>
	<CAP7+vJKa8CCyKYfvdh-BnTE=1uiHTcBpUmA94W3x1AzkzJkSQw@mail.gmail.com>
Message-ID: <e51c5e89-232e-4bdb-b6d6-0994d58dabff@email.android.com>

Guido van Rossum <guido at python.org> wrote:

>On Tue, Feb 12, 2013 at 2:50 PM, Markus Unterwaditzer
><markus at unterwaditzer.net> wrote:
>> Forgive my naiveness, but why do we need a new type for enums?
>Wouldn't a new builtin function that returns a dict suffice? Something
>like:
>>
>> def enum(*keys):
>>     return dict((k, i) for i, k in enumerate(keys))
>>
>> loglevels = enum('debug', 'warning', 'error')
>
>Because you shouldn't have to put the enum names in quotes.

Oh, I thought you were just against quoted names if they were also accessible without those:

>Whenever I see quoted names that are also used unquoted, I cringe a little (and this includes calls to hasattr() or getattr() with a string literal for the name).
-- Markus


From guido at python.org  Wed Feb 13 00:12:02 2013
From: guido at python.org (Guido van Rossum)
Date: Tue, 12 Feb 2013 15:12:02 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <e51c5e89-232e-4bdb-b6d6-0994d58dabff@email.android.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<d6bb7f2f-dda7-451f-9b11-2739d81aac8c@email.android.com>
	<CAP7+vJKa8CCyKYfvdh-BnTE=1uiHTcBpUmA94W3x1AzkzJkSQw@mail.gmail.com>
	<e51c5e89-232e-4bdb-b6d6-0994d58dabff@email.android.com>
Message-ID: <CAP7+vJK0xccHxcZfEO=sM7ShyeVkDkJe0V_=QE+atd46Jk60qw@mail.gmail.com>

Well but you will use them as loglevels.debug, right?

On Tue, Feb 12, 2013 at 3:09 PM, Markus Unterwaditzer
<markus at unterwaditzer.net> wrote:
> Guido van Rossum <guido at python.org> wrote:
>
>>On Tue, Feb 12, 2013 at 2:50 PM, Markus Unterwaditzer
>><markus at unterwaditzer.net> wrote:
>>> Forgive my naiveness, but why do we need a new type for enums?
>>Wouldn't a new builtin function that returns a dict suffice? Something
>>like:
>>>
>>> def enum(*keys):
>>>     return dict((k, i) for i, k in enumerate(keys))
>>>
>>> loglevels = enum('debug', 'warning', 'error')
>>
>>Because you shouldn't have to put the enum names in quotes.
>
> Oh, I thought you were just against quoted names if they were also accessible without those:
>
>>Whenever I see quoted names that are also used unquoted, I cringe a little (and this includes calls to hasattr() or getattr() with a string literal for the name).
> -- Markus



--
--Guido van Rossum (python.org/~guido)


From ethan at stoneleaf.us  Wed Feb 13 00:14:29 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 15:14:29 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
Message-ID: <511ACCD5.90000@stoneleaf.us>

On 02/12/2013 03:09 PM, Tim Delaney wrote:
> On 13 February 2013 09:56, Guido van Rossum wrote:
>
> Frankly, enums are not that useful in small programs. For large
> programs or libraries, and especially for public APIs, the extra cost
> of defining the enum shouldn't count against them.
>
> Let's just import Barry's enums into the stdlib.
>
>
> That's entirely your call. FWIW I probably won't use them, as they fail
> to meet my needs for an enum, #1 being not having to specify their
> values in any way if I don't want to. Once you get past 3 or 4 values,
> it's too easy to miss or reuse a value.

+1





From greg.ewing at canterbury.ac.nz  Wed Feb 13 00:26:21 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 13 Feb 2013 12:26:21 +1300
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511AC996.3080008@pearwood.info>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<20130212220412.4bd3e604@pitrou.net> <511AC996.3080008@pearwood.info>
Message-ID: <511ACF9D.4090807@canterbury.ac.nz>

Steven D'Aprano wrote:

> 99.9% of the time I don't care what value enums have, in fact it is an
> advantage if I don't know. Pascal, as I remember it, doesn't give you any
> way to find out.

Yes, it does: the ord() function.

> But I consider enums displaying as their name to be
> essential to an enum.

That's one thing that, frustratingly, standard Pascal
*doesn't* give you (although many dialects have
extensions that do).

-- 
Greg


From g.brandl at gmx.net  Wed Feb 13 00:57:38 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 13 Feb 2013 00:57:38 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212144518.3be491c5@anarchist.wooz.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
Message-ID: <kfekqs$r49$1@ger.gmane.org>

Am 12.02.2013 20:45, schrieb Barry Warsaw:
> On Feb 12, 2013, at 09:07 AM, Ethan Furman wrote:
> 
>>The result of the implementation I've been playing with looks something like:
>>
>>class Color(Enum):
>>     type = 'sequence'
>>     RED, GREEN, BLUE
>>
>>class Geometry(Enum):
>>     type = 'unique'
>>     LEFT, RIGHT, TOP, BOTTOM
>>     WEST, EAST, NORTH, SOUTH
>>
>>class FieldType(Enum):
>>     type = 'flag'
>>     BINARY, AUTOINC, NULLABLE

I am also against this.  It requires special support and heuristics in static
analysis tools in order not to flag these as errors.  It will be annoying.

> IMHO, none of these approach the simplicity and explicitness of this API:
> 
> class Color(Enum):
>     RED = 1
>     GREEN = 2
>     BLUE = 3

We could even allow

class Color(Enum):
    RED = 1
    GREEN = ...       # becomes 2
    BLUE = ...        # 3
    MAGENTA = 5
    FLUFL = ...       # 6

class Color(FlagEnum):
    RED = 1
    GREEN = ...       # becomes 2
    BLUE = ...        # 4
    MAGENTA = 16
    FLUFL = ...

class Color(StringEnum):
    RED = ...         # becomes 'red'
    GREEN = ...       # etc.
    BLUE = ...

Georg



From g.brandl at gmx.net  Wed Feb 13 00:59:08 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 13 Feb 2013 00:59:08 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511ACB63.7030901@pearwood.info>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<CAH0mxTRH3nKn+=nhOqNZHaH+jX4SuwdLTfL8g+V-oMqCA9Cing@mail.gmail.com>
	<20130212145036.32a98928@anarchist.wooz.org>
	<511ACB63.7030901@pearwood.info>
Message-ID: <kfektl$r49$2@ger.gmane.org>

Am 13.02.2013 00:08, schrieb Steven D'Aprano:
> On 13/02/13 06:50, Barry Warsaw wrote:
>> On Feb 12, 2013, at 04:05 PM, Joao S. O. Bueno wrote:
>>
>>> That is too much ' ' typing - I think it would be ok, to have it like that,
>>> but a helper function that would work just like the namedtuple call: Color =
>>> Enum("Color", "RED GREEN BLUE", int)
>>
>> How about:
>>
>>      >>>  from flufl.enum import make
>>      >>>  make('Colors', 'RED GREEN BLUE'.split())
>>      <Colors {RED: 1, GREEN: 2, BLUE: 3}>
> 
> 
> That reminds me of namedtuple. I could live with something close to that.
> 
> Like namedtuple, it should do the split for you.

And like namedtuple, it's ugly as hell :)

Georg



From breamoreboy at yahoo.co.uk  Wed Feb 13 01:11:33 2013
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Wed, 13 Feb 2013 00:11:33 +0000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAN8CLgmb=Ebwky0BAKLv50pr=+LvkoV29d6ZQ7BTK-ZZb63SqQ@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
	<20130212214655.GA20475@sleipnir.bytereef.org>
	<CAKfyG3yJS454zAq7KPLN023J1WxbLi5uqPsVgKRiaS4cYCapPQ@mail.gmail.com>
	<CAN8CLgmb=Ebwky0BAKLv50pr=+LvkoV29d6ZQ7BTK-ZZb63SqQ@mail.gmail.com>
Message-ID: <kfelmt$25f$1@ger.gmane.org>

On 12/02/2013 22:32, Tim Delaney wrote:
>
> * for my definition of "nice".

Neurotic, Insecure, Crazy, Emotional? :)

>
> Tim Delaney
>

-- 
Cheers.

Mark Lawrence



From ericsnowcurrently at gmail.com  Wed Feb 13 02:07:31 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 12 Feb 2013 18:07:31 -0700
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <kfekqs$r49$1@ger.gmane.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
Message-ID: <CALFfu7C8FQ9we+RDWy27F15bw8mpV6D0Zc+rn-Z=bWc6bD8NWg@mail.gmail.com>

On Tue, Feb 12, 2013 at 4:57 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> We could even allow
>
> class Color(Enum):
>     RED = 1
>     GREEN = ...       # becomes 2
>     BLUE = ...        # 3
>     MAGENTA = 5
>     FLUFL = ...       # 6
>
> class Color(FlagEnum):
>     RED = 1
>     GREEN = ...       # becomes 2
>     BLUE = ...        # 4
>     MAGENTA = 16
>     FLUFL = ...
>
> class Color(StringEnum):
>     RED = ...         # becomes 'red'
>     GREEN = ...       # etc.
>     BLUE = ...

Nice.  Both the use of ellipsis and having the semantics depend
directly on which specific class you inherit from.

At the same time, Steven D'Aprano's proposition of "the underlying
value is irrelevant" and "must be bitwise comparable" appeals to me.
If the underlying values matter then I'd call it a "constant" rather
than an "enum".  A proper enum type would remove the temptation to use
the underlying value.

I would say that there is room for both an enum type and one or more
constants type (e.g. what Georg demonstrated above).  Regardless, this
discussion would be well-served by recognizing the distinction.

-eric


From timothy.c.delaney at gmail.com  Wed Feb 13 02:45:24 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 12:45:24 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAN8CLgmNUHRPj0piWRz+2zCUX-NNT2AZgOiLm_0FF0qbXJo2Wg@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgmNUHRPj0piWRz+2zCUX-NNT2AZgOiLm_0FF0qbXJo2Wg@mail.gmail.com>
Message-ID: <CAN8CLgk1mfuWLxgNYNHzGCwtuuwN9QZdh5TSnU9gNk2r0CcbPA@mail.gmail.com>

On 13 February 2013 07:13, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> Well, that particular case would work (you'd get a NameError: sys) due to
> having done an attribute lookup on sys, but the following:
>
> class Color(Enum):
>     RED, GREEN, BLUE
>     if platfor == 'win32':
>         MAGENTA
>
> would create an enum value 'platfor'.
>

BTW, I've just addressed this in my latest version up on BitBucket. Any
usage of an object that is more than just assignment (basically, anything
that results in a call of a __dunder__ method on the object) will now mark
the name as not an enum value and result in a NameError for an unknown name.

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum
>>> class Color(Enum):
...     RED
...     if unknown_name == 'value':
...         PINK
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in Color
  File ".\enum.py", line 123, in __eq__
    def __eq__(self, other): return self._get(True) == other
  File ".\enum.py", line 98, in _get
    raise NameError(self.key)
NameError: unknown_name

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/4445079a/attachment.html>

From timothy.c.delaney at gmail.com  Wed Feb 13 02:51:14 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 12:51:14 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAPTjJmozAB99EO0SBtqG0QzBevex4RknFGTtQ8VeZ-=RMdSrvQ@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
	<CAN8CLgmn3ikWQR57G54z_QMDpB5dFMcpVCJvsThBBqhE6BN5GA@mail.gmail.com>
	<CAN8CLg=V7PCc4pedH=+evOf5BDKryakLH1VO24geVNx=rVmqQg@mail.gmail.com>
	<CAPTjJmozAB99EO0SBtqG0QzBevex4RknFGTtQ8VeZ-=RMdSrvQ@mail.gmail.com>
Message-ID: <CAN8CLgkoiejUDAEy79M_tGP8rqVRn7vMjDtwkOhDfStw3MKm0w@mail.gmail.com>

On 13 February 2013 08:36, Chris Angelico <rosuav at gmail.com> wrote:

> On Tue, Feb 12, 2013 at 7:06 PM, Tim Delaney
> <timothy.c.delaney at gmail.com> wrote:
> > This actually makes a kind of perverse sense. Conceptually the actual
> > EnumValues are the instances of the Enum (and in fact, I've made them
> > pretend to actually be so).
>
> Not so perverse. I think that makes very good sense (but then, I know
> C++ enums, so maybe I'm tainted). Looks good, though I've not actually
> played with the code.
>

And more ...

Firstly, handles more edge cases that might have been mis-identified as
enum values.

EnumParams is now redundant (and will probably be removed), replaced by
calling the (undefined) enum value name.

 Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum
>>> class MyEnum(Enum):
...     A, B
...     C(None, 'c1', c2='2')
...     D(10, 'd')
...     E(None, e='e')
...
>>> repr(MyEnum)
"<enum __main__.MyEnum {<EnumValue 'A': 0>, <EnumValue 'B': 1>, <EnumValue
'C': 2>, <EnumValue 'D': 10>, <EnumValue 'E': 11>}>"
>>> str(MyEnum)
'{A:0, B:1, C:2, D:10, E:11}'
>>> MyEnum.A.args, MyEnum.A.kwargs
((), {})
>>> MyEnum.C.args, MyEnum.C.kwargs
(('c1',), {'c2': '2'})

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/9ab69f6f/attachment.html>

From timothy.c.delaney at gmail.com  Wed Feb 13 03:20:31 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 13:20:31 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAN8CLgkoiejUDAEy79M_tGP8rqVRn7vMjDtwkOhDfStw3MKm0w@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<CAN8CLg=2t_S1+-jMFZ2eggmURVB6t6SHbqs4RWe-zfv6d7a3LA@mail.gmail.com>
	<CAN8CLgmn3ikWQR57G54z_QMDpB5dFMcpVCJvsThBBqhE6BN5GA@mail.gmail.com>
	<CAN8CLg=V7PCc4pedH=+evOf5BDKryakLH1VO24geVNx=rVmqQg@mail.gmail.com>
	<CAPTjJmozAB99EO0SBtqG0QzBevex4RknFGTtQ8VeZ-=RMdSrvQ@mail.gmail.com>
	<CAN8CLgkoiejUDAEy79M_tGP8rqVRn7vMjDtwkOhDfStw3MKm0w@mail.gmail.com>
Message-ID: <CAN8CLgmhweDjpzdWiNU8+8md-gawPM_qPN0SJv-Bw7vcVG08gA@mail.gmail.com>

On 13 February 2013 12:51, Tim Delaney <timothy.c.delaney at gmail.com> wrote:

> And more ...
>

And inspired by the other thread, Ellipsis (...) is now used to indicate
"use the next value". Ellipsis can either be assigned to an enum value, or
passed as the first parameter to EnumParams or when calling the enum value
during construction.

None is no longer valid.

 Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from enum import Enum
>>> class MyEnum(Enum):
...     A
...     B, C, D = ..., 5, ...
...     E = ...
...     F(..., 'f')
...
>>> MyEnum
<enum __main__.MyEnum {<EnumValue 'A': 0>, <EnumValue 'B': 1>, <EnumValue
'C': 5>, <EnumValue 'D': 6>, <EnumValue 'E': 7>, <EnumValue 'F': 8>}>

I'm stopping now.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/5c27c124/attachment.html>

From greg.ewing at canterbury.ac.nz  Wed Feb 13 00:19:16 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 13 Feb 2013 12:19:16 +1300
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511A8A82.4010602@stoneleaf.us>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
Message-ID: <511ACDF4.4040107@canterbury.ac.nz>

Ethan Furman wrote:
> If you don't mind verbose, we could always go for something like:
> 
> class Color(Enum):
>     start_defs()
>     BLACK
>     RED, GREEN, BLUE
>     CYAN, YELLOW, MAGENTA
>     end_defs()

Or

     class Color(Enum):
         values(BLACK,
            RED, GREEN, BLUE,
            CYAN, YELLOW, MAGENTA)

where values() ignores its arguments and ends the
definition phase.

This would also provide a way to avoid relying on
language guarantees of evaluation order. The initial
lookup of the names could assign arbitrary values to
them, and then values() could remap them to sequential
values based on the order it receives them in.

-- 
Greg


From raymond.hettinger at gmail.com  Wed Feb 13 04:25:53 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 12 Feb 2013 19:25:53 -0800
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
Message-ID: <71DF1DA7-06D9-42F2-8E7A-B92732C4C424@gmail.com>


On Feb 11, 2013, at 5:28 AM, Eli Bendersky <eliben at gmail.com> wrote:

> Can you elaborate on the utility of this feature? What realistic use cases do you see for it? I think that at this point it's important to weigh all benefits of features vs. implementation complexity, and there's absolutely no need to support every feature every other enum implementation has. I want to stress again that the most important characteristic of your implementation is the clean syntax which means that enums are so easy to define they don't really need special Python syntax and a library feature can do. However, there's a big leap from this to defining custom metaclasses for enums.

Well said.  I agree with you critique.
In the absence of compelling use cases,
the language is better-off without a complicated
new feature. 


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130212/2c977284/attachment.html>

From timothy.c.delaney at gmail.com  Wed Feb 13 04:38:27 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Wed, 13 Feb 2013 14:38:27 +1100
Subject: [Python-ideas] constant/enum type in stdlib
In-Reply-To: <71DF1DA7-06D9-42F2-8E7A-B92732C4C424@gmail.com>
References: <CAH0mxTQ9wfcR_R5SbXV0J63iVaB1PvYvSF06tRy=AibO2Qajfg@mail.gmail.com>
	<CADiSq7cKrdgzBgENPYYXA5S-oCtT4Jxs5xnMwr750LUyKg+v6Q@mail.gmail.com>
	<CAF-Rda8B1Cw1vNZxmwuCP0AxHxhjdNQvoCWGqJsKmZsj4g0C3g@mail.gmail.com>
	<51085AAB.6090303@canterbury.ac.nz>
	<CAF-Rda9Oup0fmJapmAg9kFv=YaGwjph2ZyMXrMB3Q0bEhWz2eg@mail.gmail.com>
	<CAP7+vJLT+KxjqNT6=c6ZYmBWw55U_88W8MFqW+_3OH7AqTVw4Q@mail.gmail.com>
	<5108A87D.9000207@canterbury.ac.nz>
	<20130130082639.0b28d7eb@pitrou.net>
	<CAKCKLWyegkHo15dcVvhC23WaY=3Mnr6P-JT5M_Lm6UKCMahUhg@mail.gmail.com>
	<CAKCKLWxTt0kwjc4dpRJUz27_0bt+u+AHkEqzTJEoZNC7rshKkQ@mail.gmail.com>
	<kec3h1$3rv$1@ger.gmane.org>
	<CAN8CLg=e9DAqdpgd4=0chU=dubVEr66LtnWUwYdJb6O2O4Vbag@mail.gmail.com>
	<CAN8CLgnNB9iOY8EX9ouJ+Jq_t1BP0BRwWR-_xQgVgL4ivZ4F1w@mail.gmail.com>
	<CAN8CLg=NTDzCEA325EZtgtt6UQc-9mtePduoZZd0a=CP=iwMWQ@mail.gmail.com>
	<CAN8CLgk=VbX9-n2HpVJbANwTPUTmi+F2OWyqxSRcFVjTiiV9fw@mail.gmail.com>
	<20130201103656.0b24c40d@anarchist.wooz.org>
	<CAN8CLgmE2=Fg1F1yrQMJH_9kVU1LzR1NETvPMhV=HRQaDD_rAw@mail.gmail.com>
	<CAN8CLgnbvdqLT2i0UMqqN0bEbJNnZiwsU0PjwiXWH1LMX+wiBg@mail.gmail.com>
	<CAN8CLgmy3UGbAx6VGw5EQx69WxJobECF+RpSaWTSGpOSjaD8xg@mail.gmail.com>
	<CAOyAWggDkJRPPDWYcNWN+_oAC+mrhYW5LarDUhPE5UKNXMAaZw@mail.gmail.com>
	<CAN8CLgmMxcyz9XskEojWofXEjyfB23t=D0P-xiFS-nmooDZTDw@mail.gmail.com>
	<CAN8CLg=a_OzDLEJPN6eB5N9zFMXmgRff91k7siafe927Eeuk0Q@mail.gmail.com>
	<CAF-Rda8+Y_iYNqMqp+kjn6o4WkSEZ+g21a7s0M2hNjg+6Fvi5Q@mail.gmail.com>
	<71DF1DA7-06D9-42F2-8E7A-B92732C4C424@gmail.com>
Message-ID: <CAN8CLgktMd+8nxVhMgCeCD-czSwvq+KdcXerwa0rre0Cxw9BnQ@mail.gmail.com>

On 13 February 2013 14:25, Raymond Hettinger <raymond.hettinger at gmail.com>wrote:

>
> On Feb 11, 2013, at 5:28 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
> Can you elaborate on the utility of this feature? What realistic use cases
> do you see for it? I think that at this point it's important to weigh all
> benefits of features vs. implementation complexity, and there's absolutely
> no need to support every feature every other enum implementation has. I
> want to stress again that the most important characteristic of your
> implementation is the clean syntax which means that enums are so easy to
> define they don't really need special Python syntax and a library feature
> can do. However, there's a big leap from this to defining custom
> metaclasses for enums.
>
>
> Well said.  I agree with you critique.
> In the absence of compelling use cases,
> the language is better-off without a complicated
> new feature.
>

Absolutely. At the moment my implementation has everything in there as I've
been incrementally finding syntactically cleaner ways of doing things. I
intend to remove a lot of the extra functionality eventually - for example,
supporting EnumMeta/EnumValue subclassing.

In any case, it looks like Guido is strongly heading towards flufl.enum -
doesn't mean I won't keep working on my implementation, but it doesn't look
like it will be the basis for a stdlib enum.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/86011aff/attachment.html>

From jsbueno at python.org.br  Wed Feb 13 04:41:12 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Wed, 13 Feb 2013 01:41:12 -0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
Message-ID: <CAH0mxTSBRCGcE+xTbrnC-Om9JGZbC8c_dzG6787EbL5stK7yHA@mail.gmail.com>

On 12 February 2013 19:29, Guido van Rossum <guido at python.org> wrote:
> Please. Anything *except*
>
> class Color(Enum):
>   RED, BLUE, GREEN
>
> And not this eiter:
>
> class Color(Enum):
>   RED
>   BLUE
>   GREEN

Just throwing in another syntax that is valid, requires no quotes and
no magic - in case someone
is interested in developing on it:

>>> Colors = make(RED=..., GREEN=..., BLUE=...)


From stephen at xemacs.org  Wed Feb 13 04:41:20 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 13 Feb 2013 12:41:20 +0900
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130212214655.GA20475@sleipnir.bytereef.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAP7+vJ+5b=62PK20BQdbN7ELTdzY=fGKhGBrmcY8Yo=YzXM7wg@mail.gmail.com>
	<20130212214655.GA20475@sleipnir.bytereef.org>
Message-ID: <87a9r8yj6n.fsf@uwakimon.sk.tsukuba.ac.jp>

Stefan Krah writes:
 > Guido van Rossum <guido at python.org> wrote:
 > > Please. Anything *except*
 > > 
 > > class Color(Enum):
 > >   RED, BLUE, GREEN

I'll take you at your word.  But maybe you should stop reading
now.<wink/>

 > Is a new enum keyword still a possibility? To me that seems to be the
 > cleanest way.

If parser support were on the table, couldn't we avoid a new keyword
by aping the Pascal syntax with

    class Color(Enum) = [RED, GREEN, BLUE]

or something like that?  I realize this is horrible because of the
missing colon, and because availability of the syntax depends on the
base class.  Maybe it will suggest something to somebody, though.  The
base class seems to be needed to support variants like OrderedEnum and
FlagEnum, as the implicit Enum

    class Color = [RED, GREEN, BLUE]

can't disambiguate common use cases (ordered enums, or-able flags).
Another possibility would be to allow the list of identifiers as the
base class:

    class Color([RED, GREEN, BLUE]):
        pass

I suppose this is better than the version using "=", which doesn't
seem to allow explicit initialization of the enumerated ids, or
operations on them, in the class body.  Perhaps the Enum base class
should be explicit:

    class Color(Enum, [RED, GREEN, BLUE]):
        pass

In all cases the point of using lists instead of (implicit) tuples is
to clearly guarantee order of evaluation to support OrderedEnum.  Sets
are out because FlagEnum doesn't need such support.



From stephen at xemacs.org  Wed Feb 13 04:14:13 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 13 Feb 2013 12:14:13 +0900
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
Message-ID: <87boboykfu.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:

 > Frankly, enums are not that useful in small programs.

This seems like a very personal thing to me.  In some sense that I
suppose can be made as objective as counting keystrokes[1], I have to
agree with you.  But I don't feel that Steven's cringing at "magic
literals" is a less valid argument for supporting a particular syntax
than your cringing at "quoted identifier names".

Of course, technically speaking, Steven could perfectly well use
"explicit-valued" enums[2] to eliminate magic literals.  But from a
beautiful language perspective, ISTM that he is just one among many to
advocate that an enum *definition* should look like an *unordered list
of identifiers*.[3]  It's not about keystroke count, it's about
correspondence of syntax to semantics.  He's not doing himself any
favors by replacing a few magic literals with a hard to read,
overdefined enum declaration.

I strongly suspect that if you put explicit-valued enums into the
stdlib now, you will find that there remains a strong demand for
"nice" enums as Tim calls them (although most RFEs will probably use
the word "real"!)

Footnotes: 
[1]  But far more important, I grant you that. ;-)

[2]  Note that EIBTI is *not* an argument for explicit-valued enums,
because an enum elements's name and class should tell you everything
you need to know about its semantics.  If it doesn't, you aren't using
it as an enum!  And "YELLOW = val()" is hardly explicit. :-(

[3]  If that "list" of names actually has more structure (a set that
allows subsetting and union as commonly done with flags, or an order
such as the spectrum of colors), it would be nice if that structure is
imposed automatically as well -- but this should be done by a subclass
such as OrderedEnum or FlagEnum which makes the structure explicit.



From ncoghlan at gmail.com  Wed Feb 13 05:07:48 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 13 Feb 2013 14:07:48 +1000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
Message-ID: <CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>

On 13 Feb 2013 09:11, "Tim Delaney" <timothy.c.delaney at gmail.com> wrote:
>
> On 13 February 2013 09:56, Guido van Rossum <guido at python.org> wrote:
>>
>> Frankly, enums are not that useful in small programs. For large
>> programs or libraries, and especially for public APIs, the extra cost
>> of defining the enum shouldn't count against them.
>>
>> Let's just import Barry's enums into the stdlib.
>
>
> That's entirely your call. FWIW I probably won't use them, as they fail
to meet my needs for an enum, #1 being not having to specify their values
in any way if I don't want to. Once you get past 3 or 4 values, it's too
easy to miss or reuse a value.

What's wrong with enum.make? That just accepts the sequence of names
namedtuple style, no values specified anywhere.

Cheers,
Nick.

>
> Tim Delaney
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/9709eabc/attachment.html>

From ethan at stoneleaf.us  Wed Feb 13 06:04:39 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 21:04:39 -0800
Subject: [Python-ideas] another enum implementation
Message-ID: <511B1EE7.4060408@stoneleaf.us>

I realize this isn't going very far, but I would still appreciate 
feedback.  :)

The code is here:
https://bitbucket.org/stoneleaf/enum

It is based on ideas from Michael Foord, Antoine Pitrou, and 
(eventually) Ted Delaney.


Enum only accepts upper-case names as enum candidates; all enum values 
must be the same: either 'sequence' (0, 1, 2, etc) or 'flag' (1, 2, 4, 
etc.) or 'unique' (north, south, east, west, etc.); and extending via 
subclassing is possible.

Sample code:

class Color(Enum):
     type = 'sequence'
     BLACK
     RED
     GREEN = 4
     BLUE
print Color
# Color(BLACK:0, RED:1, GREEN:4, BLUE:5)

class MoreColor(Color):
     MAGENTA
     YELLOW
     CYAN
print MoreColor
# MoreColor(BLACK:0, RED:1, GREEN:4, BLUE:5, MAGENTA:6, YELLOW:7,
#	    CYAN:8)


class DbfType(Enum):
     type = 'unique'
     CLP
     DB3
     VFP
--> print(enum.DbfType)
# DbfType(DB3:'db3', CLP:'clp', VFP:'vfp')

class SomeFlags(Enum):
     type = 'flag'
     ON_DISK
     HAS_MEMO
     LARGE_CHAR
     UNICODE
--> print(enum.SomeFlags)
# SomeFlags(ON_DISK:1, HAS_MEMO:2, LARGE_CHAR:4, UNICODE:8)

class Error(Enum):
     type = 'sequence'
     THIS
     THAT
     The_Other		# raises NameError
     THOSE = 1		# raises InvalidEnum

-->enum.Color.RED
Color('RED')

-->enum.Color.RED == 1
True

-->int(enum.Color.RED)
1

-->enum.SomeFlags.ON_DISK
SomeFlags('ON_DISK')

-->enum.SomeFlags.ON_DISK == 1
True

-->int(enum.SomeFlags.ON_DISK)
1

-->enum.SomeFlags.ON_DISK == enum.Color.RED
False

-->enum.MoreColor.RED == 1
True

-->enum.MoreColor.RED == enum.Color.RED
True

-->enum.Color(1)
Color('RED')

-->enum.Color('RED')
Color('RED')

--> for color in enum.Color:
...   print(color)
...
BLACK
RED
GREEN
BLUE

--> for color in enum.Color:
...   print(repr(color))
...
Color('BLACK')
Color('RED')
Color('GREEN')
Color('BLUE')


I would appreciate any comments on both the API, and the code behind it.

Thanks.

--
~Ethan~


From yselivanov.ml at gmail.com  Wed Feb 13 06:24:15 2013
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 13 Feb 2013 00:24:15 -0500
Subject: [Python-ideas] another enum implementation
In-Reply-To: <511B1EE7.4060408@stoneleaf.us>
References: <511B1EE7.4060408@stoneleaf.us>
Message-ID: <89705A92-071D-42D2-A9DF-ABBEDF9A3E7C@gmail.com>

I'm not sure why, but I didn't see anyone making a point about
documentation.  To me, it's important that stdlib's and other
libraries' enums are well documented, and I think that the best
way of providing and enforcing this is to mimic "property" builtin.

class Color(enum.Sequence):
   "Some general docs about Color enum"
   BLACK = enum.value(doc='Black Color')

or

class Color(enum.Flag):
   "..."
   BLACK = enum(doc='Black Color')
   WHITE = enum()

Maybe this is uglier than magic of just writing flags' names,
but this implementation is much simpler, backwards compatible with
python 2, and verbose.  The latter point is both good and bad, but
we don't write enums that often to make a bit of verbosity unacceptable.

Just my two cents.

-
Yury

On 2013-02-13, at 12:04 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> I realize this isn't going very far, but I would still appreciate feedback.  :)
> 
> The code is here:
> https://bitbucket.org/stoneleaf/enum
> 
> It is based on ideas from Michael Foord, Antoine Pitrou, and (eventually) Ted Delaney.
> 
> 
> Enum only accepts upper-case names as enum candidates; all enum values must be the same: either 'sequence' (0, 1, 2, etc) or 'flag' (1, 2, 4, etc.) or 'unique' (north, south, east, west, etc.); and extending via subclassing is possible.
> 
> Sample code:
> 
> class Color(Enum):
>    type = 'sequence'
>    BLACK
>    RED
>    GREEN = 4
>    BLUE
> print Color
> # Color(BLACK:0, RED:1, GREEN:4, BLUE:5)
> 
> class MoreColor(Color):
>    MAGENTA
>    YELLOW
>    CYAN
> print MoreColor
> # MoreColor(BLACK:0, RED:1, GREEN:4, BLUE:5, MAGENTA:6, YELLOW:7,
> #	    CYAN:8)
> 
> 
> class DbfType(Enum):
>    type = 'unique'
>    CLP
>    DB3
>    VFP
> --> print(enum.DbfType)
> # DbfType(DB3:'db3', CLP:'clp', VFP:'vfp')
> 
> class SomeFlags(Enum):
>    type = 'flag'
>    ON_DISK
>    HAS_MEMO
>    LARGE_CHAR
>    UNICODE
> --> print(enum.SomeFlags)
> # SomeFlags(ON_DISK:1, HAS_MEMO:2, LARGE_CHAR:4, UNICODE:8)
> 
> class Error(Enum):
>    type = 'sequence'
>    THIS
>    THAT
>    The_Other		# raises NameError
>    THOSE = 1		# raises InvalidEnum
> 
> -->enum.Color.RED
> Color('RED')
> 
> -->enum.Color.RED == 1
> True
> 
> -->int(enum.Color.RED)
> 1
> 
> -->enum.SomeFlags.ON_DISK
> SomeFlags('ON_DISK')
> 
> -->enum.SomeFlags.ON_DISK == 1
> True
> 
> -->int(enum.SomeFlags.ON_DISK)
> 1
> 
> -->enum.SomeFlags.ON_DISK == enum.Color.RED
> False
> 
> -->enum.MoreColor.RED == 1
> True
> 
> -->enum.MoreColor.RED == enum.Color.RED
> True
> 
> -->enum.Color(1)
> Color('RED')
> 
> -->enum.Color('RED')
> Color('RED')
> 
> --> for color in enum.Color:
> ...   print(color)
> ...
> BLACK
> RED
> GREEN
> BLUE
> 
> --> for color in enum.Color:
> ...   print(repr(color))
> ...
> Color('BLACK')
> Color('RED')
> Color('GREEN')
> Color('BLUE')
> 
> 
> I would appreciate any comments on both the API, and the code behind it.
> 
> Thanks.
> 
> --
> ~Ethan~
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



From senthil at uthcode.com  Wed Feb 13 07:18:03 2013
From: senthil at uthcode.com (Senthil Kumaran)
Date: Tue, 12 Feb 2013 22:18:03 -0800
Subject: [Python-ideas] another enum implementation
In-Reply-To: <511B1EE7.4060408@stoneleaf.us>
References: <511B1EE7.4060408@stoneleaf.us>
Message-ID: <CAPOVWOSyCk_7sh9KtshxTpM-9HLZNHj-_hEbM2Egyg09K6nv=w@mail.gmail.com>

Two comments.

1. When it is fully generalized, is type is the way to to specify the
type of enums? Or there will be helper functions or classes that can
specify the type of enum one desires.
2. Personally, I have resorted to using namedtuples for enum like
behavior. How would this class benefit me? Any edification would be
helpful.

On Tue, Feb 12, 2013 at 9:04 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> I realize this isn't going very far, but I would still appreciate feedback.
> :)
>
> The code is here:
> https://bitbucket.org/stoneleaf/enum
>
> It is based on ideas from Michael Foord, Antoine Pitrou, and (eventually)
> Ted Delaney.
>
>
> Enum only accepts upper-case names as enum candidates; all enum values must
> be the same: either 'sequence' (0, 1, 2, etc) or 'flag' (1, 2, 4, etc.) or
> 'unique' (north, south, east, west, etc.); and extending via subclassing is
> possible.
>
> Sample code:
>
> class Color(Enum):
>     type = 'sequence'
>     BLACK
>     RED
>     GREEN = 4
>     BLUE
> print Color
> # Color(BLACK:0, RED:1, GREEN:4, BLUE:5)
>
> class MoreColor(Color):
>     MAGENTA
>     YELLOW
>     CYAN
> print MoreColor
> # MoreColor(BLACK:0, RED:1, GREEN:4, BLUE:5, MAGENTA:6, YELLOW:7,
> #           CYAN:8)
>
>
> class DbfType(Enum):
>     type = 'unique'
>     CLP
>     DB3
>     VFP
> --> print(enum.DbfType)
> # DbfType(DB3:'db3', CLP:'clp', VFP:'vfp')
>
> class SomeFlags(Enum):
>     type = 'flag'
>     ON_DISK
>     HAS_MEMO
>     LARGE_CHAR
>     UNICODE
> --> print(enum.SomeFlags)
> # SomeFlags(ON_DISK:1, HAS_MEMO:2, LARGE_CHAR:4, UNICODE:8)
>
> class Error(Enum):
>     type = 'sequence'
>     THIS
>     THAT
>     The_Other           # raises NameError
>     THOSE = 1           # raises InvalidEnum
>
> -->enum.Color.RED
> Color('RED')
>
> -->enum.Color.RED == 1
> True
>
> -->int(enum.Color.RED)
> 1
>
> -->enum.SomeFlags.ON_DISK
> SomeFlags('ON_DISK')
>
> -->enum.SomeFlags.ON_DISK == 1
> True
>
> -->int(enum.SomeFlags.ON_DISK)
> 1
>
> -->enum.SomeFlags.ON_DISK == enum.Color.RED
> False
>
> -->enum.MoreColor.RED == 1
> True
>
> -->enum.MoreColor.RED == enum.Color.RED
> True
>
> -->enum.Color(1)
> Color('RED')
>
> -->enum.Color('RED')
> Color('RED')
>
> --> for color in enum.Color:
> ...   print(color)
> ...
> BLACK
> RED
> GREEN
> BLUE
>
> --> for color in enum.Color:
> ...   print(repr(color))
> ...
> Color('BLACK')
> Color('RED')
> Color('GREEN')
> Color('BLUE')
>
>
> I would appreciate any comments on both the API, and the code behind it.
>
> Thanks.
>
> --
> ~Ethan~
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From senthil at uthcode.com  Wed Feb 13 07:35:33 2013
From: senthil at uthcode.com (Senthil Kumaran)
Date: Tue, 12 Feb 2013 22:35:33 -0800
Subject: [Python-ideas] another enum implementation
In-Reply-To: <CAPOVWOSyCk_7sh9KtshxTpM-9HLZNHj-_hEbM2Egyg09K6nv=w@mail.gmail.com>
References: <511B1EE7.4060408@stoneleaf.us>
	<CAPOVWOSyCk_7sh9KtshxTpM-9HLZNHj-_hEbM2Egyg09K6nv=w@mail.gmail.com>
Message-ID: <CAPOVWOTdMU9ePQNW8XzA3HJmmwwxSvedj5xHzSqaTReQYC8EHQ@mail.gmail.com>

On Tue, Feb 12, 2013 at 10:18 PM, Senthil Kumaran <senthil at uthcode.com> wrote:
> Two comments.

I should say, I responded without looking at the other thread. If the
questions are relevant, please proceed, if not I shall find the answer
the other thread which seem to have discussed this topic in greater
detail.


From tjreedy at udel.edu  Wed Feb 13 07:45:54 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 13 Feb 2013 01:45:54 -0500
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAN1F8qVS-whJDUJNWz4BrmkciNu4sqh1T=B8iTuV7dDHoASWLQ@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<511A9764.2060902@stoneleaf.us>
	<CAN1F8qVS-whJDUJNWz4BrmkciNu4sqh1T=B8iTuV7dDHoASWLQ@mail.gmail.com>
Message-ID: <kffcra$nv8$1@ger.gmane.org>

On 2/12/2013 3:43 PM, Joshua Landau wrote:

> class Color(Enum):
>      values = "RED GREEN BLUE MAGENTA OTHER COLORS HERE TOO".split()
>
> so I'm OK with that, but then you're in the territory when you might as
> well go with the method Barry suggested.

A custom mapping can automatically do the split. This is an interesting 
compromise between no quotes and multiple quotes.

-- 
Terry Jan Reedy



From ericsnowcurrently at gmail.com  Wed Feb 13 07:49:01 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 12 Feb 2013 23:49:01 -0700
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CALFfu7C8FQ9we+RDWy27F15bw8mpV6D0Zc+rn-Z=bWc6bD8NWg@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
	<CALFfu7C8FQ9we+RDWy27F15bw8mpV6D0Zc+rn-Z=bWc6bD8NWg@mail.gmail.com>
Message-ID: <CALFfu7A2v2fokSECmeiq9zjvx8h451E4HrYEUYNaxNWohoeEKw@mail.gmail.com>

On Tue, Feb 12, 2013 at 6:07 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> At the same time, Steven D'Aprano's proposition of "the underlying
> value is irrelevant" and "must be bitwise comparable" appeals to me.
> If the underlying values matter then I'd call it a "constant" rather
> than an "enum".  A proper enum type would remove the temptation to use
> the underlying value.

Here's an implementation of what I'm talking about:

http://code.activestate.com/recipes/578455/

Each enum value is simply an object and not backed by any underlying
value.  The bitwise operators are implemented.  The implementation is
a little rough, but hopefully gets the idea across.

-eric


From ethan at stoneleaf.us  Wed Feb 13 08:01:30 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 23:01:30 -0800
Subject: [Python-ideas] another enum implementation
In-Reply-To: <89705A92-071D-42D2-A9DF-ABBEDF9A3E7C@gmail.com>
References: <511B1EE7.4060408@stoneleaf.us>
	<89705A92-071D-42D2-A9DF-ABBEDF9A3E7C@gmail.com>
Message-ID: <511B3A4A.2000608@stoneleaf.us>

On 02/12/2013 09:24 PM, Yury Selivanov wrote:
> I'm not sure why, but I didn't see anyone making a point about
> documentation.  To me, it's important that stdlib's and other
> libraries' enums are well documented, and I think that the best
> way of providing and enforcing this is to mimic "property" builtin.
>
> class Color(enum.Sequence):
>     "Some general docs about Color enum"
>     BLACK = enum.value(doc='Black Color')
>
> or
>
> class Color(enum.Flag):
>     "..."
>     BLACK = enum(doc='Black Color')
>     WHITE = enum()
>
> Maybe this is uglier than magic of just writing flags' names,
> but this implementation is much simpler, backwards compatible with
> python 2, and verbose.  The latter point is both good and bad, but
> we don't write enums that often to make a bit of verbosity unacceptable.
>
> Just my two cents.

You definitely have a good point about documentation, and I'm going to 
add the functionality that Tim has in his which will allow other 
attributes on enum values (the difference with his is that mine are 
actual instances of their class so all values of one class will have the 
same attributes).

However, for small enums, well chosen names should be documentation 
enough, so doc strings will not be required.  Even more importantly, if 
I'm just trying something out I do not want to be forced to add stuff I 
don't need.

--
~Ethan~


From solipsis at pitrou.net  Wed Feb 13 08:21:29 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 08:21:29 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
Message-ID: <20130213082129.5d917b00@pitrou.net>

On Wed, 13 Feb 2013 00:57:38 +0100
Georg Brandl <g.brandl at gmx.net> wrote:
> > IMHO, none of these approach the simplicity and explicitness of this API:
> > 
> > class Color(Enum):
> >     RED = 1
> >     GREEN = 2
> >     BLUE = 3
> 
> We could even allow
> 
> class Color(Enum):
>     RED = 1
>     GREEN = ...       # becomes 2
>     BLUE = ...        # 3
>     MAGENTA = 5
>     FLUFL = ...       # 6

The fact that the ellipsis variant requires *more* typing sounds a bit
wrong to me. The common case should be the easiest to type.

Regards

Antoine.




From g.brandl at gmx.net  Wed Feb 13 08:34:21 2013
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 13 Feb 2013 08:34:21 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130213082129.5d917b00@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org> <20130213082129.5d917b00@pitrou.net>
Message-ID: <kfffj6$crv$1@ger.gmane.org>

Am 13.02.2013 08:21, schrieb Antoine Pitrou:
> On Wed, 13 Feb 2013 00:57:38 +0100
> Georg Brandl <g.brandl at gmx.net> wrote:
>> > IMHO, none of these approach the simplicity and explicitness of this API:
>> > 
>> > class Color(Enum):
>> >     RED = 1
>> >     GREEN = 2
>> >     BLUE = 3
>> 
>> We could even allow
>> 
>> class Color(Enum):
>>     RED = 1
>>     GREEN = ...       # becomes 2
>>     BLUE = ...        # 3
>>     MAGENTA = 5
>>     FLUFL = ...       # 6
> 
> The fact that the ellipsis variant requires *more* typing sounds a bit
> wrong to me. The common case should be the easiest to type.

Well, nobody forbids you from writing the integers out.  But one of the
complaints was that you shouldn't need to know or write the constant
values (and "..." is shorter than "val()" which was proposed for that
purpose).

And for stringy enums that argument is void :)

Georg



From ethan at stoneleaf.us  Wed Feb 13 07:38:11 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Feb 2013 22:38:11 -0800
Subject: [Python-ideas] another enum implementation
In-Reply-To: <CAPOVWOSyCk_7sh9KtshxTpM-9HLZNHj-_hEbM2Egyg09K6nv=w@mail.gmail.com>
References: <511B1EE7.4060408@stoneleaf.us>
	<CAPOVWOSyCk_7sh9KtshxTpM-9HLZNHj-_hEbM2Egyg09K6nv=w@mail.gmail.com>
Message-ID: <511B34D3.40009@stoneleaf.us>

On 02/12/2013 10:18 PM, Senthil Kumaran wrote:
> Two comments.
>
> 1. When it is fully generalized, is type is the way to to specify the
> type of enums? Or there will be helper functions or classes that can
> specify the type of enum one desires.

Creating enums with the class method is so easy I don't see myself 
spending time on helper functions.

I'll have to think some more about using

     type = 'unique'
vs.
     class Geometry(UniqueEnum)

although I feel myself leaning towards the UniqueEnum style.


> 2. Personally, I have resorted to using namedtuples for enum like
> behavior. How would this class benefit me? Any edification would be
> helpful.

namedtuples are designed to have many instances with different values in 
the same slots; the Enum class is a singleton, and its instances are the 
values.

The Enum class' instances are derived from either int or str, so they 
can be an easy drop-in replacement to API's that currently expect ints 
or strs.

Better repr()'s.  :)

--
~Ethan~

P.S.
Definitely read the other thread, and no worries.


From solipsis at pitrou.net  Wed Feb 13 08:38:43 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 08:38:43 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org> <20130213082129.5d917b00@pitrou.net>
	<kfffj6$crv$1@ger.gmane.org>
Message-ID: <20130213083843.7dbc7598@pitrou.net>

On Wed, 13 Feb 2013 08:34:21 +0100
Georg Brandl <g.brandl at gmx.net> wrote:
> Am 13.02.2013 08:21, schrieb Antoine Pitrou:
> > On Wed, 13 Feb 2013 00:57:38 +0100
> > Georg Brandl <g.brandl at gmx.net> wrote:
> >> > IMHO, none of these approach the simplicity and explicitness of this API:
> >> > 
> >> > class Color(Enum):
> >> >     RED = 1
> >> >     GREEN = 2
> >> >     BLUE = 3
> >> 
> >> We could even allow
> >> 
> >> class Color(Enum):
> >>     RED = 1
> >>     GREEN = ...       # becomes 2
> >>     BLUE = ...        # 3
> >>     MAGENTA = 5
> >>     FLUFL = ...       # 6
> > 
> > The fact that the ellipsis variant requires *more* typing sounds a bit
> > wrong to me. The common case should be the easiest to type.
> 
> Well, nobody forbids you from writing the integers out.  But one of the
> complaints was that you shouldn't need to know or write the constant
> values (and "..." is shorter than "val()" which was proposed for that
> purpose).

It would still be shorter to type something like:

class Color(Enum):
    values = ('RED', 'GREEN', 'BLUE', 'MAGENTA', 'FLUFL')

> And for stringy enums that argument is void :)

I'm not sure how it's void. What is the proposed syntax for string
enums?

Regards

Antoine.




From brett at python.org  Wed Feb 13 17:26:26 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 13 Feb 2013 11:26:26 -0500
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
Message-ID: <CAP1=2W48dPocVUQqLX-9x3MJY7XHeS8sfF_kxV=Q4q5YoscMcA@mail.gmail.com>

On Tue, Feb 12, 2013 at 11:07 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> On 13 Feb 2013 09:11, "Tim Delaney" <timothy.c.delaney at gmail.com> wrote:
> >
> > On 13 February 2013 09:56, Guido van Rossum <guido at python.org> wrote:
> >>
> >> Frankly, enums are not that useful in small programs. For large
> >> programs or libraries, and especially for public APIs, the extra cost
> >> of defining the enum shouldn't count against them.
> >>
> >> Let's just import Barry's enums into the stdlib.
> >
> >
> > That's entirely your call. FWIW I probably won't use them, as they fail
> to meet my needs for an enum, #1 being not having to specify their values
> in any way if I don't want to. Once you get past 3 or 4 values, it's too
> easy to miss or reuse a value.
>
> What's wrong with enum.make? That just accepts the sequence of names
> namedtuple style, no values specified anywhere.
>
>
> Georg called it ugly, but that's the only complaint I remember reading. I
too find them ugly but not enough to not say this isn't the best solution
considering everyone's complaints of verbosity (e.g. Guido not liking
excessive quoting), magic (e.g. all non-explicit proposals that don't
involve calling a classmethod like val()), and conciseness (e.g. not having
to specify every value).

Heck, if it was me I would just do::

  import types

  def make_enum(keys, *, value=None):
    """For incremental numbers: ``lambda key, index: index``.
    For bitwise flags: ``lambda key, index: 2**index``.

    Could change to take an iterator instead of a lambda and simply
special-case the `None` argument for strings,
    but that seems needlessly limiting when the integer case is probably
for interfacing with some C code.
    """
    if value is None:
      value = lambda key, index: key
    items = {}
    for key, index in enumerate(keys.split()):
      items[key] = value(key, index)
    return types.SimpleNamespace(**items)  # Creating a read-only subclass
is trivial

But I know people will want fancier reprs, globally unique objects, etc.
instead of just some simple way to create an object with attributes
referencing automatically-generated, unique, predictable values (which is
all I would want from an enum).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/abf4c72b/attachment.html>

From solipsis at pitrou.net  Wed Feb 13 17:38:07 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 17:38:07 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
Message-ID: <20130213173807.1465d42b@pitrou.net>

Le Wed, 13 Feb 2013 14:07:48 +1000,
Nick Coghlan <ncoghlan at gmail.com> a
?crit :
> On 13 Feb 2013 09:11, "Tim Delaney"
> <timothy.c.delaney at gmail.com> wrote:
> >
> > On 13 February 2013 09:56, Guido van Rossum
> > <guido at python.org> wrote:
> >>
> >> Frankly, enums are not that useful in small programs. For large
> >> programs or libraries, and especially for public APIs, the extra
> >> cost of defining the enum shouldn't count against them.
> >>
> >> Let's just import Barry's enums into the stdlib.
> >
> >
> > That's entirely your call. FWIW I probably won't use them, as they
> > fail
> to meet my needs for an enum, #1 being not having to specify their
> values in any way if I don't want to. Once you get past 3 or 4
> values, it's too easy to miss or reuse a value.
> 
> What's wrong with enum.make? That just accepts the sequence of names
> namedtuple style, no values specified anywhere.

What's wrong is that TSBOOWTDI. With enum and enum.make, you have two
different idioms for declaring enums, while a single class-based
definition style should suffice.

Regards

Antoine.




From brett at python.org  Wed Feb 13 18:09:19 2013
From: brett at python.org (Brett Cannon)
Date: Wed, 13 Feb 2013 12:09:19 -0500
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130213173807.1465d42b@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
Message-ID: <CAP1=2W5kVKBMqKjy_Z4qS58p4MwA4nNShOLE+tCJ-XuCtZvHNA@mail.gmail.com>

On Wed, Feb 13, 2013 at 11:38 AM, Antoine Pitrou <solipsis at pitrou.net>wrote:

> Le Wed, 13 Feb 2013 14:07:48 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a
> ?crit :
> > On 13 Feb 2013 09:11, "Tim Delaney"
> > <timothy.c.delaney at gmail.com> wrote:
> > >
> > > On 13 February 2013 09:56, Guido van Rossum
> > > <guido at python.org> wrote:
> > >>
> > >> Frankly, enums are not that useful in small programs. For large
> > >> programs or libraries, and especially for public APIs, the extra
> > >> cost of defining the enum shouldn't count against them.
> > >>
> > >> Let's just import Barry's enums into the stdlib.
> > >
> > >
> > > That's entirely your call. FWIW I probably won't use them, as they
> > > fail
> > to meet my needs for an enum, #1 being not having to specify their
> > values in any way if I don't want to. Once you get past 3 or 4
> > values, it's too easy to miss or reuse a value.
> >
> > What's wrong with enum.make? That just accepts the sequence of names
> > namedtuple style, no values specified anywhere.
>
> What's wrong is that TSBOOWTDI. With enum and enum.make, you have two
> different idioms for declaring enums, while a single class-based
> definition style should suffice.
>

Or just a function idiom. We don't have to support the class-based
definition style either.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/ef742cbf/attachment.html>

From ubershmekel at gmail.com  Wed Feb 13 18:33:00 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 13 Feb 2013 19:33:00 +0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP1=2W5kVKBMqKjy_Z4qS58p4MwA4nNShOLE+tCJ-XuCtZvHNA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CAP1=2W5kVKBMqKjy_Z4qS58p4MwA4nNShOLE+tCJ-XuCtZvHNA@mail.gmail.com>
Message-ID: <CANSw7KyLNs5FTiHYkDMYcp_CNCgG-JXhpCEZyGcO-UGKfTC_mg@mail.gmail.com>

On Wed, Feb 13, 2013 at 7:09 PM, Brett Cannon <brett at python.org> wrote:

> Or just a function idiom. We don't have to support the class-based
> definition style either.
>

Well, namedtuple has an enum for tuple indexes. I think we should be
consistent which entails:

Colors = collections.enum('RED GREEN BLUE')


Cheers,

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/9070e52f/attachment.html>

From barry at python.org  Wed Feb 13 19:00:33 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 13:00:33 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
Message-ID: <20130213130033.61cb8aa6@anarchist.wooz.org>

On Feb 13, 2013, at 12:57 AM, Georg Brandl wrote:

>class Color(Enum):
>    RED = 1
>    GREEN = ...       # becomes 2
>    BLUE = ...        # 3
>    MAGENTA = 5
>    FLUFL = ...       # 6

That's pretty cute.

>class Color(FlagEnum):
>    RED = 1
>    GREEN = ...       # becomes 2
>    BLUE = ...        # 4
>    MAGENTA = 16
>    FLUFL = ...
>
>class Color(StringEnum):
>    RED = ...         # becomes 'red'
>    GREEN = ...       # etc.
>    BLUE = ...

I want to think about the latter two (especially the string one), but I've
filed this bug to track the feature.

https://bugs.launchpad.net/flufl.enum/+bug/1124357

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/328355c7/attachment.pgp>

From barry at python.org  Wed Feb 13 19:04:29 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 13:04:29 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<CAH0mxTRH3nKn+=nhOqNZHaH+jX4SuwdLTfL8g+V-oMqCA9Cing@mail.gmail.com>
	<20130212145036.32a98928@anarchist.wooz.org>
	<511ACB63.7030901@pearwood.info>
Message-ID: <20130213130429.6e99dc94@anarchist.wooz.org>

On Feb 13, 2013, at 10:08 AM, Steven D'Aprano wrote:

>>      >>>  from flufl.enum import make
>>      >>>  make('Colors', 'RED GREEN BLUE'.split())
>>      <Colors {RED: 1, GREEN: 2, BLUE: 3}>
>
>That reminds me of namedtuple. I could live with something close to that.
>
>Like namedtuple, it should do the split for you.

Part of the reason for defining make() the way it is, is so that you can use
any iterator in the second argument.  I suppose you could special case
splitting if the second argument is a single string, but I'm not sure this
special case is special enough.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/4efed0d2/attachment.pgp>

From barry at python.org  Wed Feb 13 19:05:17 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 13:05:17 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<CAH0mxTRH3nKn+=nhOqNZHaH+jX4SuwdLTfL8g+V-oMqCA9Cing@mail.gmail.com>
	<20130212145036.32a98928@anarchist.wooz.org>
	<511ACB63.7030901@pearwood.info> <kfektl$r49$2@ger.gmane.org>
Message-ID: <20130213130517.2ac6afc1@anarchist.wooz.org>

On Feb 13, 2013, at 12:59 AM, Georg Brandl wrote:

>>>      >>>  from flufl.enum import make
>>>      >>>  make('Colors', 'RED GREEN BLUE'.split())
>>>      <Colors {RED: 1, GREEN: 2, BLUE: 3}>

>And like namedtuple, it's ugly as hell :)

Suggestions for a better repr are welcome. ;)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/249e9796/attachment.pgp>

From barry at python.org  Wed Feb 13 19:14:03 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 13:14:03 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<20130212220412.4bd3e604@pitrou.net>
	<CAF-Rda-AiL0KQhSF3RoBAKdb8jDeAu4FuFxvyODO7h6S2wY93g@mail.gmail.com>
Message-ID: <20130213131403.3edece68@anarchist.wooz.org>

On Feb 12, 2013, at 02:13 PM, Eli Bendersky wrote:

>I agree, and Tim's implementation provides a very nice string
>representation, no matter what the value underneath is. I don't really care
>what the value underneath is, actually.

Except that sometimes you do, which is why flufl.enum has a nice concise str
and a more detailed repr.  The former doesn't include the underlying value
while the latter does.  Edible cake.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/0419c67a/attachment.pgp>

From barry at python.org  Wed Feb 13 19:19:14 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 13:19:14 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
Message-ID: <20130213131914.5a303f8b@anarchist.wooz.org>

On Feb 13, 2013, at 08:19 AM, Tim Delaney wrote:

>I'm of the opinion that the most fundamental property of an enum is that it
>can generate its values automatically. Everything else is a nice-to-have,
>but without that property it's not much use as an enum.

See, I don't care about that much because I define an enum once, but I *use*
it hundreds of times.  Thus a little more verbosity/explicitness at definition
time is a one-time cost, and a very low one at that.  It might not seems that
way when you're writing them in so many mailing list replies though ;).

In real code, having a really good str and repr, along with good semantics
makes them much more useful for debugging, and very pleasant to use.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/0a9c7a8c/attachment.pgp>

From barry at python.org  Wed Feb 13 19:12:39 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 13:12:39 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<20130212220412.4bd3e604@pitrou.net>
Message-ID: <20130213131239.421aaebd@anarchist.wooz.org>

On Feb 12, 2013, at 10:04 PM, Antoine Pitrou wrote:

>I still hope enum values are strings by default - or, if not, that they
>get nice reprs. Integer enums are only useful for interoperability with
>stubborn low-level libraries :-)

    >>> from flufl.enum import make
    >>> Colors = make('Colors', 'RED GREEN BLUE'.split())
    >>> print(Colors.RED)
    Colors.RED
    >>> Colors.RED
    <EnumValue: Colors.RED [int=1]>

Having the enum name in the str and (admittedly uglier but more informational)
repr is very important to me, because I typically use enums like so:

    from mailman.interfaces.members import DeliveryMode
    ...
    def send_plaintext(member):
        ...
        if member.delivery_mode is DeliveryMode.regular:
            ...
        elif member.delivery_mode is DeliveryMode.mime_digests:
            ...

Now, if something goes wrong, I can set a breakpoint or debug-print right
before the conditional and get the value of `member.delivery_mode`.  The fact
that it will print `DeliveryMode.regular` makes debugging the problem a *much*
better experience.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/2265b97c/attachment.pgp>

From timothy.c.delaney at gmail.com  Wed Feb 13 20:22:11 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Thu, 14 Feb 2013 06:22:11 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130213131403.3edece68@anarchist.wooz.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<20130212220412.4bd3e604@pitrou.net>
	<CAF-Rda-AiL0KQhSF3RoBAKdb8jDeAu4FuFxvyODO7h6S2wY93g@mail.gmail.com>
	<20130213131403.3edece68@anarchist.wooz.org>
Message-ID: <CAN8CLg=xijX2-_o4O5cFrGX8pJMFTy8=r-a_oV_OVyQFa96NPw@mail.gmail.com>

On 14 February 2013 05:14, Barry Warsaw <barry at python.org> wrote:

> On Feb 12, 2013, at 02:13 PM, Eli Bendersky wrote:
>
> >I agree, and Tim's implementation provides a very nice string
> >representation, no matter what the value underneath is. I don't really
> care
> >what the value underneath is, actually.
>
> Except that sometimes you do, which is why flufl.enum has a nice concise
> str
> and a more detailed repr.  The former doesn't include the underlying value
> while the latter does.  Edible cake.
>

Yes - my enum's exactly the same - simple, concise str (for both the enum
and values) and detailed repr. The actual repr is a little different to
flufl.enum repr (I think the str is the same) but conveys the same
information.

Color.RED
<EnumValue 'Color.RED': 0>

{RED:0, GREEN:1, BLUE:2, CYAN:10, MAGENTA:11, YELLOW:12, BLACK:127}
<enum __main__.Color {<EnumValue 'RED': 0>, <EnumValue 'GREEN': 1>,
<EnumValue 'BLUE': 2>, <EnumValue 'CYAN': 10>, <EnumValue 'MAGENTA': 11>,
<EnumValue 'YELLOW': 12>, <EnumValue 'BLACK': 127>}>

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130214/bb40105c/attachment.html>

From timothy.c.delaney at gmail.com  Wed Feb 13 20:28:37 2013
From: timothy.c.delaney at gmail.com (Tim Delaney)
Date: Thu, 14 Feb 2013 06:28:37 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130213131914.5a303f8b@anarchist.wooz.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
	<20130213131914.5a303f8b@anarchist.wooz.org>
Message-ID: <CAN8CLgnTS4-H_7-N=qODn_2PzGWGjo+H8f3qODRJpcHp4T19ig@mail.gmail.com>

On 14 February 2013 05:19, Barry Warsaw <barry at python.org> wrote:

> On Feb 13, 2013, at 08:19 AM, Tim Delaney wrote:
>
> >I'm of the opinion that the most fundamental property of an enum is that
> it
> >can generate its values automatically. Everything else is a nice-to-have,
> >but without that property it's not much use as an enum.
>
> See, I don't care about that much because I define an enum once, but I
> *use*
> it hundreds of times.  Thus a little more verbosity/explicitness at
> definition
> time is a one-time cost, and a very low one at that.  It might not seems
> that
> way when you're writing them in so many mailing list replies though ;).
>

My main concern regarding this is ensuring that:

1. Values don't get repeated.

2. Values don't get skipped (unless explicitly skipped);

#1 obviously is the most important property, but in practice I find #2 to
be very important as well. Whilst it's more verbose, the ... syntax isn't
bad, and I've added support to it to my enum to play with. Currently my
enum implementation includes the kitchen sink, but I'm looking to do some
renovations soon.


> In real code, having a really good str and repr, along with good semantics
> makes them much more useful for debugging, and very pleasant to use.
>

Absolutely. But it's not just that either. If you can convert from the
string name to the enum, it makes the enum suitable as a transport
mechanism. Each end can have different values for the enum in practice so
long as the names don't change - it can make things much more robust across
versions.

Tim Delaney
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130214/dc9b71c4/attachment.html>

From solipsis at pitrou.net  Wed Feb 13 20:30:05 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 20:30:05 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
	<20130213131914.5a303f8b@anarchist.wooz.org>
Message-ID: <20130213203005.7562643e@pitrou.net>

On Wed, 13 Feb 2013 13:19:14 -0500
Barry Warsaw <barry at python.org> wrote:
> On Feb 13, 2013, at 08:19 AM, Tim Delaney wrote:
> 
> >I'm of the opinion that the most fundamental property of an enum is that it
> >can generate its values automatically. Everything else is a nice-to-have,
> >but without that property it's not much use as an enum.
> 
> See, I don't care about that much because I define an enum once, but I *use*
> it hundreds of times.  Thus a little more verbosity/explicitness at definition
> time is a one-time cost, and a very low one at that.

The cost is not low when you have many values. Also, with many values
and one value per line, it can make your declaration very long
vertically.

And it's not always true that you use an enum much more often than you
define it. For example, you may define many error codes (e.g. errnos)
for compatibility with another system, but only check a few of them
explicitly in your application code.

Regards

Antoine.




From solipsis at pitrou.net  Wed Feb 13 20:27:31 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 13 Feb 2013 20:27:31 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<CAH0mxTRH3nKn+=nhOqNZHaH+jX4SuwdLTfL8g+V-oMqCA9Cing@mail.gmail.com>
	<20130212145036.32a98928@anarchist.wooz.org>
	<511ACB63.7030901@pearwood.info> <kfektl$r49$2@ger.gmane.org>
	<20130213130517.2ac6afc1@anarchist.wooz.org>
Message-ID: <20130213202731.6e52c09f@pitrou.net>

On Wed, 13 Feb 2013 13:05:17 -0500
Barry Warsaw <barry at python.org> wrote:
> On Feb 13, 2013, at 12:59 AM, Georg Brandl wrote:
> 
> >>>      >>>  from flufl.enum import make
> >>>      >>>  make('Colors', 'RED GREEN BLUE'.split())
> >>>      <Colors {RED: 1, GREEN: 2, BLUE: 3}>
> 
> >And like namedtuple, it's ugly as hell :)
> 
> Suggestions for a better repr are welcome. ;)

It's not about the repr, it's about the make(...) call.

Regards

Antoine.




From jonathan at slenders.be  Wed Feb 13 21:48:13 2013
From: jonathan at slenders.be (Jonathan Slenders)
Date: Wed, 13 Feb 2013 21:48:13 +0100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130213130033.61cb8aa6@anarchist.wooz.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
	<20130213130033.61cb8aa6@anarchist.wooz.org>
Message-ID: <CAKfyG3wQGf_p8-xxBHDYHLOu108oP7u4mu52MXpfGWdUEEx-6Q@mail.gmail.com>

How is this a bug? It can be implemented in Python 3, I think. Only
Enum.__metaclass__ would not be able to know the order of it's
elements. (But unless you want to do bitwise operations, it's
irrelevant. Is alphabetical order okay?)


2013/2/13 Barry Warsaw <barry at python.org>
>
> On Feb 13, 2013, at 12:57 AM, Georg Brandl wrote:
>
> >class Color(Enum):
> >    RED = 1
> >    GREEN = ...       # becomes 2
> >    BLUE = ...        # 3
> >    MAGENTA = 5
> >    FLUFL = ...       # 6
>
> That's pretty cute.
>
> >class Color(FlagEnum):
> >    RED = 1
> >    GREEN = ...       # becomes 2
> >    BLUE = ...        # 4
> >    MAGENTA = 16
> >    FLUFL = ...
> >
> >class Color(StringEnum):
> >    RED = ...         # becomes 'red'
> >    GREEN = ...       # etc.
> >    BLUE = ...
>
> I want to think about the latter two (especially the string one), but I've
> filed this bug to track the feature.
>
> https://bugs.launchpad.net/flufl.enum/+bug/1124357
>
> Cheers,
> -Barry


From random832 at fastmail.us  Wed Feb 13 22:02:50 2013
From: random832 at fastmail.us (random832 at fastmail.us)
Date: Wed, 13 Feb 2013 16:02:50 -0500
Subject: [Python-ideas] frozenset literals
In-Reply-To: <510D20B6.5040705@pearwood.info>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
Message-ID: <1360789370.24461.140661191087829.377CE510@webmail.messagingengine.com>

On Sat, Feb 2, 2013, at 9:20, Steven D'Aprano wrote:
> Unfortunately the proposal to use f{ ... } for frozen sets cannot work
> within the constraints of Python's lexer:
> 
> http://mail.python.org/pipermail/python-3000/2008-January/011838.html
> 
> Unfortunately we're running out of useful, easy to enter symbols for
> literals. Until such time (Python4000 perhaps, or more likely Python5000)
> as we can use a rich set of Unicode literals, I don't think there is any
> clear way to have a frozenset literal.

I was going to post about not being sure what the objection is (if it's
multiple tokens, let it be multiple tokens - the contents are multiple
tokens anyway - and saying it would block a future syntax extension
doesn't seem like a reasonable objection to a proposed syntax
extension), but I had a new idea so I'll post that instead:

{ as frozenset { ... } }

The sequence "{ as" can't occur (to my knowledge) anywhere now. So, the
thing after it is a keyword in that context (and only that context,
otherwise "frozenset" remains an identifier naming an ordinary builtin)
and specifies what kind of literal the following sequence is. You could
also extend it to alternate forms for some other builtin types - for
example { as bytes [1, 2, 3, 4, 5] } instead of b"\x1\x2\x3\x4\x5".
Or... { as set { } }


From joshua.landau.ws at gmail.com  Wed Feb 13 22:30:36 2013
From: joshua.landau.ws at gmail.com (Joshua Landau)
Date: Wed, 13 Feb 2013 21:30:36 +0000
Subject: [Python-ideas] frozenset literals
In-Reply-To: <1360789370.24461.140661191087829.377CE510@webmail.messagingengine.com>
References: <CAOOU=zgCoXG-aUL4kaPXYyMGdkmnWDn_t8_4GBr5Ni7LB3WrSA@mail.gmail.com>
	<keifi0$ej4$1@ger.gmane.org>
	<CAOOU=zj=SFk70tcr=aNxTTE6+WOSiAffPus-8iLa915+562Z7Q@mail.gmail.com>
	<kej30k$tpi$1@ger.gmane.org> <510D20B6.5040705@pearwood.info>
	<1360789370.24461.140661191087829.377CE510@webmail.messagingengine.com>
Message-ID: <CAN1F8qXu_1CY=e7FLRdCZp4KnjLvCdFoM-cubCiyKE9-FgcQcw@mail.gmail.com>

On 13 February 2013 21:02, <random832 at fastmail.us> wrote:

> On Sat, Feb 2, 2013, at 9:20, Steven D'Aprano wrote:
> > Unfortunately the proposal to use f{ ... } for frozen sets cannot work
> > within the constraints of Python's lexer:
> >
> > http://mail.python.org/pipermail/python-3000/2008-January/011838.html
> >
> > Unfortunately we're running out of useful, easy to enter symbols for
> > literals. Until such time (Python4000 perhaps, or more likely Python5000)
> > as we can use a rich set of Unicode literals, I don't think there is any
> > clear way to have a frozenset literal.
>
> I was going to post about not being sure what the objection is (if it's
> multiple tokens, let it be multiple tokens - the contents are multiple
> tokens anyway - and saying it would block a future syntax extension
> doesn't seem like a reasonable objection to a proposed syntax
> extension), but I had a new idea so I'll post that instead:
>
> { as frozenset { ... } }
>
> The sequence "{ as" can't occur (to my knowledge) anywhere now. So, the
> thing after it is a keyword in that context (and only that context,
> otherwise "frozenset" remains an identifier naming an ordinary builtin)
> and specifies what kind of literal the following sequence is. You could
> also extend it to alternate forms for some other builtin types - for
> example { as bytes [1, 2, 3, 4, 5] } instead of b"\x1\x2\x3\x4\x5".
> Or... { as set { } }


I'm really not sure I like this idea, but surely:

LITERAL as KEYWORD
{ ... } as frozenset
[1, 2, 3, 4, 5] as bytes
{} as set

would work better.

However, I'm not happy on the idea that an identifier can be a keyword in
another context.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/fcf3987e/attachment.html>

From barry at python.org  Wed Feb 13 22:57:53 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 16:57:53 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
	<20130213130033.61cb8aa6@anarchist.wooz.org>
	<CAKfyG3wQGf_p8-xxBHDYHLOu108oP7u4mu52MXpfGWdUEEx-6Q@mail.gmail.com>
Message-ID: <20130213165753.3c5b1a6c@anarchist.wooz.org>

On Feb 13, 2013, at 09:48 PM, Jonathan Slenders wrote:

>How is this a bug?

It's just an item in flufl.enum's bug tracker.  It's marked "wish list" so
it's more correct to call it a feature request.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/82c10635/attachment.pgp>

From barry at python.org  Wed Feb 13 23:00:39 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 17:00:39 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<20130212220412.4bd3e604@pitrou.net>
	<CAF-Rda-AiL0KQhSF3RoBAKdb8jDeAu4FuFxvyODO7h6S2wY93g@mail.gmail.com>
	<20130213131403.3edece68@anarchist.wooz.org>
	<CAN8CLg=xijX2-_o4O5cFrGX8pJMFTy8=r-a_oV_OVyQFa96NPw@mail.gmail.com>
Message-ID: <20130213170039.1507a75b@anarchist.wooz.org>

On Feb 14, 2013, at 06:22 AM, Tim Delaney wrote:

>Yes - my enum's exactly the same - simple, concise str (for both the enum
>and values) and detailed repr. The actual repr is a little different to
>flufl.enum repr (I think the str is the same) but conveys the same
>information.

I like that you came to the same conclusion, namely that the str should be
simple and the repr more detailed.  The exactly color of that detail can be
bikeshedded to death now. :)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/713442c7/attachment.pgp>

From barry at python.org  Wed Feb 13 23:25:49 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 17:25:49 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
	<20130213131914.5a303f8b@anarchist.wooz.org>
	<CAN8CLgnTS4-H_7-N=qODn_2PzGWGjo+H8f3qODRJpcHp4T19ig@mail.gmail.com>
Message-ID: <20130213172549.328dfc00@anarchist.wooz.org>

On Feb 14, 2013, at 06:28 AM, Tim Delaney wrote:

>1. Values don't get repeated.

This is easy, and flufl.enum guarantees this, even for subclasses:

    >>> class Colors(Enum):
    ...    red = 1
    ...    blue = 1
    ... 
    Traceback (most recent call last):
    ...
    TypeError: Multiple enum values: 1
    >>> class Colors(Enum):
    ...    red = 1
    ...    blue = 2
    ... 
    >>> class MoreColors(Colors):
    ...    green = 2
    ... 
    Traceback (most recent call last):
    ...
    TypeError: Multiple enum values: 2

>2. Values don't get skipped (unless explicitly skipped);

Is #2 is the reason why you have different subclasses for flag enums, since
the definition of "skipped" is different?

Many folks have said they don't care about the actual enum values, so for
them, skips don't matter.  Setting aside explicit skips, you can add such
verification with a class decorator, e.g.:

-----snip snip-----
from flufl.enum import Enum

def noskips(cls):
    for i, intval in enumerate(cls, start=1):
        if i != int(intval):
            raise TypeError('Skipped value: {}'.format(i))
    return cls

@noskips
class GoodColors(Enum):
    green = 3
    blue = 1
    red = 2

@noskips
class BadColors(Enum):
    green = 4
    blue = 1
    red = 2

$ python3 noskips.py
Traceback (most recent call last):
...
TypeError: Skipped value: 3
-----snip snip-----

>If you can convert from the string name to the enum, it makes the enum
>suitable as a transport mechanism.

    >>> from flufl.enum import make
    >>> Colors = make('Colors', 'red green blue'.split())
    >>> Colors['red']
    <EnumValue: Colors.red [int=1]>
    >>> Colors[2]
    <EnumValue: Colors.green [int=2]>
    >>> Colors[Colors.green.name]
    <EnumValue: Colors.green [int=2]>

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/d975fc9d/attachment.pgp>

From barry at python.org  Wed Feb 13 23:28:09 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 13 Feb 2013 17:28:09 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
	<20130213131914.5a303f8b@anarchist.wooz.org>
	<20130213203005.7562643e@pitrou.net>
Message-ID: <20130213172809.69932bcb@anarchist.wooz.org>

On Feb 13, 2013, at 08:30 PM, Antoine Pitrou wrote:

>The cost is not low when you have many values. Also, with many values
>and one value per line, it can make your declaration very long
>vertically.
>
>And it's not always true that you use an enum much more often than you
>define it. For example, you may define many error codes (e.g. errnos)
>for compatibility with another system, but only check a few of them
>explicitly in your application code.

Huge enums haven't been common in my experience, but in that case I'd probably
just use the make() helper.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130213/e267b2ca/attachment.pgp>

From ncoghlan at gmail.com  Thu Feb 14 00:11:05 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Feb 2013 09:11:05 +1000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130213173807.1465d42b@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
Message-ID: <CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>

On 14 Feb 2013 02:39, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> Le Wed, 13 Feb 2013 14:07:48 +1000,
> Nick Coghlan <ncoghlan at gmail.com> a
> ?crit :
> > On 13 Feb 2013 09:11, "Tim Delaney"
> > <timothy.c.delaney at gmail.com> wrote:
> > >
> > > On 13 February 2013 09:56, Guido van Rossum
> > > <guido at python.org> wrote:
> > >>
> > >> Frankly, enums are not that useful in small programs. For large
> > >> programs or libraries, and especially for public APIs, the extra
> > >> cost of defining the enum shouldn't count against them.
> > >>
> > >> Let's just import Barry's enums into the stdlib.
> > >
> > >
> > > That's entirely your call. FWIW I probably won't use them, as they
> > > fail
> > to meet my needs for an enum, #1 being not having to specify their
> > values in any way if I don't want to. Once you get past 3 or 4
> > values, it's too easy to miss or reuse a value.
> >
> > What's wrong with enum.make? That just accepts the sequence of names
> > namedtuple style, no values specified anywhere.
>
> What's wrong is that TSBOOWTDI. With enum and enum.make, you have two
> different idioms for declaring enums, while a single class-based
> definition style should suffice.

Really? What happened to the principle of layered API complexity, where a
simple *convenience* function is the obvious way to do it for the cases it
covers, while the full, more powerful, but also more verbose, explicit
subclassing API covers the more complex cases?

Magic namespaces and metaclasses sure don't meet *my* definition of
obvious. A normal subclassing API with a convenience function for simple
cases? That's just good layered API design.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130214/78571548/attachment.html>

From steve at pearwood.info  Thu Feb 14 00:48:23 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 14 Feb 2013 10:48:23 +1100
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130213173807.1465d42b@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
Message-ID: <511C2647.3030602@pearwood.info>

On 14/02/13 03:38, Antoine Pitrou wrote:
> Le Wed, 13 Feb 2013 14:07:48 +1000,
> Nick Coghlan<ncoghlan at gmail.com>  a ?crit :
[...]
>> What's wrong with enum.make? That just accepts the sequence of names
>> namedtuple style, no values specified anywhere.
>
> What's wrong is that TSBOOWTDI. With enum and enum.make, you have two
> different idioms for declaring enums, while a single class-based
> definition style should suffice.

I disagree. enum.make is the more obvious solution, less verbose and IMO nicer looking too. It's explicit that it makes enums, the API is familiar to anyone who has used namedtuple, and it's an expression rather than a statement so it's more flexible. You can't do this with the class-based syntax:

[enum.make(name % i, factory()) for (i, factory) in enumerate(factories)]


Besides, the presence of a second, non-obvious solution is not a violation of One Obvious Way.

I find it amusing that we as a community put so much emphasis on the Zen which ironically includes one of Tim Peter's subtle jokes.

http://bugs.python.org/issue3364

Python is not Perl, but really, there's hardly anything in Python that can't be done two ways if you really try.



-- 
Steven


From p.f.moore at gmail.com  Thu Feb 14 00:54:51 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 13 Feb 2013 23:54:51 +0000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511C2647.3030602@pearwood.info>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<511C2647.3030602@pearwood.info>
Message-ID: <CACac1F80vt760cj=0f8uXTo=BCnJG=9V8bLWWM3QBCcoZXzj8A@mail.gmail.com>

On 13 February 2013 23:48, Steven D'Aprano <steve at pearwood.info> wrote:
> I find it amusing that we as a community put so much emphasis on the Zen
> which ironically includes one of Tim Peter's subtle jokes.
>
> http://bugs.python.org/issue3364

Thanks for that link - I too had missed that joke.

Missing-Tim-even-more-now-ly y'rs
Paul.


From zuo at chopin.edu.pl  Thu Feb 14 01:46:05 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Thu, 14 Feb 2013 01:46:05 +0100
Subject: [Python-ideas] =?utf-8?q?PEP_for_enum_library_type=3F?=
In-Reply-To: <20130213083843.7dbc7598@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
	<20130213082129.5d917b00@pitrou.net> <kfffj6$crv$1@ger.gmane.org>
	<20130213083843.7dbc7598@pitrou.net>
Message-ID: <0579a31de42f8eeb4940d2d3ed2425a0@chopin.edu.pl>

13.02.2013 08:38, Antoine Pitrou wrote:

> It would still be shorter to type something like:
>
> class Color(Enum):
>     values = ('RED', 'GREEN', 'BLUE', 'MAGENTA', 'FLUFL')

Or even in a namedtuple-like manner:

     class Color(Enum):
         values = 'RED, GREEN, BLUE, MAGENTA, FLUFL'

Cheers.
*j



From zuo at chopin.edu.pl  Thu Feb 14 01:57:50 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Thu, 14 Feb 2013 01:57:50 +0100
Subject: [Python-ideas] =?utf-8?q?PEP_for_enum_library_type=3F?=
In-Reply-To: <kfekqs$r49$1@ger.gmane.org>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
Message-ID: <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl>

13.02.2013 00:57, Georg Brandl wrote:

> We could even allow
>
> class Color(Enum):
>     RED = 1
>     GREEN = ...       # becomes 2
>     BLUE = ...        # 3
>     MAGENTA = 5
>     FLUFL = ...       # 6
>
> class Color(FlagEnum):
>     RED = 1
>     GREEN = ...       # becomes 2
>     BLUE = ...        # 4
>     MAGENTA = 16
>     FLUFL = ...
>
> class Color(StringEnum):
>     RED = ...         # becomes 'red'
>     GREEN = ...       # etc.
>     BLUE = ...

It's nice. But what about synonymous items?

     class Color(Enum):
         RED = R = 1
         GREEN = ...
         BLUE = B = ...    # we ment BLUE = B = 3, but effectively
                           # we'll have BLUE = 3 and B = 4  :-|

What about:

(all comments are for explanation only)

     class Color(Enum):
         RED = 1              #  1
         GREEN = +one         #  2
         BLUE = +one          #  3
         YELLOW = Y = +one    #  4
         BLACK = B = 10       # 10
         ORANGE = -one        #  9
         VIOLET = -one        #  8

     class Flag(Enum):
         FOO = 1              #   1
         BAR = +rot           #   2
         BAZ = +rot           #   4
         BOO = +rot           #   8
         SPAM = 256           # 256
         HAM = -rot           # 128
         RAM = -rot           #  64

     class Color(Enum):
         RED = ...            # 'RED'
         GREEN = ...          # 'GREEN'
         BLUE = ...           # 'BLUE'

and maybe also:

     class Color(Enum):
         # 0     1     2     3
         RED, GREEN, BLUE, YELLOW, *end = seq()

     class Color(Enum):
         # 3     4     5     6
         RED, GREEN, BLUE, YELLOW, *end = seq(3)

     class Flag(Enum):
         # 1    2    4    8    16
         FOO, BAR, BAZ, BOO, SPAM, *end = flags()

(yes, it *is* possible to implement it without playing with stack 
frames...)


Cheers.
*j


PS. And now for something completely different: :-)

     @enum
     def Color(v):
         v.RED       # 0
         v.GREEN     # 1
         v.BLUE      # 2
         v.YELLOW    # 3

     @enum
     def Color(v):
         v.RED = 7
         v.GREEN     # 8
         v.BLUE      # 9
         v.YELLOW    # 10



From ryan at ryanhiebert.com  Thu Feb 14 02:58:57 2013
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Wed, 13 Feb 2013 17:58:57 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
	<7958702a0d603cecaeccb8146df38d61@chopin.edu.pl>
Message-ID: <3014408E-0204-4CAE-AAB7-F3CD25A734DE@ryanhiebert.com>

On Feb 13, 2013, at 4:57 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:

> and maybe also:
> 
>    class Color(Enum):
>        # 0     1     2     3
>        RED, GREEN, BLUE, YELLOW, *end = seq()
> 
>    class Color(Enum):
>        # 3     4     5     6
>        RED, GREEN, BLUE, YELLOW, *end = seq(3)
> 
>    class Flag(Enum):
>        # 1    2    4    8    16
>        FOO, BAR, BAZ, BOO, SPAM, *end = flags()
> 
> (yes, it *is* possible to implement it without playing with stack frames...)

This suggestion interests me most. Would it require language changes for that *end stuff?

From ethan at stoneleaf.us  Thu Feb 14 05:50:36 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 13 Feb 2013 20:50:36 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <3014408E-0204-4CAE-AAB7-F3CD25A734DE@ryanhiebert.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
	<7958702a0d603cecaeccb8146df38d61@chopin.edu.pl>
	<3014408E-0204-4CAE-AAB7-F3CD25A734DE@ryanhiebert.com>
Message-ID: <511C6D1C.2090206@stoneleaf.us>

On 02/13/2013 05:58 PM, Ryan Hiebert wrote:
> On Feb 13, 2013, at 4:57 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>
>> and maybe also:
>>
>>     class Color(Enum):
>>         # 0     1     2     3
>>         RED, GREEN, BLUE, YELLOW, *end = seq()
>>
>>     class Color(Enum):
>>         # 3     4     5     6
>>         RED, GREEN, BLUE, YELLOW, *end = seq(3)
>>
>>     class Flag(Enum):
>>         # 1    2    4    8    16
>>         FOO, BAR, BAZ, BOO, SPAM, *end = flags()
>>
>> (yes, it *is* possible to implement it without playing with stack frames...)
>
> This suggestion interests me most. Would it require language changes for that *end stuff?

Already is, since 3.0 I believe.

http://www.python.org/dev/peps/pep-3132/

--
~Ethan~


From ryan at ryanhiebert.com  Thu Feb 14 06:34:53 2013
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Wed, 13 Feb 2013 21:34:53 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511C6D1C.2090206@stoneleaf.us>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
	<7958702a0d603cecaeccb8146df38d61@chopin.edu.pl>
	<3014408E-0204-4CAE-AAB7-F3CD25A734DE@ryanhiebert.com>
	<511C6D1C.2090206@stoneleaf.us>
Message-ID: <4943D1D0-932F-4C14-81BE-7824B178386F@ryanhiebert.com>

On Feb 13, 2013, at 8:50 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 02/13/2013 05:58 PM, Ryan Hiebert wrote:
> 
>> On Feb 13, 2013, at 4:57 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>> 
>>> and maybe also:
>>> 
>>>    class Color(Enum):
>>>        # 0     1     2     3
>>>        RED, GREEN, BLUE, YELLOW, *end = seq()
>>> 
>>>    class Color(Enum):
>>>        # 3     4     5     6
>>>        RED, GREEN, BLUE, YELLOW, *end = seq(3)
>>> 
>>>    class Flag(Enum):
>>>        # 1    2    4    8    16
>>>        FOO, BAR, BAZ, BOO, SPAM, *end = flags()
>>> 
>>> (yes, it *is* possible to implement it without playing with stack frames...)
>> 
>> This suggestion interests me most. Would it require language changes for that *end stuff?
> 
> Already is, since 3.0 I believe.
> 
> http://www.python.org/dev/peps/pep-3132/

Wow, I've been missing out on a great feature. How is it able to avoid the stack frames? If I knew that, I'd love to write up an implementation.

From ncoghlan at gmail.com  Thu Feb 14 07:18:12 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Feb 2013 16:18:12 +1000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511C2647.3030602@pearwood.info>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<511C2647.3030602@pearwood.info>
Message-ID: <CADiSq7eS7r+p-ECg5zZtgrUXEGPJpWe+h94x73xdM9JpuR-tPw@mail.gmail.com>

On Thu, Feb 14, 2013 at 9:48 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Besides, the presence of a second, non-obvious solution is not a violation
> of One Obvious Way.

Something that is often forgotten is that having two ways to do
something is often *good* for your API design, because it lets you
design one simple API that covers a large fraction of use cases, and
then a more complex underlying API that covers all (or almost all) of
the rest.

A procedural wrapper around an object-oriented core is the classic
means of achieving this, and the standard library does it all over the
place (sometimes we don't publish the OO core, but the option is there
if our users demand the extra flexibility).

The trick to doing it well is to make sure that users are aware that
when the simple API is sufficient, that's the one they should use.
Only when that API is inadequate should they reach for the more
complex one. It's very, very easy to fall into the trap of documenting
the comprehensive core first, and then saying "oh, by the way, here's
this convenient helper function that means you will probably never
need to worry about all that underlying complexity".

The subprocess docs used to fall into that trap: call, check_call and
check_output cover many use cases, with Popen as the complex fallback,
but the old docs hit readers in the face with Popen, and only
mentioned the helper functions as an afterthought. The new docs have
the helper functions front and centre, with Popen relegated to "if you
really need it" status.

The flufl.enum docs *do* currently fall into the trap of treating
enum.make as an afterthought rather than as the preferred creation API
for the cases that it can handle, though.

Armin Ronacher has a lovely elaboration of this principle here:
http://lucumr.pocoo.org/2013/2/13/moar-classes/ (although he's mostly
complaining about the other direction, simple procedural APIs which
*don't* expose a flexible OO core, rather than the one here where an
OO core is exposed ).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From solipsis at pitrou.net  Thu Feb 14 08:35:58 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Feb 2013 08:35:58 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>
Message-ID: <20130214083558.77dc3424@pitrou.net>

On Thu, 14 Feb 2013 09:11:05 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 14 Feb 2013 02:39, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
> > >
> > > What's wrong with enum.make? That just accepts the sequence of names
> > > namedtuple style, no values specified anywhere.
> >
> > What's wrong is that TSBOOWTDI. With enum and enum.make, you have two
> > different idioms for declaring enums, while a single class-based
> > definition style should suffice.
> 
> Really? What happened to the principle of layered API complexity, where a
> simple *convenience* function is the obvious way to do it for the cases it
> covers, while the full, more powerful, but also more verbose, explicit
> subclassing API covers the more complex cases?

Yes, what happened to it?
Do you think enums are an important enough feature to warrant a
"layered API"? Why stop at 2 layers?

And why do you think subclassing is actually a complex API? It's
easily understood by any OO programmer out there, even non-Python
experts. Why do you think Python ORMs choose a subclassing API instead
of your so-called "obvious way to do it using a convenience function"?

Hint: nobody would find the convenience function API as obvious as the
subclassing API.

> Magic namespaces and metaclasses sure don't meet *my* definition of
> obvious.

Perhaps, but Python is not a supermarket where you choose your own brand
of obvious as you like. It's a programming language striving to
provide a rather consistent experience. People with bizarre tastes can
always publish their packages on PyPI.

Regards

Antoine.




From solipsis at pitrou.net  Thu Feb 14 10:19:52 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Feb 2013 10:19:52 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<CAP7+vJKK7sEqku+UzN_Ni-Z5s7O2tDRn7TtQe8pMqP9yOXM79A@mail.gmail.com>
	<511A8A82.4010602@stoneleaf.us>
	<CAP7+vJKywiMD6e+qiN1FHNkx6AjsRu043iUpboWWhC1UoA5PgA@mail.gmail.com>
	<CAN8CLgkDeHrrXwOXuRDcjaK1ERRLhKc8fYrovP5TvZYquQa3YA@mail.gmail.com>
	<CAP7+vJL9rvVnPwh4=aa-mCoOBO+sN=rYG-XVOP=g+_JDVO8iYQ@mail.gmail.com>
	<CAF-Rda9qCwNcLZV_JsF4ZKGSJPv_uSHEJ=3FmZMT1U6NtuCxfA@mail.gmail.com>
	<CAN8CLgm90fHo7L5_8H8BE6rvRO1UaB_sfp_oboNF0Z_OtLSyNw@mail.gmail.com>
	<20130213131914.5a303f8b@anarchist.wooz.org>
	<20130213203005.7562643e@pitrou.net>
	<20130213172809.69932bcb@anarchist.wooz.org>
Message-ID: <20130214101952.5534b471@pitrou.net>

Le Wed, 13 Feb 2013 17:28:09 -0500,
Barry Warsaw <barry at python.org> a ?crit :
> On Feb 13, 2013, at 08:30 PM, Antoine Pitrou wrote:
> 
> >The cost is not low when you have many values. Also, with many values
> >and one value per line, it can make your declaration very long
> >vertically.
> >
> >And it's not always true that you use an enum much more often than
> >you define it. For example, you may define many error codes (e.g.
> >errnos) for compatibility with another system, but only check a few
> >of them explicitly in your application code.
> 
> Huge enums haven't been common in my experience, but in that case I'd
> probably just use the make() helper.

One of my common use cases for wanting an enum has always been to map a
third-party protocol or library's error codes.

Really, the one screaming use case in the stdlib is in the errno
module :-) For that you have to be able to define an enum that
subclasses int, though.

Regards

Antoine.




From ncoghlan at gmail.com  Thu Feb 14 11:40:37 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Feb 2013 20:40:37 +1000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130214083558.77dc3424@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>
	<20130214083558.77dc3424@pitrou.net>
Message-ID: <CADiSq7faSd0We4xnMPKPN7mDmvwSnnHKtc1gjKsiLJ9qZL=ATg@mail.gmail.com>

On Thu, Feb 14, 2013 at 5:35 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Hint: nobody would find the convenience function API as obvious as the
> subclassing API.

# Status quo
one, two, three, four = range(4)

# flufl.enum.make -> types.new_enum
# Factory function for classes
# For when you don't care about the specific values
# Close parallel to collections.namedtuple
OneToFour = types.new_enum("OneToFour", "one two three four")

# flufl.enum.Enum -> types.Enum
# For when you do care about (some) specific values
class OneToFourAndTen(OneToFour):
    ten = 10

collections.namedtuple is the normative example here, and while the
repetition of the class name is still annoying, that's a minor
irritation compared to completely violating everyone's expectations of
normal class behaviour.

>> Magic namespaces and metaclasses sure don't meet *my* definition of
>> obvious.
>
> Perhaps, but Python is not a supermarket where you choose your own brand
> of obvious as you like. It's a programming language striving to
> provide a rather consistent experience. People with bizarre tastes can
> always publish their packages on PyPI.

In which case, they should stop discussing them on python-ideas.
flufl.enum is a perfectly fine example of normal Pythonic class
design, with similar examples already in the standard library. The
approaches being thrown around on this list lately are very cool from
a technical point of view, and impressive illustrations of what
Python's metaclass machinery can do for you, but they're also
completely unintuitive black magic (including "x = ..." having any
result other than "x is Ellipsis").

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From solipsis at pitrou.net  Thu Feb 14 12:16:02 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Feb 2013 12:16:02 +0100
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>
	<20130214083558.77dc3424@pitrou.net>
	<CADiSq7faSd0We4xnMPKPN7mDmvwSnnHKtc1gjKsiLJ9qZL=ATg@mail.gmail.com>
Message-ID: <20130214121602.3b89e23e@pitrou.net>

Le Thu, 14 Feb 2013 20:40:37 +1000,
Nick Coghlan <ncoghlan at gmail.com> a
?crit :
> 
> collections.namedtuple is the normative example here, and while the
> repetition of the class name is still annoying, that's a minor
> irritation compared to completely violating everyone's expectations of
> normal class behaviour.

I find it slightly amusing that you're complaining about a violation
one of your latest PEPs is trying to make easier to make.

But, still, you are not convincing me that namedtuple is normative in
any way. The fact that many people would prefer a class-based
declarative syntax for namedtuple is proof.

By the way, not only is the repetition of the class name annoying, but
it also makes subclassing more annoying too. Most of my uses of
namedtuple imply subclassing, because I add some behaviour (for
example additional constructors or serializers).

So, compare:

_BaseProtocolItem = namedtuple('_BaseProtocolItem',
                               ('serial_no', 'command'))

class ProtocolItem(_BaseProtocolItem):
    # snip custom methods


To the hypothetical:

class ProtocolItem(NamedTuple):
    __fields__ = ('serial_no', 'command')

    # snip custom methods


And you understand why the namedtuple() factory function is really an
inferior solution with too much typing.

> In which case, they should stop discussing them on python-ideas.
> flufl.enum is a perfectly fine example of normal Pythonic class
> design, with similar examples already in the standard library. The
> approaches being thrown around on this list lately are very cool from
> a technical point of view, and impressive illustrations of what
> Python's metaclass machinery can do for you, but they're also
> completely unintuitive black magic (including "x = ..." having any
> result other than "x is Ellipsis").

You're still ignoring the less-magic solutions such as:

class Errno(Enum):
    __enumvalues__ = ('EBADF', 'ENOENT',) 


Regards

Antoine.




From shibturn at gmail.com  Thu Feb 14 12:34:23 2013
From: shibturn at gmail.com (Richard Oudkerk)
Date: Thu, 14 Feb 2013 11:34:23 +0000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<kfekqs$r49$1@ger.gmane.org>
	<7958702a0d603cecaeccb8146df38d61@chopin.edu.pl>
Message-ID: <kfii3u$u01$1@ger.gmane.org>

On 14/02/2013 12:57am, Jan Kaliszewski wrote:
> and maybe also:
>
>      class Color(Enum):
>          # 0     1     2     3
>          RED, GREEN, BLUE, YELLOW, *end = seq()
>
>      class Color(Enum):
>          # 3     4     5     6
>          RED, GREEN, BLUE, YELLOW, *end = seq(3)
>
>      class Flag(Enum):
>          # 1    2    4    8    16
>          FOO, BAR, BAZ, BOO, SPAM, *end = flags()
>
> (yes, it *is* possible to implement it without playing with stack
> frames...)

Are seq and flags infinite generators/iterators?  If so then this won't 
work because end will be a *list* containing the contents of the tail of 
the iterator.

 >>> def gen():
...   i = 0
...   while True:
...     yield i
...     i += 1
...
 >>> a, b, *end = gen()
<Hangs, consuming more and more memory>

-- 
Richard



From ncoghlan at gmail.com  Thu Feb 14 14:02:44 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Feb 2013 23:02:44 +1000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <20130214121602.3b89e23e@pitrou.net>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>
	<20130214083558.77dc3424@pitrou.net>
	<CADiSq7faSd0We4xnMPKPN7mDmvwSnnHKtc1gjKsiLJ9qZL=ATg@mail.gmail.com>
	<20130214121602.3b89e23e@pitrou.net>
Message-ID: <CADiSq7eooBM3_0nvT5BqomEwqb-fJ+2UqWU=zzgzzE9E34B8MA@mail.gmail.com>

On Thu, Feb 14, 2013 at 9:16 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> You're still ignoring the less-magic solutions such as:
>
> class Errno(Enum):
>     __enumvalues__ = ('EBADF', 'ENOENT',)

Yes, an approach like that, or anything similarly explicit would be
fine with me (and has the virtue of potentially supporting pickle,
unlike anonymous class generators like the current namedtuple
implementation). It was the clever insanity of some of the other ideas
being kicked around in these threads that I wanted to be clear I would
fight tooth and nail if they were ever actually proposed for stdlib
inclusion.

My enthusiasm for flufl.enum in particular is mainly of the "I'm tired
of seeing almost the exact same enum discussion get rehashed every
couple of years, and that one looks good enough to me" variety, so it
probably isn't wise to pay too much attention to me when it comes to
the specific of the API :P

Your comments did give me an idea about namedtuple though - I'll start
another thread about that.

> I find it slightly amusing that you're complaining about a violation
> one of your latest PEPs is trying to make easier to make.

[Replying to this second, since it is getting into philosophy of
language design, and I actually agree with your key point above]

I have a long history of trying to demystify various aspects of Python
metaprogramming and make it easier for people to explore the language,
and create suitable meta-frameworks for their own use, and for use
within particular subcommunities.

As I see it, violating normal expectations for class behaviour *within
the context of a framework* is fine, *so long as you can't achieve
reasonable syntax in a simpler way*. Environments like Django, SQL
Alchemy, etc set up new norms about expected behaviour in different
kinds of classes, and that's the entire reason I think these
metaprograming tools are worth providing in the first place (although
I acknowledge it *can* cause problems when people learn one of these
frameworks *first*, and then later need to be taught the difference
between the automatically provided framework level behaviour for
Forms, Models, Views, etc and the underlying standard language
behaviour).

While the standard library already provides three distinct class level
behaviours (normal classes, subclasses of type and ABCs), those are
each intertwined deeply with the core type system (arguably,
inheriting from builtins other than type or object counts as an
additional case, since inheriting from a C defined class that isn't
really designed for it has interesting consequences in CPython due to
implementation limitations). Furthermore, while our variants change
the way instantiation, descriptors, isinstance and issubclass behave
after the class is created, they *don't* alter the runtime semantics
of the class body itself.

To my mind, defining enumerations doesn't even come close to reaching
the bar that would justify highly customised class behaviour that a
reader can't guess simply from knowing Python and reading the code (I
get the impression you already agree with me on this point, which is
why I moved this philosophical section after my agreement with you
above).

Cheers,
Nick

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From ncoghlan at gmail.com  Thu Feb 14 14:19:51 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Feb 2013 23:19:51 +1000
Subject: [Python-ideas] A subclassing API for named tuples?
Message-ID: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>

An exchange with Antoine in one of the enum threads sparked a thought.

A recurring suggestion for collections.namedtuple is that it would be
nice to be able to define them like this (as it not only avoids having
to repeat the class name, but also allows them to play nicely with
pickle and other name-based reference mechanisms):

    class MyTuple(collections.NamedTuple):
        __fields__ = "a b c d e".split()

However, one of Raymond's long standing objections to such a design
for namedtuple is the ugliness of people having to remember to include
the right __slots__ definition to ensure it doesn't add any storage
overhead above and beyond that for the underlying tuple.

For the intended use case as a replacement for short tuples, an unused
dict per instance is a *big* wasted overhead, so that concern can't be
dismissed as premature optimisation:

>>> import sys
>>> class Slots(tuple): __slots__ = ()
...
>>> class InstanceDict(tuple): pass
...
>>> sys.getsizeof(tuple([1, 2, 3]))
72
>>> x = Slots([1, 2, 3])
>>> sys.getsizeof(x)
72
>>> y = InstanceDict([1, 2, 3])
>>> sys.getsizeof(y)  # All good, right?
72
>>> sys.getsizeof(y.__dict__) # Yeah, not so much...
96

However, the thought that occurred to me is that the right metaclass
definition allows the default behaviour of __slots__ to be flipped, so
that you get "__slots__ = ()" defined in your class namespace
automatically, and you have to write "del __slots__" to get normal
class behaviour back:

>>> class SlotsMeta(type):
...     def __prepare__(cls, *args, **kwds):
...         return dict(__slots__=())
...
>>> class SlotsByDefault(metaclass = SlotsMeta): pass
...
>>> class Slots(tuple, SlotsByDefault): pass
...
>>> class InstanceDict(tuple, SlotsByDefault): del __slots__
...
>>> sys.getsizeof(Slots([1, 2, 3]))
72
>>> Slots().__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Slots' object has no attribute '__dict__'
>>> sys.getsizeof(InstanceDict([1, 2, 3]))
72
>>> sys.getsizeof(InstanceDict([1, 2, 3]).__dict__)
96

So, what do people think? Too much magic? Or just the right amount to
allow a cleaner syntax for named tuple definitions, without
inadvertently encouraging people to do bad things to their memory
usage? (Note: for backwards compatibility reasons, we couldn't use a
custom metaclass for the classes returned by the existing collections
namedtuple API. However, we could certainly provide a distinct
collections.NamedTuple type which used a custom metaclass to behave
this way).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From solipsis at pitrou.net  Thu Feb 14 15:56:55 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Feb 2013 15:56:55 +0100
Subject: [Python-ideas] A subclassing API for named tuples?
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
Message-ID: <20130214155655.5cc3d582@pitrou.net>

Le Thu, 14 Feb 2013 23:19:51 +1000,
Nick Coghlan <ncoghlan at gmail.com> a
?crit :
> An exchange with Antoine in one of the enum threads sparked a thought.
> 
> A recurring suggestion for collections.namedtuple is that it would be
> nice to be able to define them like this (as it not only avoids having
> to repeat the class name, but also allows them to play nicely with
> pickle and other name-based reference mechanisms):
> 
>     class MyTuple(collections.NamedTuple):
>         __fields__ = "a b c d e".split()
> 
> However, one of Raymond's long standing objections to such a design
> for namedtuple is the ugliness of people having to remember to include
> the right __slots__ definition to ensure it doesn't add any storage
> overhead above and beyond that for the underlying tuple.

You don't *have* to remember to include it. You just have to include it
if you really care about the memory footprint. Which is something that
is easy to mention in the documentation.
(also, an empty dict that never gets accessed is not much of a
problem performance-wise)

Speaking for myself, most uses of namedtuple are not
performance-critical. They are for convenience: I want an immutable,
hashable, comparable record-like class (with a useful definition of
equality, which is also very convenient for unit tests :-)) and I don't
want to write that behaviour by hand every time.

Regards

Antoine.




From ericsnowcurrently at gmail.com  Thu Feb 14 16:32:49 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 14 Feb 2013 08:32:49 -0700
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
Message-ID: <CALFfu7CDbBzjykSg9dK9zxZTvt9HEMPwYx03xY90L-s7nJ0UrQ@mail.gmail.com>

On Feb 14, 2013 6:20 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
> A recurring suggestion for collections.namedtuple is that it would be
> nice to be able to define them like this (as it not only avoids having
> to repeat the class name, but also allows them to play nicely with
> pickle and other name-based reference mechanisms):
>
>     class MyTuple(collections.NamedTuple):
>         __fields__ = "a b c d e".split()

Something similar that I've been using:

@as_namedtuple("a b c d e")
class MyTuple:
    """My namedtuple with extra stuff."""
    @property
    def something_special(self):
        """..."""
        return ...

Support for default values is also something I've added, but that's
relatively orthogonal to this discussion.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130214/a29fb103/attachment.html>

From solipsis at pitrou.net  Thu Feb 14 16:42:53 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Feb 2013 16:42:53 +0100
Subject: [Python-ideas] A subclassing API for named tuples?
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<CALFfu7CDbBzjykSg9dK9zxZTvt9HEMPwYx03xY90L-s7nJ0UrQ@mail.gmail.com>
Message-ID: <20130214164253.0c4198b9@pitrou.net>

Le Thu, 14 Feb 2013 08:32:49 -0700,
Eric Snow <ericsnowcurrently at gmail.com> a
?crit :
> 
> Something similar that I've been using:
> 
> @as_namedtuple("a b c d e")
> class MyTuple:
>     """My namedtuple with extra stuff."""
>     @property
>     def something_special(self):
>         """..."""
>         return ...
> 
> Support for default values is also something I've added, but that's
> relatively orthogonal to this discussion.

Ah, right. That's also why I often need to subclass namedtuple.

Regards

Antoine.




From steve at pearwood.info  Thu Feb 14 23:09:57 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Feb 2013 09:09:57 +1100
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
Message-ID: <511D60B5.7000903@pearwood.info>

On 15/02/13 00:19, Nick Coghlan wrote:
> An exchange with Antoine in one of the enum threads sparked a thought.
>
> A recurring suggestion for collections.namedtuple is that it would be
> nice to be able to define them like this (as it not only avoids having
> to repeat the class name, but also allows them to play nicely with
> pickle and other name-based reference mechanisms):
>
>      class MyTuple(collections.NamedTuple):
>          __fields__ = "a b c d e".split()


How would that differ from this?

class MyTuple(collections.namedtuple("MyTupleParent", "a b c d e")):
     pass


Apart from the DRY violation in the class name, I find that perfectly
acceptable, and it seems to work fine with pickling:


py> t = MyTuple(2, 4, 8, 16, 32)
py> t
MyTuple(a=2, b=4, c=8, d=16, e=32)
py> s = pickle.dumps(t)
py> u = pickle.loads(s)
py> u == t
True




-- 
Steven


From greg.ewing at canterbury.ac.nz  Thu Feb 14 23:31:47 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 15 Feb 2013 11:31:47 +1300
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
Message-ID: <511D65D3.9000707@canterbury.ac.nz>

Nick Coghlan wrote:
 >
>     class MyTuple(collections.NamedTuple):
>         __fields__ = "a b c d e".split()
> 
> However, one of Raymond's long standing objections to such a design
> for namedtuple is the ugliness of people having to remember to include
> the right __slots__ definition to ensure it doesn't add any storage
> overhead above and beyond that for the underlying tuple.

So why not kill two definitions with one stone and spell
it like this:

      class MyTuple(collections.NamedTuple):
          __slots__ = "a b c d e".split()

-- 
Greg




From solipsis at pitrou.net  Thu Feb 14 23:40:08 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 14 Feb 2013 23:40:08 +0100
Subject: [Python-ideas] A subclassing API for named tuples?
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D60B5.7000903@pearwood.info>
Message-ID: <20130214234008.17eb7ed0@pitrou.net>

On Fri, 15 Feb 2013 09:09:57 +1100
Steven D'Aprano <steve at pearwood.info> wrote:

> On 15/02/13 00:19, Nick Coghlan wrote:
> > An exchange with Antoine in one of the enum threads sparked a thought.
> >
> > A recurring suggestion for collections.namedtuple is that it would be
> > nice to be able to define them like this (as it not only avoids having
> > to repeat the class name, but also allows them to play nicely with
> > pickle and other name-based reference mechanisms):
> >
> >      class MyTuple(collections.NamedTuple):
> >          __fields__ = "a b c d e".split()
> 
> 
> How would that differ from this?
> 
> class MyTuple(collections.namedtuple("MyTupleParent", "a b c d e")):
>      pass
> 
> 
> Apart from the DRY violation in the class name, I find that perfectly
> acceptable, and it seems to work fine with pickling:

Well, it's perfectly acceptable as long as you have one-letter field
names. Try with real field names and it becomes rather unwieldy.

Regards

Antoine.




From steve at pearwood.info  Fri Feb 15 00:17:32 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Feb 2013 10:17:32 +1100
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <20130214234008.17eb7ed0@pitrou.net>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D60B5.7000903@pearwood.info>
	<20130214234008.17eb7ed0@pitrou.net>
Message-ID: <511D708C.9030208@pearwood.info>

On 15/02/13 09:40, Antoine Pitrou wrote:
> On Fri, 15 Feb 2013 09:09:57 +1100
> Steven D'Aprano<steve at pearwood.info>  wrote:
>
>> On 15/02/13 00:19, Nick Coghlan wrote:
>>> An exchange with Antoine in one of the enum threads sparked a thought.
>>>
>>> A recurring suggestion for collections.namedtuple is that it would be
>>> nice to be able to define them like this (as it not only avoids having
>>> to repeat the class name, but also allows them to play nicely with
>>> pickle and other name-based reference mechanisms):
>>>
>>>       class MyTuple(collections.NamedTuple):
>>>           __fields__ = "a b c d e".split()
>>
>>
>> How would that differ from this?
>>
>> class MyTuple(collections.namedtuple("MyTupleParent", "a b c d e")):
>>       pass
>>
>>
>> Apart from the DRY violation in the class name, I find that perfectly
>> acceptable, and it seems to work fine with pickling:
>
> Well, it's perfectly acceptable as long as you have one-letter field
> names. Try with real field names and it becomes rather unwieldy.

Exactly the same thing can be said about the __field__ line. The only difference is that in one case you reach "unwieldy" a little sooner than in the other.

There are well-known ways to deal with excessively long lines of code which don't require a new collection type.


class MyClassWithAnExtremelyLongName(
         collections.namedtuple("Cheese",
          ("cheddar swiss ricotta camembert gouda parmesan brie limburger havarti"
           " danish_blue roquefort greek_feta provolone mozzarella edam maasdam"
           " stilton wensleydale red_leicester american colby monterey_jack kapiti"
           " casu_marzu")
          )
         ):
     pass


Style arguments about the placement of closing brackets to /dev/null :-)


I'm simply not seeing enough benefit to NamedTuple to make up for the invariable confusion between NamedTuple and namedtuple.




-- 
Steven


From zuo at chopin.edu.pl  Fri Feb 15 00:33:57 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Fri, 15 Feb 2013 00:33:57 +0100
Subject: [Python-ideas] =?utf-8?q?A_subclassing_API_for_named_tuples=3F?=
In-Reply-To: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
Message-ID: <05fea7e04f1550e3a91071e9f3c72281@chopin.edu.pl>

Please remember that in the Py3k docs is a link to my namedtuple.abc 
recipe which covers most of the issues discussed in this thread (+ ABC 
registration/isinstance/issubclass checking).

Cheers.
*j

PS. If we wanted to have the "automatic-always-__slots__" feature 
(discussed in this thread), we probably need to dedent line #57 of the 
recipe one indentation level left...

PPS. As for Py3.3, the recipe is outdateed a bit as it uses deprecated 
@abstractproperty -- but it can be easily fixed.



From jbvsmo at gmail.com  Fri Feb 15 02:54:43 2013
From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=)
Date: Thu, 14 Feb 2013 23:54:43 -0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CADiSq7eooBM3_0nvT5BqomEwqb-fJ+2UqWU=zzgzzE9E34B8MA@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>
	<20130214083558.77dc3424@pitrou.net>
	<CADiSq7faSd0We4xnMPKPN7mDmvwSnnHKtc1gjKsiLJ9qZL=ATg@mail.gmail.com>
	<20130214121602.3b89e23e@pitrou.net>
	<CADiSq7eooBM3_0nvT5BqomEwqb-fJ+2UqWU=zzgzzE9E34B8MA@mail.gmail.com>
Message-ID: <CAOyAWgj5qY6VbUWXm7BQdYokjC-3u+nk7VuhxQcjB3mkf2JuVg@mail.gmail.com>

And now for something completely different:

If the Enums are used with attribute syntax, why not this?

>>> Enum.Colors.red.green.blue
<Enum: Colors>

>>> Enum.Colors.red
<Colors.red = 0>
>>> Enum.Colors.green
<Colors.green = 1>
>>> Enum.Colors.blue
<Colors.blue = 2>

This is my 5-minute implementation
<https://gist.github.com/jbvsmo/4958012>(quite fun)

Looks like everybody is fighting against the class syntax and it is very
incompatible with the notion of enum we've got from other languages.
BTW, I'm a big fan of flufl.enum


Jo?o Bernardo


2013/2/14 Nick Coghlan <ncoghlan at gmail.com>

> On Thu, Feb 14, 2013 at 9:16 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
>
> > You're still ignoring the less-magic solutions such as:
> >
> > class Errno(Enum):
> >     __enumvalues__ = ('EBADF', 'ENOENT',)
>
> Yes, an approach like that, or anything similarly explicit would be
> fine with me (and has the virtue of potentially supporting pickle,
> unlike anonymous class generators like the current namedtuple
> implementation). It was the clever insanity of some of the other ideas
> being kicked around in these threads that I wanted to be clear I would
> fight tooth and nail if they were ever actually proposed for stdlib
> inclusion.
>
> My enthusiasm for flufl.enum in particular is mainly of the "I'm tired
> of seeing almost the exact same enum discussion get rehashed every
> couple of years, and that one looks good enough to me" variety, so it
> probably isn't wise to pay too much attention to me when it comes to
> the specific of the API :P
>
> Your comments did give me an idea about namedtuple though - I'll start
> another thread about that.
>
> > I find it slightly amusing that you're complaining about a violation
> > one of your latest PEPs is trying to make easier to make.
>
> [Replying to this second, since it is getting into philosophy of
> language design, and I actually agree with your key point above]
>
> I have a long history of trying to demystify various aspects of Python
> metaprogramming and make it easier for people to explore the language,
> and create suitable meta-frameworks for their own use, and for use
> within particular subcommunities.
>
> As I see it, violating normal expectations for class behaviour *within
> the context of a framework* is fine, *so long as you can't achieve
> reasonable syntax in a simpler way*. Environments like Django, SQL
> Alchemy, etc set up new norms about expected behaviour in different
> kinds of classes, and that's the entire reason I think these
> metaprograming tools are worth providing in the first place (although
> I acknowledge it *can* cause problems when people learn one of these
> frameworks *first*, and then later need to be taught the difference
> between the automatically provided framework level behaviour for
> Forms, Models, Views, etc and the underlying standard language
> behaviour).
>
> While the standard library already provides three distinct class level
> behaviours (normal classes, subclasses of type and ABCs), those are
> each intertwined deeply with the core type system (arguably,
> inheriting from builtins other than type or object counts as an
> additional case, since inheriting from a C defined class that isn't
> really designed for it has interesting consequences in CPython due to
> implementation limitations). Furthermore, while our variants change
> the way instantiation, descriptors, isinstance and issubclass behave
> after the class is created, they *don't* alter the runtime semantics
> of the class body itself.
>
> To my mind, defining enumerations doesn't even come close to reaching
> the bar that would justify highly customised class behaviour that a
> reader can't guess simply from knowing Python and reading the code (I
> get the impression you already agree with me on this point, which is
> why I moved this philosophical section after my agreement with you
> above).
>
> Cheers,
> Nick
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130214/b15818c3/attachment.html>

From larry at hastings.org  Fri Feb 15 03:11:38 2013
From: larry at hastings.org (Larry Hastings)
Date: Thu, 14 Feb 2013 18:11:38 -0800
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
Message-ID: <511D995A.8000008@hastings.org>

On 02/14/2013 05:19 AM, Nick Coghlan wrote:
> A recurring suggestion for collections.namedtuple is that it would be
> nice to be able to define them like this [...]

FWIW, I think namedtuple is overused.  Not that there's anything 
innately wrong with namedtuple; it's just that I think too many types 
are iterable which shouldn't be.  (Nobody unpacks the stat_result 
anymore, that's like 1995 man.)  I suggest people use 
types.SimpleNamespace unless iterability is specifically required.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130214/98e33dd7/attachment.html>

From barry at python.org  Fri Feb 15 03:57:47 2013
From: barry at python.org (Barry Warsaw)
Date: Thu, 14 Feb 2013 21:57:47 -0500
Subject: [Python-ideas] PEP for enum library type?
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<511C2647.3030602@pearwood.info>
	<CADiSq7eS7r+p-ECg5zZtgrUXEGPJpWe+h94x73xdM9JpuR-tPw@mail.gmail.com>
Message-ID: <20130214215747.7af1185d@anarchist.wooz.org>

On Feb 14, 2013, at 04:18 PM, Nick Coghlan wrote:

>The flufl.enum docs *do* currently fall into the trap of treating
>enum.make as an afterthought rather than as the preferred creation API
>for the cases that it can handle, though.

I guess that's because .make() *was* an afterthought. ;)

.make() was added long after the class syntax.  IIRC, Michael Foord suggested
that API to allow creation of enums programmatically instead of statically,
which the class syntax requires, modulo exec().  I suppose I've never really
thought of .make() as a convenience because of that, and because I personally
prefer the class syntax, all things being equal.  It's interesting that other
folks view .make()'s primary benefit as being one of convenience.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130214/81e6fd76/attachment.pgp>

From greg.ewing at canterbury.ac.nz  Fri Feb 15 06:37:47 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 15 Feb 2013 18:37:47 +1300
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAOyAWgj5qY6VbUWXm7BQdYokjC-3u+nk7VuhxQcjB3mkf2JuVg@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>
	<20130214083558.77dc3424@pitrou.net>
	<CADiSq7faSd0We4xnMPKPN7mDmvwSnnHKtc1gjKsiLJ9qZL=ATg@mail.gmail.com>
	<20130214121602.3b89e23e@pitrou.net>
	<CADiSq7eooBM3_0nvT5BqomEwqb-fJ+2UqWU=zzgzzE9E34B8MA@mail.gmail.com>
	<CAOyAWgj5qY6VbUWXm7BQdYokjC-3u+nk7VuhxQcjB3mkf2JuVg@mail.gmail.com>
Message-ID: <511DC9AB.9060906@canterbury.ac.nz>

Jo?o Bernardo wrote:
> And now for something completely different:
> 
> If the Enums are used with attribute syntax, why not this?
> 
>> >> Enum.Colors.red.green.blue
> 
> <Enum: Colors>

Interesting idea, but there seem to be a couple of problems.
It looks like you intend Enum to keep a record of all the
enum types you create with it. What happens if two unrelated
modules both define an enum called Colors?

To avoid this problem, you'd want to assign the newly
created enum type to a local name, but then you need to
write the name twice:

    Colors = Enum.Colors.red.green.blue

or

    Colors = Enum("Colors").red.green.blue

Then what happens if you mistakenly write:

    print(Colors.pineapple)

Seems like this would inadvertently add a new value to the
enum instead of producing an AttributeError.

-- 
Greg


From jbvsmo at gmail.com  Fri Feb 15 07:38:22 2013
From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=)
Date: Fri, 15 Feb 2013 04:38:22 -0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511DC9AB.9060906@canterbury.ac.nz>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>
	<20130214083558.77dc3424@pitrou.net>
	<CADiSq7faSd0We4xnMPKPN7mDmvwSnnHKtc1gjKsiLJ9qZL=ATg@mail.gmail.com>
	<20130214121602.3b89e23e@pitrou.net>
	<CADiSq7eooBM3_0nvT5BqomEwqb-fJ+2UqWU=zzgzzE9E34B8MA@mail.gmail.com>
	<CAOyAWgj5qY6VbUWXm7BQdYokjC-3u+nk7VuhxQcjB3mkf2JuVg@mail.gmail.com>
	<511DC9AB.9060906@canterbury.ac.nz>
Message-ID: <CAOyAWghoG5vCek3tj48NUtu8bn+DytMPWwfEJSKuEAkgN7+u1Q@mail.gmail.com>

This is just an idea to avoid class definition, but it can be improved.

2013/2/15 Greg Ewing <greg.ewing at canterbury.ac.nz>

> Jo?o Bernardo wrote:
>
>> And now for something completely different:
>>
>> If the Enums are used with attribute syntax, why not this?
>>
>>  >> Enum.Colors.red.green.blue
>>>
>>
>> <Enum: Colors>
>>
>
> Interesting idea, but there seem to be a couple of problems.
> It looks like you intend Enum to keep a record of all the
> enum types you create with it. What happens if two unrelated
> modules both define an enum called Colors?
>
>
Enum could write it in the current namespace, just like the `class`
statement.

>>> Enum.Colors.red.green.blue
>>> Colors.red
<Colors.red = 0>


> To avoid this problem, you'd want to assign the newly
> created enum type to a local name, but then you need to
> write the name twice:
>
>    Colors = Enum.Colors.red.green.blue
>
> or
>
>    Colors = Enum("Colors").red.green.blue
>
> Then what happens if you mistakenly write:
>
>    print(Colors.pineapple)
>
>
This could be solved with a function call to tell the enum to stop
accepting new values.

>>> Enum.Colors.red.green.blue()
>>> Colors.pineapple
AttributeError: 'Color' object has no attribute 'pineaple'



> Seems like this would inadvertently add a new value to the
> enum instead of producing an AttributeError.
>
> --
> Greg
>
> ______________________________**_________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
>


Each attribute access on `Enum` would create a new class even if it's
already defined in the same namespace.
The use of the name `Enum` is just good for definition anyway.


Jo?o Bernardo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130215/ba7f2a41/attachment.html>

From solipsis at pitrou.net  Fri Feb 15 08:03:45 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 15 Feb 2013 08:03:45 +0100
Subject: [Python-ideas] A subclassing API for named tuples?
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D60B5.7000903@pearwood.info>
	<20130214234008.17eb7ed0@pitrou.net>
	<511D708C.9030208@pearwood.info>
Message-ID: <20130215080345.61d1ca8f@pitrou.net>

On Fri, 15 Feb 2013 10:17:32 +1100
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> class MyClassWithAnExtremelyLongName(
>          collections.namedtuple("Cheese",
>           ("cheddar swiss ricotta camembert gouda parmesan brie limburger havarti"
>            " danish_blue roquefort greek_feta provolone mozzarella edam maasdam"
>            " stilton wensleydale red_leicester american colby monterey_jack kapiti"
>            " casu_marzu")
>           )
>          ):
>      pass

I don't find that readable at all. Having many indentations in a
single declaration line makes things quite messy in my opinion, and
your important class declaration is now drowning in a sea of literals.

The equivalent __fields__ *would* be readable, if properly formatted.

Regards

Antoine.




From solipsis at pitrou.net  Fri Feb 15 08:07:00 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 15 Feb 2013 08:07:00 +0100
Subject: [Python-ideas] A subclassing API for named tuples?
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D995A.8000008@hastings.org>
Message-ID: <20130215080700.542770d0@pitrou.net>

On Thu, 14 Feb 2013 18:11:38 -0800
Larry Hastings <larry at hastings.org> wrote:
> On 02/14/2013 05:19 AM, Nick Coghlan wrote:
> > A recurring suggestion for collections.namedtuple is that it would be
> > nice to be able to define them like this [...]
> 
> FWIW, I think namedtuple is overused.  Not that there's anything 
> innately wrong with namedtuple; it's just that I think too many types 
> are iterable which shouldn't be.  (Nobody unpacks the stat_result 
> anymore, that's like 1995 man.)  I suggest people use 
> types.SimpleNamespace unless iterability is specifically required.

There are tons of cases where tuple unpacking is a very convenient API.

Regards

Antoine.




From ncoghlan at gmail.com  Fri Feb 15 09:25:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 15 Feb 2013 18:25:53 +1000
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <20130215080700.542770d0@pitrou.net>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D995A.8000008@hastings.org>
	<20130215080700.542770d0@pitrou.net>
Message-ID: <CADiSq7d0RHG4FS5VkGp4-KXh4=40jXDT_tK=AuiYzUYQbSPY_g@mail.gmail.com>

On Fri, Feb 15, 2013 at 5:07 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 14 Feb 2013 18:11:38 -0800
> Larry Hastings <larry at hastings.org> wrote:
>> On 02/14/2013 05:19 AM, Nick Coghlan wrote:
>> > A recurring suggestion for collections.namedtuple is that it would be
>> > nice to be able to define them like this [...]
>>
>> FWIW, I think namedtuple is overused.  Not that there's anything
>> innately wrong with namedtuple; it's just that I think too many types
>> are iterable which shouldn't be.  (Nobody unpacks the stat_result
>> anymore, that's like 1995 man.)  I suggest people use
>> types.SimpleNamespace unless iterability is specifically required.
>
> There are tons of cases where tuple unpacking is a very convenient API.

Indeed. In particular, consuming namespaces from an iterator is
substantially more annoying than consuming a tuple.

I'd put the size cutoff for switching over to a namespace somewhere
around the 5 item mark. At 2-3, the tuple is often clearly superior,
at 4 it's arguable, at 5, unpacking starts to get a bit hard to read.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From jsbueno at python.org.br  Fri Feb 15 12:11:35 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 15 Feb 2013 09:11:35 -0200
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <511DC9AB.9060906@canterbury.ac.nz>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
	<CAN8CLg==JGKYCHYEwdTw06TFDKSf__VKh6C6fU-gSC+tzqjQ5g@mail.gmail.com>
	<CADiSq7cvhiCMTEaSrb_W8P0spWLoqgK2C_-2urjs4Y7ShPabxA@mail.gmail.com>
	<20130213173807.1465d42b@pitrou.net>
	<CADiSq7ceRt3J8z2vohHryAe8UOLjfO2sxgn_+nLKPpt-J=AazQ@mail.gmail.com>
	<20130214083558.77dc3424@pitrou.net>
	<CADiSq7faSd0We4xnMPKPN7mDmvwSnnHKtc1gjKsiLJ9qZL=ATg@mail.gmail.com>
	<20130214121602.3b89e23e@pitrou.net>
	<CADiSq7eooBM3_0nvT5BqomEwqb-fJ+2UqWU=zzgzzE9E34B8MA@mail.gmail.com>
	<CAOyAWgj5qY6VbUWXm7BQdYokjC-3u+nk7VuhxQcjB3mkf2JuVg@mail.gmail.com>
	<511DC9AB.9060906@canterbury.ac.nz>
Message-ID: <CAH0mxTRvSbhUH89GOn-Wt-zr0CO99DaDM95KezxAq_wc4Onh_g@mail.gmail.com>

On 15 February 2013 03:37, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Jo?o Bernardo wrote:
>>
>> And now for something completely different:
>>
>> If the Enums are used with attribute syntax, why not this?
>>
>>> >> Enum.Colors.red.green.blue
>>
>>
>> <Enum: Colors>
>
>
> Interesting idea, but there seem to be a couple of problems.
> It looks like you intend Enum to keep a record of all the
> enum types you create with it. What happens if two unrelated
> modules both define an enum called Colors?
>

Don't be picky - it is an interesting idea, and the obvious way to avoid
such side effect would be to require:

Colors = Enum("Colors").RED.GREEN.BLUE

implementation might be tricky  this way - you don't want new values genreated
each time one fetches a new attribute from a what should be a constant  -
then maybe:

Colors = Enum("Colors").RED.GREEN.BLUE._endEnum

> To avoid this problem, you'd want to assign the newly
> created enum type to a local name, but then you need to
> write the name twice:
>
>    Colors = Enum.Colors.red.green.blue

Writing the name of the ENum set twice might be the ideal, but
it is what one can do to get away today, in Python - and  people
are used to it (I'd favor a mechanism for an object to know
its name, but that is another issue entirely.)

>
> --
> Greg
>
PS. all - please note Jo?o Bernardo and me (Jo?o Bueno) are different
people
------------------------


From steve at pearwood.info  Sat Feb 16 05:51:28 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 16 Feb 2013 15:51:28 +1100
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <20130215080345.61d1ca8f@pitrou.net>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D60B5.7000903@pearwood.info>
	<20130214234008.17eb7ed0@pitrou.net>
	<511D708C.9030208@pearwood.info>
	<20130215080345.61d1ca8f@pitrou.net>
Message-ID: <511F1050.2000809@pearwood.info>

On 15/02/13 18:03, Antoine Pitrou wrote:
> On Fri, 15 Feb 2013 10:17:32 +1100
> Steven D'Aprano<steve at pearwood.info>  wrote:
>>
>> class MyClassWithAnExtremelyLongName(
>>           collections.namedtuple("Cheese",
>>            ("cheddar swiss ricotta camembert gouda parmesan brie limburger havarti"
>>             " danish_blue roquefort greek_feta provolone mozzarella edam maasdam"
>>             " stilton wensleydale red_leicester american colby monterey_jack kapiti"
>>             " casu_marzu")
>>            )
>>           ):
>>       pass
>
> I don't find that readable at all. Having many indentations in a
> single declaration line makes things quite messy in my opinion, and
> your important class declaration is now drowning in a sea of literals.

You don't have to stick the literals in the class definition. This is an easy problem to solve with existing techniques:

from collections import namedtuple

FIELDNAMES = """..."""  # Format it however you like.

class MyClassWithAnExtremelyLongName(namedtuple("Cheese", FIELDNAMES)):
     pass


There is no need to introduce confusion and uncertainty, "Should I use namedtuple or NamedTuple?".To say nothing of the invariable cases where people use the wrong one and have to deal with the cryptic error:

py> class C(collections.namedtuple):
...     pass
...
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: function() argument 1 must be code, not str


Moving the field name definitions outside of the namedtuple into a magic dunder attribute adds complexity. To use namedtuple, I just need to remember the calling signature, which is trivial. To use the proposed NamedTuple, I have to remember a magic dunder attribute __fields__ that is not used anywhere else. I don't see this as a win for simplicity.



-- 
Steven


From solipsis at pitrou.net  Sat Feb 16 11:18:48 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 16 Feb 2013 11:18:48 +0100
Subject: [Python-ideas] A subclassing API for named tuples?
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D60B5.7000903@pearwood.info>
	<20130214234008.17eb7ed0@pitrou.net>
	<511D708C.9030208@pearwood.info>
	<20130215080345.61d1ca8f@pitrou.net>
	<511F1050.2000809@pearwood.info>
Message-ID: <20130216111848.745dd99c@pitrou.net>

On Sat, 16 Feb 2013 15:51:28 +1100
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> from collections import namedtuple
> 
> FIELDNAMES = """..."""  # Format it however you like.
> 
> class MyClassWithAnExtremelyLongName(namedtuple("Cheese", FIELDNAMES)):
>      pass

Still not very elegant IMO (and it introduces spurious globals in the
module). Parameters for class construction shouldn't be outside of the
class declaration.

Really, the class declaration syntax has been *designed* to deal with
all of this. It's counter-productive to try not to use it.

(but, yes, now the official namedtuple API is the one we know, I agree
that having two ways to do it may be one too many)

Regards

Antoine.




From zuo at chopin.edu.pl  Sat Feb 16 14:46:39 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sat, 16 Feb 2013 14:46:39 +0100
Subject: [Python-ideas] =?utf-8?q?A_subclassing_API_for_named_tuples=3F?=
In-Reply-To: <20130216111848.745dd99c@pitrou.net>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net>
	<511D708C.9030208@pearwood.info> <20130215080345.61d1ca8f@pitrou.net>
	<511F1050.2000809@pearwood.info> <20130216111848.745dd99c@pitrou.net>
Message-ID: <b55b05c33c3cfff5d598139a410ac303@chopin.edu.pl>

16.02.2013 11:18, Antoine Pitrou wrote:
> On Sat, 16 Feb 2013 15:51:28 +1100
> Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> from collections import namedtuple
>>
>> FIELDNAMES = """..."""  # Format it however you like.
>>
>> class MyClassWithAnExtremelyLongName(namedtuple("Cheese", 
>> FIELDNAMES)):
>>      pass
>
> Still not very elegant IMO

I agree.  I see some nicer (IMHO) alternatives...  Apart from the 
recipe I mentioned
in the recent post (although I am *not* convinced it should be added to 
the stdlib)
some decorator-based way may be nice, e.g.:

     @namedtuple(fields='length weight is_poisonous')
     class Snake:
         def hiss(self):
             return 'hiss' + self.length * 's'

Cheers.
*j



From greg at krypto.org  Sun Feb 17 00:41:06 2013
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 16 Feb 2013 15:41:06 -0800
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <511D995A.8000008@hastings.org>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D995A.8000008@hastings.org>
Message-ID: <CAGE7PNJB=wuKSHitBfUsf+Bu0Fpm8GBtOKm3k3QHZj6b-imFYw@mail.gmail.com>

On Thu, Feb 14, 2013 at 6:11 PM, Larry Hastings <larry at hastings.org> wrote:

>  On 02/14/2013 05:19 AM, Nick Coghlan wrote:
>
> A recurring suggestion for collections.namedtuple is that it would be
> nice to be able to define them like this [...]
>
>
> FWIW, I think namedtuple is overused.  Not that there's anything innately
> wrong with namedtuple; it's just that I think too many types are iterable
> which shouldn't be.  (Nobody unpacks the stat_result anymore, that's like
> 1995 man.)  I suggest people use types.SimpleNamespace unless iterability
> is specifically required.
>

Ironically the types.SimpleNamespace documentation says: "However, for a
structured record type use namedtuple() instead."

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130216/c7835c8f/attachment.html>

From steve at pearwood.info  Sun Feb 17 22:38:38 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 18 Feb 2013 08:38:38 +1100
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <b55b05c33c3cfff5d598139a410ac303@chopin.edu.pl>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D60B5.7000903@pearwood.info>
	<20130214234008.17eb7ed0@pitrou.net>
	<511D708C.9030208@pearwood.info>
	<20130215080345.61d1ca8f@pitrou.net>
	<511F1050.2000809@pearwood.info>
	<20130216111848.745dd99c@pitrou.net>
	<b55b05c33c3cfff5d598139a410ac303@chopin.edu.pl>
Message-ID: <51214DDE.2020506@pearwood.info>

On 17/02/13 00:46, Jan Kaliszewski wrote:
> 16.02.2013 11:18, Antoine Pitrou wrote:
>> On Sat, 16 Feb 2013 15:51:28 +1100
>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>
>>> from collections import namedtuple
>>>
>>> FIELDNAMES = """...""" # Format it however you like.
>>>
>>> class MyClassWithAnExtremelyLongName(namedtuple("Cheese", FIELDNAMES)):
>>> pass
>>
>> Still not very elegant IMO
>
> I agree. I see some nicer (IMHO) alternatives... Apart from the recipe I mentioned
> in the recent post (although I am *not* convinced it should be added to the stdlib)
> some decorator-based way may be nice, e.g.:
>
> @namedtuple(fields='length weight is_poisonous')
> class Snake:
>     def hiss(self):
>         return 'hiss' + self.length * 's'


This implies that all namedtuples must be callable.

Point3D = namedtuple('Point3D', 'x y z')
pt = Point3D(2, 4, 8)

I don't think it's appropriate for tuples, named or not, to be callable.



-- 
Steven


From zuo at chopin.edu.pl  Mon Feb 18 00:45:15 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 18 Feb 2013 00:45:15 +0100
Subject: [Python-ideas] =?utf-8?q?A_subclassing_API_for_named_tuples=3F?=
In-Reply-To: <51214DDE.2020506@pearwood.info>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net>
	<511D708C.9030208@pearwood.info> <20130215080345.61d1ca8f@pitrou.net>
	<511F1050.2000809@pearwood.info> <20130216111848.745dd99c@pitrou.net>
	<b55b05c33c3cfff5d598139a410ac303@chopin.edu.pl>
	<51214DDE.2020506@pearwood.info>
Message-ID: <2988b553eb09138a013b8a636cb9487b@chopin.edu.pl>

17.02.2013 22:38, Steven D'Aprano wrote:
> On 17/02/13 00:46, Jan Kaliszewski wrote:
>> 16.02.2013 11:18, Antoine Pitrou wrote:
>>> On Sat, 16 Feb 2013 15:51:28 +1100
>>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>>
>>>> from collections import namedtuple
>>>>
>>>> FIELDNAMES = """...""" # Format it however you like.
>>>>
>>>> class MyClassWithAnExtremelyLongName(namedtuple("Cheese", 
>>>> FIELDNAMES)):
>>>> pass
>>>
>>> Still not very elegant IMO
>>
>> I agree. I see some nicer (IMHO) alternatives... Apart from the 
>> recipe I mentioned
>> in the recent post (although I am *not* convinced it should be added 
>> to the stdlib)
>> some decorator-based way may be nice, e.g.:
>>
>> @namedtuple(fields='length weight is_poisonous')
>> class Snake:
>>     def hiss(self):
>>         return 'hiss' + self.length * 's'
>
>
> This implies that all namedtuples must be callable.

No, not at all. It only means that signature specification of the 
namedtuple factory
function would need to be extended a bit (or a separate decorator, such 
as namedtuple.subclass,
would need to be added as an attribute) to support returning a 
decorator instead of
a ready named tuple type.

> Point3D = namedtuple('Point3D', 'x y z')
> pt = Point3D(2, 4, 8)
>
> I don't think it's appropriate for tuples, named or not, to be 
> callable.

I don't understand what do you mean. It's obvious that a named tuple 
*type* must be callable
(as any other instantiable type) and that in 99% of cases named tuple 
*instances* should not be...

Cheers.
*j



From techtonik at gmail.com  Mon Feb 18 08:40:07 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 18 Feb 2013 10:40:07 +0300
Subject: [Python-ideas] Poll about -h,--help options
Message-ID: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>

Hi,

Is it interesting to know if people expect -h to work as a --help
equivalent by default?

As it directly affects best practices of using Python in command line, can
we run such poll on python.org? I thought that this stuff is obvious, but
it appears that it is not, so I'd like to see this usability study - it
will be interesting for the Python community to know itself better.
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130218/c2bd0d9d/attachment.html>

From ubershmekel at gmail.com  Mon Feb 18 09:07:21 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Mon, 18 Feb 2013 10:07:21 +0200
Subject: [Python-ideas] Poll about -h,--help options
In-Reply-To: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
References: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
Message-ID: <CANSw7Ky200C9NBaviX9LB5-pmz-V_vf82-cVFLtxuh0scTQDSA@mail.gmail.com>

On Mon, Feb 18, 2013 at 9:40 AM, anatoly techtonik <techtonik at gmail.com>wrote:

> Hi,
>
> Is it interesting to know if people expect -h to work as a --help
> equivalent by default?
>
> As it directly affects best practices of using Python in command line, can
> we run such poll on python.org? I thought that this stuff is obvious, but
> it appears that it is not, so I'd like to see this usability study - it
> will be interesting for the Python community to know itself better.
>  --
> anatoly t.
>

I hate typing "-h" only to find that just "--help" works.

I'm pretty sure it's obvious and standard.
http://en.wikipedia.org/wiki/Command-line_interface#Built-in_usage_help


Yuval Greenfield
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130218/1407601d/attachment.html>

From steve at pearwood.info  Mon Feb 18 10:39:28 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 18 Feb 2013 20:39:28 +1100
Subject: [Python-ideas] Poll about -h,--help options
In-Reply-To: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
References: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
Message-ID: <5121F6D0.2010303@pearwood.info>

On 18/02/13 18:40, anatoly techtonik wrote:
> Hi,
>
> Is it interesting to know if people expect -h to work as a --help
> equivalent by default?

Yes, absolutely.



-- 
Steven


From raymond.hettinger at gmail.com  Mon Feb 18 10:46:36 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 18 Feb 2013 01:46:36 -0800
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
Message-ID: <567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com>


On Feb 14, 2013, at 5:19 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> So, what do people think? Too much magic? Or just the right amount to
> allow a cleaner syntax for named tuple definitions, without
> inadvertently encouraging people to do bad things to their memory
> usage? (Note: for backwards compatibility reasons, we couldn't use a
> custom metaclass for the classes returned by the existing collections
> namedtuple API. However, we could certainly provide a distinct
> collections.NamedTuple type which used a custom metaclass to behave
> this way).

To me, this smells of over-engineering.   For most uses, the current
form of named tuple is simple and clean:

    CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])

The current form of namedtuple is also very flexible.  The docs for it show
how it can easily be subclassed to add computed fields, new reprs, etc.
It also works great for implementing prototype instance, enums, etc.
And it has a verbose option that makes it self-documenting.

The namedtuple API was born out of mixing the best parts of many different
implementations found in the field.  It went through extensive evaluation,
review and refinement as a recipe on ASPN.   IMO, there is zero need 
to wreck its simplicity by throwing metaclass firepower into the mix.

We really don't need a second way to do it.
For people who care about the memory used
by subclasses, it takes less effort to learn how
to use slots that it does to learn and remember 
an second API for namedtuples.  

After all, the sole justification for __slots__ is to prevent the creation of 
an instance dictionary.   That is what it's for.    I think your real issue 
doesn't have anything to do with named tuples in particular.  
Instead, your issue is with subclassing *any* class that  uses __slots__.



Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130218/c7067fa3/attachment.html>

From chris.jerdonek at gmail.com  Mon Feb 18 10:41:12 2013
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 18 Feb 2013 01:41:12 -0800
Subject: [Python-ideas] Poll about -h,--help options
In-Reply-To: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
References: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
Message-ID: <CAOTb1wc92nK0EPaAFxga+6mbTnOJxdyw7tgx3M7aWTRBqAuCwg@mail.gmail.com>

On Sun, Feb 17, 2013 at 11:40 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> Hi,
>
> Is it interesting to know if people expect -h to work as a --help equivalent
> by default?
>
> As it directly affects best practices of using Python in command line, can
> we run such poll on python.org? I thought that this stuff is obvious, but it
> appears that it is not, so I'd like to see this usability study - it will be
> interesting for the Python community to know itself better.

The argparse and optparse modules in the standard library expose help
via both -h and --help by default (and this is in the Python
documentation and in the help strings generated by those modules), so
I would expect Python users at least to be familiar with both
invocations:

http://docs.python.org/2/library/optparse.html#generating-help
http://docs.python.org/2/library/argparse.html#add-help

--Chris


> --
> anatoly t.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From ncoghlan at gmail.com  Mon Feb 18 10:56:17 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Feb 2013 19:56:17 +1000
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com>
Message-ID: <CADiSq7dbcUd4-k3G3Pd8W1NFRWbS6Le47KaAn5JgKZrd6XLzHg@mail.gmail.com>

On Mon, Feb 18, 2013 at 7:46 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> After all, the sole justification for __slots__ is to prevent the creation
> of
> an instance dictionary.   That is what it's for.    I think your real issue
> doesn't have anything to do with named tuples in particular.
> Instead, your issue is with subclassing *any* class that  uses __slots__.

My real issue is with interminable enum discussions, this thread was
merely a side effect of enum-induced frustration :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From ubershmekel at gmail.com  Mon Feb 18 11:30:54 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Mon, 18 Feb 2013 12:30:54 +0200
Subject: [Python-ideas] Poll about -h,--help options
In-Reply-To: <5121F6D0.2010303@pearwood.info>
References: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
	<5121F6D0.2010303@pearwood.info>
Message-ID: <CANSw7KwoYNLBOg0WDQErOv2x87BirQe1BGWRURM1x2+g-T-ACA@mail.gmail.com>

On Mon, Feb 18, 2013 at 11:39 AM, Steven D'Aprano <steve at pearwood.info>wrote:

> On 18/02/13 18:40, anatoly techtonik wrote:
>
>> Hi,
>>
>> Is it interesting to know if people expect -h to work as a --help
>> equivalent by default?
>>
>
> Yes, absolutely.
>
>
>
Which reminds me of the most annoying behavior in existence:

user at pc:~$ ln -h
ln: invalid option -- 'h'
Try `ln --help' for more information.
user at pc:~$


Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130218/2cae100f/attachment.html>

From stefan at drees.name  Mon Feb 18 11:59:13 2013
From: stefan at drees.name (Stefan Drees)
Date: Mon, 18 Feb 2013 11:59:13 +0100
Subject: [Python-ideas] Poll about -h,--help options
In-Reply-To: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
References: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
Message-ID: <51220981.2020709@drees.name>

On 18.02.13 08:40, Anatoly Techtonik wrote:
> ...Is it interesting to know if people expect -h to work as a --help
> equivalent by default?
>
> As it directly affects best practices of using Python in command line,
> can we run such poll on python.org <http://python.org>? I thought that
> this stuff is obvious, but it appears that it is not, so I'd like to see
> this usability study - it will be interesting for the Python community
> to know itself better.
> ...

Polls are funny. Why not :-?

In the gnu world (at least) appending " -h" MAY for some tools result in 
output transformed to "human-readable", ie. tools like ls, du or df all 
support the -h triggers meaning "human readable" output options.

Asking for help or versions seems to be not a global priority ;-)

All the best,
Stefan.



From tjreedy at udel.edu  Tue Feb 19 01:43:07 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 18 Feb 2013 19:43:07 -0500
Subject: [Python-ideas] Poll about -h,--help options
In-Reply-To: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
References: <CAPkN8x+Nj5rEzjW-1OZLAPB=TRhhS0BoZ8uEHSicDU1h4JYZmw@mail.gmail.com>
Message-ID: <kfuhr7$oq$1@ger.gmane.org>

On 2/18/2013 2:40 AM, anatoly techtonik wrote:

> Is it interesting to know if people expect -h to work as a --help
> equivalent by default?

Of course.

Given that it already does for python itself
"
-?
-h
--help

     Print a short description of all command line options.
"
and at least some of its modules with a command line interface, I am not 
sure what you are getting at.

If some python module is deficient, report it on the tracker. If you 
think the opt/argparse docs should say more than they do, make a 
suggestion here, but be prepared to accept a decision to not add 
anything, on the basis that it is mostly our policy to not tell people 
how to use Python. (PEP 8 is a style guide for the stdlib, and for other 
stuff only as people decide to use it.)

> As it directly affects best practices of using Python in command line,
> can we run such poll on python.org <http://python.org>? I thought that
> this stuff is obvious, but it appears that it is not,

I have no idea what you are referring to, but if some other software is 
deficient, take it up with the authors.

-- 
Terry Jan Reedy



From raymond.hettinger at gmail.com  Tue Feb 19 06:48:48 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 18 Feb 2013 21:48:48 -0800
Subject: [Python-ideas] A subclassing API for named tuples?
In-Reply-To: <CADiSq7dbcUd4-k3G3Pd8W1NFRWbS6Le47KaAn5JgKZrd6XLzHg@mail.gmail.com>
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com>
	<CADiSq7dbcUd4-k3G3Pd8W1NFRWbS6Le47KaAn5JgKZrd6XLzHg@mail.gmail.com>
Message-ID: <05C3D9F7-111B-4AC3-AB76-D15AEBEB01CD@gmail.com>


On Feb 18, 2013, at 1:56 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> My real issue is with interminable enum discussions, this thread was
> merely a side effect of enum-induced frustration :)

I understand.  It never ends :-)


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130218/f3eeeb74/attachment.html>

From solipsis at pitrou.net  Tue Feb 19 10:24:06 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 19 Feb 2013 10:24:06 +0100
Subject: [Python-ideas] A subclassing API for named tuples?
References: <CADiSq7ewRqAYRMrQuwkzFrApVNXM=XcGtj62CtovqPr+qgSiYw@mail.gmail.com>
	<567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com>
	<CADiSq7dbcUd4-k3G3Pd8W1NFRWbS6Le47KaAn5JgKZrd6XLzHg@mail.gmail.com>
	<05C3D9F7-111B-4AC3-AB76-D15AEBEB01CD@gmail.com>
Message-ID: <20130219102406.735059be@pitrou.net>

Le Mon, 18 Feb 2013 21:48:48 -0800,
Raymond Hettinger
<raymond.hettinger at gmail.com> a ?crit :
> 
> On Feb 18, 2013, at 1:56 AM, Nick Coghlan
> <ncoghlan at gmail.com> wrote:
> 
> > My real issue is with interminable enum discussions, this thread was
> > merely a side effect of enum-induced frustration :)
> 
> I understand.  It never ends :-)

The problem with enum is that since it doesn't unlock any particularly
interesting feature, it must at least cater to all existing use cases
to be desirable for stdlib inclusion. Hence the long discussions.

Regards

Antoine.




From miki.tebeka at gmail.com  Tue Feb 19 17:03:16 2013
From: miki.tebeka at gmail.com (Miki Tebeka)
Date: Tue, 19 Feb 2013 08:03:16 -0800 (PST)
Subject: [Python-ideas] argparse - add support for environment variables
Message-ID: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>

Greetings,

The usual way of resolving configuration is command line -> environment -> 
default.
Currently argparse supports only command line -> default, I'd like to 
suggest an optional "env" keyword to add_argument that will also resolve 
from environment. (And also optional env dictionary to the ArgumentParser 
__init__ method [or to parse_args], which will default to os.environ).

Example:
         [spam.py]

parser = ArgumentParser()

     parser.add_argument('--spam', env='SPAM', default=7)
     args = parser.parse_args()
     print(args.spam)

     ./spam.py  -> 7
     ./spam.py --spam=12 -> 12
     SPAM=9 ./spam.py -> 9
     SPAM=9 ./spam.py --spam=12 -> 12

What do you think?
--
Miki
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130219/bc6656a4/attachment.html>

From graffatcolmingov at gmail.com  Tue Feb 19 17:39:24 2013
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Tue, 19 Feb 2013 11:39:24 -0500
Subject: [Python-ideas] Fwd: argparse - add support for environment variables
In-Reply-To: <CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
Message-ID: <CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>

---------- Forwarded message ----------
From: Ian Cordasco <graffatcolmingov at gmail.com>
Date: Tue, Feb 19, 2013 at 11:35 AM
Subject: Re: [Python-ideas] argparse - add support for environment variables
To: Miki Tebeka <miki.tebeka at gmail.com>
Cc: python-ideas at googlegroups.com


Why not:

    parser.add_argument('--spam', default=os.environ.get('SPAM', 7))

This way if SPAM isn't set, your default is 7. If spam is set, your
default becomes that.

On Tue, Feb 19, 2013 at 11:03 AM, Miki Tebeka <miki.tebeka at gmail.com> wrote:
> Greetings,
>
> The usual way of resolving configuration is command line -> environment ->
> default.
> Currently argparse supports only command line -> default, I'd like to
> suggest an optional "env" keyword to add_argument that will also resolve
> from environment. (And also optional env dictionary to the ArgumentParser
> __init__ method [or to parse_args], which will default to os.environ).
>
> Example:
>          [spam.py]
>
> parser = ArgumentParser()
>
>      parser.add_argument('--spam', env='SPAM', default=7)
>      args = parser.parse_args()
>      print(args.spam)
>
>      ./spam.py  -> 7
>      ./spam.py --spam=12 -> 12
>      SPAM=9 ./spam.py -> 9
>      SPAM=9 ./spam.py --spam=12 -> 12
>
> What do you think?
> --
> Miki

GMail decided to reply to python-ideas at googlegroups.com, but this
was my response which I shamefully did not bottom post.


From miki.tebeka at gmail.com  Wed Feb 20 00:51:08 2013
From: miki.tebeka at gmail.com (Miki Tebeka)
Date: Tue, 19 Feb 2013 15:51:08 -0800 (PST)
Subject: [Python-ideas] Fwd: argparse - add support for environment
 variables
In-Reply-To: <CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
Message-ID: <f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>



> Why not: 
>
>     parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) 
>
 Mainly because then you can't swap in different dictionaries to resolve 
SPAM (when specifying "env" to ArgumentParser).

The other reason is that it's making common code shorter, which is a win 
IMO.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130219/0a88c9bc/attachment.html>

From moloney at ohsu.edu  Wed Feb 20 02:54:58 2013
From: moloney at ohsu.edu (Brendan Moloney)
Date: Tue, 19 Feb 2013 17:54:58 -0800
Subject: [Python-ideas] Fwd: argparse - add support for environment
 variables
In-Reply-To: <f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>,
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
Message-ID: <5E25C96030E66B44B9CFAA95D3DE59194675672FA1@EX-MB08.ohsu.edu>

This is something I wanted before (as well as loading defaults from per user config files). A solution was proposed a while back, but it was never merged due to a lack of consensus on the API: http://code.google.com/p/argparse/issues/detail?id=35

Not sure if there is a more recent discussion.
________________________________________
From: Python-ideas [python-ideas-bounces+moloney=ohsu.edu at python.org] On Behalf Of Miki Tebeka [miki.tebeka at gmail.com]
Sent: Tuesday, February 19, 2013 3:51 PM
To: python-ideas at googlegroups.com
Cc: python-ideas
Subject: Re: [Python-ideas] Fwd: argparse - add support for environment variables

Why not:

    parser.add_argument('--spam', default=os.environ.get('SPAM', 7))
 Mainly because then you can't swap in different dictionaries to resolve SPAM (when specifying "env" to ArgumentParser).

The other reason is that it's making common code shorter, which is a win IMO.


From rovitotv at gmail.com  Wed Feb 20 03:48:16 2013
From: rovitotv at gmail.com (Todd Rovito)
Date: Tue, 19 Feb 2013 21:48:16 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception
	for All Branches
Message-ID: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>

Please see http://www.python.org/dev/peps/pep-0434/ for the complete
PEP.  The idea for the PEP was generated because a right click menu
was added to IDLE in Python 2.7 this started a debate about bug fix VS
enhancement see the PEP references for more information.  IDLE has
many patches that already exist but are difficult to get committed.
This PEP is designed to make it easier to bring IDLE up to speed with
modern GUI standards.  The PEP is not perfect some issues that need to
be solved:
-How do we test IDLE patches to make sure they are high quality?
Apparently the build bots don't test user interface issues.  My
philosophy is keep it simple so I was thinking that a simple procedure
for testing on each of the major platforms should be performed before
a commit.  Creating a build bot that tests user interfaces seems
difficult to me and could put IDLE improvements even more behind.
-Does it make sense to separate IDLE from the stdlib for Python 3.4?
 I understand the batteries included argument, but if IDLE development
is going to continue to proceed at a more accelerated pace than
Python, it might make sense to separate it out into its own project or
a sub-project.

Please provide comments or other concerns. Thanks.


From eliben at gmail.com  Wed Feb 20 05:29:48 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 19 Feb 2013 20:29:48 -0800
Subject: [Python-ideas] Fwd: argparse - add support for environment
	variables
In-Reply-To: <f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
Message-ID: <CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>

On Tue, Feb 19, 2013 at 3:51 PM, Miki Tebeka <miki.tebeka at gmail.com> wrote:

>
> Why not:
>>
>>     parser.add_argument('--spam', default=os.environ.get('SPAM', 7))
>>
>  Mainly because then you can't swap in different dictionaries to resolve
> SPAM (when specifying "env" to ArgumentParser).
>

Can you elaborate on this requirement?


> The other reason is that it's making common code shorter, which is a win
> IMO.
>

How common is such code? YMMV here... I don't think it's nearly common
enough to warrant adding complexity to a stdlib module in order to
implement a non-orthogonal feature.

In general, I'm -1 on the idea. argparse is for parsing command line
arguments. Python provides you convenient access to env vars via
os.environ. Why mix up the two?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130219/d79ef7c1/attachment.html>

From steve at pearwood.info  Wed Feb 20 06:10:29 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 20 Feb 2013 16:10:29 +1100
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
	Exception for All Branches
In-Reply-To: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
Message-ID: <20130220051028.GA22784@ando>

On Tue, Feb 19, 2013 at 09:48:16PM -0500, Todd Rovito wrote:

> -Does it make sense to separate IDLE from the stdlib for Python 3.4?
>  I understand the batteries included argument, but if IDLE development
> is going to continue to proceed at a more accelerated pace than
> Python, it might make sense to separate it out into its own project or
> a sub-project.

Personally, I don't use IDLE, but I know many people who do. I think it 
is important for them that IDLE remains in the std lib.

We could have a stable version of IDLE in the std lib, and another 
version available on PyPI. The PyPI version could change more rapidly, 
since it doesn't have to fit in with the release schedule for the std 
lib. As the release schedule allows, the two branches could be merged.

I have no idea if this has been tried before, but I think this would 
work better for an application like IDLE than for a library.


-- 
Steven


From ericsnowcurrently at gmail.com  Wed Feb 20 06:34:53 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Tue, 19 Feb 2013 22:34:53 -0700
Subject: [Python-ideas] capturing the origin of objects
Message-ID: <CALFfu7DYLcQiS8+yG1k+Pn+qWRUsC3Vmfu3bBrkq43YN3oaDkg@mail.gmail.com>

With the removal of unbound methods in Python 3, introspecting the
class on which a method was defined is not so simple.  Python 3.3
gives us __qualname__ for classes and functions (see PEP 3155), which
helps somewhat by giving us a string composed at compile-time.
However, I was considering something a little more powerful.

I propose that what would have formerly been an unbound method get a
new attribute, __origin__.  It will be bound to class where the method
is defined.

Motivation
==========

One motivator for this proposal is http://bugs.python.org/issue15582,
allowing inspect.getdoc() to "inherit" docstrings.  Without a concrete
connection between a method and the "origin" class, dynamically
determining the docstring is basically a non-starter.  The __origin__
attribute would help.

Extended Idea
=============

The name, __origin__, is a bit generic to accommodate the possible
extension of the proposal to other objects:

  * f.__origin__ (the module or function (nested) where the function
was defined)
  * cls.__origin__ (the module, class (nested), or function where the
class was defined)
  * code.__origin__ (the function for which the code object was
originally created)
  * module.__origin__ (a.k.a. module.__file__)

For functions and classes defined in a function, __origin__ would be
bound to the function rather than the locals or the code object.  For
modules, __origin__ is more accurate in the cases that the module was
generated from something other than a file.  It would still be a
string, though.  Also, we currently have a convention of setting
__module__ to the name of the module rather than the module itself.
Whether to break with that convention for __origin__ is an open
question which relates to how __origin__ would be used.

Conceivably, each use case for __origin__ could be covered by a name
more specific to the object, e.g. module.__file__.  However, having
the one name would help make the purpose clear and consistent.

Objections
==========

The downside of binding the objects to __origin__ is in memory usage
and, particularly, in ref-counts/reference-cycles.  I expect that this
is where the biggest objections will lie.  However, my understanding
is that this is less of an issue now than it used to be.  If need be,
weakref proxies could address some of the concerns.

The status quo has a gap in the case of "unbound" methods and of code
objects, though __qualname__ does offer an improvement for methods.
The actual use case I have relates strictly to those methods.  Having
something like __origin__ for those methods, at least, would be
helpful.

-eric


From vito.detullio at gmail.com  Wed Feb 20 07:59:18 2013
From: vito.detullio at gmail.com (Vito De Tullio)
Date: Wed, 20 Feb 2013 07:59:18 +0100
Subject: [Python-ideas] Fwd: argparse - add support for environment
	variables
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
	<CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
Message-ID: <kg1s86$r94$1@ger.gmane.org>

Eli Bendersky wrote:

> In general, I'm -1 on the idea. argparse is for parsing command line
> arguments. Python provides you convenient access to env vars via
> os.environ. Why mix up the two?

well, command line arguments, environmental variables and, I add, 
configuration files are commonly intertwined.

While argparse focus on the first one, there are cases where you want some 
"layer" on top of it and on os.environ and on... ConfigParser?) to DRY.

-- 
ZeD




From chris.jerdonek at gmail.com  Wed Feb 20 09:40:57 2013
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 20 Feb 2013 00:40:57 -0800
Subject: [Python-ideas] Fwd: argparse - add support for environment
	variables
In-Reply-To: <f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
Message-ID: <CAOTb1wfiROqzEHz+K1zedrmqttLvLu19pRJM_=aaV2tqWzGA_g@mail.gmail.com>

On Tue, Feb 19, 2013 at 3:51 PM, Miki Tebeka <miki.tebeka at gmail.com> wrote:
>
>> Why not:
>>
>>     parser.add_argument('--spam', default=os.environ.get('SPAM', 7))
>
>  Mainly because then you can't swap in different dictionaries to resolve
> SPAM (when specifying "env" to ArgumentParser).

Couldn't you achieve this with a simple helper function?  Something
along the lines of--

env = {'SPAM': 9}
parser = argparse.ArgumentParser()
def add_argument(key, *args, **kwargs):
    if key in env:
        kwargs['default'] = env[key]
    parser.add_argument(*args, **kwargs)
add_argument('SPAM', '--spam', default=7)
...

This also allows env to be an arbitrary dictionary.

--Chris


From barry at python.org  Wed Feb 20 15:04:26 2013
From: barry at python.org (Barry Warsaw)
Date: Wed, 20 Feb 2013 09:04:26 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
 Exception for All Branches
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
Message-ID: <20130220090426.66ebbe8e@anarchist.wooz.org>

On Feb 20, 2013, at 04:10 PM, Steven D'Aprano wrote:

>We could have a stable version of IDLE in the std lib, and another 
>version available on PyPI. The PyPI version could change more rapidly, 
>since it doesn't have to fit in with the release schedule for the std 
>lib. As the release schedule allows, the two branches could be merged.
>
>I have no idea if this has been tried before, but I think this would 
>work better for an application like IDLE than for a library.

There's some precedence for this I think, e.g. mock, unittest2, email.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130220/ed870bc6/attachment.pgp>

From ncoghlan at gmail.com  Wed Feb 20 15:30:49 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Feb 2013 00:30:49 +1000
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
 Exception for All Branches
In-Reply-To: <20130220090426.66ebbe8e@anarchist.wooz.org>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
Message-ID: <CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>

On Thu, Feb 21, 2013 at 12:04 AM, Barry Warsaw <barry at python.org> wrote:
> On Feb 20, 2013, at 04:10 PM, Steven D'Aprano wrote:
>
>>We could have a stable version of IDLE in the std lib, and another
>>version available on PyPI. The PyPI version could change more rapidly,
>>since it doesn't have to fit in with the release schedule for the std
>>lib. As the release schedule allows, the two branches could be merged.
>>
>>I have no idea if this has been tried before, but I think this would
>>work better for an application like IDLE than for a library.
>
> There's some precedence for this I think, e.g. mock, unittest2, email.

This was the plan for packaging vs distutils2, and I'll be proposing
something similar for distlib, too.

It can be a little tricky to manage at times, but you can do truly
experimental features in the PyPI version first with a warning that
they're experimental (e.g the various incarnations
contextlib.ExitStack went through when I first added it to
contextlib2), while bug fixes and features you're more confident about
can go directly into hg.python.org and then be published early through
the downstream project.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From abarnert at yahoo.com  Wed Feb 20 18:32:48 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 20 Feb 2013 09:32:48 -0800
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
	Exception for All Branches
In-Reply-To: <20130220090426.66ebbe8e@anarchist.wooz.org>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
Message-ID: <E77C0A50-6E38-4663-A3CA-096E91E9F0D7@yahoo.com>

On Feb 20, 2013, at 6:04, Barry Warsaw <barry at python.org> wrote:

> On Feb 20, 2013, at 04:10 PM, Steven D'Aprano wrote:
> 
>> We could have a stable version of IDLE in the std lib, and another 
>> version available on PyPI. The PyPI version could change more rapidly, 
>> since it doesn't have to fit in with the release schedule for the std 
>> lib. As the release schedule allows, the two branches could be merged.
>> 
>> I have no idea if this has been tried before, but I think this would 
>> work better for an application like IDLE than for a library.
> 
> There's some precedence for this I think, e.g. mock, unittest2, email.

And sqlite3.

Personally, I've found it easier to track down certain changes in sqlite3 than in other modules. The Python docs are very clear about when a new function was added, but not always as clear about, e.g., performance characteristics or internal behavior with indirect external effects. I've found it easier in some cases to find that I'm relying on, say, pysqlite >= 2.5.0 behavior than python >= 2.6. And then mapping pysqlite >= 2.5.0 to Python versions is easy. And this might be even more relevant for IDLE than for a module.

And of course it gives me a way to deal with the changes. On a machine that has 2.5, I can just pip install pysqlite. And I can certainly see it being useful to pip-3.3 install idle to get IDLE 3.4.1 on a machine with python 3.3.

I don't think these are _major_ selling points, but they're at least minor ones, which may help counter the obvious downside (merging isn't free).

From dholth at gmail.com  Wed Feb 20 18:58:24 2013
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 20 Feb 2013 12:58:24 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
 Exception for All Branches
In-Reply-To: <E77C0A50-6E38-4663-A3CA-096E91E9F0D7@yahoo.com>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
	<E77C0A50-6E38-4663-A3CA-096E91E9F0D7@yahoo.com>
Message-ID: <CAG8k2+4OYbexjtcX4Gf8LCC+mnHD1L657Q8zmJSSaE+qr4B1gg@mail.gmail.com>

Separate set of rules for features classified as "standard applications"
versus "standard libraries"?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130220/f2ff27c7/attachment.html>

From miki.tebeka at gmail.com  Wed Feb 20 19:10:33 2013
From: miki.tebeka at gmail.com (Miki Tebeka)
Date: Wed, 20 Feb 2013 10:10:33 -0800 (PST)
Subject: [Python-ideas] Fwd: argparse - add support for environment
 variables
In-Reply-To: <CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
	<CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
Message-ID: <5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com>



On Tuesday, February 19, 2013 8:29:48 PM UTC-8, Eli Bendersky wrote:
>
> On Tue, Feb 19, 2013 at 3:51 PM, Miki Tebeka <miki.... at gmail.com<javascript:>
> > wrote:
>
>>
>> Why not: 
>>>
>>>     parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) 
>>>
>>  Mainly because then you can't swap in different dictionaries to resolve 
>> SPAM (when specifying "env" to ArgumentParser).
>>
>
> Can you elaborate on this requirement?
>

Sure. The idea is the you have some kind of "parameter resolution" (very 
much like the interpreter is resolving variables). You first look into the 
command line, and if not found you query a dictionary. Most times, this 
dictionary will be os.environ. However you can pass your own dictionary to 
resolve the parameters (for testing, configuration files). You can even use 
something like Raymonds Nested Contexts<http://code.activestate.com/recipes/577434-nested-contexts-a-chain-of-mapping-objects/>for that matter.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130220/c2e3e1c0/attachment.html>

From fuzzyman at gmail.com  Wed Feb 20 20:18:25 2013
From: fuzzyman at gmail.com (Michael Foord)
Date: Wed, 20 Feb 2013 19:18:25 +0000
Subject: [Python-ideas] Fwd: argparse - add support for environment
	variables
In-Reply-To: <kg1s86$r94$1@ger.gmane.org>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
	<CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
	<kg1s86$r94$1@ger.gmane.org>
Message-ID: <CAKCKLWzf_v45zv6CZemSJ9dPEhjzJdfz-bgPvP1_BKxq2b2RkQ@mail.gmail.com>

On 20 February 2013 06:59, Vito De Tullio <vito.detullio at gmail.com> wrote:

> Eli Bendersky wrote:
>
> > In general, I'm -1 on the idea. argparse is for parsing command line
> > arguments. Python provides you convenient access to env vars via
> > os.environ. Why mix up the two?
>
> well, command line arguments, environmental variables and, I add,
> configuration files are commonly intertwined.
>
> While argparse focus on the first one, there are cases where you want some
> "layer" on top of it and on os.environ and on... ConfigParser?) to DRY.
>
>

configglue combines support for config files (and hierarchies of config
files) with command line arguments.

https://pypi.python.org/pypi/configglue/

Michael


> --
> ZeD
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130220/73d15aa1/attachment.html>

From tjreedy at udel.edu  Wed Feb 20 23:07:57 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Feb 2013 17:07:57 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
	Exception for All Branches
In-Reply-To: <CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
	<CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>
Message-ID: <kg3hga$73s$1@ger.gmane.org>

On 2/20/2013 9:30 AM, Nick Coghlan wrote:
> On Thu, Feb 21, 2013 at 12:04 AM, Barry Warsaw <barry at python.org> wrote:
>> On Feb 20, 2013, at 04:10 PM, Steven D'Aprano wrote:
>>
>>> We could have a stable version of IDLE in the std lib, and another
>>> version available on PyPI. The PyPI version could change more rapidly,
>>> since it doesn't have to fit in with the release schedule for the std
>>> lib. As the release schedule allows, the two branches could be merged.
>>>
>>> I have no idea if this has been tried before, but I think this would
>>> work better for an application like IDLE than for a library.
>>
>> There's some precedence for this I think, e.g. mock, unittest2, email.
>
> This was the plan for packaging vs distutils2, and I'll be proposing
> something similar for distlib, too.
>
> It can be a little tricky to manage at times, but you can do truly
> experimental features in the PyPI version first with a warning that
> they're experimental (e.g the various incarnations
> contextlib.ExitStack went through when I first added it to
> contextlib2), while bug fixes and features you're more confident about
> can go directly into hg.python.org and then be published early through
> the downstream project.

Most of the actual issues that prompted this proposal are minor changes, 
such as to the find or replace dialog. Some might even be considered bug 
fixes rather than enhancements, depending on one's interpretation of the 
vague doc for such. This PEP is partly about whether we need to classify 
such changes, or just put them into whatever release we feel 
appropriate. Whatever the decision, those changes should directly go 
into whatever version they are going to go in.

On the other hand, I agree that a PyPI preview release of a rewrite of 
IDLE to use the newer ttk widgets would be a good idea. But I personally 
would base such a project on 3.3 and consider whether it should require 
tcl/tk 8.6 rather than 8.5. And I would only put it in the stdlib in a 
new release (perhaps alongside the existing IDLE for at least one 
release). So this PEP is not relevance to such a project.

-- 
Terry Jan Reedy



From solipsis at pitrou.net  Thu Feb 21 00:12:44 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 00:12:44 +0100
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
 Exception for All Branches
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
	<CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>
	<kg3hga$73s$1@ger.gmane.org>
Message-ID: <20130221001244.1b602cfd@pitrou.net>

On Wed, 20 Feb 2013 17:07:57 -0500
Terry Reedy <tjreedy at udel.edu> wrote:
> 
> On the other hand, I agree that a PyPI preview release of a rewrite of 
> IDLE to use the newer ttk widgets would be a good idea. But I personally 
> would base such a project on 3.3 and consider whether it should require 
> tcl/tk 8.6 rather than 8.5. And I would only put it in the stdlib in a 
> new release (perhaps alongside the existing IDLE for at least one 
> release). So this PEP is not relevance to such a project.

The only thing relevant to such a project is to find someone actually
motivated to do it.

What is IDLE's actual maintenance activity, exactly? I am sympathetic to
the PEP but, really, thinking IDLE's development is hampered by Python's
release process is *completely* ridiculous. If you want IDLE development
to happen, please stop talking and start reviewing / committing patches.

FTR, I'm personally +1 on yanking IDLE out of 3.4.

Regards

Antoine.




From vinay_sajip at yahoo.co.uk  Thu Feb 21 01:07:55 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 21 Feb 2013 00:07:55 +0000 (UTC)
Subject: [Python-ideas]
	=?utf-8?q?Fwd=3A_argparse_-_add_support_for_enviro?=
	=?utf-8?q?nment=09variables?=
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
	<CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
	<kg1s86$r94$1@ger.gmane.org>
	<CAKCKLWzf_v45zv6CZemSJ9dPEhjzJdfz-bgPvP1_BKxq2b2RkQ@mail.gmail.com>
Message-ID: <loom.20130221T010430-704@post.gmane.org>

Michael Foord <fuzzyman at ...> writes:

> 
> configglue combines support for config files (and hierarchies of config files)
with command line arguments.
> 
> https://pypi.python.org/pypi/configglue/
> 

So does the config module:

http://www.red-dove.com/config-doc/

Regards,

Vinay Sajip




From eliben at gmail.com  Thu Feb 21 01:18:50 2013
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 20 Feb 2013 16:18:50 -0800
Subject: [Python-ideas] Fwd: argparse - add support for environment
	variables
In-Reply-To: <5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
	<CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
	<5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com>
Message-ID: <CAF-Rda8kobVf6yceb2aSL5BxZ4U=OaExzWqRCw2=Ob5fqbcVEQ@mail.gmail.com>

On Wed, Feb 20, 2013 at 10:10 AM, Miki Tebeka <miki.tebeka at gmail.com> wrote:

>
>
> On Tuesday, February 19, 2013 8:29:48 PM UTC-8, Eli Bendersky wrote:
>
>> On Tue, Feb 19, 2013 at 3:51 PM, Miki Tebeka <miki.... at gmail.com> wrote:
>>
>>>
>>> Why not:
>>>>
>>>>     parser.add_argument('--spam', default=os.environ.get('SPAM', 7))
>>>>
>>>  Mainly because then you can't swap in different dictionaries to resolve
>>> SPAM (when specifying "env" to ArgumentParser).
>>>
>>
>> Can you elaborate on this requirement?
>>
>
> Sure. The idea is the you have some kind of "parameter resolution" (very
> much like the interpreter is resolving variables). You first look into the
> command line, and if not found you query a dictionary. Most times, this
> dictionary will be os.environ. However you can pass your own dictionary to
> resolve the parameters (for testing, configuration files). You can even use
> something like Raymonds Nested Contexts<http://code.activestate.com/recipes/577434-nested-contexts-a-chain-of-mapping-objects/>for that matter.
>

What about config files? Mix them into argparse too? Oh, and support for
.ini/XML/YAML/JSON/sqlite configs, please!

Sarcasm aside (don't take it personally ;-), my point is this: argparse is
a nice, orthogonal library for dealing with command-line arguments. Let's
keep it this way. Python gives you tools to *very easily* mix up different
configuration paths, with really minimal effort.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130220/b1e38ffb/attachment.html>

From ncoghlan at gmail.com  Thu Feb 21 01:44:44 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Feb 2013 10:44:44 +1000
Subject: [Python-ideas] Fwd: argparse - add support for environment
	variables
In-Reply-To: <CAF-Rda8kobVf6yceb2aSL5BxZ4U=OaExzWqRCw2=Ob5fqbcVEQ@mail.gmail.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
	<CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
	<5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com>
	<CAF-Rda8kobVf6yceb2aSL5BxZ4U=OaExzWqRCw2=Ob5fqbcVEQ@mail.gmail.com>
Message-ID: <CADiSq7d9GSa_gupxp_=iCfySt=rshOh3GqtziZYCe0uAG3DXJQ@mail.gmail.com>

On Thu, Feb 21, 2013 at 10:18 AM, Eli Bendersky <eliben at gmail.com> wrote:
> What about config files? Mix them into argparse too? Oh, and support for
> .ini/XML/YAML/JSON/sqlite configs, please!
>
> Sarcasm aside (don't take it personally ;-), my point is this: argparse is a
> nice, orthogonal library for dealing with command-line arguments. Let's keep
> it this way. Python gives you tools to *very easily* mix up different
> configuration paths, with really minimal effort.

Indeed - a layered architecture is the way to go here. argparse can be
a *component* of a comprehensive configuration system, but it
shouldn't try to *become* a comprehensive configuration system.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From tjreedy at udel.edu  Thu Feb 21 02:41:29 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Feb 2013 20:41:29 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
	Exception for All Branches
In-Reply-To: <20130221001244.1b602cfd@pitrou.net>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
	<CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>
	<kg3hga$73s$1@ger.gmane.org> <20130221001244.1b602cfd@pitrou.net>
Message-ID: <kg3u0n$jug$1@ger.gmane.org>

On 2/20/2013 6:12 PM, Antoine Pitrou wrote:
> On Wed, 20 Feb 2013 17:07:57 -0500
> Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> On the other hand, I agree that a PyPI preview release of a rewrite of
>> IDLE to use the newer ttk widgets would be a good idea. But I personally
>> would base such a project on 3.3 and consider whether it should require
>> tcl/tk 8.6 rather than 8.5. And I would only put it in the stdlib in a
>> new release (perhaps alongside the existing IDLE for at least one
>> release). So this PEP is not relevance to such a project.
>
> The only thing relevant to such a project is to find someone actually
> motivated to do it.

Right. That I what I was trying to say.

> What is IDLE's actual maintenance activity, exactly?

I am not sure whether you are asking about volume or focus. A year ago, 
focus was on bugs that caused IDLE to quit because of an uncaught 
exception. When IDLE is started on Windows from an icon, this looks like 
a crash because there is no visible traceback or exit message. Recently, 
other issues have been worked on. In the last 5 months, activity has 
picked up and there are about 40 issues in 3.4 misc/news whose commit 
message contains 'idle'.

 > I am sympathetic to the PEP

Even though I am willing for the PEP to be rejected, great. I say this 
because I think it better for the community at large, even though a 
strict policy might be easier, on net, for developers.

> but, really, thinking IDLE's development is hampered by Python's
> release process is *completely* ridiculous.

I do not believe I have said that and certainly have not meant to say 
that. What I have said or meant to say is that uncertainly and 
disagreement about how it does and should fit into the release process 
can be a hindrance.

 > If you want IDLE development
> to happen, please stop talking and start reviewing / committing patches.

I have, of course, done some of both in the past year, even if not 
currently. As I remember, the review process for at least one issue got 
hung up on whether a (small) change was a 'bugfix' or 'enhancement' and 
if the latter, whether it could go into all versions or just default. As 
we move from obvious bug issue to more ambiguous issues, this question 
would come up more often.

> FTR, I'm personally +1 on yanking IDLE out of 3.4.

That would tend to keep 3.4 out of some classrooms.  See, for example,
http://search.gmane.org/?query=&author=Jean-Paul.Roy%40unice.fr&group=gmane.comp.python.idle&sort=relevance&DEFAULTOP=and&query=
from a French college teacher. (The issue was fixed by a tk fix last 
March.) But of course, this is a different subject.

-- 
Terry Jan Reedy



From andre.roberge at gmail.com  Thu Feb 21 02:44:33 2013
From: andre.roberge at gmail.com (Andre Roberge)
Date: Wed, 20 Feb 2013 21:44:33 -0400
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
 Exception for All Branches
In-Reply-To: <kg3u0n$jug$1@ger.gmane.org>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
	<CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>
	<kg3hga$73s$1@ger.gmane.org> <20130221001244.1b602cfd@pitrou.net>
	<kg3u0n$jug$1@ger.gmane.org>
Message-ID: <CAGMu_=p3NxQQd-dDG+vv2zLEbL8k5Nuk_tZEKO1H1CRi6f7N-g@mail.gmail.com>

>From what I have read on edu-sig, I believe it would be a huge mistake to
remove IDLE from the Python standard library.


On Wed, Feb 20, 2013 at 9:41 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 2/20/2013 6:12 PM, Antoine Pitrou wrote:
>
>> On Wed, 20 Feb 2013 17:07:57 -0500
>> Terry Reedy <tjreedy at udel.edu> wrote:
>>
>>>
>>> On the other hand, I agree that a PyPI preview release of a rewrite of
>>> IDLE to use the newer ttk widgets would be a good idea. But I personally
>>> would base such a project on 3.3 and consider whether it should require
>>> tcl/tk 8.6 rather than 8.5. And I would only put it in the stdlib in a
>>> new release (perhaps alongside the existing IDLE for at least one
>>> release). So this PEP is not relevance to such a project.
>>>
>>
>> The only thing relevant to such a project is to find someone actually
>> motivated to do it.
>>
>
> Right. That I what I was trying to say.
>
>  What is IDLE's actual maintenance activity, exactly?
>>
>
> I am not sure whether you are asking about volume or focus. A year ago,
> focus was on bugs that caused IDLE to quit because of an uncaught
> exception. When IDLE is started on Windows from an icon, this looks like a
> crash because there is no visible traceback or exit message. Recently,
> other issues have been worked on. In the last 5 months, activity has picked
> up and there are about 40 issues in 3.4 misc/news whose commit message
> contains 'idle'.
>
> > I am sympathetic to the PEP
>
> Even though I am willing for the PEP to be rejected, great. I say this
> because I think it better for the community at large, even though a strict
> policy might be easier, on net, for developers.
>
>  but, really, thinking IDLE's development is hampered by Python's
>> release process is *completely* ridiculous.
>>
>
> I do not believe I have said that and certainly have not meant to say
> that. What I have said or meant to say is that uncertainly and disagreement
> about how it does and should fit into the release process can be a
> hindrance.
>
> > If you want IDLE development
>
>> to happen, please stop talking and start reviewing / committing patches.
>>
>
> I have, of course, done some of both in the past year, even if not
> currently. As I remember, the review process for at least one issue got
> hung up on whether a (small) change was a 'bugfix' or 'enhancement' and if
> the latter, whether it could go into all versions or just default. As we
> move from obvious bug issue to more ambiguous issues, this question would
> come up more often.
>
>  FTR, I'm personally +1 on yanking IDLE out of 3.4.
>>
>
> That would tend to keep 3.4 out of some classrooms.  See, for example,
> http://search.gmane.org/?**query=&author=Jean-Paul.Roy%**
> 40unice.fr&group=gmane.comp.**python.idle&sort=relevance&**
> DEFAULTOP=and&query=<http://search.gmane.org/?query=&author=Jean-Paul.Roy%40unice.fr&group=gmane.comp.python.idle&sort=relevance&DEFAULTOP=and&query=>
> from a French college teacher. (The issue was fixed by a tk fix last
> March.) But of course, this is a different subject.
>
> --
> Terry Jan Reedy
>
> ______________________________**_________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130220/fc10b9b9/attachment.html>

From robert2682 at verizon.net  Thu Feb 21 03:11:01 2013
From: robert2682 at verizon.net (robert2682)
Date: Wed, 20 Feb 2013 21:11:01 -0500
Subject: [Python-ideas] Functions and Unbound methods
Message-ID: <51258235.3090802@verizon.net>



Hi,

I'm new; greetings all!

I'm not sure if this is a bug or feature, but it confused me so I thought
I'd raise the issue.


class a:
   def b (self):
     pass

   foo = ('Hello', b)

class c(a):
   def d(self):
     t = type (self. __class__. foo [1])
     print t
     t = type (self. __class__. b)
     print t

e = c ()
e. d()

prints <type 'function'> for the first print,
and it seems to me it should be an instancemethod


I'm trying to something like this

class EditPage:

   additonal_buttons = ()
   def __init__ (self):
     buts = []
     for x in addional_butons:
       if isinstance (x [1], types. UnboundMethodType):     # fails 
because type (x [1]) is function, not UnboundMethod
         buts. append ((x [0], types. MethodType (x [1], self)))
       else:
         buts. append (x)

class TreePage(EditPage):
   def EditAsText (self):
      pass

   additional_buttons = (('EditAsText', EditAsText),)


Thanks

Robert Kaplan
robert2682 at verizon.net



From roger.serwy at gmail.com  Thu Feb 21 03:45:04 2013
From: roger.serwy at gmail.com (serwy)
Date: Wed, 20 Feb 2013 20:45:04 -0600
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
 Exception for All Branches
In-Reply-To: <20130221001244.1b602cfd@pitrou.net>
References: <20130221001244.1b602cfd@pitrou.net>
Message-ID: <51258A30.8090809@illinois.edu>

Hi Antoine,

I am the developer of the IdleX project (http://idlex.sourceforge.net). 
It's not a fork of IDLE, but just a collection of extensions that can be 
easily merged into IDLE.

I started the project because of the slow pace of IDLE development, 
especially when having several outstanding patches languishing in the 
bug tracker. With your help I'd like to push IDLE forward.

- Roger








From python at mrabarnett.plus.com  Thu Feb 21 04:50:21 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 21 Feb 2013 03:50:21 +0000
Subject: [Python-ideas] Functions and Unbound methods
In-Reply-To: <51258235.3090802@verizon.net>
References: <51258235.3090802@verizon.net>
Message-ID: <5125997D.4090608@mrabarnett.plus.com>

On 2013-02-21 02:11, robert2682 wrote:
>
>
> Hi,
>
> I'm new; greetings all!
>
> I'm not sure if this is a bug or feature, but it confused me so I thought
> I'd raise the issue.
>
>
> class a:
>     def b (self):
>       pass
>
>     foo = ('Hello', b)
>
> class c(a):
>     def d(self):
>       t = type (self. __class__. foo [1])
>       print t
>       t = type (self. __class__. b)
>       print t
>
> e = c ()
> e. d()
>
> prints <type 'function'> for the first print,
> and it seems to me it should be an instancemethod
>
[snip]
I think what's happening is that it's defining 'b' as a function in the
class's namespace, storing a reference to that function in the tuple,
and then, when the class definition ends, it's wrapping the function as
a method.

You'll find:

 >>> a.foo[1]
<function b at 0x00B42C30>
 >>> a.b
<unbound method a.b>
 >>> a.b.im_func is a.foo[1]
True



From tjreedy at udel.edu  Thu Feb 21 05:31:01 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Feb 2013 23:31:01 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
	Exception for All Branches
In-Reply-To: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
Message-ID: <kg47ul$7ib$1@ger.gmane.org>

On 2/19/2013 9:48 PM, Todd Rovito wrote:
> Please see http://www.python.org/dev/peps/pep-0434/ for the complete
> PEP.  The idea for the PEP was generated because a right click menu
> was added to IDLE in Python 2.7 this started a debate about bug fix VS
> enhancement see the PEP references for more information.  IDLE has
> many patches that already exist but are difficult to get committed.
> This PEP is designed to make it easier to bring IDLE up to speed with
> modern GUI standards.  The PEP is not perfect some issues that need to
> be solved:

In my view, the PEP is definitely about allowing minor enhancements to 
existing features in bugfix releases. It is maybe about backporting new 
features such as would require a new entry on the menus. The right-click 
menu does not fit either category but it is more like the latter. In any 
case, I think it is an innocuous addition and should be allowed if a 
committer wants to do the backport.

I do not think the PEP should be a blank check to backport currently 
hypothetical major revisions such a themed widgets or tabbed windows. If 
and when such a thing is tested and ready to be put into the 'next' 
version, the possibility of backporting it should be a separate discussion.

> -How do we test IDLE patches to make sure they are high quality?

This is a separate issue from the PEP. Those of us who have and are 
working on IDLE issues are aware of the need to test on Windows, Linux, 
and Mac, and to test 2.7 separately from 3.x. There is a tracker issue 
for improving IDLE tests.

> -Does it make sense to separate IDLE from the stdlib for Python 3.4?

Not to me.

> I understand the batteries included argument, but if IDLE development
> is going to continue to proceed at a more accelerated pace than
> Python, it might make sense to separate it out into its own project or
> a sub-project.

We typically release bugfixes about every 6-9 months and I think that is 
often enough to release changes. The subject of the PEP is what should 
go into those releases.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Thu Feb 21 05:37:43 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Feb 2013 23:37:43 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
	Exception for All Branches
In-Reply-To: <kg3u0n$jug$1@ger.gmane.org>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
	<CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>
	<kg3hga$73s$1@ger.gmane.org> <20130221001244.1b602cfd@pitrou.net>
	<kg3u0n$jug$1@ger.gmane.org>
Message-ID: <5125A497.9040905@udel.edu>

On 2/20/2013 8:41 PM, Terry Reedy wrote:
> On 2/20/2013 6:12 PM, Antoine Pitrou wrote:

>> but, really, thinking IDLE's development is hampered by Python's
>> release process is *completely* ridiculous.
>
> I do not believe I have said that and certainly have not meant to say
> that

Re-reading Todd's original post, I see that he did suggest that as a 
possibility. I expressed my disagreement with this red herring in a 
direct response to his post.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Thu Feb 21 05:39:38 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Feb 2013 23:39:38 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
	Exception for All Branches
In-Reply-To: <CAGMu_=p3NxQQd-dDG+vv2zLEbL8k5Nuk_tZEKO1H1CRi6f7N-g@mail.gmail.com>
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
	<CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>
	<kg3hga$73s$1@ger.gmane.org> <20130221001244.1b602cfd@pitrou.net>
	<kg3u0n$jug$1@ger.gmane.org>
	<CAGMu_=p3NxQQd-dDG+vv2zLEbL8k5Nuk_tZEKO1H1CRi6f7N-g@mail.gmail.com>
Message-ID: <kg48ep$a9b$2@ger.gmane.org>

On 2/20/2013 8:44 PM, Andre Roberge wrote:
>  From what I have read on edu-sig, I believe it would be a huge mistake
> to remove IDLE from the Python standard library.

The idea was proposed 2 1/2 years ago, partly because IDLE was then 
buggy and mostly not maintained. The idea was rejected in favor of 
improving IDLE. Since then, many bugs have been quashed and idle is 
getting active attention.

Do you have any idea what education users think about backporting IDLE 
enhancements? Would they support the PEP? Or would they rather only get 
true bugfixes in bugfix releases, even for IDLE?

-- 
Terry Jan Reedy



From solipsis at pitrou.net  Thu Feb 21 08:01:56 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 08:01:56 +0100
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
 Exception for All Branches
References: <CALZ1apz8Hm1YomczbdZWhZ0u0B9u6mdD3+jLc2j1d4m427jL1g@mail.gmail.com>
	<20130220051028.GA22784@ando>
	<20130220090426.66ebbe8e@anarchist.wooz.org>
	<CADiSq7dSgW+1aSXMuSp3HFzrj_5gZTnuke1SLPoEGokwDbgUoQ@mail.gmail.com>
	<kg3hga$73s$1@ger.gmane.org> <20130221001244.1b602cfd@pitrou.net>
	<kg3u0n$jug$1@ger.gmane.org>
Message-ID: <20130221080156.0a3947af@pitrou.net>

On Wed, 20 Feb 2013 20:41:29 -0500
Terry Reedy <tjreedy at udel.edu> wrote:
> 
> > What is IDLE's actual maintenance activity, exactly?
> 
> I am not sure whether you are asking about volume or focus. A year ago, 
> focus was on bugs that caused IDLE to quit because of an uncaught 
> exception. When IDLE is started on Windows from an icon, this looks like 
> a crash because there is no visible traceback or exit message. Recently, 
> other issues have been worked on. In the last 5 months, activity has 
> picked up and there are about 40 issues in 3.4 misc/news whose commit 
> message contains 'idle'.

Ah, fair enough. In this case IDLE should probably remain, at least as
long as it's maintained anyway.

Regards

Antoine.




From ubershmekel at gmail.com  Thu Feb 21 08:20:54 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Thu, 21 Feb 2013 09:20:54 +0200
Subject: [Python-ideas] Fwd: argparse - add support for environment
	variables
In-Reply-To: <CAF-Rda8kobVf6yceb2aSL5BxZ4U=OaExzWqRCw2=Ob5fqbcVEQ@mail.gmail.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
	<CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
	<5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com>
	<CAF-Rda8kobVf6yceb2aSL5BxZ4U=OaExzWqRCw2=Ob5fqbcVEQ@mail.gmail.com>
Message-ID: <CANSw7KyT0fi94rJy8hM78vZL_+2O4pp=V_xaV9en9pryT-2jgQ@mail.gmail.com>

On Thu, Feb 21, 2013 at 2:18 AM, Eli Bendersky <eliben at gmail.com> wrote:

> What about config files? Mix them into argparse too? Oh, and support for
> .ini/XML/YAML/JSON/sqlite configs, please!
>
> Sarcasm aside (don't take it personally ;-), my point is this: argparse is
> a nice, orthogonal library for dealing with command-line arguments. Let's
> keep it this way. Python gives you tools to *very easily* mix up different
> configuration paths, with really minimal effort.
>

Excellent point,

Yuval Greenfield
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/358886e1/attachment.html>

From solipsis at pitrou.net  Thu Feb 21 10:21:45 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 10:21:45 +0100
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
 Exception for All Branches
References: <20130221001244.1b602cfd@pitrou.net>
	<51258A30.8090809@illinois.edu>
Message-ID: <20130221102145.2b8edebd@pitrou.net>


Hello Roger,

Le Wed, 20 Feb 2013 20:45:04 -0600,
serwy <roger.serwy at gmail.com> a ?crit :
> 
> I am the developer of the IdleX project
> (http://idlex.sourceforge.net). It's not a fork of IDLE, but just a
> collection of extensions that can be easily merged into IDLE.
> 
> I started the project because of the slow pace of IDLE development, 
> especially when having several outstanding patches languishing in the 
> bug tracker. With your help I'd like to push IDLE forward.

I guess it's a good idea to try and merge improvements into IDLE, but
IDLE developers such as Terry would be in a better position to answer.

Regards

Antoine.




From jeanpierreda at gmail.com  Thu Feb 21 12:01:19 2013
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Thu, 21 Feb 2013 06:01:19 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the standard
	library
Message-ID: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>

I've been noticing a lot of security-related issues being discussed in
the Python world since the Ruby YAML problemcame out. Is it time to
consider adding an alternative to pickle that is safe(r) by default?

Pickle is usable in situations few other things are, because it can
handle cyclic references and virtually any python object. The only
stdlib alternative I'm aware of is json, which can do neither of those
things. (Or at least, not without significant extra serialization
code.) I would imagine that any alternative supplied should be easy
enough to use that pickle users would seriously consider switching,
and include at least those features.

The benefit of using a secure alternative to pickle is that it
increases the difficulty of creating an insecure application, even for
those that are aware of the risks of the pickle module. With the
pickle module, you are one mistake away from an insecure program: all
you need is to have a way for the attacker to influence input to
pickle. With a secure alternative, even if you make that mistake, it
doesn't immediately result in a compromised application. You would
need another mistake on top of that that results in the deserialized
input being used improperly.

The only third party library I'm aware of that attempts to be a
safe/usable pickle replacement is cerealizer[1]_. Would it be
worth considering adding cerealizer, or something like it, to the
stdlib?

.. [1]: http://home.gna.org/oomadness/en/cerealizer/index.html

-- Devin


From solipsis at pitrou.net  Thu Feb 21 12:11:35 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 12:11:35 +0100
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
Message-ID: <20130221121135.5a1aabd6@pitrou.net>

Le Thu, 21 Feb 2013 06:01:19 -0500,
Devin Jeanpierre <jeanpierreda at gmail.com>
a ?crit :
> I've been noticing a lot of security-related issues being discussed in
> the Python world since the Ruby YAML problemcame out. Is it time to
> consider adding an alternative to pickle that is safe(r) by default?

There's already json. Is something else needed?

Regards

Antoine.




From jeanpierreda at gmail.com  Thu Feb 21 12:47:58 2013
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Thu, 21 Feb 2013 06:47:58 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <20130221121135.5a1aabd6@pitrou.net>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
Message-ID: <CABicbJL=LSHpVGLHLb+qLBTAJKESHZfQyAjOGx=d6fWHqx82cg@mail.gmail.com>

On Thu, Feb 21, 2013 at 6:11 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> I've been noticing a lot of security-related issues being discussed in
>> the Python world since the Ruby YAML problemcame out. Is it time to
>> consider adding an alternative to pickle that is safe(r) by default?
>
> There's already json. Is something else needed?

json can't handle cyclic references, and can't handle arbitrary python
types. Even if you pass in a custom default and object_pairs_hook to
json.dump and json.load respectively, it is impossible to serialize a
subclass of (e.g.) dict as anything except the way dict is serialized,
which will generally be incorrect.

Even if this is changed, creating custom hooks in default and
object_pairs_hook is a lot of work compared to using pickle (or,
indeed, cerealizer), which handles this automatically.

In some circumstances using pickle is clearly the wrong choice (e.g.
storing data in cookies), but at the same time it is easier to do the
wrong thing than the right thing.

-- Devin


From jsbueno at python.org.br  Thu Feb 21 13:38:23 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Thu, 21 Feb 2013 09:38:23 -0300
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CABicbJL=LSHpVGLHLb+qLBTAJKESHZfQyAjOGx=d6fWHqx82cg@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<CABicbJL=LSHpVGLHLb+qLBTAJKESHZfQyAjOGx=d6fWHqx82cg@mail.gmail.com>
Message-ID: <CAH0mxTQ9-d7YHkvV=Y1nkxdKW=5PLfWXYZC-T6esWqQ3yU+nfg@mail.gmail.com>

On 21 February 2013 08:47, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
> On Thu, Feb 21, 2013 at 6:11 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> I've been noticing a lot of security-related issues being discussed in
>>> the Python world since the Ruby YAML problemcame out. Is it time to
>>> consider adding an alternative to pickle that is safe(r) by default?
>>
>> There's already json. Is something else needed?
>
> json can't handle cyclic references, and can't handle arbitrary python
> types. Even if you pass in a custom default and object_pairs_hook to
> json.dump and json.load respectively, it is impossible to serialize a
> subclass of (e.g.) dict as anything except the way dict is serialized,
> which will generally be incorrect.
>
> Even if this is changed, creating custom hooks in default and
> object_pairs_hook is a lot of work compared to using pickle (or,
> indeed, cerealizer), which handles this automatically.
>
> In some circumstances using pickle is clearly the wrong choice (e.g.
> storing data in cookies), but at the same time it is easier to do the
> wrong thing than the right thing.

Do you think a couple hleper functions to json could help? Funcitons that
would translate a complex Python object into a dictionary, containing all type
information, and object metadata - still yield a simple dictionary.
(instance attributes
of the root object would be under that dictionary's "__dict__" key,
for example)

Cyclic reference would require something more complex - but just these
could allow one to json serialize arbitrary objects.

Maybe these helper functions could be in the json module itself.



>
> -- Devin
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From eric at trueblade.com  Thu Feb 21 14:32:47 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 21 Feb 2013 08:32:47 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <20130221121135.5a1aabd6@pitrou.net>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
Message-ID: <512621FF.6070606@trueblade.com>

On 2/21/2013 6:11 AM, Antoine Pitrou wrote:
> Le Thu, 21 Feb 2013 06:01:19 -0500,
> Devin Jeanpierre <jeanpierreda at gmail.com>
> a ?crit :
>> I've been noticing a lot of security-related issues being discussed in
>> the Python world since the Ruby YAML problemcame out. Is it time to
>> consider adding an alternative to pickle that is safe(r) by default?
> 
> There's already json. Is something else needed?

As stated elsewhere, it's cycles and especially arbitrary python objects
that are the big draw for pickle.

I've always wanted a version of pickle.loads() that takes a list of
classes that are allowed to be instantiated. Often, when using pickle to
serialize over say AMQP or some other transport, I know what classes I
want to allow. Anything else is either a (not infrequent) logic error or
an attack of some sort.

I realize this isn't perfect, but it would certainly reduce the attack
surface for many of my use cases. I'm already authenticating the sender,
and when I'm really paranoid I also sign the pickles.

Just a thought.

-- 
Eric.


From wolfgang.maier at biologie.uni-freiburg.de  Thu Feb 21 14:35:17 2013
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Thu, 21 Feb 2013 13:35:17 +0000 (UTC)
Subject: [Python-ideas] range
Message-ID: <loom.20130221T120736-301@post.gmane.org>

Dear all,
what do you think about allowing the use of operators on ranges?

I thought of things like:
a = range(1,10)
b = range(5,12)
intersect = a & b     # equivalent to intersect = range(5,10)
merge = a | b         # equivalent to merge = range(1,12)

to make this work in more complex cases, like:
a = range(1,10)
b = range(12,15)
merge = a | b         # equivalent to sth. like range(range(1,10),range(12,15))

maybe range nesting would be desirable, but the full sequence of values of such
a nested range could still be calculated on demand.

Such syntax would allow for the generation of more complex sequences, and, as a
side-effect, ranges could also be checked for overlap easily:
if a & b:
  do_something()

this whole idea is reminiscent, of course, of what's implemented for sets
already, so like there you could think of
intersect = a & b as shorthand for intersect = a.intersection(b)

Opinions?





From mark.hackett at metoffice.gov.uk  Thu Feb 21 14:39:02 2013
From: mark.hackett at metoffice.gov.uk (Mark Hackett)
Date: Thu, 21 Feb 2013 13:39:02 +0000
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <512621FF.6070606@trueblade.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
Message-ID: <201302211339.02892.mark.hackett@metoffice.gov.uk>

On Thursday 21 Feb 2013, Eric V. Smith wrote:
> On 2/21/2013 6:11 AM, Antoine Pitrou wrote:
> > Le Thu, 21 Feb 2013 06:01:19 -0500,
> > Devin Jeanpierre <jeanpierreda at gmail.com>
> >
> > a ?crit :
> >> I've been noticing a lot of security-related issues being discussed in
> >> the Python world since the Ruby YAML problemcame out. Is it time to
> >> consider adding an alternative to pickle that is safe(r) by default?
> >
> > There's already json. Is something else needed?
> 
> As stated elsewhere, it's cycles and especially arbitrary python objects
> that are the big draw for pickle.
> 
> I've always wanted a version of pickle.loads() that takes a list of
> classes that are allowed to be instantiated. Often, when using pickle to
> serialize over say AMQP or some other transport, I know what classes I
> want to allow. Anything else is either a (not infrequent) logic error or
> an attack of some sort.
> 
> I realize this isn't perfect, but it would certainly reduce the attack
> surface for many of my use cases. I'm already authenticating the sender,
> and when I'm really paranoid I also sign the pickles.
> 
> Just a thought.
> 

Is this not better solved by other methods? I.e. wasteful, but effective would 
be to send it all by XML.


From ubershmekel at gmail.com  Thu Feb 21 14:50:18 2013
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Thu, 21 Feb 2013 15:50:18 +0200
Subject: [Python-ideas] range
In-Reply-To: <loom.20130221T120736-301@post.gmane.org>
References: <loom.20130221T120736-301@post.gmane.org>
Message-ID: <CANSw7Kyoo7_yh_9TEmgOCLASfvXjn6QUXojbqiJu-BVnXZyrxA@mail.gmail.com>

On Thu, Feb 21, 2013 at 3:35 PM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:

> Dear all,
> what do you think about allowing the use of operators on ranges?
>
> I thought of things like:
> a = range(1,10)
> b = range(5,12)
> intersect = a & b     # equivalent to intersect = range(5,10)
> merge = a | b         # equivalent to merge = range(1,12)
>
> to make this work in more complex cases, like:
> a = range(1,10)
> b = range(12,15)
> merge = a | b         # equivalent to sth. like
> range(range(1,10),range(12,15))
>
> maybe range nesting would be desirable, but the full sequence of values of
> such
> a nested range could still be calculated on demand.
>
> Such syntax would allow for the generation of more complex sequences, and,
> as a
> side-effect, ranges could also be checked for overlap easily:
> if a & b:
>   do_something()
>
> this whole idea is reminiscent, of course, of what's implemented for sets
> already, so like there you could think of
> intersect = a & b as shorthand for intersect = a.intersection(b)
>
> Opinions?
>
>
>
This has been suggested in the past.

http://mail.python.org/pipermail/python-bugs-list/2012-June/173343.html

I think each operation you implement can have a lot of different bikeshed
details. Since these are all very simple to begin with, it's better to not
burden the language and just let users implement whatever it is they need.

Though, maybe you can show some popular and compelling use cases in the
wild?

Yuval Greenfield
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/57f0bfe2/attachment.html>

From eric at trueblade.com  Thu Feb 21 14:53:13 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 21 Feb 2013 08:53:13 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <201302211339.02892.mark.hackett@metoffice.gov.uk>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<201302211339.02892.mark.hackett@metoffice.gov.uk>
Message-ID: <512626C9.6010105@trueblade.com>

On 2/21/2013 8:39 AM, Mark Hackett wrote:
> On Thursday 21 Feb 2013, Eric V. Smith wrote:
>> On 2/21/2013 6:11 AM, Antoine Pitrou wrote:
>>> Le Thu, 21 Feb 2013 06:01:19 -0500,
>>> Devin Jeanpierre <jeanpierreda at gmail.com>
>>>
>>> a ?crit :
>>>> I've been noticing a lot of security-related issues being discussed in
>>>> the Python world since the Ruby YAML problemcame out. Is it time to
>>>> consider adding an alternative to pickle that is safe(r) by default?
>>>
>>> There's already json. Is something else needed?
>>
>> As stated elsewhere, it's cycles and especially arbitrary python objects
>> that are the big draw for pickle.
>>
>> I've always wanted a version of pickle.loads() that takes a list of
>> classes that are allowed to be instantiated. Often, when using pickle to
>> serialize over say AMQP or some other transport, I know what classes I
>> want to allow. Anything else is either a (not infrequent) logic error or
>> an attack of some sort.
>>
>> I realize this isn't perfect, but it would certainly reduce the attack
>> surface for many of my use cases. I'm already authenticating the sender,
>> and when I'm really paranoid I also sign the pickles.
>>
>> Just a thought.
>>
> 
> Is this not better solved by other methods? I.e. wasteful, but effective would 
> be to send it all by XML.

Sure. I could write a serializer (to XML or whatever) that handles
graphs of arbitrary python objects, but then I'm duplicating most of
what pickle does. I'd rather leverage all of the work that pickle
represents. Maybe I'll write a patch just to see what's involved.

-- 
Eric.


From solipsis at pitrou.net  Thu Feb 21 15:00:49 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 15:00:49 +0100
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
Message-ID: <20130221150049.2731682e@pitrou.net>

Le Thu, 21 Feb 2013 08:32:47 -0500,
"Eric V. Smith" <eric at trueblade.com> a ?crit :
> On 2/21/2013 6:11 AM, Antoine Pitrou wrote:
> > Le Thu, 21 Feb 2013 06:01:19 -0500,
> > Devin Jeanpierre <jeanpierreda at gmail.com>
> > a ?crit :
> >> I've been noticing a lot of security-related issues being
> >> discussed in the Python world since the Ruby YAML problemcame out.
> >> Is it time to consider adding an alternative to pickle that is
> >> safe(r) by default?
> > 
> > There's already json. Is something else needed?
> 
> As stated elsewhere, it's cycles and especially arbitrary python
> objects that are the big draw for pickle.

Of course, but it's being powerful which also makes pickle dangerous.

> I've always wanted a version of pickle.loads() that takes a list of
> classes that are allowed to be instantiated.

Is the following enough for you:
http://docs.python.org/3.4/library/pickle.html#restricting-globals
?

Regards

Antoine.




From eric at trueblade.com  Thu Feb 21 15:11:20 2013
From: eric at trueblade.com (Eric V. Smith)
Date: Thu, 21 Feb 2013 09:11:20 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <20130221150049.2731682e@pitrou.net>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
Message-ID: <51262B08.3030600@trueblade.com>

On 2/21/2013 9:00 AM, Antoine Pitrou wrote:
> Le Thu, 21 Feb 2013 08:32:47 -0500,
> "Eric V. Smith" <eric at trueblade.com> a ?crit :
>> On 2/21/2013 6:11 AM, Antoine Pitrou wrote:
>>> Le Thu, 21 Feb 2013 06:01:19 -0500,
>>> Devin Jeanpierre <jeanpierreda at gmail.com>
>>> a ?crit :
>>>> I've been noticing a lot of security-related issues being
>>>> discussed in the Python world since the Ruby YAML problemcame out.
>>>> Is it time to consider adding an alternative to pickle that is
>>>> safe(r) by default?
>>>
>>> There's already json. Is something else needed?
>>
>> As stated elsewhere, it's cycles and especially arbitrary python
>> objects that are the big draw for pickle.
> 
> Of course, but it's being powerful which also makes pickle dangerous.
> 
>> I've always wanted a version of pickle.loads() that takes a list of
>> classes that are allowed to be instantiated.
> 
> Is the following enough for you:
> http://docs.python.org/3.4/library/pickle.html#restricting-globals
> ?

Indeed, it is. Thanks for pointing it out! I've never gotten past the
module interface part of the docs. Maybe the warning at the top of the
page could also mention that there are ways to mitigate the safety
concerns, and point to #restricting-globals?

-- 
Eric.


From solipsis at pitrou.net  Thu Feb 21 16:41:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 16:41:42 +0100
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
Message-ID: <20130221164142.09cbc5dd@pitrou.net>

Le Thu, 21 Feb 2013 09:11:20 -0500,
"Eric V. Smith" <eric at trueblade.com> a ?crit :
> On 2/21/2013 9:00 AM, Antoine Pitrou wrote:
> > Le Thu, 21 Feb 2013 08:32:47 -0500,
> > "Eric V. Smith" <eric at trueblade.com> a ?crit :
> >> On 2/21/2013 6:11 AM, Antoine Pitrou wrote:
> >>> Le Thu, 21 Feb 2013 06:01:19 -0500,
> >>> Devin Jeanpierre <jeanpierreda at gmail.com>
> >>> a ?crit :
> >>>> I've been noticing a lot of security-related issues being
> >>>> discussed in the Python world since the Ruby YAML problemcame
> >>>> out. Is it time to consider adding an alternative to pickle that
> >>>> is safe(r) by default?
> >>>
> >>> There's already json. Is something else needed?
> >>
> >> As stated elsewhere, it's cycles and especially arbitrary python
> >> objects that are the big draw for pickle.
> > 
> > Of course, but it's being powerful which also makes pickle
> > dangerous.
> > 
> >> I've always wanted a version of pickle.loads() that takes a list of
> >> classes that are allowed to be instantiated.
> > 
> > Is the following enough for you:
> > http://docs.python.org/3.4/library/pickle.html#restricting-globals
> > ?
> 
> Indeed, it is. Thanks for pointing it out! I've never gotten past the
> module interface part of the docs. Maybe the warning at the top of the
> page could also mention that there are ways to mitigate the safety
> concerns, and point to #restricting-globals?

Yes, that would be a good idea :-)

Regards

Antoine.




From bruce at leapyear.org  Thu Feb 21 16:43:22 2013
From: bruce at leapyear.org (Bruce Leban)
Date: Thu, 21 Feb 2013 07:43:22 -0800
Subject: [Python-ideas] range
In-Reply-To: <CANSw7Kyoo7_yh_9TEmgOCLASfvXjn6QUXojbqiJu-BVnXZyrxA@mail.gmail.com>
References: <loom.20130221T120736-301@post.gmane.org>
	<CANSw7Kyoo7_yh_9TEmgOCLASfvXjn6QUXojbqiJu-BVnXZyrxA@mail.gmail.com>
Message-ID: <CAGu0AntyyW5HidxxyPaTbrUGb8Rf9XAG=zfumO-_GMV5wk_ZEw@mail.gmail.com>

On Thu, Feb 21, 2013 at 5:50 AM, Yuval Greenfield <ubershmekel at gmail.com>wrote:

> On Thu, Feb 21, 2013 at 3:35 PM, Wolfgang Maier <
> wolfgang.maier at biologie.uni-freiburg.de> wrote:
>
>> a = range(1,10)
>> b = range(5,12)
>> intersect = a & b     # equivalent to intersect = range(5,10)
>> merge = a | b         # equivalent to merge = range(1,12)
>>
>> ...

>  this whole idea is reminiscent, of course, of what's implemented for sets
>> already, so like there you could think of
>> intersect = a & b as shorthand for intersect = a.intersection(b)
>>
>> I think each operation you implement can have a lot of different bikeshed
> details. Since these are all very simple to begin with, it's better to not
> burden the language and just let users implement whatever it is they need.
>
>
Ignoring performance issues, imagine instead a sorted set class that you
could add ranges to (or remove them). Well, actually it's trivial to
convert a range to a set; the interesting problem is performance. You could
save the range parts unexpanded so you could add huge ranges and you could
also support infinite ranges (itertools.count).

I suggest this because once you consider doing the first set of operations,
like r1 & r2, it's quite natural for someone to want

some_non_unlucky_integers = range(100) - [13]

The idea of a sorted set class is, of course, not original. See for example
http://stutzbachenterprises.com/blist/sortedset.html. If SortedSet isn't in
the stdlib, I can't imagine this enhancement being either.

--- Bruce
Latest blog post: Alice's Puzzle Page http://www.vroospeak.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/9b39ba4a/attachment.html>

From dustin at v.igoro.us  Thu Feb 21 16:50:33 2013
From: dustin at v.igoro.us (Dustin J. Mitchell)
Date: Thu, 21 Feb 2013 10:50:33 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <51262B08.3030600@trueblade.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
Message-ID: <CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>

This conversation worries me.  The security community has shown that safety
isn't something you can add to a powerful tool.  With great power comes
great expressivity, and correspondingly more difficulty reasoning about
it.  Not to mention reasoning about yhe implementation.  JSON is probably
secure against code-execution exploits, but only probably.

When you put something in the stdlib and call it "safe", even with caveats,
people will make even more brazen mistakes than with a documented-unsafe
tool like pickle.

Dustin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/4687db5b/attachment.html>

From masklinn at masklinn.net  Thu Feb 21 17:19:50 2013
From: masklinn at masklinn.net (Masklinn)
Date: Thu, 21 Feb 2013 17:19:50 +0100
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
Message-ID: <00C63493-1733-46CE-A37D-2D0477A00C54@masklinn.net>

On 2013-02-21, at 16:50 , Dustin J. Mitchell wrote:
> This conversation worries me.  The security community has shown that safety isn't something you can add to a powerful tool.  With great power comes great expressivity, and correspondingly more difficulty reasoning about it.  Not to mention reasoning about yhe implementation.  JSON is probably secure against code-execution exploits, but only probably.

Considering there's no provision whatsoever in JSON itself for directing
any kind of execution or programmatic-ish behavior (as opposed to YAML
and ? from what I understand ? XML) why "only probably"?

I could see JSON implementations having vulnerability and applications
using JSON to do unsafe things (e.g. eval'ing JSON-sourced strings), but
JSON itself?

From ned at nedbatchelder.com  Thu Feb 21 17:38:21 2013
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 21 Feb 2013 11:38:21 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <00C63493-1733-46CE-A37D-2D0477A00C54@masklinn.net>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<00C63493-1733-46CE-A37D-2D0477A00C54@masklinn.net>
Message-ID: <51264D7D.5010003@nedbatchelder.com>

On 2/21/2013 11:19 AM, Masklinn wrote:
> On 2013-02-21, at 16:50 , Dustin J. Mitchell wrote:
>> This conversation worries me.  The security community has shown that safety isn't something you can add to a powerful tool.  With great power comes great expressivity, and correspondingly more difficulty reasoning about it.  Not to mention reasoning about yhe implementation.  JSON is probably secure against code-execution exploits, but only probably.
> Considering there's no provision whatsoever in JSON itself for directing
> any kind of execution or programmatic-ish behavior (as opposed to YAML
> and ? from what I understand ? XML) why "only probably"?

I was going to say, "YAML the format does not include execution," but 
then I went to read the YAML spec about the !! notation, and I honestly 
have no idea what it means.  YAML is scary...

--Ned.

> I could see JSON implementations having vulnerability and applications
> using JSON to do unsafe things (e.g. eval'ing JSON-sourced strings), but
> JSON itself?
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



From jeanpierreda at gmail.com  Thu Feb 21 18:18:50 2013
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Thu, 21 Feb 2013 12:18:50 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
Message-ID: <CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>

On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell <dustin at v.igoro.us> wrote:
> When you put something in the stdlib and call it "safe", even with caveats,
> people will make even more brazen mistakes than with a documented-unsafe
> tool like pickle.

Then how do we improve on the status quo? The best situation can't
possibly be one in which the standard serialization tool allows for
code injection exploits out of the box, by default, and where there is
no reasonable alternative in the stdlib without such problems.

To my ears, this objection is like objecting to the inclusion of
raw_input. Surely people will make even more brazen mistakes with a
so-called "safe" input method like raw_input, than with a
documented-unsafe tool like input()?

-- Devin


From abarnert at yahoo.com  Thu Feb 21 18:20:28 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 21 Feb 2013 09:20:28 -0800
Subject: [Python-ideas] range
In-Reply-To: <loom.20130221T120736-301@post.gmane.org>
References: <loom.20130221T120736-301@post.gmane.org>
Message-ID: <1CD63256-1739-4998-B84A-97F82C198712@yahoo.com>

On Feb 21, 2013, at 5:35, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:

> Dear all,
> what do you think about allowing the use of operators on ranges?
> 
> I thought of things like:
> a = range(1,10)
> b = range(5,12)
> intersect = a & b     # equivalent to intersect = range(5,10)
> merge = a | b         # equivalent to merge = range(1,12)
> 
> to make this work in more complex cases, like:
> a = range(1,10)
> b = range(12,15)
> merge = a | b         # equivalent to sth. like range(range(1,10),range(12,15))

That means ranges are no longer constant size, no longer have constant- time contains-testing, etc.; it's linear in the number of sub ranges. (And so would intersect and union, for that matter.) 

That's more like a sparse set or an RLE-encoded set or something--which is a useful thing for numerical work (does pandas have an equivalent?), but it's not really a range anymore.

Maybe build a new IntervalSet or something, which can interact with ranges (and itertools.count?) transparently (by treating them as a set of 1), put it on PyPI, and see how many users find themselves wishing that range(1, 10) | range(12, 15) would return an IntervalSet so they wouldn't have to do IntervalSet(range(1, 10)) | range(12, 15). That seems like the best way to find out whether this would be useful in practice.

Also, I think I'd expect difference and symmetric difference if I wanted union, and if you're handling infinite ranges I'd expect complement as well. Plus interaction with single ints as well as ranges and other sets. But that may be excessive feature creep--since I don't have an obvious use for union, I may be imagining things wrong.

> maybe range nesting would be desirable, but the full sequence of values of such
> a nested range could still be calculated on demand.
> 
> Such syntax would allow for the generation of more complex sequences, and, as a
> side-effect, ranges could also be checked for overlap easily:
> if a & b:
>  do_something()

This side effect seems like it would be useful more often than the main effect. I've written a ranges_intersect function in at least two projects, but I've never written an intersect_ranges. (That is, I only need a boolean value.)

> 
> this whole idea is reminiscent, of course, of what's implemented for sets
> already, so like there you could think of
> intersect = a & b as shorthand for intersect = a.intersection(b)
> 
> Opinions?
> 
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From mark.hackett at metoffice.gov.uk  Thu Feb 21 18:22:47 2013
From: mark.hackett at metoffice.gov.uk (Mark Hackett)
Date: Thu, 21 Feb 2013 17:22:47 +0000
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
Message-ID: <201302211722.47380.mark.hackett@metoffice.gov.uk>

On Thursday 21 Feb 2013, Devin Jeanpierre wrote:
> On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell <dustin at v.igoro.us> 
wrote:
> > When you put something in the stdlib and call it "safe", even with
> > caveats, people will make even more brazen mistakes than with a
> > documented-unsafe tool like pickle.
> 
> Then how do we improve on the status quo? The best situation can't
> possibly be one in which the standard serialization tool allows for
> code injection exploits out of the box, by default, and where there is
> no reasonable alternative in the stdlib without such problems.

By writing your application for its needs, not the needs of 10000 programs yet 
to be written and making the wrong assumption and putting it in a stdlib.

If every problem could be solved with a stdlib call, there'd only have to be 
one programmer in the world...


From solipsis at pitrou.net  Thu Feb 21 18:24:52 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 21 Feb 2013 18:24:52 +0100
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<201302211722.47380.mark.hackett@metoffice.gov.uk>
Message-ID: <20130221182452.1ef90184@pitrou.net>

Le Thu, 21 Feb 2013 17:22:47 +0000,
Mark Hackett <mark.hackett at metoffice.gov.uk> a
?crit :

> On Thursday 21 Feb 2013, Devin Jeanpierre wrote:
> > On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell
> > <dustin at v.igoro.us> 
> wrote:
> > > When you put something in the stdlib and call it "safe", even with
> > > caveats, people will make even more brazen mistakes than with a
> > > documented-unsafe tool like pickle.
> > 
> > Then how do we improve on the status quo? The best situation can't
> > possibly be one in which the standard serialization tool allows for
> > code injection exploits out of the box, by default, and where there
> > is no reasonable alternative in the stdlib without such problems.
> 
> By writing your application for its needs, not the needs of 10000
> programs yet to be written and making the wrong assumption and
> putting it in a stdlib.
> 
> If every problem could be solved with a stdlib call, there'd only
> have to be one programmer in the world...

You're forgetting the millions of stdlib programmers :-)

Regards

Antoine.







From abarnert at yahoo.com  Thu Feb 21 18:32:37 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 21 Feb 2013 09:32:37 -0800
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
Message-ID: <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>

How often have you needed either cyclic references or the ability to dynamically store arbitrary classes in something like a cookie or a cache file?

It seems to me that when you're trying to do something that would be difficult to do with json (with the new helpers proposed by someone earlier in the thread), you're usually doing something too dangerous to store in a cookie anyway. If someone can construct instances of arbitrary classes, they can run code. If they can't, why did you need pickle?

I realize the overlap is just small, not zero. But I think catering to it could be an attractive nuisance that would lead to more unsafe code rather than less.

It's always easier to preserve safety while adding features to something simple, than to add safety while restricting features in something powerful.

Sorry for the top post.

Sent from a random iPhone

On Feb 21, 2013, at 9:18, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:

> On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell <dustin at v.igoro.us> wrote:
>> When you put something in the stdlib and call it "safe", even with caveats,
>> people will make even more brazen mistakes than with a documented-unsafe
>> tool like pickle.
> 
> Then how do we improve on the status quo? The best situation can't
> possibly be one in which the standard serialization tool allows for
> code injection exploits out of the box, by default, and where there is
> no reasonable alternative in the stdlib without such problems.
> 
> To my ears, this objection is like objecting to the inclusion of
> raw_input. Surely people will make even more brazen mistakes with a
> so-called "safe" input method like raw_input, than with a
> documented-unsafe tool like input()?
> 
> -- Devin
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From mark.hackett at metoffice.gov.uk  Thu Feb 21 18:33:02 2013
From: mark.hackett at metoffice.gov.uk (Mark Hackett)
Date: Thu, 21 Feb 2013 17:33:02 +0000
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <20130221182452.1ef90184@pitrou.net>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<201302211722.47380.mark.hackett@metoffice.gov.uk>
	<20130221182452.1ef90184@pitrou.net>
Message-ID: <201302211733.02145.mark.hackett@metoffice.gov.uk>

On Thursday 21 Feb 2013, Antoine Pitrou wrote:
> Le Thu, 21 Feb 2013 17:22:47 +0000,
> Mark Hackett <mark.hackett at metoffice.gov.uk> a
> 
> >
> > If every problem could be solved with a stdlib call, there'd only
> > have to be one programmer in the world...
> 
> You're forgetting the millions of stdlib programmers :-)
> 
> Regards
> 
> Antoine.
> 

But only because 10,000 different stdlib calls are needed!

(I should shush now, I don't want to give away the Programmer's Secret. Even 
if it IS very similar to the Lawyers Secret)...

Being serious, though, if your code requires a serious amount of security, 
you're better off writing your own parsing.

It does mean the poor guy taking over has to use this "stdlib", but that's 
more work for programmers.

Oh no, I said it...!

AaaarrrgghhCARRIER LOST


From abarnert at yahoo.com  Thu Feb 21 18:35:14 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 21 Feb 2013 09:35:14 -0800
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <20130221182452.1ef90184@pitrou.net>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<201302211722.47380.mark.hackett@metoffice.gov.uk>
	<20130221182452.1ef90184@pitrou.net>
Message-ID: <AE7C01A2-E1CF-4EBD-9DD8-1F4DAFB33833@yahoo.com>

On Feb 21, 2013, at 9:24, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 21 Feb 2013 17:22:47 +0000,
> Mark Hackett <mark.hackett at metoffice.gov.uk> a
> ?crit :
> 
>> On Thursday 21 Feb 2013, Devin Jeanpierre wrote:
>>> On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell
>>> <dustin at v.igoro.us>
>> wrote:
>>>> When you put something in the stdlib and call it "safe", even with
>>>> caveats, people will make even more brazen mistakes than with a
>>>> documented-unsafe tool like pickle.
>>> 
>>> Then how do we improve on the status quo? The best situation can't
>>> possibly be one in which the standard serialization tool allows for
>>> code injection exploits out of the box, by default, and where there
>>> is no reasonable alternative in the stdlib without such problems.
>> 
>> By writing your application for its needs, not the needs of 10000
>> programs yet to be written and making the wrong assumption and
>> putting it in a stdlib.
>> 
>> If every problem could be solved with a stdlib call, there'd only
>> have to be one programmer in the world...
> 
> You're forgetting the millions of stdlib programmers :-)

This is one of those "any sufficiently powerful language becomes lisp" things, isn't it. :)


> Regards
> 
> Antoine.
> 
> 
> 
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From Steve.Dower at microsoft.com  Thu Feb 21 18:55:57 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Thu, 21 Feb 2013 17:55:57 +0000
Subject: [Python-ideas] Adding a safe alternative to pickle in
	the	standard library
In-Reply-To: <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
Message-ID: <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>

> From: Andrew Barnert
> How often have you needed either cyclic references or the ability to
> dynamically store arbitrary classes in something like a cookie or a cache file?

In a past life I used pickle regularly to snapshot long-running (evolutionary) algorithms that used user-provided classes and all sorts of highly improper circular references. And there are plenty of researchers out there using Python for much crazier things than I ever did.

There is a lot more to Python than web apps...


Steve




From stephen at xemacs.org  Thu Feb 21 19:29:57 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 22 Feb 2013 03:29:57 +0900
Subject: [Python-ideas] Adding a safe alternative to pickle
	in	the	standard library
In-Reply-To: <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
Message-ID: <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>

Steve Dower writes:

 > In a past life I used pickle regularly to snapshot long-running
 > (evolutionary) algorithms that used user-provided classes

And how do you propose to prevent user-provided exploits, then?

Nobody wants to take away the power of pickle if it imposes only risks
you're happy to bear.  The question here is "is it possible to be
*safer* than pickle without giving up any of the power?"


From jeanpierreda at gmail.com  Thu Feb 21 19:48:06 2013
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Thu, 21 Feb 2013 13:48:06 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>

On Thu, Feb 21, 2013 at 1:29 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Steve Dower writes:
>
>  > In a past life I used pickle regularly to snapshot long-running
>  > (evolutionary) algorithms that used user-provided classes
>
> And how do you propose to prevent user-provided exploits, then?

Just because an application has one place where someone can inject new
code, doesn't mean it should have another. You might trust the people
that write these evolutionary algorithm classes, but not trust people
that give you snapshots of the algorithms running.

> Nobody wants to take away the power of pickle if it imposes only risks
> you're happy to bear.  The question here is "is it possible to be
> *safer* than pickle without giving up any of the power?"

I hope nobody is asking that question, because the answer is a strong
no. Pickle's ability to call arbitrary objects accessible in any
module, anywhere, is part of how powerful it is, but it is also a
fundamental source of unsafety.

That does not mean that we should not write or use safer alternatives.
We have written and do use safer alternatives, like the json module.
But it means we can't expect them to be usable exactly everywhere
pickle is. I would've said the question is how far in that direction
we should bother to go. How many features do you add before you're
increasing risk from faulty code, rather than decreasing it by making
it easier to use a secure-by-design library?

-- Devin


From Steve.Dower at microsoft.com  Thu Feb 21 19:52:31 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Thu, 21 Feb 2013 18:52:31 +0000
Subject: [Python-ideas] Adding a safe alternative to pickle
	in	the	standard library
In-Reply-To: <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com>

> From: Stephen J. Turnbull [mailto:stephen at xemacs.org]
> Sent: Thursday, February 21, 2013 1030
> To: Steve Dower
> Cc: Andrew Barnert; Devin Jeanpierre; Dustin J. Mitchell; Eric V. Smith;
> Python-Ideas
> Subject: Re: [Python-ideas] Adding a safe alternative to pickle in the standard
> library
> 
> Steve Dower writes:
> 
>  > In a past life I used pickle regularly to snapshot long-running  >
> (evolutionary) algorithms that used user-provided classes
> 
> And how do you propose to prevent user-provided exploits, then?
>
> Nobody wants to take away the power of pickle if it imposes only risks you're
> happy to bear.  The question here is "is it possible to be
> *safer* than pickle without giving up any of the power?"
> 

I was only aiming to provide perspective, rather than a proposal. The existing ability to customise the unpickler suited my needs perfectly, not that I actually used it. To make it safer I would have a restricted unpickler as the default (for load/loads) and force people to override it if they want to loosen restrictions.

To be really explicit, I would make load/loads only work with built-in types. For compatibility when reading earlier protocols we could add a type representing a class instance and its members that doesn't actually construct anything. (Maybe we override __getattr__ to make it lazily construct the instance when the program actually wants to use it?)

For convenience, I'd add a parameter to Unpickler to let the user provide a set of types that are allowed to be constructed (or a mapping from names to callables as used in find_class()).

Finally, I'd give greater exposure to overriding find_class(), as has already been suggested. People who want to unpickle any arbitrary type have to do this to opt-in. (Maybe that's a little aggressive?)

I'd expect this would come under a new protocol version, but I wouldn't be opposed to old data being unpickled under these rules, especially if the non-constructing type is actually a lazy-constructing type.


Cheers,
Steve


From foogod at gmail.com  Thu Feb 21 20:18:55 2013
From: foogod at gmail.com (Alex Stewart)
Date: Thu, 21 Feb 2013 11:18:55 -0800
Subject: [Python-ideas] PEP for enum library type?
Message-ID: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>

Hi all,

Sorry to jump into this discussion so late, but I just finished reading
through this thread and had a few thoughts..

(Sorry this message is a bit long.  TL;DR: Please check the list of
required/desired/undesired properties at the end and let me know if I've
gotten anything seriously wrong with my interpretation of the discussion
thus far)

It seems to me that this particular thread started out as a call to step
away from existing implementations and take a look at this issue from the
direction of "what do we want/need" instead, but then it quickly got
sidetracked back into discussing all the details of various
existing/proposed implementations.  I'd like to try to take a step back
(again) for a minute and raise the question:  What do we actually want to
get out of this whole endeavor?

First of all, as I see it, there are two main (and fairly distinct) use
cases for enums in Python:

   1. Predefined "unique values" for passing around within Python code
   2. API-defined constants for interacting with non-Python libraries, etc
   (i.e. C defines/enums that need to be represented in Python, or database
   field values, etc)

In non-Python code, typically enums have always been represented under the
covers as ints, and therefore must be passed back and forth as numbers.
 The fact that they have an integer value, however, is purely an
implementation artifact.  It comes from the fact that C and some other
languages don't have a rich enough type system to properly make enums their
own distinct types, but Python does not have this limitation, and I think
we should be careful not to constrain the way we do things within Python
just because of the limitations of other languages.

Where possible I believe we should conceptually be thinking of enums not as
"sequences of ints" but more as "collections of singletons".  That is, they
are simply objects, with a defined name and type, which compare equal to
themselves but not to others, and are generally related to others by some
sort of grouping mechanism (and the same name always maps to the same
object).  In this context, the idea of assigning a "value" to an enum makes
little sense and is arguably completely unnecessary.  (and if we eliminate
this aspect, it mitigates many of the issues that have been brought up
about evaluation order and accidental duplication, in addition to
potentially making the base design a lot simpler)

Obviously, the second use case does require an association between enums
and (typically int) values, but that could be viewed as simply a special
case of the larger definition of "enums", rather than the other way around.
 I do think one thing worth noting, however, is that (at least in my
experience) the cases which require associating names with values pretty
much also always require that every name has a specific value, so the value
for each and every enum within the group should probably be being defined
explicitly anyway (I have encountered very few cases where it's actually
useful to mix-and-match "I care about this value but I don't care about
those ones").  It doesn't seem unreasonable, therefore, to define two
different categories of enums: one that has no concept of "value" (for
pure-Python), and one which does have associated values but all values have
to be specified explicitly (for the "mapping constants" case).

On a related note, to be honest, I'm not really sure I can think of any
realistic use cases for "string enums" (or really anything other than ints
in general).  Does anybody have an example of where this would actually be
useful as opposed to just using "pure" (valueless) enums (which would
already have string names)?

Anyway, in the interest of trying to get the discussion back onto more
theoretical ground, I also wanted to try to summarize the more general
thoughts/impressions I've gleaned from the discussions up to this point.
 From what I can tell, these are some of the properties that there seems to
be some general consensus enums probably should or shouldn't have:

Required properties (without these, any implementation is not generally
useful, or is at least something different than an "enum"):

   1. Enums must be groupable in some way (i.e. "Colors", or "Error values")
   2. Enums within the same group must not compare equal to each other
   (unless two names are intentionally defined to map to the same enum (i.e.
   "aliases"))
   3. (Within reason and the limitations of Python) Once defined, an enum's
   properties (i.e. its name, identity, group membership, relationships to
   other objects, etc) must be treated as immutable (i.e. not change out from
   under the programmer unexpectedly).  Conceptually they should be considered
   to be "constants".

Desirable properties (which ones are more or less desirable will vary for
different people, but from what I've heard I think everybody sorta agrees
that all of these could be good things as long as they don't cause other
problems):

   1. Enums should represent themselves (str/repr) by symbolic names, not
   as ints, etc.
   2. Enums from different groups should preferably not compare equal to
   each other (even if they have the same associated int value).
   3. It should be possible to determine what group an enum belongs to.
   4. Enums/groups should be definable inline using a fairly simple Python
   syntax.
   5. It should also be relatively easy to define enums/groups
   programmatically.
   6. By default, enums should be referenceable as relatively simple
   identifiers (i.e. no need for quoting, function-calls, etc, just
   variables/attributes/etc)
   7. If the programmer doesn't care about the value of an enum, she
   shouldn't have to explicitly state a meaningless value.
   8. (If an enum does have an associated value) it should be easy to
   compare with and/or convert back and forth between enums and values (so
   that they can be used with existing APIs).
   9. It would be nice to be able to associate docstrings, and possibly
   other metadata with enums.

Undesirable properties:

   1. Enum syntax should not be "too magic".  (In particular, it's pretty
   clear at this point that creating new enums as a side-effect of name
   lookups (even as convenient as it does make the syntax) is ultimately not
   gonna fly)
   2. The syntax for defining enums should not be so onerous or verbose
   that it's significantly harder to use than the existing idioms people are
   already using.
   3. The syntax for defining enums should not be so alien that it will
   completely baffle programmers who are already used to typical Python
   constructs.
   4. It shouldn't be necessary to quote enum names when defining them
   (since they won't be quoted when they're used)

I want to check: Is this a valid summary of things?  Anything I missed, or
do people have substantial objections to any of the
required/desirable/undesirable points I mentioned?

Obviously, it may not be possible to achieve all of the desirable
properties at the same time, but I think it's useful to start with an idea
of what we'd ideally like, and then we can sit down and see how close we
can actually get to it..

(Actually, on pondering these properties, I've started to put together a
slightly different enum implementation which I think has some potential
(it's somewhat a cross between Barry's and Tim's with a couple of ideas of
my own).  I think I'll flesh it out a little more and then put it up for
comment as a separate thread, if people don't mind..)

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/9b73c9e3/attachment.html>

From miki.tebeka at gmail.com  Thu Feb 21 20:34:46 2013
From: miki.tebeka at gmail.com (Miki Tebeka)
Date: Thu, 21 Feb 2013 11:34:46 -0800 (PST)
Subject: [Python-ideas] Fwd: argparse - add support for environment
 variables
In-Reply-To: <CAF-Rda8kobVf6yceb2aSL5BxZ4U=OaExzWqRCw2=Ob5fqbcVEQ@mail.gmail.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAN-Kwu0nXzsKXmdfrxos8LvG4oKUMqCSSgMh3Wf_E8dUp6cVSw@mail.gmail.com>
	<CAN-Kwu3WcRJ5BZRxhwB1cHd=nD_cEBkbknHREcGRVQxob-vk-A@mail.gmail.com>
	<f83fb8f8-249b-42e7-b92f-47e5bedf2337@googlegroups.com>
	<CAF-Rda8CfoiA17ew5=Gw=ui49D0v3NSBm5gW6o_34oZ+RjEWyQ@mail.gmail.com>
	<5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com>
	<CAF-Rda8kobVf6yceb2aSL5BxZ4U=OaExzWqRCw2=Ob5fqbcVEQ@mail.gmail.com>
Message-ID: <eaa904e7-a7c3-4e90-bf38-b301c4bbfea7@googlegroups.com>



> What about config files? Mix them into argparse too? Oh, and support for 
> .ini/XML/YAML/JSON/sqlite configs, please!
>
Well, baby steps toward configuration solution :)
The other thing is that unlike config files which have many formats, 
environment has one uniform access to it. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/48b36d81/attachment.html>

From jsbueno at python.org.br  Thu Feb 21 20:54:49 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Thu, 21 Feb 2013 16:54:49 -0300
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
Message-ID: <CAH0mxTTiSfrEPZ98jjCLwn=RnTHSfGvs_MERTdUY=j8-AxxgKw@mail.gmail.com>

On 21 February 2013 16:18, Alex Stewart <foogod at gmail.com> wrote:
...

On a related note, to be honest, I'm not really sure I can think of
any realistic use cases for "string enums" (or really anything other
than ints in general).  Does anybody have an example of where this
would actually be useful as opposed to just using "pure" (valueless)
enums (which would already have string names)?

Backwards compatibility.
Of course, although you have restricted your wording to "enums" we
need constants.
And it would be nice to replace magic string contants in the Python
stdlib itself, by real -"unquoted" constants - and if the new
constants are equivalent to the strings existing in current code,
there is no breaking.


Think of """ my_text.encode("utf--8", errors=str.IGNORE) """, instead of
""" my_text.encode("utf--8", errors="ignore") """, for example.


...
> Anyway, in the interest of trying to get the discussion back onto more
> theoretical ground, I also wanted to try to summarize the more general
> thoughts/impressions I've gleaned from the discussions up to this point.
> From what I can tell, these are some of the properties that there seems to
> be some general consensus enums probably should or shouldn't have:
>
> Required properties (without these, any implementation is not generally
> useful, or is at least something different than an "enum"):
>
>1. Enums must be groupable in some way (i.e. "Colors", or "Error values")
> 2. Enums within the same group must not compare equal to each other (unless two
> names are intentionally defined to map to the same enum (i.e. "aliases"))
>3. (Within reason and the limitations of Python) Once defined, an enum's
> properties (i.e. its name, identity, group membership, relationships to
> other objects, etc) must be treated as immutable (i.e. not change out from
> under the programmer unexpectedly).  Conceptually they should be considered
> to be "constants".
4. They should subclass INTs in way it  is possible to interact with low level
libraries already using C Enums (only aplicable to those that need an
int value) -
their use should be as transparent as True and False are today.

>
> Desirable properties (which ones are more or less desirable will vary for
> different people, but from what I've heard I think everybody sorta agrees
> that all of these could be good things as long as they don't cause other
> problems):
>
>1. Enums should represent themselves (str/repr) by symbolic names, not as ints,
> etc.
I would say that, while not "required", as low level languages lack this,
the whole idea is simply not worth implementing  without this. This is the
real motivation - and NB. also for "Constants", not only "Enums".

>2. Enums from different groups should preferably not compare equal to each
> other (even if they have the same associated int value).
I am not shure about this requirement. I agree about "valueless" Enums, but
as far as they have values associated, they should behave just like
their values. I did not hear of anyone injured  by " True == 1" in Python.


>3. It should be possible to determine what group an enum belongs to.
> Enums/groups should be definable inline using a fairly simple Python syntax.
>4. It should also be relatively easy to define enums/groups programmatically.
>5. By default, enums should be referenceable as relatively simple identifiers
> (i.e. no need for quoting, function-calls, etc, just
> variables/attributes/etc)
>6. If the programmer doesn't care about the value of an enum, she shouldn't
> have to explicitly state a meaningless value.
>7. (If an enum does have an associated value) it should be easy to compare with
> and/or convert back and forth between enums and values (so that they can be
> used with existing APIs).
>8. It would be nice to be able to associate docstrings, and possibly other
> metadata with enums.
>
> Undesirable properties:
>
>1. Enum syntax should not be "too magic".  (In particular, it's pretty clear at
> this point that creating new enums as a side-effect of name lookups (even as
> convenient as it does make the syntax) is ultimately not gonna fly)
> 2.The syntax for defining enums should not be so onerous or verbose that it's
> significantly harder to use than the existing idioms people are already
> using.
>3. The syntax for defining enums should not be so alien that it will completely
> baffle programmers who are already used to typical Python constructs.
>4.  It shouldn't be necessary to quote enum names when defining them (since they
> won't be quoted when they're used)
>
> I want to check: Is this a valid summary of things?  Anything I missed, or
> do people have substantial objections to any of the
> required/desirable/undesirable points I mentioned?
>
> Obviously, it may not be possible to achieve all of the desirable properties
> at the same time, but I think it's useful to start with an idea of what we'd
> ideally like, and then we can sit down and see how close we can actually get
> to it..
>
> (Actually, on pondering these properties, I've started to put together a
> slightly different enum implementation which I think has some potential
> (it's somewhat a cross between Barry's and Tim's with a couple of ideas of
> my own).  I think I'll flesh it out a little more and then put it up for
> comment as a separate thread, if people don't mind..)


Ok -  I added there what I think are the most important.
What lead me to ressurect this subject was a period when I was dealing
with a couple of Image
packages related packages - PIL, GIMP-Python and Pygame -- all of them
make intense use of
Constants and Enums...all of which print as "int"s. All the three have
the common trait that
they can't rely on other 3rd party Python packages besides the stdlib
- since there
is a relatively large codebase depending only on them and Python.

  js
 -><-
> --Alex


From vinay_sajip at yahoo.co.uk  Thu Feb 21 21:16:46 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 21 Feb 2013 20:16:46 +0000 (UTC)
Subject: [Python-ideas]
	=?utf-8?q?Adding_a_safe_alternative_to_pickle_in_t?=
	=?utf-8?q?he_standard=09library?=
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
Message-ID: <loom.20130221T204800-323@post.gmane.org>

Devin Jeanpierre <jeanpierreda at ...> writes:

> Pickle is usable in situations few other things are, because it can
> handle cyclic references and virtually any python object. The only
> stdlib alternative I'm aware of is json, which can do neither of those
> things. (Or at least, not without significant extra serialization
> code.) I would imagine that any alternative supplied should be easy
> enough to use that pickle users would seriously consider switching,
> and include at least those features.

What I'm about to mention may or may not meet your use case, but there is a way
in which JSON can support cyclic references and virtually any Python object.
Not without *some* work and *some* limitations, obviously, but then,
TANSTAAFL :-)

In logging.dictConfig, we use a dict to configure logging, but the underlying
mechanism is more general. It allows you to construct objects outside the
dict using a syntax such as 'ext://sys.stderr', and objects inside the dict
using syntax such as 'cfg://xyz.abc[1]'. To illustrate, consider the following
JSON:

{
    "v1": "ext://sys.stderr",
    "v2": "cfg://v3.some_list[1]",
    "v3": {
        "some_list": [1, 2, 3],
        "some_value": "cfg://v4.key"
    },
    "v4": {
        "key": "value",
        "some_value": "cfg://v3.some_value"
    }
}

You can see that sys.stderr is the external object, and that there are various
cyclic references. Now, using Python 2.7 (where dictConfig was introduced),
you can do this:

>>> import json
>>> with open("example.json") as f:
...     d = json.load(f)
...
>>> d
{u'v1': u'ext://sys.stderr', u'v2': u'cfg://v3.some_list[1]', u'v3': {u'some_lis
t': [1, 2, 3], u'some_value': u'cfg://v4.key'}, u'v4': {u'some_value': u'cfg://v
3.some_value', u'key': u'value'}}
>>> import logging.config
>>> bc = logging.config.BaseConfigurator(d)
>>> bc.config['v1']
<open file '<stderr>', mode 'w' at 0x004750D0>
>>> bc.config['v2']
2
>>> bc.config['v3']
{u'some_list': [1, 2, 3], u'some_value': u'cfg://v4.key'}
>>> bc.config['v3']['some_value']
u'value'
>>> bc.config['v4']['some_value']
u'value'

The values are lazily evaluated (as you can see from the above example for
bc.config['v3'], but that isn't necessarily a problem.

Of course, one can build on this using keys to define classes, kwargs, etc.,
which is how logging's DictConfigurator (derived from BaseConfigurator) works.
The handlers for ext://, cfg:// are customisable, and additional schemes can
be added; it allows for fairly flexible configuration, but within bounds you
can set.

I probably should write a post about the BaseConfigurator, it is useful
outside the context of logging and I have used it for other things, but it's
hidden away under the covers a bit.

Regards,

Vinay Sajip




From foogod at gmail.com  Thu Feb 21 21:34:25 2013
From: foogod at gmail.com (Alex Stewart)
Date: Thu, 21 Feb 2013 12:34:25 -0800 (PST)
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAH0mxTTiSfrEPZ98jjCLwn=RnTHSfGvs_MERTdUY=j8-AxxgKw@mail.gmail.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<CAH0mxTTiSfrEPZ98jjCLwn=RnTHSfGvs_MERTdUY=j8-AxxgKw@mail.gmail.com>
Message-ID: <35679e42-3cd1-4fa4-b5ef-99e3fad35b79@googlegroups.com>

On Thursday, February 21, 2013 11:54:49 AM UTC-8, Joao S. O. Bueno wrote:

> 4. They should subclass INTs in way it  is possible to interact with low 
> level 
> libraries already using C Enums (only aplicable to those that need an 
> int value) - 
> their use should be as transparent as True and False are today. 
>

Ok, I can see that would be a desirable trait (and personally I'm with you 
on that one), though I'm not sure it really qualifies as "required" (there 
are potential uses for int-enums which would still be useful/valid even if 
it couldn't do that automatically), but I'll definitely add that to my 
"desirable" list..
 

> >1. Enums should represent themselves (str/repr) by symbolic names, not as 
> ints, 
> > etc. 
> I would say that, while not "required", as low level languages lack this, 
> the whole idea is simply not worth implementing  without this. This is the 
> real motivation - and NB. also for "Constants", not only "Enums".


Frankly, I think this one is so easy it's really not worth fussing about 
that much.  It's pretty much a given that any solution we come up with will 
have this functionality anyway..

>2. Enums from different groups should preferably not compare equal to each 
>
> other (even if they have the same associated int value). 
> I am not shure about this requirement. I agree about "valueless" Enums, 
> but 
> as far as they have values associated, they should behave just like 
> their values. I did not hear of anyone injured  by " True == 1" in Python.


Well, it's explicitly not a requirement (that's why it's under "desirable" 
instead of "required").  Here's the question, though:  Do you believe if a 
solution was proposed which had this property it would be a bad thing?

Note that this isn't really exactly the same as the "True == 1" case.  In 
my opinion the argument in favor of this is along the lines of:

class FooOptions (Enum):
    YES = 1
    MAYBE = 2
    NO = 3

class BarOptions (Enum):
    NO = 1
    MAYBE = 2
    YES = 3
 
def foofunc(choice):
    if choice == FooOptions.YES:
        do_something()


Now what if somebody calls foofunc(BarOptions.NO)?

Admittedly, this is a somewhat exaggerated case, but I still think it would 
be better if in these sorts of cases FooOptions.YES != BarOptions.NO (if 
one explicitly did want to test if the values were the same (regardless of 
enum type) they could do something like "if int(choice) == FooOptions.YES" 
instead)

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/5e96d408/attachment.html>

From greg.ewing at canterbury.ac.nz  Fri Feb 22 00:24:50 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 Feb 2013 12:24:50 +1300
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
Message-ID: <5126ACC2.3080904@canterbury.ac.nz>

On 22/02/13 08:18, Alex Stewart wrote:
> In non-Python code, typically enums have always been represented under the
> covers as ints, and therefore must be passed back and forth as numbers. The
> fact that they have an integer value, however, is purely an implementation
> artifact.  It comes from the fact that C and some other languages don't have
 > a rich enough type system to properly make enums their own distinct type

I don't think that's true for all languages. For example, enums in
Pascal are definitely distinct types from integers, yet the language
explicitly assigns them ordinal values and defines an ordering for
them. Wirth must have thought there was a benefit in doing that.

> It doesn't seem
> unreasonable, therefore, to define two different categories of enums: one that
> has no concept of "value" (for pure-Python), and one which does have
> associated values but all values have to be specified explicitly

That sounds reasonable. However, I'm wondering if there isn't a
third case: where you don't care about the values, but you do
want them to have a defined ordering?

>    1. Enum syntax should not be "too magic". (In particular, it's pretty clear
>       at this point that creating new enums as a side-effect of name lookups
>       (even as convenient as it does make the syntax) is ultimately not gonna fly)

>    4. It shouldn't be necessary to quote enum names when defining them (since
>       they won't be quoted when they're used)

Satisfying both of these constraints without new syntax seems
to be pretty much impossible. That seems to be the main
sticking point in these discussions.

> I want to check: Is this a valid summary of things?

It looks pretty comprehensive to me!

-- 
Greg


From bruce at leapyear.org  Fri Feb 22 00:39:19 2013
From: bruce at leapyear.org (Bruce Leban)
Date: Thu, 21 Feb 2013 15:39:19 -0800
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
Message-ID: <CAGu0AnvTFzLCxrNGa_7LfP8Gu7UFBkNTydfse1VEEYwQ5Pw-1w@mail.gmail.com>

On Thu, Feb 21, 2013 at 3:01 AM, Devin Jeanpierre <jeanpierreda at gmail.com>
wrote:

> I've been noticing a lot of security-related issues being discussed in
> the Python world since the Ruby YAML problemcame out. Is it time to
> consider adding an alternative to pickle that is safe(r) by default?
>
> Pickle is usable in situations few other things are, because it can
> handle cyclic references and virtually any python object. The only
> stdlib alternative I'm aware of is json, which can do neither of those
> things. (Or at least, not without significant extra serialization
> code.) I would imagine that any alternative supplied should be easy
> enough to use that pickle users would seriously consider switching,
> and include at least those features.
>

Pickle is unsafe if you give it untrusted input. It's safe if you pickle
something yourself and then unpickle it. If the problem is that you want to
pickle something and store it in some unsafe place (like a cookie or a db
under user control) and then read it back in later and unpickle it, then
you can mitigate the risk by using an HMAC or some other mechanism to
prevent tampering and may want to consider encrypting it too.

That said, there is one risk in pickling something yourself and unpickling
it later that you need to watch out for. If your objects change, then
unpickling might produce unexpected and even potentially unsafe results.
You can mitigate this by adding object versions to your objects (as long as
you don't forget to update that when the object changes).

There's another problem - pickling is not guaranteed to work across Python
versions. So you may find yourself having to read pickles that are no
longer readable in a future python version. Not a problem for cookies, but
a potential headache with long-lived pickles.

All of this leads me to suggest using a better format for this problem.
Json is a reasonable choice (I've used it myself) although I would still
use an HMAC. If you encrypt it then that makes attacking the object that
much harder. I'd advise against using your own format. I wrote a tutorial
on hacking web sites called Gruyere <http://j.mp/gruyere-security>. I
suggest reading the section on cookies
http://j.mp/learn-state-manipulation (although
to be honest, I recommend reading the whole thing :-)

Aside from security, using a format like json encourages you to think about
what belongs in the persisted object and what doesn't. Suppose your object
includes a url. If you pickle it, you may end up persisting the parsed url
with a dictionary of parameters and other unnecessary overhead. When you
convert to json, you're going to just copy the url.



On Thu, Feb 21, 2013 at 7:50 AM, Dustin J. Mitchell <dustin at v.igoro.us>wrote:

> This conversation worries me.  The security community has shown that
> safety isn't something you can add to a powerful tool.  With great power
> comes great expressivity, and correspondingly more difficulty reasoning
> about it.  Not to mention reasoning about yhe implementation.  JSON is
> probably secure against code-execution exploits, but only probably.
>
> When you put something in the stdlib and call it "safe", even with
> caveats, people will make even more brazen mistakes than with a
> documented-unsafe tool like pickle.
>
Yes indeed.

--- Bruce
Latest blog post: Alice's Puzzle Page http://www.vroospeak.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/67876f1d/attachment.html>

From python at mrabarnett.plus.com  Fri Feb 22 01:39:06 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 22 Feb 2013 00:39:06 +0000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <5126ACC2.3080904@canterbury.ac.nz>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
Message-ID: <5126BE2A.70804@mrabarnett.plus.com>

On 2013-02-21 23:24, Greg Ewing wrote:
> On 22/02/13 08:18, Alex Stewart wrote:
>> In non-Python code, typically enums have always been represented under the
>> covers as ints, and therefore must be passed back and forth as numbers. The
>> fact that they have an integer value, however, is purely an implementation
>> artifact.  It comes from the fact that C and some other languages don't have
>   > a rich enough type system to properly make enums their own distinct type
>
> I don't think that's true for all languages. For example, enums in
> Pascal are definitely distinct types from integers, yet the language
> explicitly assigns them ordinal values and defines an ordering for
> them. Wirth must have thought there was a benefit in doing that.
>
[snip]

It means that sets of enums can be implemented using bitsets.



From rob.cliffe at btinternet.com  Fri Feb 22 02:58:47 2013
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Fri, 22 Feb 2013 01:58:47 +0000
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <5126ACC2.3080904@canterbury.ac.nz>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
Message-ID: <5126D0D7.7040302@btinternet.com>


On 21/02/2013 23:24, Greg Ewing wrote:
> On 22/02/13 08:18, Alex Stewart wrote:
>> In non-Python code, typically enums have always been represented 
>> under the
>> covers as ints, and therefore must be passed back and forth as 
>> numbers. The
>> fact that they have an integer value, however, is purely an 
>> implementation
>> artifact.  It comes from the fact that C and some other languages 
>> don't have
> > a rich enough type system to properly make enums their own distinct 
> type
>
> I don't think that's true for all languages. For example, enums in
> Pascal are definitely distinct types from integers, yet the language
> explicitly assigns them ordinal values and defines an ordering for
> them. Wirth must have thought there was a benefit in doing that.
>
Not necessarily.  With respect to the great man, he may not have given 
the matter as much thought, or considered (or been interested in) as 
many use cases as the contributors to this thread have.
>> It doesn't seem
>> unreasonable, therefore, to define two different categories of enums: 
>> one that
>> has no concept of "value" (for pure-Python), and one which does have
>> associated values but all values have to be specified explicitly
>
> That sounds reasonable. However, I'm wondering if there isn't a
> third case: where you don't care about the values, but you do
> want them to have a defined ordering?
The same thought occurred to me, although I could not think of a use 
case on the spot.  But I think that there surely are such.
>
>>    1. Enum syntax should not be "too magic". (In particular, it's 
>> pretty clear
>>       at this point that creating new enums as a side-effect of name 
>> lookups
>>       (even as convenient as it does make the syntax) is ultimately 
>> not gonna fly)
>
>>    4. It shouldn't be necessary to quote enum names when defining 
>> them (since
>>       they won't be quoted when they're used)
>
> Satisfying both of these constraints without new syntax seems
> to be pretty much impossible. That seems to be the main
> sticking point in these discussions.
>
>> I want to check: Is this a valid summary of things?
>
> It looks pretty comprehensive to me!
>
Apart from the point about maybe wanting to order enums, I agree.  I 
think Alex is right when he says you want to create singleton enums that 
don't equate to anything else.  We don't think of RED, GREEN, BLUE as 
being identical with the ints 1,2,3, and so don't want our 
implementation of them to compare equal to those ints.
Rob Cliffe


From tjreedy at udel.edu  Fri Feb 22 03:05:25 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 21 Feb 2013 21:05:25 -0500
Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement
	Exception for All Branches
In-Reply-To: <20130221102145.2b8edebd@pitrou.net>
References: <20130221001244.1b602cfd@pitrou.net>
	<51258A30.8090809@illinois.edu>
	<20130221102145.2b8edebd@pitrou.net>
Message-ID: <kg6jpl$png$1@ger.gmane.org>

On 2/21/2013 4:21 AM, Antoine Pitrou wrote:
> Le Wed, 20 Feb 2013 20:45:04 -0600,
> serwy <roger.serwy at gmail.com> a ?crit :
>>
>> I am the developer of the IdleX project
>> (http://idlex.sourceforge.net). It's not a fork of IDLE, but just a
>> collection of extensions that can be easily merged into IDLE.
>>
>> I started the project because of the slow pace of IDLE development,
>> especially when having several outstanding patches languishing in the
>> bug tracker. With your help I'd like to push IDLE forward.

> I guess it's a good idea to try and merge improvements into IDLE, but
> IDLE developers such as Terry would be in a better position to answer.

The pace of reviewing and applying patches to the default branch is a 
functions of people's time. I admit that I have been slower than I would 
like to be. This PEP is strictly about the question of which patches 
applied to tip should also be applied to older branches. A clearer 
answer, whatever it is, will help me, at least, to get more done.

If some enhancements are backported to bug-fix releases, they may appear 
in the wild sooner. On the other hand, not backporting might mean that 
more total patches get applied to default before the next release. On 
the third hand, it can be easier to get a patch widely tested on 
existing releases than on newly built default. (This is true on Windows 
where _tkinter is not compiled without considerable extra work.)

-- 
Terry Jan Reedy




From ethan at stoneleaf.us  Fri Feb 22 02:59:00 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 21 Feb 2013 17:59:00 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
Message-ID: <5126D0E4.3090402@stoneleaf.us>

On 02/21/2013 11:18 AM, Alex Stewart wrote:
> On a related note, to be honest, I'm not really sure I can think of any realistic use cases for "string enums" (or really anything other than ints in general).  Does anybody have an example of where this would actually be useful as opposed to just using "pure" (valueless) enums (which would already have string names)?

The tk library is a good example of where string enums would be useful; they also provide an easy "valueless" entity (at least as far as numbers go).

> I want to check: Is this a valid summary of things?  Anything I missed, or do people have substantial objections to any of the required/desirable/undesirable points I mentioned?

Looks good.

--
~Ethan~


From ethan at stoneleaf.us  Fri Feb 22 03:15:55 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 21 Feb 2013 18:15:55 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <5126D0D7.7040302@btinternet.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<5126D0D7.7040302@btinternet.com>
Message-ID: <5126D4DB.5020007@stoneleaf.us>

On 02/21/2013 05:58 PM, Rob Cliffe wrote:
>
> On 21/02/2013 23:24, Greg Ewing wrote:
>> On 22/02/13 08:18, Alex Stewart wrote:
>>> It doesn't seem
>>> unreasonable, therefore, to define two different categories of enums: one that
>>> has no concept of "value" (for pure-Python), and one which does have
>>> associated values but all values have to be specified explicitly
>>
>> That sounds reasonable. However, I'm wondering if there isn't a
>> third case: where you don't care about the values, but you do
>> want them to have a defined ordering?
>
> The same thought occurred to me, although I could not think of a use case on the spot.  But I think that there surely
> are such.

If you want order, but don't care about values, let them be ints and assign names in the order you want.


>>>    1. Enum syntax should not be "too magic". (In particular, it's pretty clear
>>>       at this point that creating new enums as a side-effect of name lookups
>>>       (even as convenient as it does make the syntax) is ultimately not gonna fly)
>>
>>>    4. It shouldn't be necessary to quote enum names when defining them (since
>>>       they won't be quoted when they're used)
>>
>> Satisfying both of these constraints without new syntax seems
>> to be pretty much impossible. That seems to be the main
>> sticking point in these discussions.
>>
>>> I want to check: Is this a valid summary of things?
>>
>> It looks pretty comprehensive to me!
>>
> Apart from the point about maybe wanting to order enums, I agree.  I think Alex is right when he says you want to create
> singleton enums that don't equate to anything else.  We don't think of RED, GREEN, BLUE as being identical with the ints
> 1,2,3, and so don't want our implementation of them to compare equal to those ints.

That depends entirely an what you are interfacing with: if it's a graphical library then you surely will want to compare 
equal with an int; if something else where the int value is meaningless, base the enum on a string.  ;)

--
~Ethan~


From robert2682 at verizon.net  Fri Feb 22 03:28:02 2013
From: robert2682 at verizon.net (robert2682)
Date: Thu, 21 Feb 2013 21:28:02 -0500
Subject: [Python-ideas] Functions and Unbound methods
Message-ID: <5126D7B2.3050903@verizon.net>


Hi,
> Hi,
>
> I'm new; greetings all!
>
> I'm not sure if this is a bug or feature, but it confused me so I thought
> I'd raise the issue.
>
>
> class a:
>      def b (self):
>        pass
>
>      foo = ('Hello', b)
>
> class c(a):
>      def d(self):
>        t = type (self. __class__. foo [1])
>        print t
>        t = type (self. __class__. b)
>        print t
>
> e = c ()
> e. d()
>
> prints <type 'function'> for the first print,
> and it seems to me it should be an instancemethod
>
> [snip]
> I think what's happening is that it's defining 'b' as a function in the
> class's namespace, storing a reference to that function in the tuple,
> and then, when the class definition ends, it's wrapping the function as
> a method.
>
> You'll find:
>
>   >>> a.foo[1]
> <function b at 0x00B42C30>
>   >>> a.b
> <unbound method a.b>
>   >>> a.b.im_func is a.foo[1]
> True
>

I'm afraid this won't do what I want.  As illustrated below, I want to 
store the
reference from withing the same class, so as to be able to access in 
from a superclass.
(sorry if subclass and superclass is not the correct terminology for Python)

> I'm trying to something like this
>
> class EditPage:
>
>     additonal_buttons = ()
>     def __init__ (self):
>       buts = []
>       for x in addional_butons:
>         if isinstance (x [1], types. UnboundMethodType):     # fails
> because type (x [1]) is function, not UnboundMethod
>           buts. append ((x [0], types. MethodType (x [1], self)))
>         else:
>           buts. append (x)
>
> class TreePage(EditPage):
>     def EditAsText (self):
>        pass
>
>     additional_buttons = (('EditAsText', EditAsText),)
>
>

It might be that Python simple doesn't do exactly what I want.
This is really unusual; mostly I think it's great and highly intuitive.

No biggie; I'm sure I can find a workaround!

Bob


From greg.ewing at canterbury.ac.nz  Fri Feb 22 05:59:10 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 Feb 2013 17:59:10 +1300
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <5126BE2A.70804@mrabarnett.plus.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<5126BE2A.70804@mrabarnett.plus.com>
Message-ID: <5126FB1E.8030504@canterbury.ac.nz>

MRAB wrote:
> On 2013-02-21 23:24, Greg Ewing wrote:
> 
>> For example, enums in
>> Pascal are definitely distinct types from integers, yet the language
>> explicitly assigns them ordinal values and defines an ordering for
>> them. Wirth must have thought there was a benefit in doing that.
>
> It means that sets of enums can be implemented using bitsets.

That could still have been done without exposing the ordinal
values. Yet, Pascal provides an ord() function.

I suppose one viewpoint might be that as long as there is
a defined ordering, there's always going to be at least an
implied mapping to natural numbers, so there's not much
point in trying to hide it.

-- 
Greg


From steve at pearwood.info  Fri Feb 22 07:00:47 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 22 Feb 2013 17:00:47 +1100
Subject: [Python-ideas] Functions and Unbound methods
In-Reply-To: <51258235.3090802@verizon.net>
References: <51258235.3090802@verizon.net>
Message-ID: <5127098F.6090503@pearwood.info>

On 21/02/13 13:11, robert2682 wrote:
>
>
> Hi,
>
> I'm new; greetings all!
>
> I'm not sure if this is a bug or feature, but it confused me so I thought
> I'd raise the issue.

Not a bug, but a feature, and in fact a fundamental way that Python works.

The def statement creates functions. Always, without exception. So when you
define a method in a class, you're actually defining a function. It doesn't
get converted to a method until later, when it is retrieved via attribute
access from the class or instance. You can google on "descriptor protocol"
to learn more about it, although that is considered fairly advanced.

This mailing list is actually intended for discussing new ideas and future
changes to the Python language, standard library, and compiler, not for
generic questions like this. For general questions, you should subscribe
to the "python-list at python.org" mailing list, or "tutor@" if you are a
beginner.

If you repeat your question there, someone will reply with a more detailed
response and some suggestions for your code.


-- 
Steven


From steve at pearwood.info  Fri Feb 22 07:55:12 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 22 Feb 2013 17:55:12 +1100
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CAGu0AnvTFzLCxrNGa_7LfP8Gu7UFBkNTydfse1VEEYwQ5Pw-1w@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CAGu0AnvTFzLCxrNGa_7LfP8Gu7UFBkNTydfse1VEEYwQ5Pw-1w@mail.gmail.com>
Message-ID: <51271650.2030808@pearwood.info>

On 22/02/13 10:39, Bruce Leban wrote:

> There's another problem - pickling is not guaranteed to work across Python
> versions.

I think you are confusing pickle with marshal.

Obviously you cannot guarantee unpickling in older versions of Python,
e.g. I can't unpickle a Python 2.7 set in Python 2.1, since sets didn't
exist back then. Or a pickle created using version N of the protocol
requires a Python version recent enough to understand version N, e.g.
protocol 3 exists only in Python 3.x.

But going forward should not be a problem, and the pickle documentation
promises backwards-compatibility. Actually I believe that *forward*
compatibility is a better description, but either way, the compatibility
is guaranteed:

Python version X will be able to unpickle anything pickled by an older
version of Python; furthermore, anything pickled by version X will be
able to be unpickled by future versions.

(Modulo removal of the pickled classes, etc.)



-- 
Steven


From bruce at leapyear.org  Fri Feb 22 08:03:40 2013
From: bruce at leapyear.org (Bruce Leban)
Date: Thu, 21 Feb 2013 23:03:40 -0800
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <51271650.2030808@pearwood.info>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CAGu0AnvTFzLCxrNGa_7LfP8Gu7UFBkNTydfse1VEEYwQ5Pw-1w@mail.gmail.com>
	<51271650.2030808@pearwood.info>
Message-ID: <CAGu0Ant2eMR4DpsB+8kdj4uXu31VgDLcpSYKOvdne2taXvYV_Q@mail.gmail.com>

On Thu, Feb 21, 2013 at 10:55 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> On 22/02/13 10:39, Bruce Leban wrote:
>
>  There's another problem - pickling is not guaranteed to work across Python
>> versions.
>>
>
> I think you are confusing pickle with marshal.
>

My bad. Should have rtfm. Thanks.

--- Bruce
Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130221/2f80c4d3/attachment.html>

From steve at pearwood.info  Fri Feb 22 08:04:05 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 22 Feb 2013 18:04:05 +1100
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <201302211733.02145.mark.hackett@metoffice.gov.uk>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<201302211722.47380.mark.hackett@metoffice.gov.uk>
	<20130221182452.1ef90184@pitrou.net>
	<201302211733.02145.mark.hackett@metoffice.gov.uk>
Message-ID: <51271865.4040900@pearwood.info>

On 22/02/13 04:33, Mark Hackett wrote:

> Being serious, though, if your code requires a serious amount of security,
> you're better off writing your own parsing.

If you're serious about security, you don't want amateurs trying to build
security from scratch. And that includes yourself, if you are not a security
expert.

A programmer ought to be aware of their own limitations. I am not a security
expert, and I don't have the time or inclination to become one. I want, no,
I *need*, solutions for common problems to be safe by default, or at least for
their vulnerabilities to be documented clearly and obviously in language I
can understand, so I can write code with reasonable levels of security instead
of inventing my own insecure, unsafe solutions.

I know enough not to call eval() on data retrieved from untrusted sources. Not
everyone even knows that much. I've seen code that literally downloaded content
from a website, then eval'ed it without even a token attempt to sanitize it.
Do you expect this person to write his own secure data serialiser?

Anyone can write code with no security vulnerabilities that *they* can see.
And frequently do.



-- 
Steven


From mark.hackett at metoffice.gov.uk  Fri Feb 22 12:24:26 2013
From: mark.hackett at metoffice.gov.uk (Mark Hackett)
Date: Fri, 22 Feb 2013 11:24:26 +0000
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <51271865.4040900@pearwood.info>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<201302211733.02145.mark.hackett@metoffice.gov.uk>
	<51271865.4040900@pearwood.info>
Message-ID: <201302221124.26452.mark.hackett@metoffice.gov.uk>

On Friday 22 Feb 2013, Steven D'Aprano wrote:
> On 22/02/13 04:33, Mark Hackett wrote:
> > Being serious, though, if your code requires a serious amount of
> > security, you're better off writing your own parsing.
> 
> If you're serious about security, you don't want amateurs trying to build
> security from scratch.

And if your code needs to be secure and you aren't capable of doing so, then 
pay someone to do it who is capable. You know, get a security expert.


From mark.hackett at metoffice.gov.uk  Fri Feb 22 12:26:55 2013
From: mark.hackett at metoffice.gov.uk (Mark Hackett)
Date: Fri, 22 Feb 2013 11:26:55 +0000
Subject: [Python-ideas] Fwd: argparse - add support for environment
	variables
In-Reply-To: <eaa904e7-a7c3-4e90-bf38-b301c4bbfea7@googlegroups.com>
References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com>
	<CAF-Rda8kobVf6yceb2aSL5BxZ4U=OaExzWqRCw2=Ob5fqbcVEQ@mail.gmail.com>
	<eaa904e7-a7c3-4e90-bf38-b301c4bbfea7@googlegroups.com>
Message-ID: <201302221126.55127.mark.hackett@metoffice.gov.uk>

On Thursday 21 Feb 2013, Miki Tebeka wrote:
> > What about config files? Mix them into argparse too? Oh, and support for
> > .ini/XML/YAML/JSON/sqlite configs, please!
> 
> Well, baby steps toward configuration solution :)
> The other thing is that unlike config files which have many formats,
> environment has one uniform access to it.
> 

In the interests of doing it the UNIX way, would this be better done as a 
parser for the .ini/XML/YAML (they're ALL YAML, though. Yet ANOTHER Meta 
Language..!) that results in $ENV changing in python.

Send the result that to argparse.

Bada-bum, bada-bing, done.


From stephen at xemacs.org  Fri Feb 22 12:58:34 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 22 Feb 2013 20:58:34 +0900
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <51271865.4040900@pearwood.info>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<201302211722.47380.mark.hackett@metoffice.gov.uk>
	<20130221182452.1ef90184@pitrou.net>
	<201302211733.02145.mark.hackett@metoffice.gov.uk>
	<51271865.4040900@pearwood.info>
Message-ID: <87liagsgph.fsf@uwakimon.sk.tsukuba.ac.jp>

Steven D'Aprano writes:

 > A programmer ought to be aware of their own limitations. I am not a security
 > expert, and I don't have the time or inclination to become one. I want, no,
 > I *need*, solutions for common problems to be safe by default, or at least for
 > their vulnerabilities to be documented clearly and obviously in language I
 > can understand, so I can write code with reasonable levels of security instead
 > of inventing my own insecure, unsafe solutions.

Sure.  So just use JSON where it will do, and avoid pickle.  No?

Sure, you can make a case that "restricted pickle" would give you a
trivial upgrade path if you find you really need it later.  But it
seems to me that if you think you need a protocol that executes
serialized code automatically, you've got a heck of a lot of security
work to do, beside which the effort to port from JSON API to pickle
API is tiny.


From stephen at xemacs.org  Fri Feb 22 13:19:33 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 22 Feb 2013 21:19:33 +0900
Subject: [Python-ideas] Adding a safe alternative to pickle
	in	the	standard library
In-Reply-To: <769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com>
Message-ID: <87k3q0sfqi.fsf@uwakimon.sk.tsukuba.ac.jp>

Steve Dower writes:

 > I was only aiming to provide perspective, rather than a proposal.

Sure, but I didn't think there was a need for more "general"
perspective.  Pickle is a well-established protocol with certain risks
and certain benefits, of which the python-dev community is basically
well-aware.  This is coming up *now* because recent events (Ruby/YAML
has been mentioned) are causing some people to reevaluate the risks.
This is a "quantitative" issue, and needs concrete proposals.

 > To be really explicit, I would make load/loads only work with
 > built-in types. For compatibility when reading earlier protocols we
 > could add a type representing a class instance and its members that
 > doesn't actually construct anything. (Maybe we override __getattr__
 > to make it lazily construct the instance when the program actually
 > wants to use it?)

I am not a security expert, but it seems to me that's going in the
wrong direction.  Unpickler would *still* run constructor code
automatically under some circumstances -- but those circumstances
become murkier.

 > For convenience, I'd add a parameter to Unpickler to let the user
 > provide a set of types that are allowed to be constructed (or a
 > mapping from names to callables as used in find_class()).

And this is secure, why?  There's no way to decorate the allowed types
to add nasty stuff to the pickled class definitions (including
built-in types), right?  There are no bugs that allow a back door,
right?  Is the API sufficiently well-designed that users will easily
figure out how to do what they need, and *only* what they need, and
therefore won't be tempted to simply turn on permission to do
*everything*?  And they won't give up, and write their own?

Isn't it better just to give users the advice to use JSON where it
will do?  Perhaps the difference in APIs will give them pause to think
again if they're starting to think about unpickling classes?

Granted, I don't have answers to those questions (except for myself!)
But I think some thought should be given to them before trying to
create a restricted pickle protocol and make it default.  Restricted
modes/protocols/sublanguages are hard to get right.



From stephen at xemacs.org  Fri Feb 22 13:29:38 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 22 Feb 2013 21:29:38 +0900
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
Message-ID: <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>

Devin Jeanpierre writes:

 > That does not mean that we should not write or use safer alternatives.
 > We have written and do use safer alternatives, like the json module.

Then why do we need a "safe alternative to pickle" when json is
already in the standard library?

 > But it means we can't expect them to be usable exactly everywhere
 > pickle is. I would've said the question is how far in that direction
 > we should bother to go.

OK, that is a better way to put what I have in mind.

Well, we've already gone as far as json, which is pretty powerful (but
still subject to attacks using "relatively secure" json to transport
"insecure" data!)  Why do we need an alternative *between* pickle and
json?  Maybe we should advocate that users think seriously about
securing channels, and validating the pickles before doing anything
with them, if they think they need more features than json offers?



From jsbueno at python.org.br  Fri Feb 22 13:53:27 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 22 Feb 2013 09:53:27 -0300
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <5126D0D7.7040302@btinternet.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<5126D0D7.7040302@btinternet.com>
Message-ID: <CAH0mxTT_nOrix=zh0Ey-CJTGrLj9stjWR0PK_8v34hiYvwLspw@mail.gmail.com>

On 21 February 2013 22:58, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
> Apart from the point about maybe wanting to order enums, I agree.  I think
> Alex is right when he says you want to create singleton enums that don't
> equate to anything else.  We don't think of RED, GREEN, BLUE as being
> identical with the ints 1,2,3, and so don't want our implementation of them
> to compare equal to those ints.


Again, I thinnkt hat my 2002 code that expected a (1) working when it
got a (True) instead was a great, great thing. And I think that lots
of places where magical strings today ("str.decode" errors argument,
"compile" mode argument) should be lifted up to expect constants and
still keep working if one pass them the strings "replace" or "eval".

Requiring that things compare differently, IMHO, is too much
statically-typed-minded - it is not the way of Python. As for Alex's
surreal example of  someone defining "NO" and "YES" with different
values in different Enums and passing the wrong one to a given call -
this person will have it comming. Think of this case instead:

>>> disk_io.IDDLE == "iddle"
True
>>> network.IDDLE == "iddle"
True
>>> disk_io.IDDLE == network.IDDLE
False

So, if the compared enums do have an explicit value assigned to them
(and I agree there are some cases the value is not needed) , they
should compare just as that value - anything else is more close to C++
than Python.

And no one is preventing people from using decorators or use other
mecanisms for certain function calls to accept only values of a given
Enum, just as no one ever was prohibited from statically verifying any
other values in a call.


From jeanpierreda at gmail.com  Fri Feb 22 14:26:06 2013
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Fri, 22 Feb 2013 08:26:06 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
	<87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>

On Fri, Feb 22, 2013 at 7:29 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>  > That does not mean that we should not write or use safer alternatives.
>  > We have written and do use safer alternatives, like the json module.
>
> Then why do we need a "safe alternative to pickle" when json is
> already in the standard library?

json can't handle cycles or new types -- at least, not without a
rather significant amount of preprocessing and postprocessing that is
little less work than writing your own serialization library from
scratch.

I do believe that support for these things is important. People do
want to store frozensets of tuples, and objects which reference
themselves, and so on. The easy way to do this is with pickle, and
that encourages people to use pickle even where it is a bad idea. The
easier it is to do something safer, the more likely it is people will
do something safer. I've seen people use pickle even when advised that
it was a security risk, because they didn't want to go through the
effort of using the json module. To me that signals that something is
wrong.

> Well, we've already gone as far as json, which is pretty powerful (but
> still subject to attacks using "relatively secure" json to transport
> "insecure" data!)

Of course a serialization library can't protect against
eval(deserialize(foo)) running arbitrary code. That doesn't mean we
shouldn't bother with security.

> Why do we need an alternative *between* pickle and
> json?  Maybe we should advocate that users think seriously about
> securing channels, and validating the pickles before doing anything
> with them, if they think they need more features than json offers?

Signed pickles and secured channels and so on don't solve the problem
of trying to get data from an an untrusted user. Moreover I think it
isn't an uncommon opinion that, even in that case, it'd be better to
use json or some other nominally secure library instead of pickle,
since it increases the number of mistakes necessary to compromise your
security.

-- Devin


From solipsis at pitrou.net  Fri Feb 22 15:05:57 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 22 Feb 2013 15:05:57 +0100
Subject: [Python-ideas] range
References: <loom.20130221T120736-301@post.gmane.org>
Message-ID: <20130222150557.36884c3f@pitrou.net>

Le Thu, 21 Feb 2013 13:35:17 +0000 (UTC),
Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> a
?crit :
> Dear all,
> what do you think about allowing the use of operators on ranges?
> 
> I thought of things like:
> a = range(1,10)
> b = range(5,12)
> intersect = a & b     # equivalent to intersect = range(5,10)
> merge = a | b         # equivalent to merge = range(1,12)
> 
> to make this work in more complex cases, like:
> a = range(1,10)
> b = range(12,15)
> merge = a | b         # equivalent to sth. like
> range(range(1,10),range(12,15))

-1. range() is a contiguous sequence, not an arbitrary container.
Simplicity is part of its virtues.

Regards

Antoine.




From ncoghlan at gmail.com  Fri Feb 22 16:57:36 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Feb 2013 01:57:36 +1000
Subject: [Python-ideas] range
In-Reply-To: <20130222150557.36884c3f@pitrou.net>
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
Message-ID: <CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>

On Sat, Feb 23, 2013 at 12:05 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Thu, 21 Feb 2013 13:35:17 +0000 (UTC),
> Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de> a
> ?crit :
>> Dear all,
>> what do you think about allowing the use of operators on ranges?
>>
>> I thought of things like:
>> a = range(1,10)
>> b = range(5,12)
>> intersect = a & b     # equivalent to intersect = range(5,10)
>> merge = a | b         # equivalent to merge = range(1,12)
>>
>> to make this work in more complex cases, like:
>> a = range(1,10)
>> b = range(12,15)
>> merge = a | b         # equivalent to sth. like
>> range(range(1,10),range(12,15))
>
> -1. range() is a contiguous sequence, not an arbitrary container.
> Simplicity is part of its virtues.

-1 from me as well. We don't even implement concatenation and
repetition for ranges in order to keep them simple - we have to caveat
the range docs to point out they don't *quite* implement the full
sequence API due to this restriction.

However, I will note that starting in Python 3.3, range objects expose
their "start", "stop" and "step" attributes. That makes it much easier
for third party software to manipulate them arithmetically.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From Steve.Dower at microsoft.com  Fri Feb 22 17:59:32 2013
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Fri, 22 Feb 2013 16:59:32 +0000
Subject: [Python-ideas] Adding a safe alternative to pickle
	in	the	standard library
In-Reply-To: <87k3q0sfqi.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com>
	<87k3q0sfqi.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <065d6242ab4940638811c9f3071e68b7@BLUPR03MB035.namprd03.prod.outlook.com>

> From: Stephen J. Turnbull [mailto:stephen at xemacs.org]
> Steve Dower writes:
>  > To be really explicit, I would make load/loads only work with  
> > built-in types. For compatibility when reading earlier protocols we  
> > could add a type representing a class instance and its members that 
> > doesn't actually construct anything. (Maybe we override __getattr__ 
> > to make it lazily construct the instance when the program actually 
> > wants to use it?)
> 
> I am not a security expert, but it seems to me that's going in the wrong
> direction.  Unpickler would *still* run constructor code automatically under
> some circumstances -- but those circumstances become murkier.

Agreed on the bit in parentheses, that's probably questionable enough to ignore. However, if it only works with built-in types then there is no user code that will run. IIUC we already have a C implementation of pickle that is immune to users redefining builtins (if not, we should do this too). Pickled objects would be unpickled as (effectively) a tuple of the members - ('system', ("echo Hello World")) does not execute any code. (And yeah, wrap that tuple up in a type that can be tested.)

> > For convenience, I'd add a parameter to Unpickler to let the user 
> > provide a set of types that are allowed to be constructed (or a  
> > mapping from names to callables as used in find_class()).
> 
> And this is secure, why?  There's no way to decorate the allowed types to
> add nasty stuff to the pickled class definitions (including built-in types), right?

Code is only pickled by name. Unpickler resolves the names and returns the class or function reference in the current environment. If it can't find the module or name in its current environment, it raises an error.

> There are no bugs that allow a back door, right? 

Of course not. That's why we never see security patches or updates for operating systems or platforms. This is a silly argument.

> Is the API sufficiently well-designed that users will easily figure out how to
> do what they need, and *only* what they need, and therefore won't be
> tempted to simply turn on permission to do *everything*? 

All we can ever do is provide instructions to keep the developer safe and make it clear that ignoring those rules will reduce the security of their program. It's up to the developer to make the right decisions.

> And they won't give up, and write their own?

In my experience, people more often write their own out of ignorance rather than frustration (same for unnecessarily using XML). Or they'll switch to an earlier version of Python that doesn't have this change in it. Again, we can encourage, but not dictate.

> Isn't it better just to give users the advice to use JSON where it will do?
> Perhaps the difference in APIs will give them pause to think again if they're
> starting to think about unpickling classes?

Maybe, though since pickle is literally the Python equivalent (should it have been called Python Object Notation (PON)? Probably not...) we should be ensuring that it is the best it can be.

> Granted, I don't have answers to those questions (except for myself!) But I
> think some thought should be given to them before trying to create a
> restricted pickle protocol and make it default.  Restricted
> modes/protocols/sublanguages are hard to get right.

Agreed. I don't think we need a new protocol though, just a less permissive default implementation of Unpickler.find_class().


From abarnert at yahoo.com  Fri Feb 22 18:07:32 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 22 Feb 2013 09:07:32 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <5126ACC2.3080904@canterbury.ac.nz>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
Message-ID: <A15CBBA7-6426-47BC-8E43-B74094B59332@yahoo.com>

On Feb 21, 2013, at 15:24, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> On 22/02/13 08:18, Alex Stewart wrote:
>> 
>> It doesn't seem
>> unreasonable, therefore, to define two different categories of enums: one that
>> has no concept of "value" (for pure-Python), and one which does have
>> associated values but all values have to be specified explicitly
> 
> That sounds reasonable. However, I'm wondering if there isn't a
> third case: where you don't care about the values, but you do
> want them to have a defined ordering?

And a fourth case: You don't care about the values, but you want to be able to | them together into a set.

For example, consider file open flags, mmap access flags, chmod flags, etc. You need some way to pass READ | TEXT, WRITE | SHARED, etc. occasionally, but most of the time you just pass READ or WRITE. It wouldn't be too onerous to require an explicit {READ, SHARED} set for the uncommon case, but set(READ) for the common case would be horrible. And you don't want every function that takes enums to typeswitch so it can handle a single value as if it were a set. The obvious answer to this is the one every other language uses: take READ | SHARED.

This doesn't have to mean flag enums have integral values. However, the only alternative I can think of is that enums have set syntax, so READ | SHARED returns something like READ.__class__({READ, SHARED}), which of course also means that set has to automatically be a valid value for the type. Which is pretty magical and complex.

Of course the answer could be not to allow that. Call mmap(file, access=WRITE, contention=SHARED), not mmap(WRITE | SHARED). But if we're talking about changing the stdlib and hopefully most popular third party libs, that would be a pretty drastic change.

Finally, the answer could be: if you want that, you have to give them int values, and accept ints instead of enums in your APIs, at which point I think the benefit of valueless enums has gone way down, because many places you'd think you want them (including all over the stdlib), you don't.



From g.rodola at gmail.com  Fri Feb 22 18:15:43 2013
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Fri, 22 Feb 2013 18:15:43 +0100
Subject: [Python-ideas] My wishlist for Tulip
Message-ID: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>

I understand it's still beta but anyways, here's my little wish list for Tulip.

* provide a 'demo' / 'example' directory containing very simple
scripts showing the most basic usages such as:

- echo_tcp_client.py
- echo_tcp_server.py
- echo_tcp_server_w_timeout.py  (same as echo_server.py but also
disconnects the client after a certain time of inactivity)
- echo_tcp_ssl_client.py
- echo_tcp_ssl_server.py
- echo_udp_client.py
- echo_udp_server.py

* move all *test*.py scripts in a separate 'test' directory

* if it's not part of the API intended to be public move
tulip/http_client.py elsewhere ('examples'/'demo' or a brand new
'scripts'/'tools' directory)

* (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks
like they belong elsewhere

* write a simple benchmark framework testing (at least) sending,
receiving and the internal scheduler  (I'd like to help with this one)


--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/


From abarnert at yahoo.com  Fri Feb 22 18:29:36 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 22 Feb 2013 09:29:36 -0800
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
Message-ID: <CD64C69C-3335-453C-9BA1-2985A4684F89@yahoo.com>

On Feb 21, 2013, at 9:55, Steve Dower <Steve.Dower at microsoft.com> wrote:

>> From: Andrew Barnert
>> How often have you needed either cyclic references or the ability to
>> dynamically store arbitrary classes in something like a cookie or a cache file?
> 
> In a past life I used pickle regularly to snapshot long-running (evolutionary) algorithms that used user-provided classes and all sorts of highly improper circular references. And there are plenty of researchers out there using Python for much crazier things than I ever did.
> 
> There is a lot more to Python than web apps...

But you're not storing those pickles in a cookie, which is exactly my point. There are many cases where you need the power of pickle. There are also many cases where you need safe serialization. But there's not much overlap.

There are plenty of cases where you need safety, and also need a little more power than JSON--but you still don't usually need the full power of pickle for those cases, and making it easier to extend the json lib is a much cleaner way forward than making it easier to restrict pickle.

It's true that "not much overlap" != "no overlap". But you can't cover everything. If you're building, say, an online interactive python interpreter that saves and restores its state between sessions, you're going to have to think through the security implications. That doesn't mean someone who wants to just store scientific data and doesn't have untested sources, or someone building a web app who doesn't care about storing arbitrary dynamically defined types, should have the same burden.

From wolfgang.maier at biologie.uni-freiburg.de  Fri Feb 22 18:40:29 2013
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Fri, 22 Feb 2013 17:40:29 +0000 (UTC)
Subject: [Python-ideas] range
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
	<CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>
Message-ID: <loom.20130222T183719-360@post.gmane.org>

Nick Coghlan <ncoghlan at ...> writes:

> -1 from me as well. We don't even implement concatenation and
> repetition for ranges in order to keep them simple - we have to caveat
> the range docs to point out they don't *quite* implement the full
> sequence API due to this restriction.
> 
> However, I will note that starting in Python 3.3, range objects expose
> their "start", "stop" and "step" attributes. That makes it much easier
> for third party software to manipulate them arithmetically.
> 
> Cheers,
> Nick.
> 

Any plans to make range a valid base class to build upon in future releases?

Best,
Wolfgang





From abarnert at yahoo.com  Fri Feb 22 18:41:02 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 22 Feb 2013 09:41:02 -0800
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
	<87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
Message-ID: <C330DB1C-A165-4C69-A5FB-4CDBA2697063@yahoo.com>

On Feb 22, 2013, at 5:26, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:

>> Well, we've already gone as far as json, which is pretty powerful (but
>> still subject to attacks using "relatively secure" json to transport
>> "insecure" data!)
> 
> Of course a serialization library can't protect against
> eval(deserialize(foo)) running arbitrary code. That doesn't mean we
> shouldn't bother with security.

The difference is that json.loads is just deserialize(foo), which pickle.loads inherently has some eval mixed in.

That's why I think for most use cases, the answer is making json easier to extend, not making pickle easier to secure.

The biggest problem people have with the json library isn't that you have to do the extending explicitly and externally, but that it's a huge pain to do so. There was a suggestion earlier in this thread (I forget the author) that would go a long way toward relieving that pain.

Some people also want it to be implicitly extensible, to have some way to create an instance of a new empty class named Foo with given attributes (but not an existing builtin or user-defined class named Foo). I'm not sure what their use case is, and I'm not sure it's a good idea--but if it is, there was also a suggestion for that.

From jeanpierreda at gmail.com  Fri Feb 22 19:14:16 2013
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Fri, 22 Feb 2013 13:14:16 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <C330DB1C-A165-4C69-A5FB-4CDBA2697063@yahoo.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
	<87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
	<C330DB1C-A165-4C69-A5FB-4CDBA2697063@yahoo.com>
Message-ID: <CABicbJKr8g8_yN5KYhF3CLwcLDBn9g2B5y9s99LU_sFhorw6pA@mail.gmail.com>

On Fri, Feb 22, 2013 at 12:41 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> The difference is that json.loads is just deserialize(foo), which pickle.loads inherently has some eval mixed in.
>
> That's why I think for most use cases, the answer is making json easier to extend, not making pickle easier to secure.

My original suggestion was to add a third thing, such as cerealizer,
not to restrict pickle or extend json. Some others have talked about
restricting pickle, but I don't know how one could do that and still
be confident in the safety of the end product. You usually build
things to be safe from the ground up, not as some afterthought with a
few restrictions.

> The biggest problem people have with the json library isn't that you have to do the extending explicitly and externally, but that it's a huge pain to do so. There was a suggestion earlier in this thread (I forget the author) that would go a long way toward relieving that pain.

I feel that'd be very helpful, yes. Obviously not as helpful as
something that can handle cyclic references, but those aren't really
as important. Besides which, a yaml module could synthesize something
more complete out of these pieces (YAML is like JSON, but with support
for cyclic references and some extra syntax).

My issue is making safe serialization easier, so that not using pickle
is a viable option. As you say, we can go a long way towards this
using the json module.

-- Devin


From foogod at gmail.com  Fri Feb 22 19:21:57 2013
From: foogod at gmail.com (Alex Stewart)
Date: Fri, 22 Feb 2013 10:21:57 -0800 (PST)
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAH0mxTT_nOrix=zh0Ey-CJTGrLj9stjWR0PK_8v34hiYvwLspw@mail.gmail.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<5126D0D7.7040302@btinternet.com>
	<CAH0mxTT_nOrix=zh0Ey-CJTGrLj9stjWR0PK_8v34hiYvwLspw@mail.gmail.com>
Message-ID: <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com>

On Friday, February 22, 2013 4:53:27 AM UTC-8, Joao S. O. Bueno wrote:

> Again, I thinnkt hat my 2002 code that expected a (1) working when it 
> got a (True) instead was a great, great thing. And I think that lots 
> of places where magical strings today ("str.decode" errors argument, 
> "compile" mode argument) should be lifted up to expect constants and 
> still keep working if one pass them the strings "replace" or "eval". 
>

Ok, first of all, nobody's suggesting that wouldn't work.  I'm not talking 
about comparing enums with base types (which I have no problem working like 
this), I'm talking specifically about comparing enums directly with other 
enums from a different category.

Requiring that things compare differently, IMHO, is too much 
> statically-typed-minded - it is not the way of Python. As for Alex's 
> surreal example of  someone defining "NO" and "YES" with different 
> values in different Enums and passing the wrong one to a given call - 
> this person will have it comming.


I think this is unnecessarily dismissive.  Obviously, my example was 
simplified, but what if these two enums were created by two completely 
different people in different modules that just happen to be used in the 
same program?  Does the hapless programmer "have it coming" then?

You say that my suggestion is more like C++ than Python, but I counter that 
your suggestion seems to me a lot more like Perl than Python.  It smacks of 
one of the things I dislike most about Perl, which also happens to be, in 
my opinion, one of the language features most responsible for (often 
subtle) bugs in Perl code even by experienced programmers:  The contents of 
variables can have very different semantic meanings depending solely on the 
context in which they're evaluated (scalar vs array, etc).  In this case 
what you're saying is that just because entirely unrelated programmers 
happened to use the same int value underneath, that Align.TOP and 
Errno.EPERM and Choice.OK should all be considered to mean exactly the same 
thing everywhere (even if none of the programmers actually intended or 
foresaw that possibility), but what that meaning is actually depends 
entirely on what context they happen to be evaluated in.

Another way of looking at this is that what you're proposing is effectively 
implicit type-casting between enums of different types.  Python does 
support implicit casting in a very few well-defined cases, but not most of 
the time, and only when doing so does not discard important semantic 
information (i.e. "meaning").  For ints, floats, etc, they really only have 
one piece of semantic information ("a number value").  Enums, however, have 
more than that.  They have (potentially) a number value, but they also have 
names, and they (preferably) have information about "what type of thing am 
I".  Automatically throwing out that extra information on a comparison 
where both objects have the extra info and it's not the same on both 
objects is, in my opinion, just wrong.  If there were no extra semantic 
meaning to enums, then this wouldn't be an issue, but the symbolic 
information they carry is clearly important, because it's the whole reason 
we're having this discussion in the first place instead of just using ints 
everywhere.

(Indeed, it could easily be argued that the entire point to things like 
int-enums is that the programmer isn't supposed to have to care what the 
underlying value is, but should be able to think of things entirely 
symbolically.)

In pretty much all the cases I can imagine (and that have been suggested 
thus far), it seems to me that passing a different class of enum than the 
other side is testing against is almost always going to be an accident, or 
at the very least sloppy code (if there are cases where it is intentional, 
it's unlikely to be common, and the programmer should be able to explicitly 
cast between them somehow).  Given this, I think that the Principle of 
Least Surprise enters into things here as well.  If somebody makes a coding 
mistake and uses the wrong type of enum somewhere, which is likely to be 
the least surprising outcome?:

   1. If you use an enum of the wrong type, it won't be considered 
   equivalent to any of the expected options.
   2. If you use an enum of the wrong type, it might not be considered to 
   be equivalent to any of the expected options, or it might match one of them 
   and do something either the same or completely different than you expect, 
   but you have no way to tell what will happen without looking at an 
   underlying constant which you're not supposed to have to care about.

 --Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/9f293ef3/attachment.html>

From jsbueno at python.org.br  Fri Feb 22 19:32:28 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Fri, 22 Feb 2013 15:32:28 -0300
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<5126D0D7.7040302@btinternet.com>
	<CAH0mxTT_nOrix=zh0Ey-CJTGrLj9stjWR0PK_8v34hiYvwLspw@mail.gmail.com>
	<2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com>
Message-ID: <CAH0mxTR38af7MNqw9ZXba9ykouLQGpzHBgdEWQ_0CUb_wGLz1Q@mail.gmail.com>

On 22 February 2013 15:21, Alex Stewart <foogod at gmail.com> wrote:
> On Friday, February 22, 2013 4:53:27 AM UTC-8, Joao S. O. Bueno wrote:
>>
>> Again, I thinnkt hat my 2002 code that expected a (1) working when it
>> got a (True) instead was a great, great thing. And I think that lots
>> of places where magical strings today ("str.decode" errors argument,
>> "compile" mode argument) should be lifted up to expect constants and
>> still keep working if one pass them the strings "replace" or "eval".
>
>
> Ok, first of all, nobody's suggesting that wouldn't work.  I'm not talking
> about comparing enums with base types (which I have no problem working like
> this), I'm talking specifically about comparing enums directly with other
> enums from a different category.
(big snip)

Ok. I do agree with your argumentation. And in fact,
my previous example of
----------
>>> disk_io.IDDLE == "iddle"
True
>>> network.IDDLE == "iddle"
True
>>> disk_io.IDDLE == network.IDDLE
False
-----
If the final behavior in some cases is this, no one will get harmed.
We alread hav e NaN == Nan -> False  and the language does
not explode. :-)

So, indeed - values in distinct enums should compare differently,
even if they compare equal to their underlying value.

(sorry for making you write the extensive e-mail)

  js
 -><-


From foogod at gmail.com  Fri Feb 22 19:43:41 2013
From: foogod at gmail.com (Alex Stewart)
Date: Fri, 22 Feb 2013 10:43:41 -0800 (PST)
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAH0mxTR38af7MNqw9ZXba9ykouLQGpzHBgdEWQ_0CUb_wGLz1Q@mail.gmail.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<5126D0D7.7040302@btinternet.com>
	<CAH0mxTT_nOrix=zh0Ey-CJTGrLj9stjWR0PK_8v34hiYvwLspw@mail.gmail.com>
	<2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com>
	<CAH0mxTR38af7MNqw9ZXba9ykouLQGpzHBgdEWQ_0CUb_wGLz1Q@mail.gmail.com>
Message-ID: <b404f8f4-c6b9-46b3-b3fa-21fee01d7152@googlegroups.com>



> (sorry for making you write the extensive e-mail) 
>

No worries..  Having to write it all out actually helped me clarify my own 
thoughts and make sure that my intuitive position did actually make logical 
sense to me, so it was probably a good thing to do anyway :)

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/5c34e87c/attachment.html>

From ethan at stoneleaf.us  Fri Feb 22 19:46:27 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 22 Feb 2013 10:46:27 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <A15CBBA7-6426-47BC-8E43-B74094B59332@yahoo.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<A15CBBA7-6426-47BC-8E43-B74094B59332@yahoo.com>
Message-ID: <5127BD03.5030703@stoneleaf.us>

On 02/22/2013 09:07 AM, Andrew Barnert wrote:
> On Feb 21, 2013, at 15:24, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
>> On 22/02/13 08:18, Alex Stewart wrote:
>>>
>>> It doesn't seem
>>> unreasonable, therefore, to define two different categories of enums: one that
>>> has no concept of "value" (for pure-Python), and one which does have
>>> associated values but all values have to be specified explicitly
>>
>> That sounds reasonable. However, I'm wondering if there isn't a
>> third case: where you don't care about the values, but you do
>> want them to have a defined ordering?
>
> And a fourth case: You don't care about the values, but you want to be able to | them together into a set.
>
> For example, consider file open flags, mmap access flags, chmod flags, etc. You need some way to pass READ | TEXT, WRITE | SHARED, etc. occasionally, but most of the time you just pass READ or WRITE. It wouldn't be too onerous to require an explicit {READ, SHARED} set for the uncommon case, but set(READ) for the common case would be horrible. And you don't want every function that takes enums to typeswitch so it can handle a single value as if it were a set. The obvious answer to this is the one every other language uses: take READ | SHARED.
>
> This doesn't have to mean flag enums have integral values. However, the only alternative I can think of is that enums have set syntax, so READ | SHARED returns something like READ.__class__({READ, SHARED}), which of course also means that set has to automatically be a valid value for the type. Which is pretty magical and complex.
>
> Of course the answer could be not to allow that. Call mmap(file, access=WRITE, contention=SHARED), not mmap(WRITE | SHARED). But if we're talking about changing the stdlib and hopefully most popular third party libs, that would be a pretty drastic change.
>
> Finally, the answer could be: if you want that, you have to give them int values, and accept ints instead of enums in your APIs, at which point I think the benefit of valueless enums has gone way down, because many places you'd think you want them (including all over the stdlib), you don't.

We shouldn't get hung up on "don't care about the values" == "no value assigned".  I personally don't see any value in a 
valueless enum; I do see value in three other types: sequence (based on int), bitmask (based on int), and unique (or 
string -- based on str).  It seems to me that we're throwing around "valueless" because we don't want some enums to 
support math operations -- so use the string version for that type.

As for your example above:

Python 3.2.3 (default, Oct 19 2012, 19:53:16)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

-->from yaenum import BitMaskEnum, enum

-->BitMaskEnum.create('MMap', 'READ WRITE TEXT SHARED', export=globals())
<class 'yaenum.MMap'>

-->READ, WRITE, TEXT, SHARED
(MMap('READ', value=1), MMap('WRITE', value=2), MMap('TEXT', value=4), MMap('SHARED', value=8))

-->READ | TEXT
MMap('READ|TEXT', value=5)

-->WRITE | SHARED
MMap('WRITE|SHARED', value=10)


From ethan at stoneleaf.us  Fri Feb 22 19:33:29 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 22 Feb 2013 10:33:29 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<5126D0D7.7040302@btinternet.com>
	<CAH0mxTT_nOrix=zh0Ey-CJTGrLj9stjWR0PK_8v34hiYvwLspw@mail.gmail.com>
	<2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com>
Message-ID: <5127B9F9.9020704@stoneleaf.us>

On 02/22/2013 10:21 AM, Alex Stewart wrote:
> On Friday, February 22, 2013 4:53:27 AM UTC-8, Joao S. O. Bueno wrote:
>     Requiring that things compare differently, IMHO, is too much
>     statically-typed-minded - it is not the way of Python. As for Alex's
>     surreal example of  someone defining "NO" and "YES" with different
>     values in different Enums and passing the wrong one to a given call -
>     this person will have it comming.
>
>
> I think this is unnecessarily dismissive.  Obviously, my example was simplified, but what if these two enums were
> created by two completely different people in different modules that just happen to be used in the same program?  Does
> the hapless programmer "have it coming" then?

+1


> You say that my suggestion is more like C++ than Python, but I counter that your suggestion seems to me a lot more like
> Perl than Python.  It smacks of one of the things I dislike most about Perl, which also happens to be, in my opinion,
> one of the language features most responsible for (often subtle) bugs in Perl code even by experienced programmers:  The
> contents of variables can have very different semantic meanings depending solely on the context in which they're
> evaluated (scalar vs array, etc).  In this case what you're saying is that just because entirely unrelated programmers
> happened to use the same int value underneath, that Align.TOP and Errno.EPERM and Choice.OK should all be considered to
> mean exactly the same thing everywhere (even if none of the programmers actually intended or foresaw that possibility),
> but what that meaning is actually depends entirely on what context they happen to be evaluated in.

+1


> Another way of looking at this is that what you're proposing is effectively implicit type-casting between enums of
> different types.  Python does support implicit casting in a very few well-defined cases, but not most of the time, and
> only when doing so does not discard important semantic information (i.e. "meaning").  For ints, floats, etc, they really
> only have one piece of semantic information ("a number value").  Enums, however, have more than that.  They have
> (potentially) a number value, but they also have names, and they (preferably) have information about "what type of thing
> am I".  Automatically throwing out that extra information on a comparison where both objects have the extra info and
> it's not the same on both objects is, in my opinion, just wrong.  If there were no extra semantic meaning to enums, then
> this wouldn't be an issue, but the symbolic information they carry is clearly important, because it's the whole reason
> we're having this discussion in the first place instead of just using ints everywhere.

+1


> In pretty much all the cases I can imagine (and that have been suggested thus far), it seems to me that passing a
> different class of enum than the other side is testing against is almost always going to be an accident, or at the very
> least sloppy code (if there are cases where it is intentional, it's unlikely to be common, and the programmer should be
> able to explicitly cast between them somehow).  Given this, I think that the Principle of Least Surprise enters into
> things here as well.  If somebody makes a coding mistake and uses the wrong type of enum somewhere, which is likely to
> be the least surprising outcome?:
>
>  1. If you use an enum of the wrong type, it won't be considered equivalent to any of the expected options.
>  2. If you use an enum of the wrong type, it might not be considered to be equivalent to any of the expected options, or
>     it might match one of them and do something either the same or completely different than you expect, but you have no
>     way to tell what will happen without looking at an underlying constant which you're not supposed to have to care about.

+1000

Just to be clear, option 1 is what should happen.

--
~Ethan~


From solipsis at pitrou.net  Fri Feb 22 19:43:10 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 22 Feb 2013 19:43:10 +0100
Subject: [Python-ideas] range
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
	<CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>
	<loom.20130222T183719-360@post.gmane.org>
Message-ID: <20130222194310.75129c1f@pitrou.net>

On Fri, 22 Feb 2013 17:40:29 +0000 (UTC)
Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de>
wrote:
> Nick Coghlan <ncoghlan at ...> writes:
> 
> > -1 from me as well. We don't even implement concatenation and
> > repetition for ranges in order to keep them simple - we have to caveat
> > the range docs to point out they don't *quite* implement the full
> > sequence API due to this restriction.
> > 
> > However, I will note that starting in Python 3.3, range objects expose
> > their "start", "stop" and "step" attributes. That makes it much easier
> > for third party software to manipulate them arithmetically.
> > 
> > Cheers,
> > Nick.
> > 
> 
> Any plans to make range a valid base class to build upon in future releases?

I suppose it wouldn't very difficult to make range subclassable.
You can try writing a patch if you want:
http://docs.python.org/devguide/

Regards

Antoine.




From bruce at leapyear.org  Fri Feb 22 20:08:34 2013
From: bruce at leapyear.org (Bruce Leban)
Date: Fri, 22 Feb 2013 11:08:34 -0800
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CABicbJKr8g8_yN5KYhF3CLwcLDBn9g2B5y9s99LU_sFhorw6pA@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
	<87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
	<C330DB1C-A165-4C69-A5FB-4CDBA2697063@yahoo.com>
	<CABicbJKr8g8_yN5KYhF3CLwcLDBn9g2B5y9s99LU_sFhorw6pA@mail.gmail.com>
Message-ID: <CAGu0AnurHcM2vb3WTqFW5v744Q_wgiC5K1jfH34Ci3fks0k1QQ@mail.gmail.com>

To take this back to the ideas stage, one idea might be to integrate hmac
into pickle. At a minimum, provide some sample code showing how to wrap an
hmac around a pickled object. Related to this I note that it would be
helpful if the hmac docs gave some advice on key generation (e.g.,
suggested length).

Or to be a little more convenient, add new methods like these:

pickle.set_hmac_key([key])
    Sets the key used when pickle hmacs are generated. The key is as
expected by hmac.new. If key is not provided or this function is not called
before using an hmac, a random key is generated that will vary each time
the program is run. This is useful if you do not want pickles to be
reusable between different runs of the program.

pickle.dump_hmac(obj, file[, protocol])
    Same as pickle.dump except that it attaches an hmac to the pickled data.

pickle.dumps_hmac(obj[, protocol])
    Same as pickle.dumps except that it attaches an hmac to the pickled
data.

pickle.load_hmac(file)
    Same as pickle.load except that it verifies and removes an hmac as
attached by dump_hmac. Raises UnpicklingHmacError if the hmac cannot be
verified.

pickle.loads_hmac(string)
    Same as pickle.loads except that it (1) verifies and removes an hmac as
attached by dump_hmac and (2) it does not ignore extra characters. Raises
UnpicklingHmacError if the hmac cannot be verified.

The reason I suggest setting the hmac key at a global level rather than in
each call is that this eliminates the need for either passing around the
key or generating keys at multiple points in the code. If a key were passed
in each call, it would have the benefit that a program could use multiple
keys to ensure that pickles from one part of the program were not unpickled
in other parts. This seems like a heavy-handed use of the feature.

The reason I suggest using new methods rather than adding a keyword arg to
the current methods is that this facilitates wholesale replacement of
pickle.dump
with pickle.dump_hmac and I don't envision an explosion of variations.
Usually I'm an advocate of doing it the other way around. :-)

--- Bruce
Latest blog post: Alice's Puzzle Page http://www.vroospeak.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/e4f56b14/attachment.html>

From dholth at gmail.com  Fri Feb 22 20:09:12 2013
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 22 Feb 2013 14:09:12 -0500
Subject: [Python-ideas] range
In-Reply-To: <20130222194310.75129c1f@pitrou.net>
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
	<CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>
	<loom.20130222T183719-360@post.gmane.org>
	<20130222194310.75129c1f@pitrou.net>
Message-ID: <CAG8k2+4tUSuPfXW1oB=B8gVAdwO-rqRgQFj08Gj-VFh9RVp4xg@mail.gmail.com>

On Fri, Feb 22, 2013 at 1:43 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Fri, 22 Feb 2013 17:40:29 +0000 (UTC)
> Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de>
> wrote:
> > Nick Coghlan <ncoghlan at ...> writes:
> >
> > > -1 from me as well. We don't even implement concatenation and
> > > repetition for ranges in order to keep them simple - we have to caveat
> > > the range docs to point out they don't *quite* implement the full
> > > sequence API due to this restriction.
> > >
> > > However, I will note that starting in Python 3.3, range objects expose
> > > their "start", "stop" and "step" attributes. That makes it much easier
> > > for third party software to manipulate them arithmetically.
> > >
> > > Cheers,
> > > Nick.
> > >
> >
> > Any plans to make range a valid base class to build upon in future
> releases?
>
> I suppose it wouldn't very difficult to make range subclassable.
> You can try writing a patch if you want:
> http://docs.python.org/devguide/
>
> Regards


I would be very happy to see a similar data structure available (on pypi).
I was writing an HTTP-backed file object that used partial GET requests to
just fetch the parts of the file that were actually read; it would have
benefited from an efficient set-of-ranges (start+stop but not step)
implementation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/6189da8c/attachment.html>

From foogod at gmail.com  Fri Feb 22 20:14:52 2013
From: foogod at gmail.com (Alex Stewart)
Date: Fri, 22 Feb 2013 11:14:52 -0800 (PST)
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <A15CBBA7-6426-47BC-8E43-B74094B59332@yahoo.com>
References: <CACxLcFcsQbnqtdqSPmf_AqK319bwLDOPONgQEYcBr+Ce=WFxGw@mail.gmail.com>
	<5126ACC2.3080904@canterbury.ac.nz>
	<A15CBBA7-6426-47BC-8E43-B74094B59332@yahoo.com>
Message-ID: <892ca40c-474b-4e34-ad94-94051a0b4182@googlegroups.com>



On Friday, February 22, 2013 9:07:32 AM UTC-8, Andrew Barnert wrote:
>
> On Feb 21, 2013, at 15:24, Greg Ewing <greg.... at canterbury.ac.nz<javascript:>> 
> wrote: 
>
> That sounds reasonable. However, I'm wondering if there isn't a 
> > third case: where you don't care about the values, but you do 
> > want them to have a defined ordering? 
>

This is a valid point.  I think there are a lot of cases where enums are 
useful without any ordering, but there are arguably some cases where order 
would be useful (the immediate example that comes to mind is "logging 
levels", where DEBUG, INFO, and ERROR don't necessarily need to have any 
numerical meaning, but it would be nice to be able to say that DEBUG < INFO 
< ERROR)..  I think it should be possible to build this into an existing 
enum implementation which doesn't inherently rely on underlying numbers, 
though..  I'll look into it..

And a fourth case: You don't care about the values, but you want to be able 
> to | them together into a set. 
>

Heh.. I was kinda hoping nobody was going to bring this up yet :)  Good 
point, though, and FWIW, I had considered this too but didn't bring it up 
initially because I was afraid it would muddy the discussion too much too 
early.   I do however agree that this is a use case that will ultimately be 
important to support, and it's been in the back of my mind.  This actually 
applies both to the "valueless" case and the "valued" case, but in somewhat 
different ways.

As you mentioned, in the "valueless" case, the most obvious way to deal 
with this is with sets.  I think we could probably ameliorate the 
single-value case a bit by just having enum objects also behave like 
single-item sets if used with set operations (which I don't think is too 
magic), and then we can define the '|' operator to just create (or add to) 
an enum-set..

The valued case is actually more complicated, because ideally if READ = 
enum(1) and SHARED = enum(4), then saying "READ | SHARED" should produce 
something that has an int() value of the or'd values (5), but it would also 
be nice if it still represented itself symbolically (as "READ | SHARED", 
for example, instead of "5"), and though not strictly required, it would 
probably also be nice if they could be inspected using the same set-type 
operations as valueless enums/sets ("if READ in openmode", for example). 
 This would probably require some sort of new "enum-set" class/type which 
supported amalgomated valued enums, but I think it would still be doable 
without too much magic (I hope).  Then of course there's extra issues like: 
if we also support string-enums, what does 'or' mean for string constants? 
(etc, etc..)

But in summary, I think the valueless case is actually pretty easy to 
implement, but doing it well with valued-enums is much more work, which 
once again reinforces my opinion that valueless enums are useful to have, 
and preferable to use when lowlevel-type-compatibility is not explicitly 
required by some API..

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/a1ea449b/attachment.html>

From foogod at gmail.com  Fri Feb 22 20:33:41 2013
From: foogod at gmail.com (Alex Stewart)
Date: Fri, 22 Feb 2013 11:33:41 -0800 (PST)
Subject: [Python-ideas] iterable.__unpack__ method
Message-ID: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>

So when putting together my enum implementation I came up against something 
I've run into a couple of times in a few different situations, and I 
thought warranted some discussion as its own proposal:  There are times 
where it would be extremely convenient if there was a way when unpacking an 
object (using the "a, b, c = foo" notation) for that object to know how 
many values it needed to provide.

I would like to propose the following:

When the unpack notation is used, the interpreter will automatically look 
for an __unpack__ method on the rvalue, and if it exists will call it as:

    rvalue.__unpack__(min_count, max_count)

... and use the returned value as the iterable to unpack into the variables 
instead of the original object.  (If no __unpack__ method exists, it will 
just use the rvalue itself, as it does currently)

In the case of simple unpacking ("a, b, c = foo"), min_count and max_count 
will be the same value (3).  In the case of extended unpacking ("a, b, c, 
*d = foo"), min_count would be the smallest length that will not cause an 
unpacking exception (3, in this case), and max_count would be None.

By effectively separating the notion of "unpackable" from "iterable", this 
would allow for a few useful things:

   1. It would allow otherwise non-iterable objects to support the unpack 
   notation if it actually makes sense to do so
   2. It would allow for "smart" unpackable objects (such as __ in my enum 
   example), which would "just work" no matter how many values they're 
   required to produce.
   3. It would allow (properly designed) infinite (or 
   unknown-possibly-large-size) iterators to be unpackable into a finite 
   sequence of variables if desired (which can only be done with extra 
   special-case code currently)
   4. It could make backwards-compatibility easier for APIs which return 
   iterables (i.e. if you want to change a function that used to return a 
   3-tuple to return a 5-tuple, instead of inherently breaking all existing 
   code that chose to unpack the return value (assuming only 3 items would 
   ever be returned), you could return an unpackable object which will work 
   correctly with both old and new code)
   
Thoughts?

--Alex

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/12731dd0/attachment.html>

From foogod at gmail.com  Fri Feb 22 20:25:08 2013
From: foogod at gmail.com (Alex Stewart)
Date: Fri, 22 Feb 2013 11:25:08 -0800 (PST)
Subject: [Python-ideas] Yet another enum proposal :)
Message-ID: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>

Ok, so at the risk of muddying the waters even more, I've put together yet 
another possible way to do enums, and would be interested to hear comments..

Based on the list of required/desired/undesired properties laid out in the 
enum PEP thread, I believe a lot of the back and forth to date has been due 
to the fact that all of the proposed implementations fall short of 
fulfilling at least some of the desired properties for a general-purpose 
enum implementation in different ways.  I've put together something based 
on ideas from Tim's, Barry's, other things thrown out in discussion, and a 
few of my own which I think comes closer to ticking off most of the boxes.

(The code and some examples are available at 
https://github.com/foogod/pyenum)

Enums/groups are defined by creating subclasses of Enum, similarly to 
Barry's implementation.  The distinction is that the base "Enum" class does 
not have any associated values (they're just singleton objects with names). 
 At a basic level, the values themselves are defined like so:

class Color (Enum):
    RED = __
    GREEN = __
    BLUE = __

As has been (quite correctly) pointed out before, the single-underscore (_) 
has a well-established meaning in most circles related to gettext/etc, so 
for this application I picked the next best thing, the double-underscore 
(__) instead.  I think this is reasonably mnemonic as a "fill in the blank" 
placeholder, and also not unduly verbose.  One advantage of using __ over, 
say, ellipsis (...) is that since it involves a name resolution, we can add 
(just a little!) magic to generate distinct __ objects on each reference, 
so, for example, the following actually works as the user probably expects 
it to:

class Color (Enum):
    RED = CRIMSON = __
    BLUE = __

(RED and CRIMSON actually become aliases referring to the same enum value, 
but BLUE is a different enum value)

One other rather notable advantage to __ is that we can define a special 
multiplication behavior for it which allows us to make the syntax much more 
compact:

class Color (Enum):
    RED, GREEN, BLUE, ORANGE, VIOLET, BEIGE, ULTRAVIOLET, ULTRABEIGE = __ * 
8

(as an aside, I have an idea about how we might be able to get rid of the 
"* 8" altogether, but it requires a different (I think generally useful) 
change to the language which I will propose separately)

Each enum value is actually an instance of its defining class, so you can 
determine what type of enum something is with simple inheritance checks:

>>> isinstance(Color.RED, Color)
True

For enums which need to have int values, we can use IntEnum instead of Enum:

class Errno (IntEnum):
    EPERM = 1
    ENOENT = 2
    ESRCH = 3
    EINTR = 4

(Technically, IntEnum is just a special case of the more generic TypeEnum 
class:

class IntEnum (TypeEnum, basetype=int):
    pass

..which means that theoretically you could define enums based on (almost) 
any base type (examples.py has an example using floats))

Anyway, as expected, enums have reasonable strs/reprs, and can be easily 
translated to/from base values using traditional "casting" syntax:

>>> Errno.EPERM
<__main__.Errno.EPERM (1)>
>>> str(Errno.EPERM)
'EPERM'
>>> int(Errno.EPERM)
1
>>> Errno(1)
<__main__.Errno.EPERM (1)>

You can also lookup enums by name using index notation:

>>> Errno['EPERM']
<__main__.Errno.EPERM (1)>

It's actually worth noting here that TypeEnums are actually subclasses of 
their basetype, so IntEnums are also ints, and can be used as drop-in 
replacements for any existing code which is expecting an int argument. 
 They can also be compared directly as if they were ints:

if exc.errno == Errno.EPERM:
   do_something()

TypeEnums enforce uniqueness:

>>> class Foo (IntEnum):
...     A = 1
...     B = 1
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  ...
ValueError: Duplicate enum value: 1

However, you can define "aliases" for enums, if you want to:

>>> class Foo (IntEnum):
...     A = 1
...     B = A
... 
>>> Foo.A
<__main__.Foo.A (1)>
>>> Foo.B
<__main__.Foo.A (1)>

Of course, pure (valueless) Enum instances are logical singletons (i.e. 
they only compare equal to themselves), so there's no potential for 
duplication there.

Finally, a special feature of __ is that it can also be called with 
parameters to create enum values with docstrings or other metadata:

>>> class Errno (IntEnum):
...     EPERM = __(1, doc="Permission denied")
... 
>>> Errno.EPERM.__doc__
'Permission denied'
>>> class Color (Enum):
...     RED = __(doc="The color red", rgb=(1., 0., 0.))
... 
>>> Color.RED.__doc__
'The color red'
>>> Color.RED.rgb
(1.0, 0.0, 0.0)

A few other notable properties:

   - Enum values are hashable, so they can be used as dict keys, etc.
   - Though enum values will compare equal to their associated basetype 
   values (Errno.EPERM == 1), enum values from different classes never compare 
   equal to each other (Foo.A != Errno.EPERM).  This prevents enum-aware code 
   from accidentally mistaking one enum for a completely different enum (with 
   a completely different meaning) simply because they map to the same int 
   value.
   - Using the class constructor does not create new enums, it just returns 
   the existing singleton enum associated with the passed value (Errno(1) 
   returns Errno.EPERM, not a new Errno).  If there is no predefined enum 
   matching that value, a ValueError is raised, thus using the constructor is 
   basically a "cast" operation, and only works within the supported range of 
   values.  (New enum values can be created using the .new() method on the 
   class, however, if desired)

So... thoughts?

--Alex

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/05725d90/attachment.html>

From guido at python.org  Fri Feb 22 22:17:59 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 22 Feb 2013 13:17:59 -0800
Subject: [Python-ideas] My wishlist for Tulip
In-Reply-To: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>
References: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>
Message-ID: <CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>

Those are all good ideas! Now I just have to find time to implement them...

On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
> I understand it's still beta but anyways, here's my little wish list for Tulip.
>
> * provide a 'demo' / 'example' directory containing very simple
> scripts showing the most basic usages such as:
>
> - echo_tcp_client.py
> - echo_tcp_server.py
> - echo_tcp_server_w_timeout.py  (same as echo_server.py but also
> disconnects the client after a certain time of inactivity)
> - echo_tcp_ssl_client.py
> - echo_tcp_ssl_server.py
> - echo_udp_client.py
> - echo_udp_server.py
>
> * move all *test*.py scripts in a separate 'test' directory
>
> * if it's not part of the API intended to be public move
> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new
> 'scripts'/'tools' directory)
>
> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks
> like they belong elsewhere
>
> * write a simple benchmark framework testing (at least) sending,
> receiving and the internal scheduler  (I'd like to help with this one)
>
>
> --- Giampaolo
> http://code.google.com/p/pyftpdlib/
> http://code.google.com/p/psutil/
> http://code.google.com/p/pysendfile/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



--
--Guido van Rossum (python.org/~guido)


From fafhrd91 at gmail.com  Fri Feb 22 23:37:44 2013
From: fafhrd91 at gmail.com (Nikolay Kim)
Date: Fri, 22 Feb 2013 14:37:44 -0800
Subject: [Python-ideas] My wishlist for Tulip
In-Reply-To: <CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>
References: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>
	<CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>
Message-ID: <D20535FF-84AC-42BD-8D27-76A6AA76D2A9@gmail.com>

I'm planing to work on udp support sometime next week. i'll implement echo examples except udp, later today.


On Feb 22, 2013, at 1:17 PM, Guido van Rossum <guido at python.org> wrote:

> Those are all good ideas! Now I just have to find time to implement them...
> 
> On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
>> I understand it's still beta but anyways, here's my little wish list for Tulip.
>> 
>> * provide a 'demo' / 'example' directory containing very simple
>> scripts showing the most basic usages such as:
>> 
>> - echo_tcp_client.py
>> - echo_tcp_server.py
>> - echo_tcp_server_w_timeout.py  (same as echo_server.py but also
>> disconnects the client after a certain time of inactivity)
>> - echo_tcp_ssl_client.py
>> - echo_tcp_ssl_server.py
>> - echo_udp_client.py
>> - echo_udp_server.py
>> 
>> * move all *test*.py scripts in a separate 'test' directory
>> 
>> * if it's not part of the API intended to be public move
>> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new
>> 'scripts'/'tools' directory)
>> 
>> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks
>> like they belong elsewhere
>> 
>> * write a simple benchmark framework testing (at least) sending,
>> receiving and the internal scheduler  (I'd like to help with this one)
>> 

we can test performance on gunicorn with 3 different workers (sync, gevent, gtulip)



From ethan at stoneleaf.us  Sat Feb 23 00:16:45 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 22 Feb 2013 15:16:45 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
Message-ID: <5127FC5D.3040609@stoneleaf.us>

On 02/22/2013 11:25 AM, Alex Stewart wrote:
> Ok, so at the risk of muddying the waters even more, I've put together yet another possible way to do enums, and would
> be interested to hear comments..

Largely similar to my own implementation -- so of course I like it!  :)


> A few other notable properties:
>
>   * Enum values are hashable, so they can be used as dict keys, etc.

Problem here:  should we have our enums hash the same as the underlying value?  Consider:

--> import yaenum

--> class Color(yaenum.Enum):
...     black
...     red
...     green
...     blue
...

--> class Literature(yaenum.Enum):
...    scifi
...    fantasy
...    mystery
...    pop
...

--> Color.black
Color('black', value=0)

--> Literature.scifi
Literature('scifi', value=0)

--> black = Color.black

--> scifi = Literature.scifi

--> black == 0
True

--> hash(black)
0

--> scifi == 0
True

--> hash(scifi)
0

--> black == scifi
False

--> hash(0)
0

--> huh = dict()
--> huh[black] = 9
--> huh
{Color('black', value=0): 9}

--> huh[0]
9
--> huh[scifi]
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
KeyError: Literature('scifi', value=0)

--> huh[scifi] = 11
--> huh
{Color('black', value=0): 9, Literature('scifi', value=0): 11}

--> huh[0]
9

--> del huh[0]
--> huh[0]
11

 From a practicality standpoint the question is:  How likely is it to use different enum classes as keys?

--
~Ethan~


From alexandre.zani at gmail.com  Sat Feb 23 00:45:32 2013
From: alexandre.zani at gmail.com (Alexandre Zani)
Date: Fri, 22 Feb 2013 15:45:32 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <5127FC5D.3040609@stoneleaf.us>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
	<5127FC5D.3040609@stoneleaf.us>
Message-ID: <CAJVMpooDJdfp=Jf0gHFVnUwUzwz-9eO5B=zvn=zxLAi1BvV_UA@mail.gmail.com>

On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 02/22/2013 11:25 AM, Alex Stewart wrote:
>
>> Ok, so at the risk of muddying the waters even more, I've put together
>> yet another possible way to do enums, and would
>> be interested to hear comments..
>>
>
> Largely similar to my own implementation -- so of course I like it!  :)
>
>
>  A few other notable properties:
>>
>>   * Enum values are hashable, so they can be used as dict keys, etc.
>>
>
> Problem here:  should we have our enums hash the same as the underlying
> value?  Consider:
>

One feature of the implementation is that enum values from different enum
classes will never equal each other. It would be very weird for them to
have identical hashes.


>
> --> import yaenum
>
> --> class Color(yaenum.Enum):
> ...     black
> ...     red
> ...     green
> ...     blue
> ...
>
> --> class Literature(yaenum.Enum):
> ...    scifi
> ...    fantasy
> ...    mystery
> ...    pop
> ...
>
> --> Color.black
> Color('black', value=0)
>
> --> Literature.scifi
> Literature('scifi', value=0)
>
> --> black = Color.black
>
> --> scifi = Literature.scifi
>
> --> black == 0
> True
>
> --> hash(black)
> 0
>
> --> scifi == 0
> True
>
> --> hash(scifi)
> 0
>
> --> black == scifi
> False
>
> --> hash(0)
> 0
>
> --> huh = dict()
> --> huh[black] = 9
> --> huh
> {Color('black', value=0): 9}
>
> --> huh[0]
> 9
> --> huh[scifi]
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> KeyError: Literature('scifi', value=0)
>
> --> huh[scifi] = 11
> --> huh
> {Color('black', value=0): 9, Literature('scifi', value=0): 11}
>
> --> huh[0]
> 9
>
> --> del huh[0]
> --> huh[0]
> 11
>
> From a practicality standpoint the question is:  How likely is it to use
> different enum classes as keys?
>
> --
> ~Ethan~
> ______________________________**_________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/5403a8a3/attachment.html>

From bruce at leapyear.org  Sat Feb 23 00:57:25 2013
From: bruce at leapyear.org (Bruce Leban)
Date: Fri, 22 Feb 2013 15:57:25 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <5127FC5D.3040609@stoneleaf.us>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
	<5127FC5D.3040609@stoneleaf.us>
Message-ID: <CAGu0Anutxxor6mnOBt3wi8JrraKiViSNOBmMLi4cmGEH8103bQ@mail.gmail.com>

On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

>
> Problem here:  should we have our enums hash the same as the underlying
> value?  Consider:
>
> <snip>

>
> From a practicality standpoint the question is:  How likely is it to use
> different enum classes as keys?
>
>
Having the same hash value isn't the problem.

>>> hash(-1) == hash(-2)
True
>>> d={}
>>> d[-1] = 'm1'
>>> d[-2] = 'm2'
>>> d
{-2: 'm2', -1: 'm1'}
>>> d[-2]
'm2'
>>> del d[-2]
>>> d[-2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: -2
>>>


The problem is that black == 0 and scifi == 0. So when the hash values
collide, it chains them and then uses == to compare the 0 to find the
matching value in the table. To avoid this problem ensure that hash(enum)
!= hash(int(enum)) [or whatever the base type of the enum is]. Actually,
I'm not sure that works without looking at the implementation of dict. When
there are multiple values in a bucket, does it compare hash values first or
does it jump right to ==?

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/0bc89270/attachment.html>

From foogod at gmail.com  Sat Feb 23 01:01:30 2013
From: foogod at gmail.com (Alex Stewart)
Date: Fri, 22 Feb 2013 16:01:30 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <CAJVMpooDJdfp=Jf0gHFVnUwUzwz-9eO5B=zvn=zxLAi1BvV_UA@mail.gmail.com>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
	<5127FC5D.3040609@stoneleaf.us>
	<CAJVMpooDJdfp=Jf0gHFVnUwUzwz-9eO5B=zvn=zxLAi1BvV_UA@mail.gmail.com>
Message-ID: <CACxLcFdzDNgdhn2xzKru9Go3x2XxYGD+PFbrbZ7pwOGDFmvWxA@mail.gmail.com>

On Fri, Feb 22, 2013 at 3:45 PM, Alexandre Zani <alexandre.zani at gmail.com>wrote:

> On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>
>>   * Enum values are hashable, so they can be used as dict keys, etc.
>>>
>>
>> Problem here:  should we have our enums hash the same as the underlying
>> value?  Consider:
>>
>
> One feature of the implementation is that enum values from different enum
> classes will never equal each other. It would be very weird for them to
> have identical hashes.
>

Actually, not really.  The only requirement in Python is that if two
objects compare equal, they need to have the same hash value, but there is
no requirement the other direction (indeed, that would be a problem in many
cases as the domain of hash values is inherently much smaller than the
domain of all possible object data, so it must be expected that you will
get hash duplication from time to time, no matter what method you use for
calculating the hashes).

In the case of pure Enums, they're just singleton object instances, so they
hash the same as any other object, which works pretty much as one would
expect.

In the case of TypeEnums, since they do compare equal to the underlying
value, they must also hash the same. However hashes are just a "hint":  Two
objects can hash the same but not compare equal, in which case they will be
considered to be different objects when used as dictionary keys, etc.
 There is actually an example of this in the examples.py I put up in github
with the reference code.  It creates a dictionary with two keys, which are
two different Enums, which both use the same int value.  Because they do
not compare equal, they are treated as different keys by Python, as I think
most people would expect..

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/b6894bcf/attachment.html>

From alexandre.zani at gmail.com  Sat Feb 23 01:09:32 2013
From: alexandre.zani at gmail.com (Alexandre Zani)
Date: Fri, 22 Feb 2013 16:09:32 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <CACxLcFdzDNgdhn2xzKru9Go3x2XxYGD+PFbrbZ7pwOGDFmvWxA@mail.gmail.com>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
	<5127FC5D.3040609@stoneleaf.us>
	<CAJVMpooDJdfp=Jf0gHFVnUwUzwz-9eO5B=zvn=zxLAi1BvV_UA@mail.gmail.com>
	<CACxLcFdzDNgdhn2xzKru9Go3x2XxYGD+PFbrbZ7pwOGDFmvWxA@mail.gmail.com>
Message-ID: <CAJVMpopWztXhmLJEYhbrKO7oyyL2V_wwJuuw6DeM7wKEWBka6g@mail.gmail.com>

On Fri, Feb 22, 2013 at 4:01 PM, Alex Stewart <foogod at gmail.com> wrote:

> On Fri, Feb 22, 2013 at 3:45 PM, Alexandre Zani <alexandre.zani at gmail.com>wrote:
>
>> On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>>>   * Enum values are hashable, so they can be used as dict keys, etc.
>>>>
>>>
>>> Problem here:  should we have our enums hash the same as the underlying
>>> value?  Consider:
>>>
>>
>> One feature of the implementation is that enum values from different enum
>> classes will never equal each other. It would be very weird for them to
>> have identical hashes.
>>
>
> Actually, not really.  The only requirement in Python is that if two
> objects compare equal, they need to have the same hash value, but there is
> no requirement the other direction (indeed, that would be a problem in many
> cases as the domain of hash values is inherently much smaller than the
> domain of all possible object data, so it must be expected that you will
> get hash duplication from time to time, no matter what method you use for
> calculating the hashes).
>

My bad. Please forget I ever mentioned it.

>
> In the case of pure Enums, they're just singleton object instances, so
> they hash the same as any other object, which works pretty much as one
> would expect.
>
> In the case of TypeEnums, since they do compare equal to the underlying
> value, they must also hash the same. However hashes are just a "hint":  Two
> objects can hash the same but not compare equal, in which case they will be
> considered to be different objects when used as dictionary keys, etc.
>  There is actually an example of this in the examples.py I put up in github
> with the reference code.  It creates a dictionary with two keys, which are
> two different Enums, which both use the same int value.  Because they do
> not compare equal, they are treated as different keys by Python, as I think
> most people would expect..
>
> --Alex
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/defabbe1/attachment.html>

From bruce at leapyear.org  Sat Feb 23 01:38:45 2013
From: bruce at leapyear.org (Bruce Leban)
Date: Fri, 22 Feb 2013 16:38:45 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <CAGu0Anutxxor6mnOBt3wi8JrraKiViSNOBmMLi4cmGEH8103bQ@mail.gmail.com>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
	<5127FC5D.3040609@stoneleaf.us>
	<CAGu0Anutxxor6mnOBt3wi8JrraKiViSNOBmMLi4cmGEH8103bQ@mail.gmail.com>
Message-ID: <CAGu0Anv2bvSY-pUumieMgzx9o0QQS1jgD1_SxhACWyGo89n3hQ@mail.gmail.com>

On Fri, Feb 22, 2013 at 3:57 PM, Bruce Leban <bruce at leapyear.org> wrote:

> To avoid this problem ensure that hash(enum) != hash(int(enum)) [or
> whatever the base type of the enum is].
>

Never mind. That's a bad idea. As others pointed out, if two objects
compare equal, they have to have the same hash value. This weird behavior
is a side effect of having non-transitive equality (a == b, b == c but a !=
c) and objects like dicts are not designed to work with objects with
non-transitive equality. Not having transitivity when it's expected leads
to weird behavior. For example, if you have non-transitive inequality  (a <
b < c < a, as is the case in PHP, for example) then sorting may not work
and some sorting code can even get into infinite loops.

--- Bruce
Latest blot post: Alice's Puzzle Page http://www.vroospeak.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/f9efaa42/attachment.html>

From ethan at stoneleaf.us  Sat Feb 23 02:12:49 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 22 Feb 2013 17:12:49 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <CAGu0Anv2bvSY-pUumieMgzx9o0QQS1jgD1_SxhACWyGo89n3hQ@mail.gmail.com>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
	<5127FC5D.3040609@stoneleaf.us>
	<CAGu0Anutxxor6mnOBt3wi8JrraKiViSNOBmMLi4cmGEH8103bQ@mail.gmail.com>
	<CAGu0Anv2bvSY-pUumieMgzx9o0QQS1jgD1_SxhACWyGo89n3hQ@mail.gmail.com>
Message-ID: <51281791.4040000@stoneleaf.us>

On 02/22/2013 04:38 PM, Bruce Leban wrote:
>
> On Fri, Feb 22, 2013 at 3:57 PM, Bruce Leban <bruce at leapyear.org <mailto:bruce at leapyear.org>> wrote:
>
>     To avoid this problem ensure that hash(enum) != hash(int(enum)) [or whatever the base type of the enum is].
>
>
> Never mind. That's a bad idea. As others pointed out, if two objects compare equal, they have to have the same hash
> value. This weird behavior is a side effect of having non-transitive equality (a == b, b == c but a != c) and objects
> like dicts are not designed to work with objects with non-transitive equality. Not having transitivity when it's
> expected leads to weird behavior. For example, if you have non-transitive inequality  (a < b < c < a, as is the case in
> PHP, for example) then sorting may not work and some sorting code can even get into infinite loops.

Yeah, I can easily see this issue keeping an enum implementation out of the stdlib; perhaps an acceptable breakage
would be to have enum instances not be hashable?

--
~Ethan~


From ethan at stoneleaf.us  Sat Feb 23 01:41:00 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 22 Feb 2013 16:41:00 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <CAGu0Anutxxor6mnOBt3wi8JrraKiViSNOBmMLi4cmGEH8103bQ@mail.gmail.com>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
	<5127FC5D.3040609@stoneleaf.us>
	<CAGu0Anutxxor6mnOBt3wi8JrraKiViSNOBmMLi4cmGEH8103bQ@mail.gmail.com>
Message-ID: <5128101C.3070803@stoneleaf.us>

On 02/22/2013 03:57 PM, Bruce Leban wrote:
>
> On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman <ethan at stoneleaf.us <mailto:ethan at stoneleaf.us>> wrote:
>
>
>     Problem here:  should we have our enums hash the same as the underlying value?  Consider:
>
> <snip>
>
>
>      From a practicality standpoint the question is:  How likely is it to use different enum classes as keys?
>
>
> Having the same hash value isn't the problem.

True -- the problem is using two different enum classes with the same underlying int value as keys and then using
ints instead of the enums to try to access the values.


> The problem is that black == 0 and scifi == 0. So when the hash values collide, it chains them and then uses == to
> compare the 0 to find the matching value in the table. To avoid this problem ensure that hash(enum) != hash(int(enum))
> [or whatever the base type of the enum is].

If we do this, then a plain int won't be able to look up an enum key, which would mean that int enums are not drop-in
replacements for int's currently be used in enum-type contexts.

I'm inclined to leave the problem as is, though, unless somebody has a use case where they are using two different
enum classes in a single dictionary?


>  Actually, I'm not sure that works without looking at the implementation of
> dict. When there are multiple values in a bucket, does it compare hash values first or does it jump right to ==?

dict uses hash first -- after all, it's a hash table.  :)

--
~Ethan~


From g.rodola at gmail.com  Sat Feb 23 03:14:15 2013
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Sat, 23 Feb 2013 03:14:15 +0100
Subject: [Python-ideas] My wishlist for Tulip
In-Reply-To: <D20535FF-84AC-42BD-8D27-76A6AA76D2A9@gmail.com>
References: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>
	<CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>
	<D20535FF-84AC-42BD-8D27-76A6AA76D2A9@gmail.com>
Message-ID: <CAFYqXL9nFabMKvVpDPvt87YcP0Mp4QcDrfS7mLCQORW0SZebhQ@mail.gmail.com>

Also, I had some troubles figuring out how to run tests because
setup.py doesn't actually install Tulip and "import tulip" can be used
only from within the root directory.

> I'm planing to work on udp support sometime next week.
> i'll implement echo examples except udp, later today.

Thanks.

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/


From foogod at gmail.com  Sat Feb 23 03:57:36 2013
From: foogod at gmail.com (Alex Stewart)
Date: Fri, 22 Feb 2013 18:57:36 -0800 (PST)
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <5128101C.3070803@stoneleaf.us>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
	<5127FC5D.3040609@stoneleaf.us>
	<CAGu0Anutxxor6mnOBt3wi8JrraKiViSNOBmMLi4cmGEH8103bQ@mail.gmail.com>
	<5128101C.3070803@stoneleaf.us>
Message-ID: <378431d8-7927-4da4-9ca0-280a1733d36f@googlegroups.com>



On Friday, February 22, 2013 4:41:00 PM UTC-8, stoneleaf wrote:
>
> I'm inclined to leave the problem as is, though, unless somebody has a use 
> case where they are using two different 
> enum classes in a single dictionary?


I'll admit the behavior in this case is a little unconventional, but to be 
honest I think it's something that most people would find made a reasonable 
amount of sense if they stopped and thought about it for a minute, so I 
don't think it's that horrible (and seems like it'd be pretty rare anyway)..

In my opinion it's far less disturbing than some of the behavior we already 
have in the language for certain types of keys:

>>> x = {True: 'foo', 1: 'bar'}
>>> x
{True: 'bar'}

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130222/dcc8267e/attachment.html>

From guido at python.org  Sat Feb 23 06:17:15 2013
From: guido at python.org (Guido van Rossum)
Date: Fri, 22 Feb 2013 21:17:15 -0800
Subject: [Python-ideas] My wishlist for Tulip
In-Reply-To: <CAFYqXL9nFabMKvVpDPvt87YcP0Mp4QcDrfS7mLCQORW0SZebhQ@mail.gmail.com>
References: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>
	<CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>
	<D20535FF-84AC-42BD-8D27-76A6AA76D2A9@gmail.com>
	<CAFYqXL9nFabMKvVpDPvt87YcP0Mp4QcDrfS7mLCQORW0SZebhQ@mail.gmail.com>
Message-ID: <CAP7+vJJLH6202-hU7nZMvFA9eczEpbYSscDmKTqBLg5bsnn8GA@mail.gmail.com>

Personally I think you'd be crazy to install this version of tulip
(it's not beta, it's way before alpha!). The setup.py file only exists
to build the C extension for Windows. (How to run tests is explained
in README BTW. There is no boilerplate there. :-)


From tjreedy at udel.edu  Sat Feb 23 10:14:52 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 23 Feb 2013 04:14:52 -0500
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
Message-ID: <kga1b9$apn$1@ger.gmane.org>

On 2/22/2013 2:33 PM, Alex Stewart wrote:
> So when putting together my enum implementation I came up against
> something I've run into a couple of times in a few different situations,
> and I thought warranted some discussion as its own proposal:  There are
> times where it would be extremely convenient if there was a way when
> unpacking an object (using the "a, b, c = foo" notation) for that object
> to know how many values it needed to provide.

Such statements always have a definite number of targets, so you can 
always arrange for the iterable to produce the required number of items 
-- unless it cannot. The count can be used as a multiplier, to slice or 
islice, or as an argument to an iterator class or function (such as 
generator functions).

> I would like to propose the following:
>
> When the unpack notation is used, the interpreter will automatically
> look for an __unpack__ method on the rvalue, and if it exists will call
> it as:
>
>      rvalue.__unpack__(min_count, max_count)
>
> ... and use the returned value as the iterable to unpack into the
> variables instead of the original object.  (If no __unpack__ method
> exists, it will just use the rvalue itself, as it does currently)

It would be much easier, and have much the same effect, if unpacking 
simply requested the minumum number of items and stopped raising a 
ValueError if the iterator has more items. No new protocol is needed. 
Guido rejected this as silently masking errors.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Sat Feb 23 10:21:08 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 23 Feb 2013 04:21:08 -0500
Subject: [Python-ideas] range
In-Reply-To: <20130222194310.75129c1f@pitrou.net>
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
	<CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>
	<loom.20130222T183719-360@post.gmane.org>
	<20130222194310.75129c1f@pitrou.net>
Message-ID: <kga1n0$apn$2@ger.gmane.org>

On 2/22/2013 1:43 PM, Antoine Pitrou wrote:
> On Fri, 22 Feb 2013 17:40:29 +0000 (UTC)
> Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de>

>> Any plans to make range a valid base class to build upon in future releases?
>
> I suppose it wouldn't very difficult to make range subclassable.
> You can try writing a patch if you want:
> http://docs.python.org/devguide/

I was just about to suggest that making range subclassable would be an 
alternative to extending the current class. As far as I know, there is 
no principled reason why not, unlike Nonetype and bool.

-- 
Terry Jan Reedy



From rosuav at gmail.com  Sat Feb 23 11:18:49 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 23 Feb 2013 21:18:49 +1100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <kga1b9$apn$1@ger.gmane.org>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
Message-ID: <CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>

On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> It would be much easier, and have much the same effect, if unpacking simply
> requested the minumum number of items and stopped raising a ValueError if
> the iterator has more items. No new protocol is needed. Guido rejected this
> as silently masking errors.

What if it were done explicitly, though?

Currently:
>>> a,b,c=range(3)
>>> a,b,c=range(4)
Traceback (most recent call last):
  File "<pyshell#55>", line 1, in <module>
    a,b,c=range(4)
ValueError: too many values to unpack (expected 3)
>>> a,b,c,*d=range(6)
>>> a,b,c,* =range(4)
SyntaxError: invalid syntax

Suppose the last notation were interpreted as "request values for a,
b, and c, and then ignore the rest". This would support infinite
iterators, and would be an explicit statement from the caller that
it's not an error to have more elements. This doesn't solve the
backward-compatibility issue, but it would allow a client to
specifically engage a forward-compatibility mode (by putting ,* after
any unpack that might introduce more elements - of course, you'd need
to be confident that you won't care about those elements).

It can be done with current code, of course.
>>> a,b,c=itertools.islice(range(6),3)
But this has significant overhead in verbiage, where a simple ",*"
would carry the same sort of meaning it does in other contexts.

ChrisA


From ncoghlan at gmail.com  Sat Feb 23 12:11:11 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Feb 2013 21:11:11 +1000
Subject: [Python-ideas] range
In-Reply-To: <kga1n0$apn$2@ger.gmane.org>
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
	<CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>
	<loom.20130222T183719-360@post.gmane.org>
	<20130222194310.75129c1f@pitrou.net> <kga1n0$apn$2@ger.gmane.org>
Message-ID: <CADiSq7f5MMyEU6sBUw3UJcuyK1XuEDwJ5H3+yDUpG6xLtf09fA@mail.gmail.com>

On Sat, Feb 23, 2013 at 7:21 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 2/22/2013 1:43 PM, Antoine Pitrou wrote:
>>
>> On Fri, 22 Feb 2013 17:40:29 +0000 (UTC)
>> Wolfgang Maier
>> <wolfgang.maier at biologie.uni-freiburg.de>
>
>
>>> Any plans to make range a valid base class to build upon in future
>>> releases?
>>
>>
>> I suppose it wouldn't very difficult to make range subclassable.
>> You can try writing a patch if you want:
>> http://docs.python.org/devguide/
>
>
> I was just about to suggest that making range subclassable would be an
> alternative to extending the current class. As far as I know, there is no
> principled reason why not, unlike Nonetype and bool.

I wasn't even aware you *couldn't* subclass it (I'd never tried). As
far as I am aware, that's just a quirk inherited from the old xrange
implementation rather than a deliberate design decision.

On the other hand, I suspect for many cases involving more advanced
range variants, containment would be a better option than inheritance,
particular if you want to implement a type that can't be reliably
described through a (start, stop, step) triple.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From ncoghlan at gmail.com  Sat Feb 23 12:58:32 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Feb 2013 21:58:32 +1000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
Message-ID: <CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>

On Sat, Feb 23, 2013 at 8:18 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> It would be much easier, and have much the same effect, if unpacking simply
>> requested the minumum number of items and stopped raising a ValueError if
>> the iterator has more items. No new protocol is needed. Guido rejected this
>> as silently masking errors.
>
> What if it were done explicitly, though?
>
> Currently:
>>>> a,b,c=range(3)
>>>> a,b,c=range(4)
> Traceback (most recent call last):
>   File "<pyshell#55>", line 1, in <module>
>     a,b,c=range(4)
> ValueError: too many values to unpack (expected 3)
>>>> a,b,c,*d=range(6)
>>>> a,b,c,* =range(4)
> SyntaxError: invalid syntax
>
> Suppose the last notation were interpreted as "request values for a,
> b, and c, and then ignore the rest".

It highly unlikely it will ever be interpreted that way, because it
contradicts the way the unnamed star is used in function headers (that
is, to indicate the start of the keyword only arguments *without*
accepting an arbitrary number of positional arguments). If you want to
ignore excess values in unpacking, a double-underscore is the best
currently available option (even though it has the downside of not
working well with infinite iterators or large sequences):

>>> a,b,c,*__ =range(4)

However, Alex's original idea of an "unpack protocol" (distinct from,
but falling back to, the ordinary iterable protocol) definitely has at
least a few points in its favour.

1. Iterating over heterogeneous types doesn't really make sense, but
unpacking them does. A separate protocol lets a type support unpacking
assignment without supporting normal iteration.
2. The protocol could potentially be designed to allow an *iterable*
to be assigned to the star target rather than requiring it to be
unpacked into a tuple. This could be used not only to make unpacking
assignment safe to use with infinite iterators, but also to make it
cheaper with large sequences (through appropriate use of
itertools.islice in a custom container's __unpack__ implementation)
and with virtual sequences like range() objects.
3. As Alex noted, if a function that previously returned a 2-tuple
wants to start returning a 3-tuple, that's currently a backwards
incompatible change because it will break unpacking assignment. With
an unpack protocol, such a function can return an object that behaves
like a 3-tuple most of the time, but also transparently supports
unpacking assignment to two targets rather than only supporting three.

I would suggest a different signature for "__unpack__", though, built
around the fact the star target can be used at most once, but in an
arbitrary location:

    def __unpack__(target_len, star_index=None):
        """Unpack values into an iterable of the specified length.

        If star_index is not None, the value at that index will be
        an iterable representing the remaining contents that were not
        unpacked.
        """
        ...

While I'm somewhere between +0 and +1 on the idea myself, there are
some open questions/potential downsides to the idea:

- this change increases the complexity of the language, by explicitly
separating the concept of heterogeneous unpacking from homogeneous
iteration. Those *are* two distinct concepts though, so the increased
complexity may be justifiable.
- for function parameter unpacking, would we allow "*args" to be an
arbitrary iterable rather than promising that it is always a tuple? We
would probably want to do so for consistency with unpacking assignment
under this scheme, which may lead to some confusing error messages
thrown by existing functions which expect it to always be a tuple.
- we would want to document very clearly that it remains possible to
bypass custom unpacking by explicitly calling iter(obj) rather than
using obj directly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From stephen at xemacs.org  Sat Feb 23 13:37:31 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 23 Feb 2013 21:37:31 +0900
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
	<87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
Message-ID: <87bobbryt0.fsf@uwakimon.sk.tsukuba.ac.jp>

Devin Jeanpierre writes:

 > > Well, we've already gone as far as json, which is pretty powerful (but
 > > still subject to attacks using "relatively secure" json to transport
 > > "insecure" data!)
 > 
 > Of course a serialization library can't protect against
 > eval(deserialize(foo)) running arbitrary code. That doesn't mean we
 > shouldn't bother with security.

Nobody's saying we shouldn't bother with security.  Any answer needs
to be informed by the recognition that nothing we can design is proof
against the Sufficiently Stupid/Lazy User, that's all I'm trying to
say.

But security probably does have a cost in terms of inconvenience and
restriction on capabilities.  My question is "given that people can
and will do stupid things with relatively safe libraries like json,
what is the point of providing something intermediate between json and
pickle?"  In more detail, what features can we provide that don't
involve the known risks of pickle that would be sufficiently
attractive to users that they don't go to pickle anyway?

You mention handling cycles, which adds minimal risk (unprepared code
could infloop on the unpacked data, but that's not the serializer's
fault), but also "new" types which isn't clear to me.  If you mean new
built-in types, can't the json module be extended?  (That would apply
to cycles as well, since we know it's possible it should be
automatable.)  If you mean user-defined types, we're back where we
started, with merely unpacking data running code whose provenance we
don't know.

 > > Why do we need an alternative *between* pickle and json?  Maybe
 > > we should advocate that users think seriously about securing
 > > channels, and validating the pickles before doing anything with
 > > them, if they think they need more features than json offers?
 > 
 > Signed pickles and secured channels and so on don't solve the problem
 > of trying to get data from an an untrusted user.

Yeah yeah, sorry about the red herring.  It's the first question that
matters.




From stephen at xemacs.org  Sat Feb 23 14:11:25 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 23 Feb 2013 22:11:25 +0900
Subject: [Python-ideas] Adding a safe alternative to pickle in
	the	standard library
In-Reply-To: <065d6242ab4940638811c9f3071e68b7@BLUPR03MB035.namprd03.prod.outlook.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com>
	<87k3q0sfqi.fsf@uwakimon.sk.tsukuba.ac.jp>
	<065d6242ab4940638811c9f3071e68b7@BLUPR03MB035.namprd03.prod.outlook.com>
Message-ID: <87a9qvrx8i.fsf@uwakimon.sk.tsukuba.ac.jp>

Steve Dower writes:

 > > There are no bugs that allow a back door, right? 
 > 
 > Of course not. That's why we never see security patches or updates
 > for operating systems or platforms. This is a silly argument.

"Of course not."  The right answer is that "it's been audited and
we're as sure as we ever are."  The problem is that (as Devin
Jeanpierre wrote IIRC) pickle was not designed for security from the
ground up.  Removing execution of untrusted code from it may not be as
easy as just saying so.  Maybe it is.

 > > Is the API sufficiently well-designed that users will easily
 > > figure out how to do what they need, and *only* what they need,
 > > and therefore won't be tempted to simply turn on permission to do
 > > *everything*?
 > 
 > All we can ever do is provide instructions to keep the developer
 > safe and make it clear that ignoring those rules will reduce the
 > security of their program. It's up to the developer to make the
 > right decisions.

But that's not enough.  What others have said here is that json
doesn't get used when it would be perfectly suitable (and as secure as
serialization gets!) because the API doesn't provide convenient access
to the features they need.

 > Agreed. I don't think we need a new protocol though, just a less
 > permissive default implementation of Unpickler.find_class().

The proof of the pudding is in the auditing, I guess.



From jeanpierreda at gmail.com  Sat Feb 23 15:14:24 2013
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sat, 23 Feb 2013 09:14:24 -0500
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <87bobbryt0.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
	<87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
	<87bobbryt0.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CABicbJ+UtiBMA+ZN8gthCjzqj2XqMRwMyypF9WOUbZFeNtq7YQ@mail.gmail.com>

On Sat, Feb 23, 2013 at 7:37 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Devin Jeanpierre writes:
> Nobody's saying we shouldn't bother with security.  Any answer needs
> to be informed by the recognition that nothing we can design is proof
> against the Sufficiently Stupid/Lazy User, that's all I'm trying to
> say.

Sorry. Fair enough.

> But security probably does have a cost in terms of inconvenience and
> restriction on capabilities.  My question is "given that people can
> and will do stupid things with relatively safe libraries like json,
> what is the point of providing something intermediate between json and
> pickle?"  In more detail, what features can we provide that don't
> involve the known risks of pickle that would be sufficiently
> attractive to users that they don't go to pickle anyway?

I believe that the features I'm suggesting meet that criterion (but
see below for discussion of risk).

Nothing will ever be sufficient to drive away all unwarranted use of
pickle, but I feel like these two features are really big ones that
would go a long way towards making the secure thing almost as easy in
almost every circumstance. As long as I've ever personally wanted,
although I can't speak for others.

> You mention handling cycles, which adds minimal risk (unprepared code
> could infloop on the unpacked data, but that's not the serializer's
> fault), but also "new" types which isn't clear to me.  If you mean new
> built-in types, can't the json module be extended?  (That would apply
> to cycles as well, since we know it's possible it should be
> automatable.)

It can. This brings up an interesting point. YAML already extends JSON
with cycle support (via aliases) and support for a notation for
marking up nonstandard types (via tagging). For example:

    >>> yaml.load('&mydict {"a": !!python/tuple ["b", *mydict]}')
    {'a': ('b', {...})}

PyYAML is useless security-wise, but if we're going to extend the json
module, this would probably be the direction to go.

> If you mean user-defined types, we're back where we
> started, with merely unpacking data running code whose provenance we
> don't know.

That actually isn't where we started. We started with a serialization
format that includes such data as
""c__builtin__\neval\n(c__builtin__\nraw_input\n(S'py> '\ntRtR." (try
running pickle.loads on that in Python 2).

What I had in mind from the start was something where only whitelisted
constructors are used to reconstitute python values from the
serialized code. Then we're moved from trusting the input, to trusting
the competence of authors of our objects in modules that we imported.
In cerealizer there is a global registry of classes that profess to
handle input securely. Obviously, they might be wrong, and maybe a
user of a serialization library would want to provide a much smaller
whitelist. Maybe even the bigger whitelist should be disabled by
default, if we really want to be careful, and there should be a
security warning in the docs if you try to use the global registry.

So for example, there's the following things:

    # nominally safe; module authors only register if they believe
    # their deserialization code is safe even with untrusted input
    my_unserializer.loads("...",
whitelist=my_unserializer.PSEUDOSAFE_GLOBAL_REGISTRY)

    # nominally safe; if not, then a security bug in python
    my_unserializer.loads("...", whitelist=set())

-- Devin


From stephen at xemacs.org  Sat Feb 23 16:57:52 2013
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 24 Feb 2013 00:57:52 +0900
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CABicbJ+UtiBMA+ZN8gthCjzqj2XqMRwMyypF9WOUbZFeNtq7YQ@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
	<87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
	<87bobbryt0.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+UtiBMA+ZN8gthCjzqj2XqMRwMyypF9WOUbZFeNtq7YQ@mail.gmail.com>
Message-ID: <87621jrpj3.fsf@uwakimon.sk.tsukuba.ac.jp>

Devin Jeanpierre writes:

 > I believe that the features I'm suggesting meet that criterion (but
 > see below for discussion of risk).

OK, thanks for the discussion.  I need to get back to $DAYJOB (at 1am
on a Saturday :-P ), but I'll chew over that for a bit before
interjecting again. ;-)



From ethan at stoneleaf.us  Sat Feb 23 19:00:54 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 23 Feb 2013 10:00:54 -0800
Subject: [Python-ideas] PEP for enum library type?
In-Reply-To: <CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
References: <CAF-Rda_8j2QnaeFeXwerYV0Jhc5K9TisOiLT6TBAtSTcs2LZGw@mail.gmail.com>
	<20130212163102.50dbf570@pitrou.net>
	<511A76C0.4070005@stoneleaf.us>
	<20130212144518.3be491c5@anarchist.wooz.org>
	<511AC786.80807@pearwood.info>
	<CAP7+vJJ7QZoMeQDdv2BGH0038s38R2O8bws3tbLJ9tg65Nns7g@mail.gmail.com>
Message-ID: <512903D6.1000804@stoneleaf.us>

On 02/12/2013 02:56 PM, Guido van Rossum wrote:
> Frankly, enums are not that useful in small programs. For large
> programs or libraries, and especially for public APIs, the extra cost
> of defining the enum shouldn't count against them.

I have to disagree.  I'm working on converting data from program A to program B
and it involves many small scripts for the different input files.  Each script
ranges from 10 - 50 lines long, and enums -- which I use to access the various
columns from the incoming file -- are incredibly useful.


> Let's just import Barry's enums into the stdlib.

Weren't you at one point unhappy that Barry's enums were not int based?

At any rate, non-int based enums would absolutely *not* work for me.

for line in open('somefile.txt'):
     fields = line.split('\t')
     name = NameCase(fields[int(NAME)])
     business = BsnsCase(fields[int(COMPANY)])
     street = AddrCase(fields[int(ADDRESS)])

etc., etc., and stab me now with my keyboard!  ;)

--
~Ethan~


From masklinn at masklinn.net  Sat Feb 23 19:19:18 2013
From: masklinn at masklinn.net (Masklinn)
Date: Sat, 23 Feb 2013 19:19:18 +0100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
Message-ID: <BBA66F80-15DB-49F5-9380-4EC8043AE58B@masklinn.net>


On 2013-02-23, at 12:58 , Nick Coghlan wrote:

> On Sat, Feb 23, 2013 at 8:18 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>> It would be much easier, and have much the same effect, if unpacking simply
>>> requested the minumum number of items and stopped raising a ValueError if
>>> the iterator has more items. No new protocol is needed. Guido rejected this
>>> as silently masking errors.
>> 
>> What if it were done explicitly, though?
>> 
>> Currently:
>>>>> a,b,c=range(3)
>>>>> a,b,c=range(4)
>> Traceback (most recent call last):
>>  File "<pyshell#55>", line 1, in <module>
>>    a,b,c=range(4)
>> ValueError: too many values to unpack (expected 3)
>>>>> a,b,c,*d=range(6)
>>>>> a,b,c,* =range(4)
>> SyntaxError: invalid syntax
>> 
>> Suppose the last notation were interpreted as "request values for a,
>> b, and c, and then ignore the rest".
> 
> It highly unlikely it will ever be interpreted that way, because it
> contradicts the way the unnamed star is used in function headers (that
> is, to indicate the start of the keyword only arguments *without*
> accepting an arbitrary number of positional arguments).

Then again, *args in (Python 3) unpacking is already completely
incompatible with *args in functions. That does not strike me as much of
an issue.

> However, Alex's original idea of an "unpack protocol" (distinct from,
> but falling back to, the ordinary iterable protocol) definitely has at
> least a few points in its favour.
> 
> 1. Iterating over heterogeneous types doesn't really make sense, but
> unpacking them does. A separate protocol lets a type support unpacking
> assignment without supporting normal iteration.

It would also allow preventing unpacking of iterables for which
unpacking makes no sense and is most likely an error, such as
sets (well there are limited use cases for unpacking a set, but
they are few and far between and wrapping the set in an iter()
call would allow them).

> 2. The protocol could potentially be designed to allow an *iterable*
> to be assigned to the star target rather than requiring it to be
> unpacked into a tuple. This could be used not only to make unpacking
> assignment safe to use with infinite iterators, but also to make it
> cheaper with large sequences (through appropriate use of
> itertools.islice in a custom container's __unpack__ implementation)
> and with virtual sequences like range() objects.

And with immutable sequences through plain sharing.

In fact, though that's not supported by either existing __unpack__
proposal, it could even allow for mapping unpacking (and more generally
named unpacking) by adding something which tends to be missing from
python API: names awareness. Altering your proposal by replacing
`target_len` by a sequence of the names on LHS would allow implementing
__unpack__ for objects or mappings rather than have to use the (quite
unwieldy and backwards-looking) operator.itemgetter and
operator.attrgetter.

> 3. As Alex noted, if a function that previously returned a 2-tuple
> wants to start returning a 3-tuple, that's currently a backwards
> incompatible change because it will break unpacking assignment. With
> an unpack protocol, such a function can return an object that behaves
> like a 3-tuple most of the time, but also transparently supports
> unpacking assignment to two targets rather than only supporting three.
> 
> I would suggest a different signature for "__unpack__", though, built
> around the fact the star target can be used at most once, but in an
> arbitrary location:
> 
>    def __unpack__(target_len, star_index=None):
>        """Unpack values into an iterable of the specified length.
> 
>        If star_index is not None, the value at that index will be
>        an iterable representing the remaining contents that were not
>        unpacked.
>        """
>        ...
> 
> While I'm somewhere between +0 and +1 on the idea myself, there are
> some open questions/potential downsides to the idea:
> 
> - this change increases the complexity of the language, by explicitly
> separating the concept of heterogeneous unpacking from homogeneous
> iteration. Those *are* two distinct concepts though, so the increased
> complexity may be justifiable.

I'm +1 on these grounds, while the current unpacking "works" it makes
for odd limitations or corner-cases.

> - for function parameter unpacking, would we allow "*args" to be an
> arbitrary iterable rather than promising that it is always a tuple? We
> would probably want to do so for consistency with unpacking assignment
> under this scheme, which may lead to some confusing error messages
> thrown by existing functions which expect it to always be a tuple.

I don't think so, they already have a significant semantic split on
the meaning of what follows an *args, and I've got the gut feeling
that *args being a tuple is a much more common assumption/expectation
for functions than for unpacking (especially since *args in unpacking
was only introduced in Python 3)

> - we would want to document very clearly that it remains possible to
> bypass custom unpacking by explicitly calling iter(obj) rather than
> using obj directly.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



From zuo at chopin.edu.pl  Sat Feb 23 19:41:36 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sat, 23 Feb 2013 19:41:36 +0100
Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?=
In-Reply-To: <BBA66F80-15DB-49F5-9380-4EC8043AE58B@masklinn.net>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<BBA66F80-15DB-49F5-9380-4EC8043AE58B@masklinn.net>
Message-ID: <5862b5c0c1859c5b7f0ccb20d028f05d@chopin.edu.pl>

On 23.02.2013 19:19, Masklinn wrote:

> On 2013-02-23, at 12:58 , Nick Coghlan wrote:
[...]
>> - for function parameter unpacking, would we allow "*args" to be an
>> arbitrary iterable rather than promising that it is always a tuple? 
>> We
>> would probably want to do so for consistency with unpacking 
>> assignment
>> under this scheme, which may lead to some confusing error messages
>> thrown by existing functions which expect it to always be a tuple.
>
> I don't think so, they already have a significant semantic split on
> the meaning of what follows an *args, and I've got the gut feeling
> that *args being a tuple is a much more common assumption/expectation
> for functions than for unpacking (especially since *args in unpacking
> was only introduced in Python 3)

Especially that for unpacking it is *a list, not a tuple*.
  and
Cheers.
*j

PS. And would it be really possible to apply the __unpack__-idea to 
calls?
There had not been and still there is no real consistency between 
unpacking
and call argument binding, e.g.

     def fun(a, b, c, *args):
         ...
     fun(1, 2, with_dunder_unpack)

     # ^ OK: a = 1; b = 2; c, *args = with_dunder_unpack
     # but what about:

     def fun(a, b, c, *args):
         ...
     fun(1, 2, 3, 4, 5, with_dunder_unpack)

     # ^ a = 1; b = 2; c = 3; args = ???

I believe these concepts (unpacking and call argument binding) are 
simply not
really parallel in Python.

Yes, `a, b, *seq` in unpacking is *somehow* similar -- on the level of 
a rough
intuition (and that is nice) but IMHO it should not stop us from 
extending
unpacking without worrying much about call binding which was, is and 
must be
(in Python) different in many ways.



From ethan at stoneleaf.us  Sat Feb 23 20:43:29 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 23 Feb 2013 11:43:29 -0800
Subject: [Python-ideas] Yet another enum proposal :)
In-Reply-To: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com>
Message-ID: <51291BE1.6040604@stoneleaf.us>

On 02/22/2013 11:25 AM, Alex Stewart wrote:
> So... thoughts?

Having thought about this some more, I really like your idea of allowing types
such as float to be enums as well.

--
~Ethan~


From ericsnowcurrently at gmail.com  Sat Feb 23 21:10:56 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 23 Feb 2013 13:10:56 -0700
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
Message-ID: <CALFfu7Afd9jFWY4wsH8oGBK8VZPfsw0G3ccyKXT6bxjwNr0JOg@mail.gmail.com>

On Sat, Feb 23, 2013 at 4:58 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> However, Alex's original idea of an "unpack protocol" (distinct from,
> but falling back to, the ordinary iterable protocol) definitely has at
> least a few points in its favour.
>
> 1. Iterating over heterogeneous types doesn't really make sense, but
> unpacking them does. A separate protocol lets a type support unpacking
> assignment without supporting normal iteration.

This is the big one for me.  And as masklinn pointed, it works the
other way too (keep an iterable from unpacking).

> 2. The protocol could potentially be designed to allow an *iterable*
> to be assigned to the star target rather than requiring it to be
> unpacked into a tuple. This could be used not only to make unpacking
> assignment safe to use with infinite iterators, but also to make it
> cheaper with large sequences (through appropriate use of
> itertools.islice in a custom container's __unpack__ implementation)
> and with virtual sequences like range() objects.

Cool, though it would be complicated when the star target isn't the last target.

> 3. As Alex noted, if a function that previously returned a 2-tuple
> wants to start returning a 3-tuple, that's currently a backwards
> incompatible change because it will break unpacking assignment. With
> an unpack protocol, such a function can return an object that behaves
> like a 3-tuple most of the time, but also transparently supports
> unpacking assignment to two targets rather than only supporting three.
>
> I would suggest a different signature for "__unpack__", though, built
> around the fact the star target can be used at most once, but in an
> arbitrary location:
>
>     def __unpack__(target_len, star_index=None):
>         """Unpack values into an iterable of the specified length.
>
>         If star_index is not None, the value at that index will be
>         an iterable representing the remaining contents that were not
>         unpacked.
>         """
>         ...

+1

>
> While I'm somewhere between +0 and +1 on the idea myself, there are
> some open questions/potential downsides to the idea:
>
> - this change increases the complexity of the language, by explicitly
> separating the concept of heterogeneous unpacking from homogeneous
> iteration. Those *are* two distinct concepts though, so the increased
> complexity may be justifiable.

Agreed.  Also, I expect that __unpack__ would be an opt-in API that
falls back to iteration.

> - for function parameter unpacking, would we allow "*args" to be an
> arbitrary iterable rather than promising that it is always a tuple? We
> would probably want to do so for consistency with unpacking assignment
> under this scheme, which may lead to some confusing error messages
> thrown by existing functions which expect it to always be a tuple.

This should be handled/proposed separately, though it is dependent on
an unpacking protocol.

> - we would want to document very clearly that it remains possible to
> bypass custom unpacking by explicitly calling iter(obj) rather than
> using obj directly.

An unpacking protocol would add power to the language.  We really
don't have a good way to separate unpacking from iteration.
Structured record types like namedtuples aren't exactly sequences, and
could make use of the new protocol.  (I'm not advocating for the
change to namedtuple itself.)

-eric


From tjreedy at udel.edu  Sun Feb 24 01:53:30 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 23 Feb 2013 19:53:30 -0500
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
Message-ID: <kgboat$jhs$1@ger.gmane.org>

On 2/23/2013 6:58 AM, Nick Coghlan wrote:
> On Sat, Feb 23, 2013 at 8:18 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>> It would be much easier, and have much the same effect, if unpacking simply
>>> requested the minumum number of items and stopped raising a ValueError if
>>> the iterator has more items. No new protocol is needed. Guido rejected this
>>> as silently masking errors.
>>
>> What if it were done explicitly, though?
>>
>> Currently:
>>>>> a,b,c=range(3)
>>>>> a,b,c=range(4)
>> Traceback (most recent call last):
>>    File "<pyshell#55>", line 1, in <module>
>>      a,b,c=range(4)
>> ValueError: too many values to unpack (expected 3)
>>>>> a,b,c,*d=range(6)
>>>>> a,b,c,* =range(4)
>> SyntaxError: invalid syntax
>>
>> Suppose the last notation were interpreted as "request values for a,
>> b, and c, and then ignore the rest".

I think this is fundamentally the right idea. The 'problem' to be 
solved, such as it is, is that the multiple assignment unpacker, after 
requesting the number of values it needs, requests one more that it does 
not need, does not want, and will not use. If it gets it, it says 'too 
bad', undoes any work it has done, and raises an error. The only point 
in doing this is to uncover possible bugs. But some people say they want 
to knowingly provide extra values and have it not be considered a bug.

The solution, if there is to be one, and if that extra behavior is not 
to be completely undone, is to be able to tell the unpacker to skip the 
extra check. I strongly feel that the right place to do that is on the 
target side. This fixes the problem in one place rather than requiring 
that the solution be duplicated everywhere.

> It highly unlikely it will ever be interpreted that way, because it
> contradicts the way the unnamed star is used in function headers (that
> is, to indicate the start of the keyword only arguments *without*
> accepting an arbitrary number of positional arguments). If you want to
> ignore excess values in unpacking, a double-underscore is the best
> currently available option (even though it has the downside of not
> working well with infinite iterators or large sequences):

 > >>> a,b,c,*__ =range(4)

If ,* is not acceptible, how about ,** or ,... or ,None or <take your 
pick>. I rather like 'a, b, c, ... =' as it clearly implies that we are 
picking and naming the first three values from 3 or more; '...' clearly 
cannot be an assignment target.


> However, Alex's original idea of an "unpack protocol" (distinct from,
> but falling back to, the ordinary iterable protocol) definitely has at
> least a few points in its favour.

I strongly disagree as I think the two major points are wrong.

> 1. Iterating over heterogeneous types doesn't really make sense,  but
> unpacking them does. A separate protocol lets a type support unpacking
> assignment without supporting normal iteration.

This, repeated in this

 > - this change increases the complexity of the language, by explicitly
 > separating the concept of heterogeneous unpacking from homogeneous
 > iteration. Those *are* two distinct concepts though, so the increased
 > complexity may be justifiable.

attempt to reintroduce 'heterogeneous type' as a fundamental concept in 
the language, after being removed with the addition of .count and .index 
as tuple methods. Since I consider the pairing 'heterogeneous type', to 
be wrong, especially in Python, I consider this to be a serious 
regression. Let me repeat some of the previous discussion.

In Python 3, every object is an instance of class count. At that level, 
every collection is homogeneous. At other levels, and for particular 
purposes, any plural collection might be considered heterogeneous. That 
is a function of the objects or values in the collection, and not of the 
collection class itself. So I claim that 'heterogeneous' has not much 
meaning as an absolute attribute of a 'type'.

In any assignment, targets (mostly commonly names) are untyped, or have 
type 'object' -- take your pick. So iterable (of objects) is the 
appropriate source.

In any case, 'iteration' and 'unpacking' both mean 'accessing the items 
of a collection one at a time, as individual items rather than as part 
of a collection'. I do not see any important distinction at all and no 
justification for complexifying the language again.

> 3. As Alex noted, if a function that previously returned a 2-tuple
> wants to start returning a 3-tuple, that's currently a backwards
> incompatible change because it will break unpacking assignment. With
> an unpack protocol, such a function can return an object that behaves
> like a 3-tuple most of the time, but also transparently supports
> unpacking assignment to two targets rather than only supporting three.

This seems to claiming that it is sensible to change the return type of 
a function to a somewhat incompatible return type. (I am obviously 
including fixed tuple length in 'return type', as would be explicit in 
other languages). I believe many/most/all design and interface experts 
would disagree and would say it would be better to give the new function 
a new name.

The statement "that's currently a backwards incompatible change because 
it will break unpacking assignment." is a gross understatement that 
glosses over the fact that returning a 3-tuple instead of a 2-tuple will 
breaks lots of things. Just two examples:

def foo(): return 1,2
def bar(): return tuple('abc')
foobar = foo() + bar()
oof = reversed(foo())

Changing foo to return 1,2,'x' will mess up both foobar and oof for most 
uses.

--
 > 2. The protocol could potentially be designed to allow an *iterable*
 > to be assigned to the star target rather than requiring it to be
 > unpacked into a tuple. This could be used not only to make unpacking
 > assignment safe to use with infinite iterators, but also to make it
 > cheaper with large sequences (through appropriate use of
 > itertools.islice in a custom container's __unpack__ implementation)
 > and with virtual sequences like range() objects.

The problem with *x, that it is inefficient, not applicable to infinite 
iterators, and that assigning the iterable directly to x when *x is in 
final position is more likely what one wants anyway, is a different 
issue from the unwanted bug check and exception. Again, I think the 
solution is an explicit notation on the target side. Perhaps '**x' or 
'x*' or something based on _. If you do not like any of those, suggest 
another.

-- 
Terry Jan Reedy



From rosuav at gmail.com  Sun Feb 24 02:36:59 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 24 Feb 2013 12:36:59 +1100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <kgboat$jhs$1@ger.gmane.org>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
Message-ID: <CAPTjJmr-miY15f9EhudwY5+nhejueDVKzr4eUN=_XL3913pWRQ@mail.gmail.com>

On Sun, Feb 24, 2013 at 11:53 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> If ,* is not acceptible, how about ,** or ,... or ,None or <take your pick>.
> I rather like 'a, b, c, ... =' as it clearly implies that we are picking and
> naming the first three values from 3 or more; '...' clearly cannot be an
> assignment target.

I like that. It avoids the confusion with augmented assignment that ,*
has (yes, you can avoid the *syntactic* confusion by putting spaces
around the equals sign, but it's still visually similar). The only
problem might be in documentation, where it might be taken to mean "a,
b, c, d, e, and as many more variables as you want", eg indicating
that tuple unpacking works with any number of targets.

ChrisA


From ncoghlan at gmail.com  Sun Feb 24 03:12:13 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 12:12:13 +1000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <kgboat$jhs$1@ger.gmane.org>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
Message-ID: <CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>

I definitely like Terry's idea of using ", ..." to say "and ignore the
rest". Simple, elegant and can be restricted to the last element so it
works with infinite iterators and large sequences. It also allows
incremental unpacking of ordinary iterators.

I still like the idea of an unpacking protocol as well, but the above would
cover a lot of use cases without needing a new protocol, and thus should be
considered first (since a new protocol is a higher impact change than
Terry's syntax suggestion).

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130224/5af02024/attachment.html>

From python at mrabarnett.plus.com  Sun Feb 24 04:01:45 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 24 Feb 2013 03:01:45 +0000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <kgboat$jhs$1@ger.gmane.org>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
Message-ID: <51298299.9050109@mrabarnett.plus.com>

On 2013-02-24 00:53, Terry Reedy wrote:
[snip]
> If ,* is not acceptible, how about ,** or ,... or ,None or <take your
> pick>. I rather like 'a, b, c, ... =' as it clearly implies that we are
> picking and naming the first three values from 3 or more; '...' clearly
> cannot be an assignment target.
>
[snip]
,** reminds me too much of dict packing/unpacking. -1

,None looks odd because it looks like you're binding to None. -1

,... looks good to me. +1



From ncoghlan at gmail.com  Sun Feb 24 06:40:49 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 15:40:49 +1000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
Message-ID: <CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>

On Sun, Feb 24, 2013 at 12:12 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I definitely like Terry's idea of using ", ..." to say "and ignore the
> rest". Simple, elegant and can be restricted to the last element so it works
> with infinite iterators and large sequences. It also allows incremental
> unpacking of ordinary iterators.

I want to expand on this a bit now I'm back on a real computer, since
it wasn't immediately obvious to me how well the ", ..." proposal
supports incremental unpacking, but it became clear once I thought of
this simple example:

    iterargs = iter(args)
    command, ... = iterargs # Grab the first element, leave the rest
in the iterator
    commands[command](*iterargs) # Pass the rest to the command

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From guido at python.org  Sun Feb 24 07:24:50 2013
From: guido at python.org (Guido van Rossum)
Date: Sat, 23 Feb 2013 22:24:50 -0800
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
Message-ID: <CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>

But is that really so much better than

  command = next(iterargs)
  # etc.

?

And to me the ... Looks too much like something that consumes the rest,
rather than leaving it.

--Guido (not on a real keyboard)

On Saturday, February 23, 2013, Nick Coghlan wrote:

> On Sun, Feb 24, 2013 at 12:12 PM, Nick Coghlan <ncoghlan at gmail.com<javascript:;>>
> wrote:
> > I definitely like Terry's idea of using ", ..." to say "and ignore the
> > rest". Simple, elegant and can be restricted to the last element so it
> works
> > with infinite iterators and large sequences. It also allows incremental
> > unpacking of ordinary iterators.
>
> I want to expand on this a bit now I'm back on a real computer, since
> it wasn't immediately obvious to me how well the ", ..." proposal
> supports incremental unpacking, but it became clear once I thought of
> this simple example:
>
>     iterargs = iter(args)
>     command, ... = iterargs # Grab the first element, leave the rest
> in the iterator
>     commands[command](*iterargs) # Pass the rest to the command
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com <javascript:;>   |   Brisbane,
> Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <javascript:;>
> http://mail.python.org/mailman/listinfo/python-ideas
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130223/697dbc08/attachment.html>

From ncoghlan at gmail.com  Sun Feb 24 08:38:39 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Feb 2013 17:38:39 +1000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
Message-ID: <CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>

On Sun, Feb 24, 2013 at 4:24 PM, Guido van Rossum <guido at python.org> wrote:
> But is that really so much better than
>
>   command = next(iterargs)
>   # etc.
>
> ?

For a single item, using next() instead is OK, the problem is that
itertools islice is relatively ugly and unintuitive, and requires you
to manually count the number of targets. The next + islice combination
also creates a discontinuity between the way you handle getting just
the first item, versus getting multiple items.

Status quo, 1 item:

    iterargs = iter(args)
    command = next(iterargs) # Grab the first, leave the rest
    commands[command](*iterargs) # Pass the rest to the command


Status quo, 2 items (etc):

    from itertools import islice
    iterargs = iter(args)
    command, subcommand = islice(iterargs, 2) # Grab the first two,
leave the rest
    commands[command][subcommand](*iterargs) # Pass the rest to the subcommand

Proposal, 1 item:

    iterargs = iter(args)
    command, ... = iterargs # Grab the first, leave the rest
    commands[command](*iterargs) # Pass the rest to the command

Proposal, 2 items (etc):

    iterargs = iter(args)
    command, subcommand, ... = iterargs # Grab the first two, leave the rest
    commands[command][subcommand](*iterargs) # Pass the rest to the subcommand

> And to me the ... Looks too much like something that consumes the rest,
> rather than leaving it.

As far as the possible interpretation being "consume and discard the
rest" goes, I think it's one of those cases where once you know which
it is, it won't be hard to remember.

It's certainly something we could continue to live without, but I do
think it's a nice way to expand the unpacking syntax to cover
additional use cases far more elegantly than the current reliance on a
combination of next and itertools.islice.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From bruce at leapyear.org  Sun Feb 24 09:04:57 2013
From: bruce at leapyear.org (Bruce Leban)
Date: Sun, 24 Feb 2013 00:04:57 -0800
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
Message-ID: <CAGu0AnvfSXv6oc9_jzRZuO2S3f4voDhdCMjW=zTg=KZn3Rr0Eg@mail.gmail.com>

On Sat, Feb 23, 2013 at 10:24 PM, Guido van Rossum <guido at python.org> wrote:

> But is that really so much better than
>
>   command = next(iterargs)
>   # etc.
>
> ?
>
> And to me the ... Looks too much like something that consumes the rest,
> rather than leaving it.
>
> --Guido (not on a real keyboard)
>

And hypothetically

a, b, c = next(iterargs, count=3)

or

a, b, c = (next * 3)(iterargs)

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130224/358835a1/attachment.html>

From steve at pearwood.info  Sun Feb 24 10:08:57 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Feb 2013 20:08:57 +1100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAGu0AnvfSXv6oc9_jzRZuO2S3f4voDhdCMjW=zTg=KZn3Rr0Eg@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAGu0AnvfSXv6oc9_jzRZuO2S3f4voDhdCMjW=zTg=KZn3Rr0Eg@mail.gmail.com>
Message-ID: <5129D8A9.6000000@pearwood.info>

On 24/02/13 19:04, Bruce Leban wrote:

> And hypothetically
[...]
> a, b, c = (next * 3)(iterargs)

That looks like, well, I'm not sure what it looks like. Haskell maybe? But not Python.


-- 
Steven


From larry at hastings.org  Sun Feb 24 12:25:56 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 24 Feb 2013 03:25:56 -0800
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>
Message-ID: <5129F8C4.4080307@hastings.org>


On 02/23/2013 11:38 PM, Nick Coghlan wrote:
> Status quo, 2 items (etc):
>
>      from itertools import islice
>      iterargs = iter(args)
>      command, subcommand = islice(iterargs, 2) # Grab the first two,
> leave the rest
>      commands[command][subcommand](*iterargs) # Pass the rest to the subcommand

Or

     command, subcommand = next(iterargs), next(iterargs)

or

     command = next(iterargs)
     subcommand = next(iterargs)

These don't require the manual counting, nor do they feature the 
"discontinuity" you mentioned.  FWIW I don't think this problem is bad 
enough, nor this idiom used frequently enough, to warrant new syntax.  
But I've been wrong before.


I was musing on this topic and came up with what is probably a terrible 
alternate approach.  What if there was some way for the RHS to know what 
the LHS was asking for in a destructuring assignment? Something like

     def __next_n__(self, n):
         "Returns a tuple of the next n items from this iterator."

and if the object doesn't provide __next_n__ you fall back to the 
current explicit PyIter_Next() calls and failing if there aren't enough 
/ there are too many.  If we had this, we could make the second argument 
to islice() optional; it could infer how many items you wanted.  (I 
don't think the "a, b, *c, d = rhs" form is relevant here as you'd 
always consume everything from rhs.)

I'm not sure how bad an idea this is.  But complicating the iterator 
protocol for this marginal problem seems like a pretty bad idea.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130224/49e12b1a/attachment.html>

From rosuav at gmail.com  Sun Feb 24 13:59:20 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 24 Feb 2013 23:59:20 +1100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <5129F8C4.4080307@hastings.org>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>
	<5129F8C4.4080307@hastings.org>
Message-ID: <CAPTjJmq+imANaad7roL_Vfdn02qLVmYLgLkMfcPJbaNVfXtOKA@mail.gmail.com>

On Sun, Feb 24, 2013 at 10:25 PM, Larry Hastings <larry at hastings.org> wrote:
> Or
>
>     command, subcommand = next(iterargs), next(iterargs)

Err.... is there a language guarantee of the order of evaluation in a
tuple, or is this just a "CPython happens to evaluate independent
expressions left-to-right"? This is totally broken if the next() calls
could be done in either order.

ChrisA


From steve at pearwood.info  Sun Feb 24 15:16:03 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 25 Feb 2013 01:16:03 +1100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAPTjJmq+imANaad7roL_Vfdn02qLVmYLgLkMfcPJbaNVfXtOKA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>
	<5129F8C4.4080307@hastings.org>
	<CAPTjJmq+imANaad7roL_Vfdn02qLVmYLgLkMfcPJbaNVfXtOKA@mail.gmail.com>
Message-ID: <512A20A3.1020006@pearwood.info>

On 24/02/13 23:59, Chris Angelico wrote:
> On Sun, Feb 24, 2013 at 10:25 PM, Larry Hastings <larry at hastings.org> wrote:
>> Or
>>
>>      command, subcommand = next(iterargs), next(iterargs)
>
> Err.... is there a language guarantee of the order of evaluation in a
> tuple, or is this just a "CPython happens to evaluate independent
> expressions left-to-right"? This is totally broken if the next() calls
> could be done in either order.

It's a language guarantee.

http://docs.python.org/2/reference/expressions.html#evaluation-order


-- 
Steven


From rosuav at gmail.com  Sun Feb 24 15:52:48 2013
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 25 Feb 2013 01:52:48 +1100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <512A20A3.1020006@pearwood.info>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>
	<5129F8C4.4080307@hastings.org>
	<CAPTjJmq+imANaad7roL_Vfdn02qLVmYLgLkMfcPJbaNVfXtOKA@mail.gmail.com>
	<512A20A3.1020006@pearwood.info>
Message-ID: <CAPTjJmockPfzvy78N=MOfc=_+TMAdWSVc+O-RqKxYhYcwoNm+g@mail.gmail.com>

On Mon, Feb 25, 2013 at 1:16 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On 24/02/13 23:59, Chris Angelico wrote:
>>
>> On Sun, Feb 24, 2013 at 10:25 PM, Larry Hastings <larry at hastings.org>
>> wrote:
>>>
>>> Or
>>>
>>>      command, subcommand = next(iterargs), next(iterargs)
>>
>>
>> Err.... is there a language guarantee of the order of evaluation in a
>> tuple, or is this just a "CPython happens to evaluate independent
>> expressions left-to-right"? This is totally broken if the next() calls
>> could be done in either order.
>
> It's a language guarantee.
>
> http://docs.python.org/2/reference/expressions.html#evaluation-order

Ah, so it is. My bad, sorry! In that case, sure, this works. It still
violates DRY though, naming the iterable twice and relying on the
reader noticing that that means "take two off this one". But that's a
much weaker concern.

ChrisA


From vito.detullio at gmail.com  Sun Feb 24 16:45:16 2013
From: vito.detullio at gmail.com (Vito De Tullio)
Date: Sun, 24 Feb 2013 16:45:16 +0100
Subject: [Python-ideas] iterable.__unpack__ method
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>
	<5129F8C4.4080307@hastings.org>
	<CAPTjJmq+imANaad7roL_Vfdn02qLVmYLgLkMfcPJbaNVfXtOKA@mail.gmail.com>
	<512A20A3.1020006@pearwood.info>
	<CAPTjJmockPfzvy78N=MOfc=_+TMAdWSVc+O-RqKxYhYcwoNm+g@mail.gmail.com>
Message-ID: <kgdcie$pcn$1@ger.gmane.org>

Chris Angelico wrote:

>>>>      command, subcommand = next(iterargs), next(iterargs)

> Ah, so it is. My bad, sorry! In that case, sure, this works. It still
> violates DRY though, naming the iterable twice and relying on the
> reader noticing that that means "take two off this one".

well, about the DRY it's the same, but I found the "old stupid"

    command    = next(iterargs)
    subcommand = next(iterargs)

mostly more explicit in the "take two off this one"

maybe it's... too much explicit (and it's not a single expression!!! 
anathema!) but I found slightly more readable

-- 
ZeD



From guido at python.org  Sun Feb 24 17:07:19 2013
From: guido at python.org (Guido van Rossum)
Date: Sun, 24 Feb 2013 08:07:19 -0800
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAPTjJmockPfzvy78N=MOfc=_+TMAdWSVc+O-RqKxYhYcwoNm+g@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>
	<5129F8C4.4080307@hastings.org>
	<CAPTjJmq+imANaad7roL_Vfdn02qLVmYLgLkMfcPJbaNVfXtOKA@mail.gmail.com>
	<512A20A3.1020006@pearwood.info>
	<CAPTjJmockPfzvy78N=MOfc=_+TMAdWSVc+O-RqKxYhYcwoNm+g@mail.gmail.com>
Message-ID: <CAP7+vJJd37TMqN6NE5ys=SKRnZK+bt8bNHNsrJhiZ8xXA1-8Sg@mail.gmail.com>

DRY is not the only principle to consider here.

On Sunday, February 24, 2013, Chris Angelico wrote:

> On Mon, Feb 25, 2013 at 1:16 AM, Steven D'Aprano <steve at pearwood.info<javascript:;>>
> wrote:
> > On 24/02/13 23:59, Chris Angelico wrote:
> >>
> >> On Sun, Feb 24, 2013 at 10:25 PM, Larry Hastings <larry at hastings.org<javascript:;>
> >
> >> wrote:
> >>>
> >>> Or
> >>>
> >>>      command, subcommand = next(iterargs), next(iterargs)
> >>
> >>
> >> Err.... is there a language guarantee of the order of evaluation in a
> >> tuple, or is this just a "CPython happens to evaluate independent
> >> expressions left-to-right"? This is totally broken if the next() calls
> >> could be done in either order.
> >
> > It's a language guarantee.
> >
> > http://docs.python.org/2/reference/expressions.html#evaluation-order
>
> Ah, so it is. My bad, sorry! In that case, sure, this works. It still
> violates DRY though, naming the iterable twice and relying on the
> reader noticing that that means "take two off this one". But that's a
> much weaker concern.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <javascript:;>
> http://mail.python.org/mailman/listinfo/python-ideas
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130224/0fc5813f/attachment.html>

From oscar.j.benjamin at gmail.com  Sun Feb 24 17:42:20 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sun, 24 Feb 2013 16:42:20 +0000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
Message-ID: <CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>

On 24 February 2013 06:24, Guido van Rossum <guido at python.org> wrote:
> On Saturday, February 23, 2013, Nick Coghlan wrote:
>> On Sun, Feb 24, 2013 at 12:12 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> > I definitely like Terry's idea of using ", ..." to say "and ignore the
>> > rest". Simple, elegant and can be restricted to the last element so it
>> > works
>> > with infinite iterators and large sequences. It also allows incremental
>> > unpacking of ordinary iterators.
>>
>> I want to expand on this a bit now I'm back on a real computer, since
>> it wasn't immediately obvious to me how well the ", ..." proposal
>> supports incremental unpacking, but it became clear once I thought of
>> this simple example:
>>
>>     iterargs = iter(args)
>>     command, ... = iterargs # Grab the first element, leave the rest
>> in the iterator
>>     commands[command](*iterargs) # Pass the rest to the command
>
> But is that really so much better than
>
>   command = next(iterargs)
>   # etc.

Calls to next() need to be wrapped in try/except StopIteration (it's a
bad idea to leak these exceptions). So then it becomes:

  try:
       command = next(iterator)
  except StopIteration:
      raise ValueError('need more than 0 values to unpack')

Presumably with this proposal

  command, ... = iterator

would raise a ValueError if the iterator is empty.


Oscar


From zuo at chopin.edu.pl  Sun Feb 24 21:14:28 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sun, 24 Feb 2013 21:14:28 +0100
Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?=
In-Reply-To: <CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>
Message-ID: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl>

24.02.2013 17:42, Oscar Benjamin wrote:

> Calls to next() need to be wrapped in try/except StopIteration (it's 
> a
> bad idea to leak these exceptions). So then it becomes:
>
>   try:
>        command = next(iterator)
>   except StopIteration:
>       raise ValueError('need more than 0 values to unpack')

Not necessarily. You can use:

     command = next(iterator, None)

Cheers.
*j


From python at mrabarnett.plus.com  Sun Feb 24 21:34:47 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 24 Feb 2013 20:34:47 +0000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>
	<44528db0ff81ea147dfc71d97e472798@chopin.edu.pl>
Message-ID: <512A7967.1060709@mrabarnett.plus.com>

On 2013-02-24 20:14, Jan Kaliszewski wrote:
> 24.02.2013 17:42, Oscar Benjamin wrote:
>
>> Calls to next() need to be wrapped in try/except StopIteration (it's
>> a
>> bad idea to leak these exceptions). So then it becomes:
>>
>>   try:
>>        command = next(iterator)
>>   except StopIteration:
>>       raise ValueError('need more than 0 values to unpack')
>
> Not necessarily. You can use:
>
>       command = next(iterator, None)
>
That's not the same thing.

Normally when unpacking, it raises ValueError if there are too few
items. That's what we want to preserve.



From foogod at gmail.com  Sun Feb 24 23:07:24 2013
From: foogod at gmail.com (Alex Stewart)
Date: Sun, 24 Feb 2013 14:07:24 -0800 (PST)
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <512A7967.1060709@mrabarnett.plus.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>
	<44528db0ff81ea147dfc71d97e472798@chopin.edu.pl>
	<512A7967.1060709@mrabarnett.plus.com>
Message-ID: <d31a51ec-9ca5-4653-ae4f-0328d096a79a@googlegroups.com>

It seems to me this thread has actually split into discussing several 
different, fairly independent issues.

==== Unpacking protocol

First, there is the question I originally raised about separating the 
"unpacking protocol" from the "iterator protocol".  I would like to clarify 
a couple of points on this:

I proposed __unpack__ the way I did specifically for a few reasons:

   1. It is a fairly small change to the language (it does not require any 
   new syntax, does not change the behavior of any current code, and can be 
   completely ignored by programmers who don't care about the issues it 
   addresses)
   2. There are (in my opinion) already pretty good parallels with this 
   sort of change that have already been implemented in Python (for example, 
   separating the concept of "iterable" from the concept of "indexable"). 
    This seemed to be a fairly natural and easy-to-explain extension of the 
   same sort of philosophies that Python has already largely adopted in other 
   contexts.
   3. It is consistent with the general idea in Python that an object can 
   choose to emulate whichever behaviors of other objects make sense, and not 
   emulate other ones that do not make sense.  In my opinion, "iterating", and 
   "unpacking" are two related-but-not-the-same language concepts, which 
   currently are unnecessarily entangled.
   4. It would help significantly with several types of problems that do 
   not currently have great solutions.  People have proposed various 
   alternatives that would help with one or two of these, but none of the 
   proposed alternatives deal with all of them (or really even most of them).

In summary, some of the other proposals have other useful features, and 
might be worth considering, but many of them seem far more of a stretch 
than my original proposal, and I think there are other good reasons why the 
original proposal is still worth looking at on its own merits.

Also, regarding the method signature "(min_count, max_count)" vs 
"(target_len, star_index)".  I specifically discarded the latter in favor 
of the former when I was putting together the proposal because (a) the 
"star_index" way does not really gain us anything useful over the "min/max" 
way, (b) "min/max" seemed a bit more future-proof (if, in the future, for 
example, somebody came up with an extended-extended unpack syntax which 
allowed for (for example) optional parameters, or the ability to specify an 
upper-bound on the *extra stuff, it would not require any change to the 
__unpack__ protocol to support it), (c) "min/max" is less about the 
particular implementation and more about conceptually what it means to 
unpack something, which seemed more appropriate when defining the concept 
of an "unpack protocol", and (d) this also means that it could conceivably 
be called in other cases (or directly by Python code that knew what it was 
doing) to produce more sophisticated results for some objects.

Ultimately, I could live with "target_len, star_index", but I still think 
"min_count, max_count" is a better way to do things, and I'd be curious 
whether anybody can come up with an argument why "target_len, star_index" 
would be actually more useful or preferable in any way..

==== Partial unpacking

Second, there is the question of allowing explicit (or implicit) "partial 
unpacking" (i.e. allowing the programmer to say "I only want to unpack the 
first n items, and leave any others in the iterator)

I agree with the general sentiment that doing this implicitly (i.e. just 
not checking for "too many values to unpack") is silently masking errors, 
which is a bad idea, and anyway it would potentially cause problems with 
all kinds of existing code, and we just shouldn't go down that road.

I do think it might be nice to have a way to specify in the syntax to just 
leave any extra data un-iterated, but to be honest I haven't really felt 
any particular warm-fuzzies about any of the proposed syntaxes thus far. 
 (The one I like best is Terry's "a, b, c, ... = foo", though.)

However, this is really fundamentally a completely different question, 
which addresses a different use-case, than the unpacking protocol question, 
and should not be confused as addressing the same issues (because it 
doesn't).

Specifically, lots of folks seem to be ignoring the fact that any 
"unpacking protocol" (whether it's the implicit one we have now, or a more 
explicit alternative like __unpack__) has two sides to it (the provider and 
the consumer).  These two sides are not always written by the same person 
or expecting the same things (for example, when unpacking an object which 
was passed into a routine or returned by some other code).  There are times 
when the person writing the assignment statement does know "I only want 
these items and not the rest", but there are also times when the object 
itself potentially knows better "only these items make sense in this 
context".  I would emphasize that there are currently ways (even if they're 
less than ideal) for the former to be done (i.e. calling next() or using 
itertools), but there is currently no way to do the latter at all (because 
the object is currently not given any way to get any information about the 
context it's being used in).  We can make the former case more convenient 
if we want to, but it really doesn't fix the other side of the problem at 
all.

==== Iterator end-testing

The third issue that has been brought up is the slightly-related issue of 
how unpacking currently attempts to consume an extra item from an iterator, 
just to check for length.  This is, in my opinion, a valid flaw which is 
worth discussing, but it is actually not a flaw with unpacking, or really 
related in any way to unpacking except by happenstance.

The real issue here is not in how unpacking is done but a limitation in the 
iteration protocol itself.  The problem is that there is no standard way to 
query an iterator to find out if it has more data available without 
automatically consuming that next data element in the process.  In my 
opinion, if we want to deal with this issue properly, we should not be 
trying to hack around it in the single case of unpacking but instead fix 
the larger core problem by adding some way to check for the end of an 
iterator without side-effects (and then of course change unpacking to use 
that instead).

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130224/cce13a6a/attachment.html>

From tjreedy at udel.edu  Mon Feb 25 01:57:51 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 24 Feb 2013 19:57:51 -0500
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
Message-ID: <kgecue$9vc$1@ger.gmane.org>

On 2/24/2013 1:24 AM, Guido van Rossum wrote:
> But is that really so much better than
>
>    command = next(iterargs)
>    # etc.
>
> ?
>
> And to me the ... Looks too much like something that consumes the rest,
> rather than leaving it.

I considered $ for $top, but I was sure you would not want to waste $ in 
this. In my original suggestion, I also suggested None as in
a, b, c, None = source
as another possibly to signal

"I want source iteration to stop even if there are more items, and I 
don't want an exception raised even if there are, because I think I know 
what I am doing and I do not want it to be considered a bug."

Any other short spelling/signal would be fine, too.

-- 
Terry Jan Reedy



From random832 at fastmail.us  Mon Feb 25 04:09:28 2013
From: random832 at fastmail.us (Random832)
Date: Sun, 24 Feb 2013 22:09:28 -0500
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
Message-ID: <512AD5E8.4030204@fastmail.us>

On 02/22/2013 02:33 PM, Alex Stewart wrote:
>
> In the case of simple unpacking ("a, b, c = foo"), min_count and 
> max_count will be the same value (3).  In the case of extended 
> unpacking ("a, b, c, *d = foo"), min_count would be the smallest 
> length that will not cause an unpacking exception (3, in this case), 
> and max_count would be None.
>
>  1. It would allow for "smart" unpackable objects (such as __ in my
>     enum example), which would "just work" no matter how many values
>     they're required to produce.
>
If you want "smart" unpackable objects, why not allow d in your *d 
example to be something other than a list?

Like, if you have a,*b = range(1,5); b could literally be range(2,5).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130224/d8144b93/attachment.html>

From tjreedy at udel.edu  Mon Feb 25 04:41:48 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 24 Feb 2013 22:41:48 -0500
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
Message-ID: <kgemhs$ldh$1@ger.gmane.org>

An iterator iter represents the remainder of some collection, concrete 
or not, finite or not. If the remainder is not empty, its .__next__ 
method selects one object, mutates iter to represent the reduced 
remainder (now possibly empty), and returns the one item.

At various times, people have asked for the ability to determine whether 
an iterator is exhausted, whether a next() call will return or raise. If 
it will return, people have also asked for the ability to peek at what 
the return would be, without disturbing what the return will be. For 
instance, on the 'iterable.__unpack__ method' Alex Stewart today wrote:
 > The problem is that there is no standard way to query an iterator
 > to find out if it has more data available without automatically
 > consuming that next data element in the rocess.

It turns out that there is a solution that gives the ability to both 
test emptiness (in the standard way) and peek ahead, without modifying 
the iterator protocol. It merely require a wrapper iterator, much like 
the ones in itertools. I have posted one before and give my current 
version below.

Does anyone else this should be added to itertools? It seems to not be 
completely obvious to everyone, is more complex that some of the 
existing itertools, and cannot be composed from them either. (Nor can it 
be written as a generator function.)

Any of the names can be changed. Perhaps the class should be 'peek' and 
the lookahead object something else. The sentinel should be read-only if 
possible. I considered whether the peek object should be read-only, but 
someone would say that they *want* be able to replace the next object to 
be yielded. Peeking into an exhausted iterable could raise instead of 
returning the sentinel, but I don't know if that would be more useful.

----------------
class lookahead():
     "Wrap iterator with lookahead to both peek and test exhausted"

     _NONE = object()
     def __init__(self, iterable):
         self._it = iter(iterable)
         self._set_peek()
     def __iter__(self):
         return self
     def __next__(self):
         ret = self.peek
         self._set_peek()
         return ret
     def _set_peek(self):
         try:
             self.peek = next(self._it)
         except StopIteration:
             self.peek = self._NONE
     def __bool__(self):
         return self.peek is not self._NONE

def test_lookahead():
     it = lookahead('abc')
     while it:
         a = it.peek
         b = next(it)
         print('next:', b, '; is peek:', a is b )

test_lookahead()
--------------------
 >>>
next: a ; is peek: True
next: b ; is peek: True
next: c ; is peek: True

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Mon Feb 25 04:43:58 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 24 Feb 2013 22:43:58 -0500
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <d31a51ec-9ca5-4653-ae4f-0328d096a79a@googlegroups.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>
	<44528db0ff81ea147dfc71d97e472798@chopin.edu.pl>
	<512A7967.1060709@mrabarnett.plus.com>
	<d31a51ec-9ca5-4653-ae4f-0328d096a79a@googlegroups.com>
Message-ID: <kgemlt$ldh$2@ger.gmane.org>

On 2/24/2013 5:07 PM, Alex Stewart wrote:

> ==== Unpacking protocol
>
> First, there is the question I originally raised about separating the
> "unpacking protocol" from the "iterator protocol".  I would like to
> clarify a couple of points on this:
>
> I proposed __unpack__ the way I did specifically for a few reasons:
>
>  1. It is a fairly small change to the language

To me, taking something simple and elegant and complexifying it, to 
little benefit I can see other than to solve a problem in the wrong 
place, in no small matter. Certainly, adding a new special method to 
some category of objects has not effect unless changes are made to the 
interpreter elsewhere to make automatic use of that method. Otherwise, 
you could just define a normal method to be used by user code.

>  2. There are (in my opinion) already pretty good parallels with this
>     sort of change that have already been implemented in Python (for
>     example, separating the concept of "iterable" from the concept of
>     "indexable").

The related but distinct concepts of sequential access and random access 
are basic to computation and were *always* distinct in Python. For loop 
statements were always the way to sequentially access objects in a 
collection (by default, all of them). It happens that the original 
*implementation* of sequential access, use by for loops, was by means of 
pseudo random access. This worked ok for randomly accessible sequences 
but was clumsy for anything else. (Sequential access requires memory, 
random access does not.) The addition of __iter__ along with __next__ 
added more benefits than __next__ alone would have.

>  3. It is consistent with the general idea in Python that an object can
>     choose to emulate whichever behaviors of other objects make sense,
>     and not emulate other ones that do not make sense.  In my opinion,
>     "iterating", and "unpacking" are two related-but-not-the-same
>     language concepts, which currently are unnecessarily entangled.

I do not see that at all. As near as I can tell, both mean 'sequential 
access'. You are that one that seems to me to be entangling 'bind 
objects to targets one at a time' with 'provide objects one at a time'. 
The point of both old and new iterator protocols was and is to decouple 
(disentangle) consumers from providers. At one time, target-list binding 
*was* entangled -- with tuples -- because it did not use the iterator 
protocol that for statement did.

I admit I do not understand your __unpack__ proposal, since it seemed 
vague and incomplete to me. But until I see your conceptual distinction, 
the details do not matter to me.

>  4. It would help significantly with several types of problems that do
>     not currently have great solutions.

I obviously do not see the same pile of problems that would justify a 
new special method.

...
> no way to query an iterator to find out if it has more data available

See how in my separate post:
"Add lookahead iterator (peeker) to itertools"

-- 
Terry Jan Reedy



From larry at hastings.org  Mon Feb 25 05:06:39 2013
From: larry at hastings.org (Larry Hastings)
Date: Sun, 24 Feb 2013 20:06:39 -0800
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <kgecue$9vc$1@ger.gmane.org>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<kgecue$9vc$1@ger.gmane.org>
Message-ID: <512AE34F.9020902@hastings.org>

On 02/24/2013 04:57 PM, Terry Reedy wrote:
> On 2/24/2013 1:24 AM, Guido van Rossum wrote:
>> But is that really so much better than
>>
>>    command = next(iterargs)
>>    # etc.
>>
>> ?
>>
>> And to me the ... Looks too much like something that consumes the rest,
>> rather than leaving it.
>
> I considered $ for $top, but I was sure you would not want to waste $ 
> in this.

If we are seriously considering this language addition, may I 
counter-propose the syntax

     a, b *= iterable

because star already has something vaguely to do with unpacking, 
something something. *handwave*


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130224/47c139d6/attachment.html>

From wolfgang.maier at biologie.uni-freiburg.de  Mon Feb 25 10:51:17 2013
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 25 Feb 2013 09:51:17 +0000 (UTC)
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
References: <kgemhs$ldh$1@ger.gmane.org>
Message-ID: <loom.20130225T103042-482@post.gmane.org>

Terry Reedy <tjreedy at ...> writes:

> class lookahead():
>      "Wrap iterator with lookahead to both peek and test exhausted"
...

+1 That's a nice tool that I'd love to have in itertools.

Cheers,
Wolfgang





From solipsis at pitrou.net  Mon Feb 25 11:07:17 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 11:07:17 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
References: <kgemhs$ldh$1@ger.gmane.org>
Message-ID: <20130225110717.232ce082@pitrou.net>

Le Sun, 24 Feb 2013 22:41:48 -0500,
Terry Reedy <tjreedy at udel.edu> a ?crit :
> 
> def test_lookahead():
>      it = lookahead('abc')
>      while it:
>          a = it.peek
>          b = next(it)
>          print('next:', b, '; is peek:', a is b )

def test_lookahead():
    it = iter('abc')
    while True:
        it, peeking = itertools.tee(it)
        try:
            a = next(peeking)
        except StopIteration:
            break
        b = next(it)
        print('next:', b, '; is peek:', a is b )


Regards

Antoine.




From p.f.moore at gmail.com  Mon Feb 25 11:08:42 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 25 Feb 2013 10:08:42 +0000
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <loom.20130225T103042-482@post.gmane.org>
References: <kgemhs$ldh$1@ger.gmane.org>
	<loom.20130225T103042-482@post.gmane.org>
Message-ID: <CACac1F_k450BCU8kWc88xAh-fJkcA8GRGSNFS4BJ96b7e30TFw@mail.gmail.com>

On 25 February 2013 09:51, Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> wrote:
> Terry Reedy <tjreedy at ...> writes:
>
>> class lookahead():
>>      "Wrap iterator with lookahead to both peek and test exhausted"
> ...
>
> +1 That's a nice tool that I'd love to have in itertools.

It's not a bad idea, but I don't like the fact that as written it
turns a finite iterator into an infinite one (returning an endless
sequence of sentinels after  the underlying iterator is exhausted).
That seems to me to be very prone to errors if you don't keep it very
clear whether you're dealing with iterators or lookahead-wrapped
iterators:

>>> from lookahead import lookahead
>>> it = lookahead(range(3))
>>> list(it)
<hangs>

The problem is that once you wrap an iterator with lookahead() you
can't use the underlying iterator directly any more without losing
data, as you've consumed a value from it. So you have to use the
wrapped version, and the differing behaviour means you need to write
your code differently.

I'd prefer it if lookahead(it) behaved exactly like it, except that it
had a new peek() method for getting the lookahead value, and maybe a
finished() method to tell if next() will raise StopIteration or not.
Bikeshed away over what should happen if peek() is called when
finished() is true :-)

(Disclaimer: I have no real-world use cases for this feature, so the
comments above are largely theoretical objections...)
Paul.


From wolfgang.maier at biologie.uni-freiburg.de  Mon Feb 25 11:32:34 2013
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 25 Feb 2013 10:32:34 +0000 (UTC)
Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?=
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CADiSq7f=s49N9p+2haawW8Q5585+mKvXYv1NynYGgtKToOL41w@mail.gmail.com>
Message-ID: <loom.20130225T112130-387@post.gmane.org>

Nick Coghlan <ncoghlan at ...> writes:

> Status quo, 2 items (etc):
> 
>     from itertools import islice
>     iterargs = iter(args)
>     command, subcommand = islice(iterargs, 2) # Grab the first two,
> leave the rest
>     commands[command][subcommand](*iterargs) # Pass the rest to the subcommand

When it comes to getting multiple items from an iterator, I prefer wrapping
things in my own generator function:

def x_iter(iterator,n):
    """Return n items from iterator."""
    
    i=[iterator]*n
    while True:
        try:
            result=[next(i[0])]
        except StopIteration:
            # iterator exhausted immediately, end the generator
            break
        for e in i[1:]:
            try:
                result.append(next(e))
            except StopIteration:
                # iterator exhausted after returning at least one item, but
before returning n
                raise ValueError("only %d value(s) left in iterator, expected
%d" % (len(result),n))
        yield result

Compared to islice, this has the advantage of working properly in for loops:

>>> it=iter(range(1,11))
>>> for c,d in x_iter(it,2):
	print(c,d)

	
1 2
3 4
5 6
7 8
9 10

Maybe one could improve itertools.islice accordingly??

> Proposal, 2 items (etc):
> 
>     iterargs = iter(args)
>     command, subcommand, ... = iterargs # Grab the first two, leave the rest
>     commands[command][subcommand](*iterargs) # Pass the rest to the subcommand
> 

+1  for this. I think it's very readable. I think it should raise differently
though depending on whether iterargs is exhausted right away (StopIteration) or
during unpacking (ValueError).

Best,
Wolfgang 






From mark.hackett at metoffice.gov.uk  Mon Feb 25 12:29:02 2013
From: mark.hackett at metoffice.gov.uk (Mark Hackett)
Date: Mon, 25 Feb 2013 11:29:02 +0000
Subject: [Python-ideas] Adding a safe alternative to pickle in the
	standard library
In-Reply-To: <CABicbJKr8g8_yN5KYhF3CLwcLDBn9g2B5y9s99LU_sFhorw6pA@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<C330DB1C-A165-4C69-A5FB-4CDBA2697063@yahoo.com>
	<CABicbJKr8g8_yN5KYhF3CLwcLDBn9g2B5y9s99LU_sFhorw6pA@mail.gmail.com>
Message-ID: <201302251129.02804.mark.hackett@metoffice.gov.uk>

On Friday 22 Feb 2013, Devin Jeanpierre wrote:
> On Fri, Feb 22, 2013 at 12:41 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> > The difference is that json.loads is just deserialize(foo), which
> > pickle.loads inherently has some eval mixed in.
> >
> > That's why I think for most use cases, the answer is making json easier
> > to extend, not making pickle easier to secure.
> 
> My original suggestion was to add a third thing, such as cerealizer,

Maybe we can ask Wil WHEATon...

Apologies...


From storchaka at gmail.com  Mon Feb 25 12:58:59 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 25 Feb 2013 13:58:59 +0200
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <20130225110717.232ce082@pitrou.net>
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
Message-ID: <kgfjm3$93f$1@ger.gmane.org>

On 25.02.13 12:07, Antoine Pitrou wrote:
> Le Sun, 24 Feb 2013 22:41:48 -0500,
> Terry Reedy <tjreedy at udel.edu> a ?crit :
>>
>> def test_lookahead():
>>       it = lookahead('abc')
>>       while it:
>>           a = it.peek
>>           b = next(it)
>>           print('next:', b, '; is peek:', a is b )
>
> def test_lookahead():
>      it = iter('abc')
>      while True:
>          it, peeking = itertools.tee(it)

This should be outside a loop.

>          try:
>              a = next(peeking)
>          except StopIteration:
>              break
>          b = next(it)
>          print('next:', b, '; is peek:', a is b )




From masklinn at masklinn.net  Mon Feb 25 13:02:30 2013
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 25 Feb 2013 13:02:30 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <CACac1F_k450BCU8kWc88xAh-fJkcA8GRGSNFS4BJ96b7e30TFw@mail.gmail.com>
References: <kgemhs$ldh$1@ger.gmane.org>
	<loom.20130225T103042-482@post.gmane.org>
	<CACac1F_k450BCU8kWc88xAh-fJkcA8GRGSNFS4BJ96b7e30TFw@mail.gmail.com>
Message-ID: <F0297BAE-A401-4EF6-9AF0-CF7B979C9CA0@masklinn.net>

On 2013-02-25, at 11:08 , Paul Moore wrote:
> 
>>>> from lookahead import lookahead
>>>> it = lookahead(range(3))
>>>> list(it)
> <hangs>
> 
> The problem is that once you wrap an iterator with lookahead() you
> can't use the underlying iterator directly any more without losing
> data, as you've consumed a value from it. So you have to use the
> wrapped version

That's a drawback to most itertools wrappers though, and one which
makes sense since Python iterators and generators are not
replayable/restartable.


From tjreedy at udel.edu  Mon Feb 25 13:03:12 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Feb 2013 07:03:12 -0500
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <CACac1F_k450BCU8kWc88xAh-fJkcA8GRGSNFS4BJ96b7e30TFw@mail.gmail.com>
References: <kgemhs$ldh$1@ger.gmane.org>
	<loom.20130225T103042-482@post.gmane.org>
	<CACac1F_k450BCU8kWc88xAh-fJkcA8GRGSNFS4BJ96b7e30TFw@mail.gmail.com>
Message-ID: <kgfju3$b6s$1@ger.gmane.org>

On 2/25/2013 5:08 AM, Paul Moore wrote:
> On 25 February 2013 09:51, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>> Terry Reedy <tjreedy at ...> writes:
>>
>>> class lookahead():
>>>       "Wrap iterator with lookahead to both peek and test exhausted"
>> ...
>>
>> +1 That's a nice tool that I'd love to have in itertools.
>
> It's not a bad idea, but I don't like the fact that as written it
> turns a finite iterator into an infinite one (returning an endless
> sequence of sentinels after  the underlying iterator is exhausted).

This is a bug in this re-write. The corrected .__next__

     def __next__(self):
         if self:
             ret = self.peek
             self._set_peek()
             return ret
         else:
             raise StopIteration()

passes the test with this addition

     try:
         next(it)
         assert False, "Next should have raised StopIteration"
     except StopIteration:
         pass

 >>> list(lookahead('abc')) == list('abc')
True

-- 
Terry Jan Reedy



From jsbueno at python.org.br  Mon Feb 25 13:37:11 2013
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Mon, 25 Feb 2013 09:37:11 -0300
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <kgfju3$b6s$1@ger.gmane.org>
References: <kgemhs$ldh$1@ger.gmane.org>
	<loom.20130225T103042-482@post.gmane.org>
	<CACac1F_k450BCU8kWc88xAh-fJkcA8GRGSNFS4BJ96b7e30TFw@mail.gmail.com>
	<kgfju3$b6s$1@ger.gmane.org>
Message-ID: <CAH0mxTRf--LpzB6qFZY951unr329=zxYF59=niyJjDTvd8jTfw@mail.gmail.com>

Hi all!

What is the problem with iterttols.tee even?
It is almots the samething - although much more flexible, than what is
proposed on this thread.


>>> from itertools import tee
>>> my_iter, peek = tee(iter(range(3)), 2)
>>> next(peek)
0
>>> next(my_iter)
0
>>>

  js
 -><-

On 25 February 2013 09:03, Terry Reedy <tjreedy at udel.edu> wrote:
> On 2/25/2013 5:08 AM, Paul Moore wrote:
>>
>> On 25 February 2013 09:51, Wolfgang Maier
>> <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>>>
>>> Terry Reedy <tjreedy at ...> writes:
>>>
>>>> class lookahead():
>>>>       "Wrap iterator with lookahead to both peek and test exhausted"
>>>
>>> ...
>>>
>>> +1 That's a nice tool that I'd love to have in itertools.
>>
>>
>> It's not a bad idea, but I don't like the fact that as written it
>> turns a finite iterator into an infinite one (returning an endless
>> sequence of sentinels after  the underlying iterator is exhausted).
>
>
> This is a bug in this re-write. The corrected .__next__
>
>     def __next__(self):
>         if self:
>
>             ret = self.peek
>             self._set_peek()
>             return ret
>         else:
>             raise StopIteration()
>
> passes the test with this addition
>
>     try:
>         next(it)
>         assert False, "Next should have raised StopIteration"
>     except StopIteration:
>         pass
>
>>>> list(lookahead('abc')) == list('abc')
>
> True
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From zuo at chopin.edu.pl  Mon Feb 25 13:45:50 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 25 Feb 2013 13:45:50 +0100
Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?=
In-Reply-To: <CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
Message-ID: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>

24.02.2013 07:24, Guido van Rossum wrote:
[...]
> And to me the ... Looks too much like something that consumes the
> rest, rather than leaving it.

To me either. And we already have the syntax for consuming the rest:

     a, b, *seq = iterable

But maybe it could be extended to include the following variant:

     a, b, *() = iterable

-- expressing the "leave the rest untouched" behaviour? Doesn't such
an empty-tuple-literal-like syntax suggest strongly enough that
no items are consumed?

Cheers.
*j



From masklinn at masklinn.net  Mon Feb 25 13:56:12 2013
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 25 Feb 2013 13:56:12 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <CAH0mxTRf--LpzB6qFZY951unr329=zxYF59=niyJjDTvd8jTfw@mail.gmail.com>
References: <kgemhs$ldh$1@ger.gmane.org>
	<loom.20130225T103042-482@post.gmane.org>
	<CACac1F_k450BCU8kWc88xAh-fJkcA8GRGSNFS4BJ96b7e30TFw@mail.gmail.com>
	<kgfju3$b6s$1@ger.gmane.org>
	<CAH0mxTRf--LpzB6qFZY951unr329=zxYF59=niyJjDTvd8jTfw@mail.gmail.com>
Message-ID: <300C301B-D25A-4BA0-8F9C-01390330C629@masklinn.net>

On 2013-02-25, at 13:37 , Joao S. O. Bueno wrote:

> Hi all!
> 
> What is the problem with iterttols.tee even?
> It is almots the samething - although much more flexible, than what is
> proposed on this thread.
> 
> 
>>>> from itertools import tee
>>>> my_iter, peek = tee(iter(range(3)), 2)
>>>> next(peek)
> 0
>>>> next(my_iter)
> 0
>>>> 

Verbose way to do the same thing: you have to tee() + next() once per
"peek()". And splitting to subroutines means these subroutines will have
to return an iterator alongside any potential value as they'll have to
tee() themselves.


From ron3200 at gmail.com  Mon Feb 25 16:29:52 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 25 Feb 2013 09:29:52 -0600
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <kgemhs$ldh$1@ger.gmane.org>
References: <kgemhs$ldh$1@ger.gmane.org>
Message-ID: <512B8370.6040807@gmail.com>



On 02/24/2013 09:41 PM, Terry Reedy wrote:
> Does anyone else this should be added to itertools? It seems to not be
> completely obvious to everyone, is more complex that some of the existing
> itertools, and cannot be composed from them either. (Nor can it be written
> as a generator function.)
>
> Any of the names can be changed. Perhaps the class should be 'peek' and the
> lookahead object something else. The sentinel should be read-only if
> possible. I considered whether the peek object should be read-only, but
> someone would say that they *want* be able to replace the next object to be
> yielded. Peeking into an exhausted iterable could raise instead of
> returning the sentinel, but I don't know if that would be more useful.
>
> ----------------
> class lookahead():
>      "Wrap iterator with lookahead to both peek and test exhausted"
>
>      _NONE = object()
>      def __init__(self, iterable):
>          self._it = iter(iterable)
>          self._set_peek()
>      def __iter__(self):
>          return self
>      def __next__(self):
>          ret = self.peek
>          self._set_peek()
>          return ret
>      def _set_peek(self):
>          try:
>              self.peek = next(self._it)
>          except StopIteration:
>              self.peek = self._NONE
>      def __bool__(self):
>          return self.peek is not self._NONE

>
> def test_lookahead():
>     it = lookahead('abc')
>     while it:
>         a = it.peek
>         b = next(it)
>         print('next:', b, '; is peek:', a is b )
>
> test_lookahead()


I think with a few small changes I would find it useful.

The key feature here is that the result is pre calculated and held until 
it's needed, rather than calculated when it's asked for.

You should catch any exception and hold that as well.  On the next .next() 
call, it should raise the exception if there was one, or emit the value.

I'm not sure if using the __bool__ attribute is the best choice.  I would 
prefer a .error flag, along with a .next_value attribute.  It would make 
the code using it easier to follow.

      it.error        <--    True if next(it) will raise an exception.
      it.next_value   <--    The next value, or the exception to raise.


Note that iterating a list of exceptions will still work.



About it.error.  If it was a concurrent version, then it.error could have 
three values.

      it.error == True   # Will raise an exception
      it.error == False  # Will not raise an exception
      it.error == None   # Still calculating


I wonder how this type of generator will behave with "yield from".  And if 
there would be any advantages for writing concurrent (or concurrent acting) 
code.


Of course you really need to think about weather or not this fits the 
problem being solved.

Cheers,
    Ron



From yoavglazner at gmail.com  Mon Feb 25 16:45:17 2013
From: yoavglazner at gmail.com (yoav glazner)
Date: Mon, 25 Feb 2013 17:45:17 +0200
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <512B8370.6040807@gmail.com>
References: <kgemhs$ldh$1@ger.gmane.org>
	<512B8370.6040807@gmail.com>
Message-ID: <CAJ78kjOJmAP5q4E9zvaWjoqY94AU3yvgKkGK0cW7tqH7hhQEYw@mail.gmail.com>

On Mon, Feb 25, 2013 at 5:29 PM, Ron Adam <ron3200 at gmail.com> wrote:

>
>
> On 02/24/2013 09:41 PM, Terry Reedy wrote:
>
>> Does anyone else this should be added to itertools? It seems to not be
>> completely obvious to everyone, is more complex that some of the existing
>> itertools, and cannot be composed from them either. (Nor can it be written
>> as a generator function.)
>>
>> Any of the names can be changed. Perhaps the class should be 'peek' and
>> the
>> lookahead object something else. The sentinel should be read-only if
>> possible. I considered whether the peek object should be read-only, but
>> someone would say that they *want* be able to replace the next object to
>> be
>> yielded. Peeking into an exhausted iterable could raise instead of
>> returning the sentinel, but I don't know if that would be more useful.
>>
>> ----------------
>> class lookahead():
>>      "Wrap iterator with lookahead to both peek and test exhausted"
>>
>>      _NONE = object()
>>      def __init__(self, iterable):
>>          self._it = iter(iterable)
>>          self._set_peek()
>>      def __iter__(self):
>>          return self
>>      def __next__(self):
>>          ret = self.peek
>>          self._set_peek()
>>          return ret
>>      def _set_peek(self):
>>          try:
>>              self.peek = next(self._it)
>>          except StopIteration:
>>              self.peek = self._NONE
>>      def __bool__(self):
>>          return self.peek is not self._NONE
>>
>
>
>> def test_lookahead():
>>     it = lookahead('abc')
>>     while it:
>>         a = it.peek
>>         b = next(it)
>>         print('next:', b, '; is peek:', a is b )
>>
>> test_lookahead()
>>
>
>
> I think with a few small changes I would find it useful.
>
> The key feature here is that the result is pre calculated and held until
> it's needed, rather than calculated when it's asked for.
>
It seems like a buggy feature, think a DB cursor, next has side
effects(like row locks and such)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130225/6dbeff8e/attachment.html>

From oscar.j.benjamin at gmail.com  Mon Feb 25 17:20:41 2013
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Mon, 25 Feb 2013 16:20:41 +0000
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <kgemhs$ldh$1@ger.gmane.org>
References: <kgemhs$ldh$1@ger.gmane.org>
Message-ID: <CAHVvXxRhbJmyEZL5_kqLXMDfOQHQMUJqX+8X20D-vHdquZD8=A@mail.gmail.com>

On 25 February 2013 03:41, Terry Reedy <tjreedy at udel.edu> wrote:
> An iterator iter represents the remainder of some collection, concrete or
> not, finite or not. If the remainder is not empty, its .__next__ method
> selects one object, mutates iter to represent the reduced remainder (now
> possibly empty), and returns the one item.
>
> At various times, people have asked for the ability to determine whether an
> iterator is exhausted, whether a next() call will return or raise. If it
> will return, people have also asked for the ability to peek at what the
> return would be, without disturbing what the return will be. For instance,
> on the 'iterable.__unpack__ method' Alex Stewart today wrote:
>> The problem is that there is no standard way to query an iterator
>> to find out if it has more data available without automatically
>> consuming that next data element in the rocess.
[SNIP]

At times I thought I wanted the ability to query an iterator without
necessarily popping an element. I've generally found that I ended up
solving the problem in a different way. My own solution is to have a
pushable iterator so that after inspecting a value I can push it back
onto the iterator ready for a subsequent next() call. Example code for
this looks like:

def _pushable(iterable):
    '''Helper for pushable'''
    iterator = iter(iterable)
    stack = []
    yield lambda x: stack.append(x)
    while True:
        while stack:
            yield stack.pop()
        yield next(iterator)

def pushable(iterable):
    '''Make an iterable pushable.

    >>> iterable, push = pushable(range(9))
    >>> next(iterable)
    0
    >>> next(iterable)
    1
    >>> push(1) # Push the 1 back on to the iterable
    >>> list(iterable)
    [1, 2, 3, 4, 5, 6, 7, 8]
    '''
    gen = _pushable(iterable)
    push = next(gen)
    return gen, push


Oscar


From jeanpierreda at gmail.com  Mon Feb 25 17:20:30 2013
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Mon, 25 Feb 2013 11:20:30 -0500
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
Message-ID: <CABicbJLt45hDCg4hNUVqUKu+Xmpnw6srQqeK+JHU2pa+YqEMww@mail.gmail.com>

On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> But maybe it could be extended to include the following variant:
>
>     a, b, *() = iterable
>
> -- expressing the "leave the rest untouched" behaviour? Doesn't such
> an empty-tuple-literal-like syntax suggest strongly enough that
> no items are consumed?

I would've interpreted it as unpacking the rest of the iterable into
() -- i.e., I'd assume it has the current behavior of failing if the
rest of the iterable has anything at all.

Of course, you can't unpack anything into (), because Python never had
that syntax, but you get the idea.

-- Devin


From jbvsmo at gmail.com  Mon Feb 25 17:29:07 2013
From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=)
Date: Mon, 25 Feb 2013 13:29:07 -0300
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
Message-ID: <CAOyAWgjVZuuXFGcdxXsUUr9s7T6kTv99bBOd-BDBRT5hMy2S7g@mail.gmail.com>

> But maybe it could be extended to include the following variant:
>
>     a, b, *() = iterable
>
>
Python already supports this odd syntax

    a, b, *[] = iterable

because it interprets the [] not as an empty list, but as an empty "list of
identifiers". Maybe it could be used for something useful.

BTW, the del syntax has the same "problem"

    del a, b, (c,), [d], []
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130225/4e7a890a/attachment.html>

From solipsis at pitrou.net  Mon Feb 25 19:53:02 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 19:53:02 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org>
Message-ID: <20130225195302.45704b36@pitrou.net>

On Mon, 25 Feb 2013 13:58:59 +0200
Serhiy Storchaka <storchaka at gmail.com>
wrote:
> On 25.02.13 12:07, Antoine Pitrou wrote:
> > Le Sun, 24 Feb 2013 22:41:48 -0500,
> > Terry Reedy <tjreedy at udel.edu> a ?crit :
> >>
> >> def test_lookahead():
> >>       it = lookahead('abc')
> >>       while it:
> >>           a = it.peek
> >>           b = next(it)
> >>           print('next:', b, '; is peek:', a is b )
> >
> > def test_lookahead():
> >      it = iter('abc')
> >      while True:
> >          it, peeking = itertools.tee(it)
> 
> This should be outside a loop.

Only if you restrict yourself to access peeking each time you access it.
(which, I suppose, is not the general use case for the lookahead
proposal)

Regards

Antoine.




From storchaka at gmail.com  Mon Feb 25 21:21:03 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 25 Feb 2013 22:21:03 +0200
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <20130225195302.45704b36@pitrou.net>
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
Message-ID: <kggh3g$o9t$1@ger.gmane.org>

On 25.02.13 20:53, Antoine Pitrou wrote:
>>> def test_lookahead():
>>>       it = iter('abc')
>>>       while True:
>>>           it, peeking = itertools.tee(it)
>>
>> This should be outside a loop.
>
> Only if you restrict yourself to access peeking each time you access it.
> (which, I suppose, is not the general use case for the lookahead
> proposal)

Only if your do not want to consume O(N) memory and spend O(N**2) time.




From solipsis at pitrou.net  Mon Feb 25 21:27:22 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 25 Feb 2013 21:27:22 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
	<kggh3g$o9t$1@ger.gmane.org>
Message-ID: <20130225212722.0d6de65b@pitrou.net>

On Mon, 25 Feb 2013 22:21:03 +0200
Serhiy Storchaka <storchaka at gmail.com>
wrote:
> On 25.02.13 20:53, Antoine Pitrou wrote:
> >>> def test_lookahead():
> >>>       it = iter('abc')
> >>>       while True:
> >>>           it, peeking = itertools.tee(it)
> >>
> >> This should be outside a loop.
> >
> > Only if you restrict yourself to access peeking each time you access it.
> > (which, I suppose, is not the general use case for the lookahead
> > proposal)
> 
> Only if your do not want to consume O(N) memory and spend O(N**2) time.

No, that's beside the point. If you don't consume "peeking" in
lock-step with "it", then "peeking" and "it" become desynchronized and
therefore the semantics are wrong w.r.t to the original feature
request (where "peeking" is supposed to be some proxy to "it", not an
independently-running iterator).

Regards

Antoine.




From barry at python.org  Mon Feb 25 22:08:11 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Feb 2013 16:08:11 -0500
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
References: <kgemhs$ldh$1@ger.gmane.org>
	<CAHVvXxRhbJmyEZL5_kqLXMDfOQHQMUJqX+8X20D-vHdquZD8=A@mail.gmail.com>
Message-ID: <20130225160811.680ecc42@anarchist.wooz.org>

On Feb 25, 2013, at 04:20 PM, Oscar Benjamin wrote:

>At times I thought I wanted the ability to query an iterator without
>necessarily popping an element. I've generally found that I ended up
>solving the problem in a different way. My own solution is to have a
>pushable iterator so that after inspecting a value I can push it back
>onto the iterator ready for a subsequent next() call.

The email package has one of these in its feedparser module, called
BufferedSubFile.  It needs to be able to push lines of text back onto the
stack that the normal iteration pops off.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130225/68cd81d4/attachment.pgp>

From foogod at gmail.com  Mon Feb 25 22:52:32 2013
From: foogod at gmail.com (Alex Stewart)
Date: Mon, 25 Feb 2013 13:52:32 -0800 (PST)
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <512AD5E8.4030204@fastmail.us>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<512AD5E8.4030204@fastmail.us>
Message-ID: <68bec4db-8e8a-4113-ad90-247193ad8a86@googlegroups.com>

On Sunday, February 24, 2013 7:09:28 PM UTC-8, Random832 wrote:

>  If you want "smart" unpackable objects, why not allow d in your *d 
> example to be something other than a list?
>
> Like, if you have a,*b = range(1,5); b could literally be range(2,5).
>

Well, primarily for two reasons:

   1. It does not actually do anything to support smart unpackables (i.e. 
   objects that want to be able to produce different values depending on how 
   many items they're asked to produce).  This might be useful for the 
   "infinite iterator" case (and I think somebody proposed it for that 
   earlier), but that's a completely different issue than smart-unpacking..
   2. More importantly, it would almost certainly break some existing code 
   that relies on the fact that the contents of the extended-unpacking-term 
   will always be a list, which would be bad.
   
--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130225/c6b32d70/attachment.html>

From foogod at gmail.com  Mon Feb 25 23:25:03 2013
From: foogod at gmail.com (Alex Stewart)
Date: Mon, 25 Feb 2013 14:25:03 -0800 (PST)
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <kgemlt$ldh$2@ger.gmane.org>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>
	<44528db0ff81ea147dfc71d97e472798@chopin.edu.pl>
	<512A7967.1060709@mrabarnett.plus.com>
	<d31a51ec-9ca5-4653-ae4f-0328d096a79a@googlegroups.com>
	<kgemlt$ldh$2@ger.gmane.org>
Message-ID: <a9935a92-d93b-4d28-9db8-fdad86a127a1@googlegroups.com>

On Sunday, February 24, 2013 7:43:58 PM UTC-8, Terry Reedy wrote:

> On 2/24/2013 5:07 PM, Alex Stewart wrote: 
> >  1. It is a fairly small change to the language 
>
> To me, taking something simple and elegant and complexifying it, to 
> little benefit I can see other than to solve a problem in the wrong 
> place, in no small matter.


I did not say it was a small matter.  I said it was a small change to the 
language.  From a purely practical perspective of what changes would be 
required to the language itself in order to support this proposal, in my 
opinion, it would not require that much to implement (it would probably 
only require a couple of lines of new code and some doc changes), and is 
very unlikely to affect any existing code or tools.

(Syntax changes are, in my opinion, far more disruptive changes to the 
language:  They require a change to the parser as well as the interpreter, 
and potentially changes to all manner of other utilities 
(debuggers/code-checkers/AST-manipulators/etc) out there in the wild.  They 
make it much harder to write Python code that supports older Python 
releases (with method changes, you can check for the existence of a feature 
in the interpreter and selectively use or not use it, but you typically 
can't use new syntax constructs in anything that might be run in an older 
Python release, because the parser will bomb out instead).  They also 
require programmers to learn new syntax (if nothing else, to understand 
what it means when they come across it in somebody else's code).  What's 
more, every syntax change potentially limits what can be done with the 
syntax in the future (for example, by using up special characters or 
constructs so they're not available for other things), and so on. 
 Personally, I think there should be a much higher bar for syntax changes 
than for adding a new special method, not the other way around.)

You also seem to be defining "in the wrong place" as "in a different way 
than I personally have a use for"..  As I pointed out (which you seem to 
have completely skipped over in your reply), there are use cases where the 
consumer side is the right (or only) place, and there are other different 
use cases where the producer side is the right (or only) place.  The two 
are not mutually-exclusive, and improving one does not really address the 
other issue at all.

I want to emphasize that I am not saying that we shouldn't extend unpacking 
notation to support partial-unpacking; in fact I think it might well be a 
good idea.  What I am saying, however, is that if we do, I do think it's a 
much bigger change to the language than what I proposed, and it still 
doesn't actually address most of the issues that __unpack__ was intended to 
deal with, so it's really not an either-or sort of thing anyway.

Certainly, adding a new special method to 
> some category of objects has not effect unless changes are made to the 
> interpreter elsewhere to make automatic use of that method. Otherwise, 
> you could just define a normal method to be used by user code.


My proposal does not include adding a new special method to any existing 
objects.  It adds the _ability_ for people to add a special method to 
objects *in the future* and have the interpreter take advantage of it.  It 
therefore should have no effect for any existing code.

>  2. There are (in my opinion) already pretty good parallels with this 
> >     sort of change that have already been implemented in Python (for 
> >     example, separating the concept of "iterable" from the concept of 
> >     "indexable"). 
>
> The related but distinct concepts of sequential access and random access 
> are basic to computation and were *always* distinct in Python.


Oh really?  Then how, prior to the development of the iterator protocol, 
did one define an object which was accessible sequentially but not randomly 
in the Python language?  If you couldn't do that, you can't claim that the 
concepts were really distinct in Python, in my opinion.

The truth is that the concept of an iterable object vs. an indexable object 
may have been distinct *in some people's minds*, but it was *not* originally 
distinct in the Python language itself.  People decided that that should 
change, so the language was extended to make that happen.  It should also 
be noted that the two concepts were not distinct in *everyone's* minds (I 
remember straightening out confusion on this point in numerous people's 
understanding (and still need to occasionally), particularly those who did 
not come from a classical computer-science background).  However, just 
because there were some people who didn't see them as distinct did not mean 
that it was an invalid way for other people to view them, or that they 
shouldn't ultimately be distinct in the language..

With all respect (and I do actually mean that), your primary argument 
against it seems to be one of intellectual complacency: "I've never 
considered that they might be different, so they shouldn't ever be 
separated, because then I don't have to change the way I think about 
things".

The addition of __iter__ along with __next__ 
> added more benefits than __next__ alone would have. 
>

You seem to be making my point for me.  In essence, __unpack__ is really 
just the equivalent for the unpacking operation to what __iter__ is for 
"for" loops.  Is it strictly required in order to support the basic 
behavior?  No (you could do for loops over simple iterables with just a 
variant of __next__, if you really wanted to), but it allows some more 
sophisticated objects to provide more useful behaviors by giving them more 
context about how they are being invoked (in the case of __iter__, by 
telling them ahead of time that they are going to be used in an iteration 
context, and allowing them to create or alter the initial object that the 
for loop interacts with).  This provides "more benefits than __next__ alone 
would have", so it was added to the language.

Logically, by the same argument, and in almost exactly the same way, 
__unpack__ provides more benefits than __iter__ alone would have (by 
telling them that they are going to be used in a particular unpacking 
context, and allowing them to alter the object used for that type of 
iteration), and therefore should arguably be part of the language as well.

>  3. It is consistent with the general idea in Python that an object can 
> >     choose to emulate whichever behaviors of other objects make sense, 
> >     and not emulate other ones that do not make sense.  In my opinion, 
> >     "iterating", and "unpacking" are two related-but-not-the-same 
> >     language concepts, which currently are unnecessarily entangled. 
>
> I do not see that at all. As near as I can tell, both mean 'sequential 
> access'.


Not true.  Iteration means "sequential access".  Unpacking means 
"sequential access with inherent length constraints", which is the bit you 
seem to be ignoring.  The most notable consequence of this (but definitely 
not the only one) is that a for loop (for example) will never raise 
ValueError, but the unpacking operation will, if it gets what it considers 
"wrong" results back from the unpacked object.  In this way the two 
operations do not interpret the data that the object in question provides 
in the same way, and it does not really have the same meaning.

Yes, from a purely mechanical perspective, they perform similar operations, 
but programming interfaces are not solely defined by mechanics.  The 
difference here is one of implied contracts.  In simple iteration, there is 
explicitly no minimum or maximum number of values that the iterable is 
required to produce, and in fact the entire protocol is based on the 
assumed perspective that returning a value is always the "success" 
condition and not being able to return one is the "failure" case, and in 
that context it makes sense to design iterators so that they should always 
return more data if they're able to do so.  In unpacking, however, when it 
gets to the end of the variable list, the conditions are reversed, and 
actually returning a value becomes a failure condition.  This change of 
meaning, however, is not communicated in any way to the iterator (this is 
just one example: there are similar contract issues inherent in extended 
unpacking as well).  This is, in my opinion, a flaw in the interface, 
because the language is misrepresenting (bounded) unpacking as (unbounded) 
iteration when calling the iterable object.

I admit I do not understand your __unpack__ proposal, since it seemed 
> vague and incomplete to me. But until I see your conceptual distinction, 
> the details do not matter to me.
>

I'm not exactly sure why you consider it vague or incomplete.  I provided a 
specific description of exactly what changes to the language I was 
proposing, with method signatures, a description of the new interpreter 
behavior and examples of data values in typical scenarios, and accompanied 
it with a list of several different use-cases which I believed it would 
provide significant utility.  I'm not really sure how I could have gotten 
more specific without providing a diff against the CPython sources (which 
seems a bit much for an initial proposal)..

>  4. It would help significantly with several types of problems that do 
> >     not currently have great solutions. 
>
> I obviously do not see the same pile of problems that would justify a 
> new special method.


Obviously, although I'm not really sure why not because I did explicitly 
state a bunch of them in my original post.  You seem to have just ignored 
all of them except for the issue of partial unpacking, which frankly was 
not even the most important or interesting use case I presented, in my 
opinion.  I would recommend you might want to go back and read the last 
part of my initial post where I presented several potential use cases for 
all this.

I do get the distinct impression, however, that you've already made up your 
mind and, right or wrong, there's nothing I could possibly say which would 
ever change it, so at this point I'm not sure whether it really matters 
(sigh)...

> no way to query an iterator to find out if it has more data available  

 

See how in my separate post: 
> "Add lookahead iterator (peeker) to itertools" 


I'm not necessarily opposed to that proposal, but I should point out that 
it does not actually address the problem I mentioned in this thread at all, 
so it's really kinda irrelevant to that discussion.

(Even if that is added to itertools, unpacking still won't use it, nor does 
it provide any way to safely query iterables which produce side-effects, 
even if there might be a way for them to provide that info without the 
side-effects, nor does it provide a way to test for end conditions without 
changing the underlying iterable (which might be used elsewhere in scopes 
outside the lookahead-wrapper), etc..)

--Alex

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130225/b6de1651/attachment.html>

From greg.ewing at canterbury.ac.nz  Tue Feb 26 00:30:03 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 26 Feb 2013 12:30:03 +1300
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CABicbJLt45hDCg4hNUVqUKu+Xmpnw6srQqeK+JHU2pa+YqEMww@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
	<CABicbJLt45hDCg4hNUVqUKu+Xmpnw6srQqeK+JHU2pa+YqEMww@mail.gmail.com>
Message-ID: <512BF3FB.9090803@canterbury.ac.nz>

Devin Jeanpierre wrote:
> On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>>    a, b, *() = iterable
> 
> Of course, you can't unpack anything into (), because Python never had
> that syntax, but you get the idea.

-1, this is just as arbitrary as ... or a lone *.

I prefer ... myself, because visually it has a low profile
and doesn't draw undue attention to something that you're
saying you don't care about.

Maybe ... could be blessed as an official "don't care"
assignment target, like _ in Haskell. That would make this
usage less of a special case (although it would still be
partially special, since it suppresses unpacking of the
rest of the iterator).

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Feb 26 00:37:48 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 26 Feb 2013 12:37:48 +1300
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAOyAWgjVZuuXFGcdxXsUUr9s7T6kTv99bBOd-BDBRT5hMy2S7g@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
	<CAOyAWgjVZuuXFGcdxXsUUr9s7T6kTv99bBOd-BDBRT5hMy2S7g@mail.gmail.com>
Message-ID: <512BF5CC.4050602@canterbury.ac.nz>

Jo?o Bernardo wrote:
> Python already supports this odd syntax
> 
>     a, b, *[] = iterable
> 
> because it interprets the [] not as an empty list, but as an empty "list 
> of identifiers". Maybe it could be used for something useful.

No, because it already has a meaning: there must be no more
values left in the sequence.

> BTW, the del syntax has the same "problem"
> 
>     del a, b, (c,), [d], []

Or just

    [] = iterable

The surprising thing is that a special case seems to be
made for ():

 >>> () = []
   File "<stdin>", line 1
SyntaxError: can't assign to ()

It's surprising because () and [] are otherwise completely
interchangeable for unpacking purposes.

-- 
Greg


From steve at pearwood.info  Tue Feb 26 01:07:25 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 26 Feb 2013 11:07:25 +1100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <512BF3FB.9090803@canterbury.ac.nz>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
	<CABicbJLt45hDCg4hNUVqUKu+Xmpnw6srQqeK+JHU2pa+YqEMww@mail.gmail.com>
	<512BF3FB.9090803@canterbury.ac.nz>
Message-ID: <512BFCBD.40702@pearwood.info>

On 26/02/13 10:30, Greg Ewing wrote:
> Devin Jeanpierre wrote:
>> On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>>>    a, b, *() = iterable
>>
>> Of course, you can't unpack anything into (), because Python never had
>> that syntax, but you get the idea.
>
> -1, this is just as arbitrary as ... or a lone *.
>
> I prefer ... myself, because visually it has a low profile
> and doesn't draw undue attention to something that you're
> saying you don't care about.
>
> Maybe ... could be blessed as an official "don't care"
> assignment target, like _ in Haskell. That would make this
> usage less of a special case (although it would still be
> partially special, since it suppresses unpacking of the
> rest of the iterator).

Please no. _ is already overloaded too much. In i18n contexts, _ is
conventionally used as the function for getting display text. In the
interactive interpreter, _ is used for the last result. And in source
code, _ is often used by convention as a "don't care" variable.

Turning _ into a magic "stop unpacking" symbol would break code that
uses it as a "don't care" target when unpacking:

spam, _, _, ham, eggs = stuff

and would be ambiguous when there is an underscore as the right-most
target. Would it mean, unpack but I don't care about it, or don't
unpack?

I'm still not seeing why this is important enough to add magic syntax.
If you want to unpack the first five values from an iterator, without
exhausting it, we already have some good solutions:

spam, ham, eggs, cheese, tomato = (next(it) for i in range(5))

or

spam, ham, eggs, cheese, tomato = itertools.islice(it, 5)


Neither of these are so confusing to read or onerous to use as to
justify new magic syntax:


spam, ham, eggs, cheese, tomato, $$$$ = it  # for some value of $$$




-- 
Steven


From abarnert at yahoo.com  Tue Feb 26 01:12:04 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Feb 2013 16:12:04 -0800
Subject: [Python-ideas] My wishlist for Tulip
In-Reply-To: <CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>
References: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>
	<CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>
Message-ID: <0660447D-BC71-4B49-A2F6-9CE00643A958@yahoo.com>

Sorry for the top post, but I'm on a phone.

I'd love to see a trivial chat server in the demos. There are many designs for async servers that work great with echo and even http but fall down with chat (because of interactions between clients), and it would be very useful to see how tulip makes it easy to do it right.

Sent from a random iPhone.

On Feb 22, 2013, at 13:17, Guido van Rossum <guido at python.org> wrote:

> Those are all good ideas! Now I just have to find time to implement them...
> 
> On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
>> I understand it's still beta but anyways, here's my little wish list for Tulip.
>> 
>> * provide a 'demo' / 'example' directory containing very simple
>> scripts showing the most basic usages such as:
>> 
>> - echo_tcp_client.py
>> - echo_tcp_server.py
>> - echo_tcp_server_w_timeout.py  (same as echo_server.py but also
>> disconnects the client after a certain time of inactivity)
>> - echo_tcp_ssl_client.py
>> - echo_tcp_ssl_server.py
>> - echo_udp_client.py
>> - echo_udp_server.py
>> 
>> * move all *test*.py scripts in a separate 'test' directory
>> 
>> * if it's not part of the API intended to be public move
>> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new
>> 'scripts'/'tools' directory)
>> 
>> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks
>> like they belong elsewhere
>> 
>> * write a simple benchmark framework testing (at least) sending,
>> receiving and the internal scheduler  (I'd like to help with this one)
>> 
>> 
>> --- Giampaolo
>> http://code.google.com/p/pyftpdlib/
>> http://code.google.com/p/psutil/
>> http://code.google.com/p/pysendfile/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> 
> 
> 
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas


From zuo at chopin.edu.pl  Tue Feb 26 01:13:30 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 26 Feb 2013 01:13:30 +0100
Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?=
In-Reply-To: <512BF5CC.4050602@canterbury.ac.nz>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
	<CAOyAWgjVZuuXFGcdxXsUUr9s7T6kTv99bBOd-BDBRT5hMy2S7g@mail.gmail.com>
	<512BF5CC.4050602@canterbury.ac.nz>
Message-ID: <366654f2dcc7518781582d9e7eacca91@chopin.edu.pl>

26.02.2013 00:37, Greg Ewing wrote:

> Jo?o Bernardo wrote:
>> Python already supports this odd syntax
>> a, b, *[] = iterable

Indeed, I didn't know that. :-|

>> because it interprets the [] not as an empty list, but as an empty 
>> "list of identifiers". Maybe it could be used for something useful.
>
> No, because it already has a meaning: there must be no more
> values left in the sequence.
>
>> BTW, the del syntax has the same "problem"
>> del a, b, (c,), [d], []
>
> Or just
>
>    [] = iterable
>
> The surprising thing is that a special case seems to be
> made for ():
>
>>>> () = []
>   File "<stdin>", line 1
> SyntaxError: can't assign to ()
>
> It's surprising because () and [] are otherwise completely
> interchangeable for unpacking purposes.

Not entirely...

     >>> a, b, [c, d] = 1, 2, (3, 4)
     >>> a, b, (c, d) = 1, 2, (3, 4)

OK.

     >>> a, b, *[c, d] = 1, 2, 3, 4
     >>> a, b, *(c, d) = 1, 2, 3, 4

OK as well -- *but*:

     >>> a, b, [] = 1, 2, []   # it's ok
     >>> a, b, () = 1, 2, ()   # but it's not
       File "<stdin>", line 1
     SyntaxError: can't assign to ()

...and:

     >>> a, b, *[] = 1, 2    # it's ok
     >>> a, b, *() = 1, 2    # but it's not
       File "<stdin>", line 1
     SyntaxError: can't assign to ()

Strange... (inb4: http://www.youtube.com/watch?v=5IgB8XKR23c#t=118s ).

*j



From python at mrabarnett.plus.com  Tue Feb 26 01:19:54 2013
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 26 Feb 2013 00:19:54 +0000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <512BFCBD.40702@pearwood.info>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
	<CABicbJLt45hDCg4hNUVqUKu+Xmpnw6srQqeK+JHU2pa+YqEMww@mail.gmail.com>
	<512BF3FB.9090803@canterbury.ac.nz> <512BFCBD.40702@pearwood.info>
Message-ID: <512BFFAA.2020706@mrabarnett.plus.com>

On 2013-02-26 00:07, Steven D'Aprano wrote:
> On 26/02/13 10:30, Greg Ewing wrote:
>> Devin Jeanpierre wrote:
>>> On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>>>>    a, b, *() = iterable
>>>
>>> Of course, you can't unpack anything into (), because Python never had
>>> that syntax, but you get the idea.
>>
>> -1, this is just as arbitrary as ... or a lone *.
>>
>> I prefer ... myself, because visually it has a low profile
>> and doesn't draw undue attention to something that you're
>> saying you don't care about.
>>
>> Maybe ... could be blessed as an official "don't care"
>> assignment target, like _ in Haskell. That would make this
>> usage less of a special case (although it would still be
>> partially special, since it suppresses unpacking of the
>> rest of the iterator).
>
> Please no. _ is already overloaded too much. In i18n contexts, _ is
> conventionally used as the function for getting display text. In the
> interactive interpreter, _ is used for the last result. And in source
> code, _ is often used by convention as a "don't care" variable.
>
[snip]
He didn't say that we should use "_", he said that "..." would be like
"_" in Haskell.



From fafhrd91 at gmail.com  Tue Feb 26 01:41:33 2013
From: fafhrd91 at gmail.com (Nikolay Kim)
Date: Mon, 25 Feb 2013 16:41:33 -0800
Subject: [Python-ideas] My wishlist for Tulip
In-Reply-To: <0660447D-BC71-4B49-A2F6-9CE00643A958@yahoo.com>
References: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>
	<CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>
	<0660447D-BC71-4B49-A2F6-9CE00643A958@yahoo.com>
Message-ID: <18F634D4-9C75-4FCC-9C33-917DCAE72CC8@gmail.com>

You can check my experimental httpclient code. it has web socket example, server + cmd client.

https://github.com/fafhrd91/httpclient


On Feb 25, 2013, at 4:12 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> Sorry for the top post, but I'm on a phone.
> 
> I'd love to see a trivial chat server in the demos. There are many designs for async servers that work great with echo and even http but fall down with chat (because of interactions between clients), and it would be very useful to see how tulip makes it easy to do it right.
> 
> Sent from a random iPhone.
> 
> On Feb 22, 2013, at 13:17, Guido van Rossum <guido at python.org> wrote:
> 
>> Those are all good ideas! Now I just have to find time to implement them...
>> 
>> On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
>>> I understand it's still beta but anyways, here's my little wish list for Tulip.
>>> 
>>> * provide a 'demo' / 'example' directory containing very simple
>>> scripts showing the most basic usages such as:
>>> 
>>> - echo_tcp_client.py
>>> - echo_tcp_server.py
>>> - echo_tcp_server_w_timeout.py  (same as echo_server.py but also
>>> disconnects the client after a certain time of inactivity)
>>> - echo_tcp_ssl_client.py
>>> - echo_tcp_ssl_server.py
>>> - echo_udp_client.py
>>> - echo_udp_server.py
>>> 
>>> * move all *test*.py scripts in a separate 'test' directory
>>> 
>>> * if it's not part of the API intended to be public move
>>> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new
>>> 'scripts'/'tools' directory)
>>> 
>>> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks
>>> like they belong elsewhere
>>> 
>>> * write a simple benchmark framework testing (at least) sending,
>>> receiving and the internal scheduler  (I'd like to help with this one)
>>> 
>>> 
>>> --- Giampaolo
>>> http://code.google.com/p/pyftpdlib/
>>> http://code.google.com/p/psutil/
>>> http://code.google.com/p/pysendfile/
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> http://mail.python.org/mailman/listinfo/python-ideas
>> 
>> 
>> 
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas



From jbvsmo at gmail.com  Tue Feb 26 02:20:40 2013
From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=)
Date: Mon, 25 Feb 2013 22:20:40 -0300
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <512BF5CC.4050602@canterbury.ac.nz>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
	<CAOyAWgjVZuuXFGcdxXsUUr9s7T6kTv99bBOd-BDBRT5hMy2S7g@mail.gmail.com>
	<512BF5CC.4050602@canterbury.ac.nz>
Message-ID: <CAOyAWgjduM8-4LYRKH9iRyDhAmHQbC6axhLuF0P29wx=7sevmw@mail.gmail.com>

2013/2/25 Greg Ewing <greg.ewing at canterbury.ac.nz>

> Jo?o Bernardo wrote:
>
>> Python already supports this odd syntax
>>
>>     a, b, *[] = iterable
>>
>> because it interprets the [] not as an empty list, but as an empty "list
>> of identifiers". Maybe it could be used for something useful.
>>
>
> No, because it already has a meaning: there must be no more
> values left in the sequence.
>
>
Why have two things with the same meaning?

    a, b = iterable
    a, b, *[] = iterable

Both are the same... The *[] thing is about 100% useless right now.

And the empty "list" syntax is so not used it was possible to segfault
pypy<https://bugs.pypy.org/issue1364>
by putting that in a for loop.



>
>  BTW, the del syntax has the same "problem"
>>
>>     del a, b, (c,), [d], []
>>
>
>

Or just
>
>    [] = iterable
>
> The surprising thing is that a special case seems to be
> made for ():
>
> >>> () = []
>   File "<stdin>", line 1
> SyntaxError: can't assign to ()
>
> It's surprising because () and [] are otherwise completely
> interchangeable for unpacking purposes


That's what I'm saying... [] and () during unpaking aren't the same thing
when they normally should...
The same happens with the "del" syntax!

One can think about this to see if an iterable is empty and raise an error
if it is not:
     [] = iterable

But
    *[], = iterable

Has the exact same meaning and is very cryptic.


BTW:
This is the code golf winner to make a list of chars:
    *s, = 'string'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130225/667de0fe/attachment.html>

From vinay_sajip at yahoo.co.uk  Tue Feb 26 02:29:23 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Tue, 26 Feb 2013 01:29:23 +0000 (UTC)
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
References: <kgemhs$ldh$1@ger.gmane.org>
	<CAHVvXxRhbJmyEZL5_kqLXMDfOQHQMUJqX+8X20D-vHdquZD8=A@mail.gmail.com>
	<20130225160811.680ecc42@anarchist.wooz.org>
Message-ID: <loom.20130226T022753-156@post.gmane.org>

Barry Warsaw <barry at ...> writes:

> 
> On Feb 25, 2013, at 04:20 PM, Oscar Benjamin wrote:
> 
> >At times I thought I wanted the ability to query an iterator without
> >necessarily popping an element. I've generally found that I ended up
> >solving the problem in a different way. My own solution is to have a
> >pushable iterator so that after inspecting a value I can push it back
> >onto the iterator ready for a subsequent next() call.
> 
> The email package has one of these in its feedparser module, called
> BufferedSubFile.  It needs to be able to push lines of text back onto the
> stack that the normal iteration pops off.

It's also a common pattern in lexers/parsers/compilers where you need to push
back characters/tokens while doing lookahead for e.g. disambiguation.

Regards,

Vinay Sajip






From abarnert at yahoo.com  Tue Feb 26 03:56:24 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Feb 2013 18:56:24 -0800
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <5129D8A9.6000000@pearwood.info>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAGu0AnvfSXv6oc9_jzRZuO2S3f4voDhdCMjW=zTg=KZn3Rr0Eg@mail.gmail.com>
	<5129D8A9.6000000@pearwood.info>
Message-ID: <FE58B543-0D60-43AD-A851-D4C8E6C2DEE5@yahoo.com>

On Feb 24, 2013, at 1:08, Steven D'Aprano <steve at pearwood.info> wrote:

> On 24/02/13 19:04, Bruce Leban wrote:
> 
>> And hypothetically
> [...]
>> a, b, c = (next * 3)(iterargs)
> 
> That looks like, well, I'm not sure what it looks like. Haskell maybe? But not Python.

Not Haskell either. If you told me this was an expression in a language that's like Haskell but with python (or C) syntax, I'd expect it to mean something like compose_n(next, 3)(iterargs), aka next(next(next(iterargs))), not map(next, iterargs * 3).

Maybe next(3, iterargs) or next(3)(iterargs), but that's just islice.




From abarnert at yahoo.com  Tue Feb 26 04:21:06 2013
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Feb 2013 19:21:06 -0800
Subject: [Python-ideas] My wishlist for Tulip
In-Reply-To: <18F634D4-9C75-4FCC-9C33-917DCAE72CC8@gmail.com>
References: <CAFYqXL_GL_H3YiFO6yDAi_MV_u=SOesw-OP+51W21g0oyPix4Q@mail.gmail.com>
	<CAP7+vJJ8xM1EAAvDu9-Aq=NbLU-DHbF7wUUjdH22Yc72vHRKNA@mail.gmail.com>
	<0660447D-BC71-4B49-A2F6-9CE00643A958@yahoo.com>
	<18F634D4-9C75-4FCC-9C33-917DCAE72CC8@gmail.com>
Message-ID: <E7F71BF7-4EE2-4C36-A585-EDC5CAED9D77@yahoo.com>

On Feb 25, 2013, at 16:41, Nikolay Kim <fafhrd91 at gmail.com> wrote:

> You can check my experimental httpclient code. it has web socket example, server + cmd client.
> 
> https://github.com/fafhrd91/httpclient

How does that answer my request? 

The whole point was to show how passing information between socket handlers (or, more generally, coro chains) is easier with tulip than with traditional async solutions. An http client doesn't do that. (And it's probably much more complex, too.)

> On Feb 25, 2013, at 4:12 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> 
>> Sorry for the top post, but I'm on a phone.
>> 
>> I'd love to see a trivial chat server in the demos. There are many designs for async servers that work great with echo and even http but fall down with chat (because of interactions between clients), and it would be very useful to see how tulip makes it easy to do it right.
>> 
>> Sent from a random iPhone.
>> 
>> On Feb 22, 2013, at 13:17, Guido van Rossum <guido at python.org> wrote:
>> 
>>> Those are all good ideas! Now I just have to find time to implement them...
>>> 
>>> On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
>>>> I understand it's still beta but anyways, here's my little wish list for Tulip.
>>>> 
>>>> * provide a 'demo' / 'example' directory containing very simple
>>>> scripts showing the most basic usages such as:
>>>> 
>>>> - echo_tcp_client.py
>>>> - echo_tcp_server.py
>>>> - echo_tcp_server_w_timeout.py  (same as echo_server.py but also
>>>> disconnects the client after a certain time of inactivity)
>>>> - echo_tcp_ssl_client.py
>>>> - echo_tcp_ssl_server.py
>>>> - echo_udp_client.py
>>>> - echo_udp_server.py
>>>> 
>>>> * move all *test*.py scripts in a separate 'test' directory
>>>> 
>>>> * if it's not part of the API intended to be public move
>>>> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new
>>>> 'scripts'/'tools' directory)
>>>> 
>>>> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks
>>>> like they belong elsewhere
>>>> 
>>>> * write a simple benchmark framework testing (at least) sending,
>>>> receiving and the internal scheduler  (I'd like to help with this one)
>>>> 
>>>> 
>>>> --- Giampaolo
>>>> http://code.google.com/p/pyftpdlib/
>>>> http://code.google.com/p/psutil/
>>>> http://code.google.com/p/pysendfile/
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> http://mail.python.org/mailman/listinfo/python-ideas
>>> 
>>> 
>>> 
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> http://mail.python.org/mailman/listinfo/python-ideas
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> 


From ncoghlan at gmail.com  Tue Feb 26 04:50:18 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 26 Feb 2013 13:50:18 +1000
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <CAOyAWgjduM8-4LYRKH9iRyDhAmHQbC6axhLuF0P29wx=7sevmw@mail.gmail.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
	<CAOyAWgjVZuuXFGcdxXsUUr9s7T6kTv99bBOd-BDBRT5hMy2S7g@mail.gmail.com>
	<512BF5CC.4050602@canterbury.ac.nz>
	<CAOyAWgjduM8-4LYRKH9iRyDhAmHQbC6axhLuF0P29wx=7sevmw@mail.gmail.com>
Message-ID: <CADiSq7f39gr-QszVSg+jiwn9eAFZkWy3Xp7QgcRRHCqq6R2Xqw@mail.gmail.com>

On Tue, Feb 26, 2013 at 11:20 AM, Jo?o Bernardo <jbvsmo at gmail.com> wrote:
>
> 2013/2/25 Greg Ewing <greg.ewing at canterbury.ac.nz>
>>
>> Jo?o Bernardo wrote:
>>>
>>> Python already supports this odd syntax
>>>
>>>     a, b, *[] = iterable
>>>
>>> because it interprets the [] not as an empty list, but as an empty "list
>>> of identifiers". Maybe it could be used for something useful.
>>
>>
>> No, because it already has a meaning: there must be no more
>> values left in the sequence.
>>
>
> Why have two things with the same meaning?
>
>     a, b = iterable
>
>     a, b, *[] = iterable
>
> Both are the same... The *[] thing is about 100% useless right now.

And almost certainly an unintentional quirk of the way the Grammar is
constructed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From steve at pearwood.info  Tue Feb 26 05:03:22 2013
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 26 Feb 2013 15:03:22 +1100
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <512BFFAA.2020706@mrabarnett.plus.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<66901ffd88e8837eb58590472ad06fee@chopin.edu.pl>
	<CABicbJLt45hDCg4hNUVqUKu+Xmpnw6srQqeK+JHU2pa+YqEMww@mail.gmail.com>
	<512BF3FB.9090803@canterbury.ac.nz> <512BFCBD.40702@pearwood.info>
	<512BFFAA.2020706@mrabarnett.plus.com>
Message-ID: <512C340A.304@pearwood.info>

On 26/02/13 11:19, MRAB wrote:
> On 2013-02-26 00:07, Steven D'Aprano wrote:
>> On 26/02/13 10:30, Greg Ewing wrote:
>>> Devin Jeanpierre wrote:
>>>> On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>>>>>    a, b, *() = iterable
>>>>
>>>> Of course, you can't unpack anything into (), because Python never had
>>>> that syntax, but you get the idea.
>>>
>>> -1, this is just as arbitrary as ... or a lone *.
>>>
>>> I prefer ... myself, because visually it has a low profile
>>> and doesn't draw undue attention to something that you're
>>> saying you don't care about.
>>>
>>> Maybe ... could be blessed as an official "don't care"
>>> assignment target, like _ in Haskell. That would make this
>>> usage less of a special case (although it would still be
>>> partially special, since it suppresses unpacking of the
>>> rest of the iterator).
>>
>> Please no. _ is already overloaded too much. In i18n contexts, _ is
>> conventionally used as the function for getting display text. In the
>> interactive interpreter, _ is used for the last result. And in source
>> code, _ is often used by convention as a "don't care" variable.
>>
> [snip]
> He didn't say that we should use "_", he said that "..." would be like
> "_" in Haskell.


/facepalm

And so he did. Sorry about that Greg!

But still, I think this does help demonstrate that symbols are not
always the easiest to read at a glance. It's too easy for the eyes to
slide right off them, unless they are as familiar as basic arithmetic.


-- 
Steven


From pyideas at rebertia.com  Tue Feb 26 05:51:24 2013
From: pyideas at rebertia.com (Chris Rebert)
Date: Mon, 25 Feb 2013 20:51:24 -0800
Subject: [Python-ideas] Adding a safe alternative to pickle in the
 standard library
In-Reply-To: <CAGu0AnurHcM2vb3WTqFW5v744Q_wgiC5K1jfH34Ci3fks0k1QQ@mail.gmail.com>
References: <CABicbJJuUtygQ-btZvK2virFeks+5NWnXzDBqQrDC8y+yz8FUA@mail.gmail.com>
	<20130221121135.5a1aabd6@pitrou.net>
	<512621FF.6070606@trueblade.com>
	<20130221150049.2731682e@pitrou.net>
	<51262B08.3030600@trueblade.com>
	<CAJtE5vRRM6-Jr-My=Cu0e=DAab=YtUY9rdO9vqkU1QNiyhm8Qg@mail.gmail.com>
	<CABicbJ+1vs-BYrLrMG_RZhLhkLG3tbyCru+ivSnvhrYiBhH-aw@mail.gmail.com>
	<1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com>
	<0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com>
	<87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJ+AAFJ1QoGsvFx4_eBcWg+ODszNzgEuL=G80120GUaiQg@mail.gmail.com>
	<87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CABicbJK4DUzTT9T07c2Vpo0140Dx2ZyCE-QtZHyrA52u0JaH7Q@mail.gmail.com>
	<C330DB1C-A165-4C69-A5FB-4CDBA2697063@yahoo.com>
	<CABicbJKr8g8_yN5KYhF3CLwcLDBn9g2B5y9s99LU_sFhorw6pA@mail.gmail.com>
	<CAGu0AnurHcM2vb3WTqFW5v744Q_wgiC5K1jfH34Ci3fks0k1QQ@mail.gmail.com>
Message-ID: <CAMZYqRRVZZuwWzcbT+aAY3gr18m+STYPFyg8OV=-dCFxXY1-Qg@mail.gmail.com>

On Fri, Feb 22, 2013 at 11:08 AM, Bruce Leban <bruce at leapyear.org> wrote:
> To take this back to the ideas stage, one idea might be to integrate hmac
> into pickle. At a minimum, provide some sample code showing how to wrap an
> hmac around a pickled object.

This sounds very much like the `itsdangerous` library (which uses JSON
by default, but the serializer backend is pluggable):
http://pythonhosted.org/itsdangerous/

Cheers,
Chris


From storchaka at gmail.com  Tue Feb 26 10:19:00 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 26 Feb 2013 11:19:00 +0200
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <20130225212722.0d6de65b@pitrou.net>
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
	<kggh3g$o9t$1@ger.gmane.org> <20130225212722.0d6de65b@pitrou.net>
Message-ID: <kghum6$3pf$1@ger.gmane.org>

On 25.02.13 22:27, Antoine Pitrou wrote:
> On Mon, 25 Feb 2013 22:21:03 +0200
> Serhiy Storchaka <storchaka at gmail.com>
> wrote:
>> On 25.02.13 20:53, Antoine Pitrou wrote:
>>>>> def test_lookahead():
>>>>>        it = iter('abc')
>>>>>        while True:
>>>>>            it, peeking = itertools.tee(it)
>>>>
>>>> This should be outside a loop.
>>>
>>> Only if you restrict yourself to access peeking each time you access it.
>>> (which, I suppose, is not the general use case for the lookahead
>>> proposal)
>>
>> Only if your do not want to consume O(N) memory and spend O(N**2) time.
>
> No, that's beside the point. If you don't consume "peeking" in
> lock-step with "it", then "peeking" and "it" become desynchronized and
> therefore the semantics are wrong w.r.t to the original feature
> request (where "peeking" is supposed to be some proxy to "it", not an
> independently-running iterator).

Yes, of course, you should consume "peeking" in lock-step with "it".

My note is that if you create tee every iteration, this will lead to an 
linear increase in memory consumption and degradation of speed.



From masklinn at masklinn.net  Tue Feb 26 10:23:03 2013
From: masklinn at masklinn.net (Masklinn)
Date: Tue, 26 Feb 2013 10:23:03 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <kghum6$3pf$1@ger.gmane.org>
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
	<kggh3g$o9t$1@ger.gmane.org> <20130225212722.0d6de65b@pitrou.net>
	<kghum6$3pf$1@ger.gmane.org>
Message-ID: <FFC010F8-4199-429C-B6A8-EBEF897E1A5F@masklinn.net>


On 2013-02-26, at 10:19 , Serhiy Storchaka wrote:

> On 25.02.13 22:27, Antoine Pitrou wrote:
>> On Mon, 25 Feb 2013 22:21:03 +0200
>> Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>> On 25.02.13 20:53, Antoine Pitrou wrote:
>>>>>> def test_lookahead():
>>>>>>       it = iter('abc')
>>>>>>       while True:
>>>>>>           it, peeking = itertools.tee(it)
>>>>> 
>>>>> This should be outside a loop.
>>>> 
>>>> Only if you restrict yourself to access peeking each time you access it.
>>>> (which, I suppose, is not the general use case for the lookahead
>>>> proposal)
>>> 
>>> Only if your do not want to consume O(N) memory and spend O(N**2) time.
>> 
>> No, that's beside the point. If you don't consume "peeking" in
>> lock-step with "it", then "peeking" and "it" become desynchronized and
>> therefore the semantics are wrong w.r.t to the original feature
>> request (where "peeking" is supposed to be some proxy to "it", not an
>> independently-running iterator).
> 
> Yes, of course, you should consume "peeking" in lock-step with "it".
> 
> My note is that if you create tee every iteration, this will lead to an linear increase in memory consumption and degradation of speed.

I believe I also saw situations where I blew through the recursion limit
by stacking too many tees, but I can't reproduce it right now.

From storchaka at gmail.com  Tue Feb 26 10:44:36 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 26 Feb 2013 11:44:36 +0200
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <FFC010F8-4199-429C-B6A8-EBEF897E1A5F@masklinn.net>
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
	<kggh3g$o9t$1@ger.gmane.org> <20130225212722.0d6de65b@pitrou.net>
	<kghum6$3pf$1@ger.gmane.org>
	<FFC010F8-4199-429C-B6A8-EBEF897E1A5F@masklinn.net>
Message-ID: <kgi066$hgo$1@ger.gmane.org>

On 26.02.13 11:23, Masklinn wrote:
> On 2013-02-26, at 10:19 , Serhiy Storchaka wrote:
>> My note is that if you create tee every iteration, this will lead to an linear increase in memory consumption and degradation of speed.
>
> I believe I also saw situations where I blew through the recursion limit
> by stacking too many tees, but I can't reproduce it right now.

http://bugs.python.org/issue13454 ?



From masklinn at masklinn.net  Tue Feb 26 11:06:45 2013
From: masklinn at masklinn.net (Masklinn)
Date: Tue, 26 Feb 2013 11:06:45 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <kgi066$hgo$1@ger.gmane.org>
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
	<kggh3g$o9t$1@ger.gmane.org> <20130225212722.0d6de65b@pitrou.net>
	<kghum6$3pf$1@ger.gmane.org>
	<FFC010F8-4199-429C-B6A8-EBEF897E1A5F@masklinn.net>
	<kgi066$hgo$1@ger.gmane.org>
Message-ID: <5C12F640-6C24-49E7-A6CD-56A0DFC9199E@masklinn.net>

On 2013-02-26, at 10:44 , Serhiy Storchaka wrote:

> On 26.02.13 11:23, Masklinn wrote:
>> On 2013-02-26, at 10:19 , Serhiy Storchaka wrote:
>>> My note is that if you create tee every iteration, this will lead to an linear increase in memory consumption and degradation of speed.
>> 
>> I believe I also saw situations where I blew through the recursion limit
>> by stacking too many tees, but I can't reproduce it right now.
> 
> http://bugs.python.org/issue13454 ?

No, it was a recursion limit, I was trying to use iterators in a
situation where I needed multiline lookaheads and skipping so I stacked
at least 1 tee + 1 islice per item and it blew up at one point. I can't
reproduce it anymore and I can't find the original code (replacing it
with straight lists ended up being simpler)

Although I can trivially get a segfault with islice:

    from itertools import islice, count

    it = count()
    while True:
        it = islice(it, 0)

Run this, C-c at some point, every CPython version on my machine
segfaults (pypy does not)

From storchaka at gmail.com  Tue Feb 26 11:37:25 2013
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 26 Feb 2013 12:37:25 +0200
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <5C12F640-6C24-49E7-A6CD-56A0DFC9199E@masklinn.net>
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
	<kggh3g$o9t$1@ger.gmane.org> <20130225212722.0d6de65b@pitrou.net>
	<kghum6$3pf$1@ger.gmane.org>
	<FFC010F8-4199-429C-B6A8-EBEF897E1A5F@masklinn.net>
	<kgi066$hgo$1@ger.gmane.org>
	<5C12F640-6C24-49E7-A6CD-56A0DFC9199E@masklinn.net>
Message-ID: <kgi397$g0n$1@ger.gmane.org>

On 26.02.13 12:06, Masklinn wrote:
> Although I can trivially get a segfault with islice:
>
>      from itertools import islice, count
>
>      it = count()
>      while True:
>          it = islice(it, 0)
>
> Run this, C-c at some point, every CPython version on my machine
> segfaults (pypy does not)

Thank you. This looks as pretty same issue as with tee(). I knew that 
there must be more such catches, but could not find them.

http://bugs.python.org/issue17300




From masklinn at masklinn.net  Tue Feb 26 11:57:45 2013
From: masklinn at masklinn.net (Masklinn)
Date: Tue, 26 Feb 2013 11:57:45 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <kgi397$g0n$1@ger.gmane.org>
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
	<kggh3g$o9t$1@ger.gmane.org> <20130225212722.0d6de65b@pitrou.net>
	<kghum6$3pf$1@ger.gmane.org>
	<FFC010F8-4199-429C-B6A8-EBEF897E1A5F@masklinn.net>
	<kgi066$hgo$1@ger.gmane.org>
	<5C12F640-6C24-49E7-A6CD-56A0DFC9199E@masklinn.net>
	<kgi397$g0n$1@ger.gmane.org>
Message-ID: <361FF751-43F2-48D1-B4CB-3C4D76E70608@masklinn.net>


On 2013-02-26, at 11:37 , Serhiy Storchaka wrote:

> On 26.02.13 12:06, Masklinn wrote:
>> Although I can trivially get a segfault with islice:
>> 
>>     from itertools import islice, count
>> 
>>     it = count()
>>     while True:
>>         it = islice(it, 0)
>> 
>> Run this, C-c at some point, every CPython version on my machine
>> segfaults (pypy does not)
> 
> Thank you. This looks as pretty same issue as with tee(). I knew that there must be more such catches, but could not find them.
> 
> http://bugs.python.org/issue17300
> 

Cool.

And I looked back in the VCS, turns out the code hasn't been lost but
the issue was not in the stdlib, it was a custom iterator (used as a
wrapper for a bunch of operations) which needed to be reapplied very
often (code basically went Iterator -> mix of tee, chain and
dropwhile -> Iterator -> same mix), essentially doing the following:

import itertools

_placeholder = object()
class It(object):
    def __init__(self, stream):
        self.stream = iter(stream)
        self.stopped = False
    def __iter__(self):
        return self
    def __next__(self):
        if self.stopped: raise StopIteration()
        val = next(self.stream, _placeholder)
        if val is _placeholder:
            self.stopped = True
            raise StopIteration()
        return val

it = itertools.count()
while True:
    it = It(it)
    next(it)

I'm not sure if there's any way to implement such a wrapping iterator in
a way which does not ultimately blow the stack (save in C taking after
itertools implementations I guess as they don't seem to have the issue)

From solipsis at pitrou.net  Tue Feb 26 12:15:22 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Feb 2013 12:15:22 +0100
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
References: <kgemhs$ldh$1@ger.gmane.org> <20130225110717.232ce082@pitrou.net>
	<kgfjm3$93f$1@ger.gmane.org> <20130225195302.45704b36@pitrou.net>
	<kggh3g$o9t$1@ger.gmane.org> <20130225212722.0d6de65b@pitrou.net>
	<kghum6$3pf$1@ger.gmane.org>
Message-ID: <20130226121522.55d3fcae@pitrou.net>

Le Tue, 26 Feb 2013 11:19:00 +0200,
Serhiy Storchaka <storchaka at gmail.com> a
?crit :
> On 25.02.13 22:27, Antoine Pitrou wrote:
> > On Mon, 25 Feb 2013 22:21:03 +0200
> > Serhiy Storchaka <storchaka at gmail.com>
> > wrote:
> >> On 25.02.13 20:53, Antoine Pitrou wrote:
> >>>>> def test_lookahead():
> >>>>>        it = iter('abc')
> >>>>>        while True:
> >>>>>            it, peeking = itertools.tee(it)
> >>>>
> >>>> This should be outside a loop.
> >>>
> >>> Only if you restrict yourself to access peeking each time you
> >>> access it. (which, I suppose, is not the general use case for the
> >>> lookahead proposal)
> >>
> >> Only if your do not want to consume O(N) memory and spend O(N**2)
> >> time.
> >
> > No, that's beside the point. If you don't consume "peeking" in
> > lock-step with "it", then "peeking" and "it" become desynchronized
> > and therefore the semantics are wrong w.r.t to the original feature
> > request (where "peeking" is supposed to be some proxy to "it", not
> > an independently-running iterator).
> 
> Yes, of course, you should consume "peeking" in lock-step with "it".
> 
> My note is that if you create tee every iteration, this will lead to
> an linear increase in memory consumption and degradation of speed.

Apparently, itertools.tee() is optimized for tee'ing a tee:

>>> it = iter("abc")
>>> id(it)
31927760
>>> it, peeking = itertools.tee(it)
>>> id(it)
31977128
>>> it, peeking = itertools.tee(it)
>>> id(it)
31977128

Regards

Antoine.




From demianbrecht at gmail.com  Tue Feb 26 22:18:21 2013
From: demianbrecht at gmail.com (Demian Brecht)
Date: Tue, 26 Feb 2013 13:18:21 -0800
Subject: [Python-ideas] FileCookieJar (and concrete implementations).
Message-ID: <CAE+T62aDwPZwM4G=R76UxCiAmb_st4SQrFF2dzn_8_irF3zfGg@mail.gmail.com>

Context: http://bugs.python.org/issue16942 (my patch, changing
FileCookieJar to be an abc, defining the interfaces for
*FileCookieJar).

This pertains to Terry's question about whether or not it makes sense
that an abstract base class extends a concrete class. After putting in
a little thought, he's right. It doesn't make sense.

After further thought, I'm relatively confident that the hierarchy as
it stands should be changed. Currently what's implemented in the
stdlib looks like this:

CookieJar
|
FileCookieJar
| |
| MozillaCookieJar
LWPCookieJar

What I'm proposing is that the structure is broken to be the following:

FileCookieJarProcessor             CookieJar
| |
| MozillaCookieJarProcessor
LWPCookieJarProcessor

The intention here is to have processors that operate /on/ a cookiejar
object via composition rather than inheritance. This aligns with how
urllib.request.HTTPCookieProcessor works (which has the added bonus of
cross-module consistency).

The only attributes that concrete FileCookieJarProcessor classes touch
(at least, in the stdlib) are _cookies and _cookies_lock. I have mixed
feelings about whether these should continue to be noted as
"non-public" with the _ prefix or not as keeping the _ would break
convention of operating on non-public fields, but am unsure of the
ramifications of changing them to public.

Making this change then allows for FileCookieJar(Processor) to be an
abstract base class without inheriting from CookieJar which doesn't
make a whole lot of sense from an architecture standpoint.

I have yet to see what impact these changes have to the cookiejar
extensions at http://wwwsearch.sf.net but plan on doing so if this
approach seems sound.

This will obviously break backwards compatibility, so I'm not entirely
sure what best practice is around that: leave well enough alone even
though it might not make sense, keep the old implementations around
and deprecate them to be eventually replaced by the processors, or
other ideas?

--
Demian Brecht
http://demianbrecht.github.com


From tjreedy at udel.edu  Tue Feb 26 22:57:36 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Feb 2013 16:57:36 -0500
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <512B8370.6040807@gmail.com>
References: <kgemhs$ldh$1@ger.gmane.org> <512B8370.6040807@gmail.com>
Message-ID: <kgjb4j$fam$1@ger.gmane.org>

On 2/25/2013 10:29 AM, Ron Adam wrote:
>
>
> On 02/24/2013 09:41 PM, Terry Reedy wrote:

>> class lookahead():
>>      "Wrap iterator with lookahead to both peek and test exhausted"
>>
>>      _NONE = object()
>>      def __init__(self, iterable):
>>          self._it = iter(iterable)
>>          self._set_peek()
>>      def __iter__(self):
>>          return self
>>      def __next__(self):
>>          ret = self.peek
>>          self._set_peek()
>>          return ret
>>      def _set_peek(self):
>>          try:
>>              self.peek = next(self._it)
>>          except StopIteration:
>>              self.peek = self._NONE
>>      def __bool__(self):
>>          return self.peek is not self._NONE
>
>>
>> def test_lookahead():
>>     it = lookahead('abc')
>>     while it:
>>         a = it.peek
>>         b = next(it)
>>         print('next:', b, '; is peek:', a is b )
>>
>> test_lookahead()

Since revised.

> I think with a few small changes I would find it useful.
>
> The key feature here is that the result is pre calculated and held until
> it's needed, rather than calculated when it's asked for.
>
> You should catch any exception and hold that as well.  On the next
> .next() call, it should raise the exception if there was one, or emit
> the value.

My revised version has

     def _set_peek(self):
         try:
             self.peek = next(self._it)
         except StopIteration:
             self.peek = self._NONE

That could easily be changed to e

         except Exception as e:
             self.peek = self._NONE
             self._error = e

__next would then raise self._error instead of explicitly StopIteration. 
I do not especially like the redundancy of two 'exhausted' indicators 
and thought of storing e as self.peek, but an iterator can legitimately 
yield exception classes and instances.

I think some argument can be made that if the iterator is broken, the 
exception should be raised immediately even if it means not returning 
the last item. No user should be expecting anything other than 
StopIteration.

> I'm not sure if using the __bool__ attribute is the best choice.

I am ;-). It hides the test for exhaustion, which could change, without 
complication.

> I would prefer a .error flag, along with a .next_value attribute.  It
> would make the code using it easier to follow.

Not clear to me, but a minor detail.

>       it.error        <--    True if next(it) will raise an exception.
>       it.next_value   <--    The next value, or the exception to raise.


> About it.error.  If it was a concurrent version, then it.error could
> have three values.
>
>       it.error == True   # Will raise an exception
>       it.error == False  # Will not raise an exception
>       it.error == None   # Still calculating
>
> I wonder how this type of generator will behave with "yield from".

lookaheads are iterators, but not generators and 'yield from' requires a 
generator. A generator function could recompute items to yield, but it 
cannot add methods or attributes to the generator instances it will produce.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Wed Feb 27 01:44:26 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Feb 2013 19:44:26 -0500
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <a9935a92-d93b-4d28-9db8-fdad86a127a1@googlegroups.com>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>
	<44528db0ff81ea147dfc71d97e472798@chopin.edu.pl>
	<512A7967.1060709@mrabarnett.plus.com>
	<d31a51ec-9ca5-4653-ae4f-0328d096a79a@googlegroups.com>
	<kgemlt$ldh$2@ger.gmane.org>
	<a9935a92-d93b-4d28-9db8-fdad86a127a1@googlegroups.com>
Message-ID: <kgjktf$7m6$1@ger.gmane.org>

On 2/25/2013 5:25 PM, Alex Stewart wrote:

Negative putdowns are off-topic and not persuasive. I starting with 1.3 
in March 1997 and first posted a month later. While cautious about 
changes and additions, I have mostly been in favor of those that have 
been released. I started using 3.0 in beta and starting using 3.3.0 with 
the first alpha to get the new stuff.

> On Sunday, February 24, 2013 7:43:58 PM UTC-8, Terry Reedy wrote:
>     The related but distinct concepts of sequential access and random
>     access
>     are basic to computation and were *always* distinct in Python.
>
> Oh really?

Yes really!

 > Then how, prior to the development of the iterator protocol,
> did one define an object which was accessible sequentially but not
> randomly in the Python language?

As I said, by using the original fake-getitem iterator protocol, which 
still works, instead of the newer iter-next iterator protocol.

Take any python-coded iterator class, such as my lookahead class. Remove 
or comment out the 'def __iter__(self): return self' statement. Change 
the header line 'def __next__(self):' to 'def __getitem__(self, n):'. 
Instances of the revised class will *today* work with for statements. 
Doing this with lookahead (3.3):

 >>> for item in lookahead('abc'): print(item)

a
b
c
 >>> dir(lookahead)
['_NONE', '__bool__', '__class__', '__delattr__', '__dict__', '__dir__', 
'__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', 
'__getitem__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', 
'__module__', '__ne__', '__new__', '__qualname__', '__reduce__', 
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', 
'__subclasshook__', '__weakref__', '_set_peek']

See, no __iter__, no __next__, but a __getitem__ that ignores the index 
passed. Here is where it gets a bit crazy.

 >>> s = lookahead('abc')
 >>> s[0]
'a'
 >>> s[0], s[300]
('b', 'c')
 >>> s[0]
Traceback...
StopIteration

Still, I think enabling generators was more of a motivation than 
discouraging nonsense like the above (which, obviously, is still 
possible! but also which people did not intentionally do except as a 
demonstration).

A practical difference between sequential and random access is that 
sequential access usually requires history ('state'), which random 
access does not. The __iter__ method and iter() function allows the 
separation of iterators, with iteration state, from an underlying 
concrete collections (if there is one), which usually does not need the 
iteration state. (Files, which *are* iterators and which do not have a 
separate iterator class, are unusual among builtins.) This enables 
multiple stateful iterators and nested for loops like this:

 >>> s = 'abc'
 >>> for c in lookahead(s):
	for d in lookahead(s):
		print((c,d))
		
('a', 'a')
('a', 'b')
('a', 'c')
('b', 'a')
('b', 'b')
('b', 'c')
('c', 'a')
('c', 'b')
('c', 'c')


 > If you couldn't do that, you can't
> claim that the concepts were really distinct in Python, in my opinion.

But I and anyone could and still can, so I can and will make the claim 
and reject arguments based on the false counter-claim.

---
Back to unpack and informing the source as to the number n of items 
expected to be requested.

Case 1. The desired response of the source to n is generic.

Example a: we want the source to simply refuse to produce more than the 
number specified at the start. The current generic solutions are to 
either make the exact number of explicit next() calls needed or to wrap 
the iterator in islice(iterator, n) (which in turn will make the number 
of next() calls needed).

Example b : if the source does yield n items, we want it to then yield 
the residual iterator. Again, a simple generic wrapper does the job.

import itertools

def slice_it(iterable, n):
     "Yield up to n items from iterable and if successful, the residual 
iterator:"

     it = iter(iterable)
     for i in range(n):
         yield next(it)
     else:
         yield it

a, b, c, rest = slice_it(itertools.count(), 3)
print(a, b, c, rest)
d, e = itertools.islice(rest, 2)
print(d, e)
 >>>
0 1 2 count(3)
3 4

Conclusion: generic iterator behavior modification should be done by 
generic wrappers. This is the philosophy and practice of comprehensions, 
built-in wrappers (enumerate, filter, map, reversed, and zip), and 
itertools.

Case 2. The desired response is specific to a class or even each instance.

Example: object has attributes a, b, c, d (of lesser importance), and 
can calculate e. The pieces and structures it should yield depend on n 
as in the following table.

1  ((a,b),c)
2  (a,b), c
3  a, b, c
4  a, b, c, d
5  a, b, c, d, e

Solution: write a method, call it .unpack(n), that returns an iterator 
that will produce the objects specified in the table. This can be done 
today with no change to Python. It can be done whether or not there is a 
.__iter__ method to produce a generic default iterator for the object. 
And, of course, xxx.unpack can have whatever signature is appropriate to 
xxx. It seems to me that this procedure can handle any special 
collection or structure breakup need.

Comment 1: if people can pass explicit counts to islice (or slice_it), 
they can pass explicit counts to .unpack.

Comment 2: we are not disagreeing that people might want to do custom 
count-dependent disassembly or that they should be able to do so. It can 
already be done.

Areas of disagreement:

1. consumer-source interdependency: you seem to think there is something 
special about the consumer assigning items to multiple targets in one 
statement, as opposed to doing anything else, including doing the 
multiple assignments in multiple statements. I do not. Moreover, I so 
far consider introducing such dependency in the core to be a regression.

2. general usefulness: you want .unpack to be standardized and made a 
special method. I think it is inherently variable enough and and the 
need rare enough to not justify that.

I retrieved your original post and plan to look at it sometime to see if 
I missed anything not covered by the above. But enough for today.

-- 
Terry Jan Reedy



From ron3200 at gmail.com  Wed Feb 27 02:50:17 2013
From: ron3200 at gmail.com (Ron Adam)
Date: Tue, 26 Feb 2013 19:50:17 -0600
Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools
In-Reply-To: <kgjb4j$fam$1@ger.gmane.org>
References: <kgemhs$ldh$1@ger.gmane.org> <512B8370.6040807@gmail.com>
	<kgjb4j$fam$1@ger.gmane.org>
Message-ID: <512D6659.1080802@gmail.com>



On 02/26/2013 03:57 PM, Terry Reedy wrote:

>> I would prefer a .error flag, along with a .next_value attribute.  It
>> would make the code using it easier to follow.
>
> Not clear to me, but a minor detail.
>
>>       it.error        <--    True if next(it) will raise an exception.
>>       it.next_value   <--    The next value, or the exception to raise.
>
>
>> About it.error.  If it was a concurrent version, then it.error could
>> have three values.
>>
>>       it.error == True   # Will raise an exception
>>       it.error == False  # Will not raise an exception
>>       it.error == None   # Still calculating
>>
>> I wonder how this type of generator will behave with "yield from".
>
> lookaheads are iterators, but not generators and 'yield from' requires a
> generator. A generator function could recompute items to yield, but it
> cannot add methods or attributes to the generator instances it will produce.

Yep, I missed that point.

I was thinking it may be more useful in the case of generators along with 
yield from, but it isn't as straight forward as the iterator case as you 
correctly pointed out.

(More or less just thinking out loud at this point.)

My intuition/feelings was that in both the cases of a look-ahead iterator 
and a calc-ahead generator, (if we could manage that), should act the same 
in as many ways as possible.   And a calc-ahead generator should act like a 
concurrent instance.   The reason for thinking that is, it would allow the 
calc-ahead generator instance to be replaced with a concurrent instance 
without changing anything.

So if (in theory) a concurrent generator instance wouldn't hold exceptions, 
then your look-ahead iterator shouldn't also.  But if everything happens on 
the next and send calls, then it makes handling exceptions a bit easier as 
you don't have to special case the instance creation part.

But iterators don't necessarily need to match all generator behaviours, 
It's just may preference that they do as much as possible.  :-)

Of course Guido may have something in what he's doing now that would be 
very much like this.  A kind of "futures" generator.  I've been away from 
these boards for a while and haven't caught up with everything yet.

Cheers,
    Ron



From raymond.hettinger at gmail.com  Wed Feb 27 05:46:29 2013
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 26 Feb 2013 20:46:29 -0800
Subject: [Python-ideas] range
In-Reply-To: <CADiSq7f5MMyEU6sBUw3UJcuyK1XuEDwJ5H3+yDUpG6xLtf09fA@mail.gmail.com>
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
	<CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>
	<loom.20130222T183719-360@post.gmane.org>
	<20130222194310.75129c1f@pitrou.net> <kga1n0$apn$2@ger.gmane.org>
	<CADiSq7f5MMyEU6sBUw3UJcuyK1XuEDwJ5H3+yDUpG6xLtf09fA@mail.gmail.com>
Message-ID: <F417726D-E68F-4184-8D7A-3FF19B58FEB9@gmail.com>


On Feb 23, 2013, at 3:11 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>  I suspect for many cases involving more advanced
> range variants, containment would be a better option than inheritance,
> particular if you want to implement a type that can't be reliably
> described through a (start, stop, step) triple.

I concur.  Subclassing range() is just asking for trouble.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130226/7495e108/attachment.html>

From tkluck at infty.nl  Wed Feb 27 12:27:14 2013
From: tkluck at infty.nl (Timo Kluck)
Date: Wed, 27 Feb 2013 12:27:14 +0100
Subject: [Python-ideas] easier subprocess piping and redirection
Message-ID: <CAGK+T_kxgzFpu8KS_Wxbq0cYscw0De0+5ea+-LrRr0UX8MdgYA@mail.gmail.com>

Dear python enthousiasts,

While replacing some of my bash scripts by python scripts, I found the
following useful. I wrote a small utility module that allows piping
input and output in a very similar way to how you do it in bash. The
major advantage over just using shell=True is that this does not
expose us to the latter's security risks and quoting hazards.

The major advantage of using a pipe instead of just manually feeding
the output of one process to the other is that with a pipe, it doesn't
all have to fit into memory at once. However, the Python way of doing
that is rather cumbersome:

    >>> from subprocess import Popen, PIPE
    >>> p1 = Popen(['echo', 'a\nb'], stdout=PIPE)
    >>> p2 = Popen([head], '-n1'], stdin=p1.stdout, stdout=PIPE)
    >>> p2.communicate()
    ('a\n', None)

I've been able to replace this by:

    >>> from pipeutils import call
    >>> (call('echo', 'a\nb') | call('head', '-n1')).output()
    'a\n'

And similarly, I can do direct file redirects like this:

    >>> (call('echo', 'Hello world!') > 'test.txt').do()
    0
    >>> (call('cat') < 'test.txt').output()
    'Hello world!\n'

I think that this is a lot more concise and readable. As far as I'm
concerned, this was the only reason I would ever use bash scripts
instead of python scripts.

I would love to hear opinions on this. How would people like this as a library?

The source is online at https://bitbucket.org/tkluck/pipeutils

Best, Timo


From stefan at drees.name  Wed Feb 27 12:57:41 2013
From: stefan at drees.name (Stefan Drees)
Date: Wed, 27 Feb 2013 12:57:41 +0100
Subject: [Python-ideas] easier subprocess piping and redirection
In-Reply-To: <CAGK+T_kxgzFpu8KS_Wxbq0cYscw0De0+5ea+-LrRr0UX8MdgYA@mail.gmail.com>
References: <CAGK+T_kxgzFpu8KS_Wxbq0cYscw0De0+5ea+-LrRr0UX8MdgYA@mail.gmail.com>
Message-ID: <512DF4B5.5040707@drees.name>

Hi Timo,

On 27.02.13 12:27, Timo Kluck wrote:
> Dear python enthousiasts,
>
> While replacing some of my bash scripts by python scripts, I found the
> following useful. I wrote a small utility module that allows piping
> input and output in a very similar way to how you do it in bash. The
> major advantage over just using shell=True is that this does not
> expose us to the latter's security risks and quoting hazards.
>
> The major advantage of using a pipe instead of just manually feeding
> the output of one process to the other is that with a pipe, it doesn't
> all have to fit into memory at once. However, the Python way of doing
> that is rather cumbersome:
>
>      >>> from subprocess import Popen, PIPE
>      >>> p1 = Popen(['echo', 'a\nb'], stdout=PIPE)
>      >>> p2 = Popen([head], '-n1'], stdin=p1.stdout, stdout=PIPE)
>      >>> p2.communicate()
>      ('a\n', None)
>
> I've been able to replace this by:
>
>      >>> from pipeutils import call
>      >>> (call('echo', 'a\nb') | call('head', '-n1')).output()
>      'a\n'
>

thanks a lot for sharing this. How does this relate to plumbum :-?

	>>> from plumbum.cmd import echo, head
	>>> chain = echo['a\nb'] | head['-n1']
	>>> chain()
	u'a\n'

($> pip install plumbum) ... http://plumbum.readthedocs.org/en/latest/

> And similarly, I can do direct file redirects like this:
>
>      >>> (call('echo', 'Hello world!') > 'test.txt').do()
>      0
>      >>> (call('cat') < 'test.txt').output()
>      'Hello world!\n'
>
> I think that this is a lot more concise and readable. As far as I'm
> concerned, this was the only reason I would ever use bash scripts
> instead of python scripts.
>
> I would love to hear opinions on this. How would people like this as a library?
>
> The source is online at https://bitbucket.org/tkluck/pipeutils
> ...

All the best,
Stefan.


From tkluck at infty.nl  Wed Feb 27 13:20:17 2013
From: tkluck at infty.nl (Timo Kluck)
Date: Wed, 27 Feb 2013 13:20:17 +0100
Subject: [Python-ideas] easier subprocess piping and redirection
In-Reply-To: <512DF4B5.5040707@drees.name>
References: <CAGK+T_kxgzFpu8KS_Wxbq0cYscw0De0+5ea+-LrRr0UX8MdgYA@mail.gmail.com>
	<512DF4B5.5040707@drees.name>
Message-ID: <CAGK+T_kZ-GpMoaKMua5v2JJ33PuAWT7ebOAvs40+=XxKTQJjLw@mail.gmail.com>

2013/2/27 Stefan Drees <stefan at drees.name>:
>
> thanks a lot for sharing this. How does this relate to plumbum :-?
>
>         >>> from plumbum.cmd import echo, head
>         >>> chain = echo['a\nb'] | head['-n1']
>         >>> chain()
>         u'a\n'

Fantastic! I kind of guessed that it would have existed already, but
didn't know where to find it. Thanks for pointing this out!

Timo


From phd at phdru.name  Wed Feb 27 14:23:29 2013
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 27 Feb 2013 17:23:29 +0400
Subject: [Python-ideas] easier subprocess piping and redirection
In-Reply-To: <CAGK+T_kZ-GpMoaKMua5v2JJ33PuAWT7ebOAvs40+=XxKTQJjLw@mail.gmail.com>
References: <CAGK+T_kxgzFpu8KS_Wxbq0cYscw0De0+5ea+-LrRr0UX8MdgYA@mail.gmail.com>
	<512DF4B5.5040707@drees.name>
	<CAGK+T_kZ-GpMoaKMua5v2JJ33PuAWT7ebOAvs40+=XxKTQJjLw@mail.gmail.com>
Message-ID: <20130227132329.GA18897@iskra.aviel.ru>

On Wed, Feb 27, 2013 at 01:20:17PM +0100, Timo Kluck <tkluck at infty.nl> wrote:
> 2013/2/27 Stefan Drees <stefan at drees.name>:
> >
> > thanks a lot for sharing this. How does this relate to plumbum :-?
> >
> >         >>> from plumbum.cmd import echo, head
> >         >>> chain = echo['a\nb'] | head['-n1']
> >         >>> chain()
> >         u'a\n'
> 
> Fantastic! I kind of guessed that it would have existed already, but
> didn't know where to find it. Thanks for pointing this out!

   Very easy -- search for packages at Python Package Index, PyPI:

https://pypi.python.org/pypi
https://pypi.python.org/pypi?%3Aaction=search&term=pipe&submit=search

   Add your package to the Index so other people can find it.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.


From tjreedy at udel.edu  Wed Feb 27 21:43:39 2013
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 27 Feb 2013 15:43:39 -0500
Subject: [Python-ideas] range
In-Reply-To: <F417726D-E68F-4184-8D7A-3FF19B58FEB9@gmail.com>
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
	<CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>
	<loom.20130222T183719-360@post.gmane.org>
	<20130222194310.75129c1f@pitrou.net> <kga1n0$apn$2@ger.gmane.org>
	<CADiSq7f5MMyEU6sBUw3UJcuyK1XuEDwJ5H3+yDUpG6xLtf09fA@mail.gmail.com>
	<F417726D-E68F-4184-8D7A-3FF19B58FEB9@gmail.com>
Message-ID: <kglr5u$6mn$1@ger.gmane.org>

On 2/26/2013 11:46 PM, Raymond Hettinger wrote:
>
> On Feb 23, 2013, at 3:11 AM, Nick Coghlan
> <ncoghlan at gmail.com
> <mailto:ncoghlan at gmail.com>> wrote:
>
>> I suspect for many cases involving more advanced
>> range variants, containment would be a better option than inheritance,
>> particular if you want to implement a type that can't be reliably
>> described through a (start, stop, step) triple.
>
> I concur.  Subclassing range() is just asking for trouble.

The four builtins that cannot be subclassed are bool, memoryview, range, 
slice. This is documented for bool; I propose to do so for the others in 
http://bugs.python.org/issue17279. Would it be correct to say (now) that 
all 4 are intentional omissions? and not merely oversights?

-- 
Terry Jan Reedy



From foogod at gmail.com  Wed Feb 27 21:58:47 2013
From: foogod at gmail.com (Alex Stewart)
Date: Wed, 27 Feb 2013 12:58:47 -0800
Subject: [Python-ideas] iterable.__unpack__ method
In-Reply-To: <kgjktf$7m6$1@ger.gmane.org>
References: <c2d617bf-e2e1-40fe-b50b-7746aba06946@googlegroups.com>
	<kga1b9$apn$1@ger.gmane.org>
	<CAPTjJmp==jidQSzK6yPXY2NY9Mi0QqpUuhTX3iy9MVaG5yAUrQ@mail.gmail.com>
	<CADiSq7fgqhefNGDBQALEvTwh_8WvVKqgCWYgyrOwopMLXde0dA@mail.gmail.com>
	<kgboat$jhs$1@ger.gmane.org>
	<CADiSq7dyzeferPZoPj-MmMepa3bHqSA6m+7kgnin+-2pjZQv3g@mail.gmail.com>
	<CADiSq7dhcJggX96Nn_Q_YQHfQNO+fGZYGv2LCHpqAYGPjGBkgQ@mail.gmail.com>
	<CAP7+vJJYH5Yb02HPjRbE5Vo14__XP4cPqOVUQ_ysO9bdS+pYyA@mail.gmail.com>
	<CAHVvXxRBupZgh7xzai+DP=rr8-6V_wUJ5jrvWiBike+n0E1JRQ@mail.gmail.com>
	<44528db0ff81ea147dfc71d97e472798@chopin.edu.pl>
	<512A7967.1060709@mrabarnett.plus.com>
	<d31a51ec-9ca5-4653-ae4f-0328d096a79a@googlegroups.com>
	<kgemlt$ldh$2@ger.gmane.org>
	<a9935a92-d93b-4d28-9db8-fdad86a127a1@googlegroups.com>
	<kgjktf$7m6$1@ger.gmane.org>
Message-ID: <CACxLcFf0DeZSXUuKKG-gaRAZKLC6HUZjJzV7o+FDZdW14dhLcQ@mail.gmail.com>

On Tue, Feb 26, 2013 at 4:44 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 2/25/2013 5:25 PM, Alex Stewart wrote:
>
> Negative putdowns are off-topic and not persuasive.


None of my statements were intended to be put-downs, and if it came across
that way I apologize.  It is possible that my frustration showed through on
occasion and my language was harsher than it should have been..


> I starting with 1.3 in March 1997 and first posted a month later.


For what it's worth (not that I think it's terribly important, really),
while I have not generally participated much in online discussions over the
years, I actually started using Python about the same time you did, and
have watched it evolve over a similar timespan..

> Then how, prior to the development of the iterator protocol,
>
> did one define an object which was accessible sequentially but not
>> randomly in the Python language?
>>
>
> As I said, by using the original fake-getitem iterator protocol, which
> still works, instead of the newer iter-next iterator protocol.
>

[large snip]

Ok, yes, it was technically possible with ugly hacks to do it, but I don't
consider that to be really the same as "supported by the language".  The
only way you could do it was by making an object "fake" a random-access
interface that actually likely did the wrong thing if accessed randomly.
 That is not a separation of concepts, that is a twisting of one concept in
an incompatible way into the semblance of another as a workaround to the
fact that the two concepts are not actually distinct in the language
design.  Once again, you're confusing "how it's thought of in the
particular programmer's mind" with "how the language is actually designed",
which are not the same thing.

Back to unpack and informing the source as to the number n of items
> expected to be requested.
>
> Case 1. The desired response of the source to n is generic.
>

If I'm reading you right, I believe this is basically equivalent to "the
consumer decides what behavior is most useful".  In this case, yes, the
correct place to do this is on the consumer side, which is not what
__unpack__ is intended to be for anyway (fairly obviously, since it is not
a consumer-side change).

(For what it's worth, you seem to do a decent job here of arguing against
your own propositions to change the unpack syntax, though..  Not sure if
that was intentional..)


> Case 2. The desired response is specific to a class or even each instance.
>

[...]

Solution: write a method, call it .unpack(n), that returns an iterator that
> will produce the objects specified in the table. This can be done today
> with no change to Python. It can be done whether or not there is a
> .__iter__ method to produce a generic default iterator for the object. And,
> of course, xxx.unpack can have whatever signature is appropriate to xxx. It
> seems to me that this procedure can handle any special collection or
> structure breakup need.
>

This solution works fine, in the very restrictive case where the programmer
knows exactly what type of object they've been given to work with, is aware
that it provides this interface, and knows that it's important in that
particular case to use it for that particular type of object.  If the
consumer wants to be able to use both this type of object and other ones
that don't provide that interface, then their code suddenly becomes a lot
more complicated (having to check the type, or methods of the object, and
conditionally do different things).  Even if they decide to do that, since
there is no established standard (or even convention) for this sort of
thing, they then run the risk of being given an object which has an
"unpack" method with a completely different signature, or worse yet an
object which defines "unpack" that does some completely different
operation, unrelated to this ad-hoc protocol.  Any time somebody produces a
new "smart unpackable" object that doesn't work quite the same as the
others (either deliberately, or just because the programmer didn't know
that other people were already doing the same thing in a different way), it
is quite likely that all of the existing consumer code everywhere will have
to be rewritten to support it, or (much more likely) it will be supported
haphazardly some places and not others, leading to inconsistent behavior or
outright bugs.

Even ignoring all of this, it still isn't possible to write an object which
has advanced unpacking behavior that works with any existing unpacking
consumers, such as libraries or other code not under the object-designer's
control.

In short, yes, it solves things, in a very limited, incompatible, painful,
and largely useless way.

Comment 2: we are not disagreeing that people might want to do custom
> count-dependent disassembly or that they should be able to do so. It can
> already be done.
>

I disagree that it can already be done in the manner I'm describing, as
I've explained.  There is, frankly, no existing mechanism which allows an
unpacking-producer to do this sort of thing in a standard, consistent, and
interchangeable way, and there is also no way at all to do it in a way that
is compatible with existing consumer code.


> Areas of disagreement:
>
> 1. consumer-source interdependency: you seem to think there is something
> special about the consumer assigning items to multiple targets in one
> statement, as opposed to doing anything else, including doing the multiple
> assignments in multiple statements.


This is not a matter of opinion.  It is a *fact* that there is a difference
in this case.  The difference is, quite simply, that through the use of the
unpacking construct, the programmer has given the Python interpreter
additional information (which the interpreter is not providing to the
producing object).  The disagreement appears to be that you believe for
some reason it's a good thing to silently discard this information so that
nobody can make use of it, whereas I believe it would be beneficial to make
it available for those who want to use it.

Again, I feel compelled to point out that as far as I can tell, your entire
objection on this count boils down to "unpacking must always be the same
thing as iteration because that's the way I've always thought about it".
 Unpacking *currently* uses iteration under the covers because it is a
convenient interface that already exists in the language, but there is
absolutely no reason why unpacking *must* inherently be defined as the same
thing as iteration.  You talk about it as if this is a foregone conclusion,
but as far as I can tell it's only foregone because you've already
arbitrarily decided it to be one way and just won't listen to anybody
suggesting anything else.

Alternately, if you just can't manage to get past this "unpacking must mean
iteration" thing, then don't look at this as a change to unpacking.  Look
at it instead as an extension to the iterator protocol, which allows an
iteration consumer to tell the iteration producer more information about
the number of items they are wanting to obtain from it.  Heck, I actually
wouldn't be opposed to making this a general feature of iter(), if it could
be done in a backwards-compatible way..

2. general usefulness: you want .unpack to be standardized and made a
> special method. I think it is inherently variable enough and and the need
> rare enough to not justify that.
>

I think I already spoke to this above..  Put simply, if it is not
standardized and utilized by the corresponding language constructs, it is
essentially useless as a general-purpose solution and only works in very
limited cases.  Your alternative solutions just aren't solutions to the
more general problems.

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130227/b951e158/attachment.html>

From ericsnowcurrently at gmail.com  Thu Feb 28 11:15:14 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 28 Feb 2013 03:15:14 -0700
Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace
Message-ID: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>

Just wanted to put out some feelers for the feasibility of these two
features:

* have the **kwargs param be an OrderedDict rather than a dict
* have class definitions happen relative to an OrderedDict by default
rather than a dict, and still overridable by a metaclass's __prepare__().

Both of these will need OrderedDict in C, which is getting close (issue
#16991).

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130228/7ea005f7/attachment.html>

From solipsis at pitrou.net  Thu Feb 28 11:27:42 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 11:27:42 +0100
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
Message-ID: <20130228112742.09f5500b@pitrou.net>

Le Thu, 28 Feb 2013 03:15:14 -0700,
Eric Snow <ericsnowcurrently at gmail.com> a
?crit :
> Just wanted to put out some feelers for the feasibility of these two
> features:
> 
> * have the **kwargs param be an OrderedDict rather than a dict

Rather than just feasibility, I would like performance not to regress
here.

> Both of these will need OrderedDict in C, which is getting close
> (issue #16991).

Really? Last time I looked, it wasn't getting really close.

Regards

Antoine.




From ericsnowcurrently at gmail.com  Thu Feb 28 15:55:03 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 28 Feb 2013 07:55:03 -0700
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <20130228112742.09f5500b@pitrou.net>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<20130228112742.09f5500b@pitrou.net>
Message-ID: <CALFfu7BMDYWp2gMc7o_--+BySOhA7nHubjOVTNQCPhAtbivKOw@mail.gmail.com>

On Feb 28, 2013 3:28 AM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> Le Thu, 28 Feb 2013 03:15:14 -0700,
> Eric Snow <ericsnowcurrently at gmail.com> a
> ?crit :
> > Just wanted to put out some feelers for the feasibility of these two
> > features:
> >
> > * have the **kwargs param be an OrderedDict rather than a dict
>
> Rather than just feasibility, I would like performance not to regress
> here.
>
> > Both of these will need OrderedDict in C, which is getting close
> > (issue #16991).
>
> Really? Last time I looked, it wasn't getting really close.

Everything's there and there are just a few lingering memory-related issues
to iron out.  So 50% done then...<wink>

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130228/a1799a52/attachment.html>

From solipsis at pitrou.net  Thu Feb 28 16:14:37 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 16:14:37 +0100
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<20130228112742.09f5500b@pitrou.net>
	<CALFfu7BMDYWp2gMc7o_--+BySOhA7nHubjOVTNQCPhAtbivKOw@mail.gmail.com>
Message-ID: <20130228161437.57f15090@pitrou.net>

Le Thu, 28 Feb 2013 07:55:03 -0700,
Eric Snow <ericsnowcurrently at gmail.com> a
?crit :

> On Feb 28, 2013 3:28 AM, "Antoine Pitrou"
> <solipsis at pitrou.net> wrote:
> >
> > Le Thu, 28 Feb 2013 03:15:14 -0700,
> > Eric Snow
> > <ericsnowcurrently at gmail.com> a
> > ?crit :
> > > Just wanted to put out some feelers for the feasibility of these
> > > two features:
> > >
> > > * have the **kwargs param be an OrderedDict rather than a dict
> >
> > Rather than just feasibility, I would like performance not to
> > regress here.
> >
> > > Both of these will need OrderedDict in C, which is getting close
> > > (issue #16991).
> >
> > Really? Last time I looked, it wasn't getting really close.
> 
> Everything's there and there are just a few lingering memory-related
> issues to iron out.  So 50% done then...<wink>

And it also has to be reviewed in deep.
To quote you: "The memory-related issues are pushing well past my
experience".

Regards

Antoine.




From ncoghlan at gmail.com  Thu Feb 28 16:20:02 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 1 Mar 2013 01:20:02 +1000
Subject: [Python-ideas] range
In-Reply-To: <kglr5u$6mn$1@ger.gmane.org>
References: <loom.20130221T120736-301@post.gmane.org>
	<20130222150557.36884c3f@pitrou.net>
	<CADiSq7eMhkA+Rdd3BHus2-fpRy5sxwTwuv1xfkJcsCika+2bUg@mail.gmail.com>
	<loom.20130222T183719-360@post.gmane.org>
	<20130222194310.75129c1f@pitrou.net> <kga1n0$apn$2@ger.gmane.org>
	<CADiSq7f5MMyEU6sBUw3UJcuyK1XuEDwJ5H3+yDUpG6xLtf09fA@mail.gmail.com>
	<F417726D-E68F-4184-8D7A-3FF19B58FEB9@gmail.com>
	<kglr5u$6mn$1@ger.gmane.org>
Message-ID: <CADiSq7eErP3+oxkRWBeRGUm1+P_56YsVoVqgzaWO128Bxz74tA@mail.gmail.com>

On Thu, Feb 28, 2013 at 6:43 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 2/26/2013 11:46 PM, Raymond Hettinger wrote:
>>
>>
>> On Feb 23, 2013, at 3:11 AM, Nick Coghlan
>> <ncoghlan at gmail.com
>> <mailto:ncoghlan at gmail.com>> wrote:
>>
>>> I suspect for many cases involving more advanced
>>> range variants, containment would be a better option than inheritance,
>>> particular if you want to implement a type that can't be reliably
>>> described through a (start, stop, step) triple.
>>
>>
>> I concur.  Subclassing range() is just asking for trouble.
>
>
> The four builtins that cannot be subclassed are bool, memoryview, range,
> slice. This is documented for bool; I propose to do so for the others in
> http://bugs.python.org/issue17279. Would it be correct to say (now) that all
> 4 are intentional omissions? and not merely oversights?

Yes, I think so. People will have to be *real* convincing to explain a
case where composition isn't a more appropriate solution

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From donspauldingii at gmail.com  Thu Feb 28 16:30:50 2013
From: donspauldingii at gmail.com (Don Spaulding)
Date: Thu, 28 Feb 2013 09:30:50 -0600
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <20130228112742.09f5500b@pitrou.net>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<20130228112742.09f5500b@pitrou.net>
Message-ID: <CAMaNpgXSjYf778ONf5RLLTN4s1_S2719dTi4Qweq2=oiG_1G6Q@mail.gmail.com>

On Thu, Feb 28, 2013 at 4:27 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 28 Feb 2013 03:15:14 -0700,
> Eric Snow <ericsnowcurrently at gmail.com> a
> ?crit :
> > Just wanted to put out some feelers for the feasibility of these two
> > features:
> >
> > * have the **kwargs param be an OrderedDict rather than a dict
>
> Rather than just feasibility, I would like performance not to regress
> here.
>
>
I can appreciate not wanting to see "Python 3.4 killed our performance"
articles all over news.ycombinator.com.  But isn't it worth it to have a
(small, but acceptable) performance hit for the sake of opening up use
cases which are not currently possible in Python without ugly hacks?

For an example of the "recommended" way to get the ordering of your class
attributes:
http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order

It seems to me that the "right thing" for python to do when given an
ordered list of key=value pairs in a function call or class definition, is
to retain the order.  So what's an acceptable level of performance
regression for the sake of doing things the "right way" here?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130228/5d3971a6/attachment.html>

From solipsis at pitrou.net  Thu Feb 28 16:48:10 2013
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Feb 2013 16:48:10 +0100
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<20130228112742.09f5500b@pitrou.net>
	<CAMaNpgXSjYf778ONf5RLLTN4s1_S2719dTi4Qweq2=oiG_1G6Q@mail.gmail.com>
Message-ID: <20130228164810.74f2d367@pitrou.net>

Le Thu, 28 Feb 2013 09:30:50 -0600,
Don Spaulding <donspauldingii at gmail.com>
a ?crit :
> 
> For an example of the "recommended" way to get the ordering of your
> class attributes:
> http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order

This is already possible with the __prepare__ magic method.
http://docs.python.org/3.4/reference/datamodel.html#preparing-the-class-namespace

> It seems to me that the "right thing" for python to do when given an
> ordered list of key=value pairs in a function call or class
> definition, is to retain the order.  So what's an acceptable level of
> performance regression for the sake of doing things the "right way"
> here?

Or, rather, what is the benefit of doing things "the right way"? There
are incredibly few cases for relying on the order of key=value pairs in
function calls.

Regards

Antoine.




From donspauldingii at gmail.com  Thu Feb 28 17:14:48 2013
From: donspauldingii at gmail.com (Don Spaulding)
Date: Thu, 28 Feb 2013 10:14:48 -0600
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <20130228164810.74f2d367@pitrou.net>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<20130228112742.09f5500b@pitrou.net>
	<CAMaNpgXSjYf778ONf5RLLTN4s1_S2719dTi4Qweq2=oiG_1G6Q@mail.gmail.com>
	<20130228164810.74f2d367@pitrou.net>
Message-ID: <CAMaNpgV+BXUv6W+s9NUKX9CEnpgT3CVFmGL28hUa24RoA6=9-w@mail.gmail.com>

On Thu, Feb 28, 2013 at 9:48 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Thu, 28 Feb 2013 09:30:50 -0600,
> Don Spaulding <donspauldingii at gmail.com>
> a ?crit :
> >
> > For an example of the "recommended" way to get the ordering of your
> > class attributes:
> >
> http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order
>
> This is already possible with the __prepare__ magic method.
>
> http://docs.python.org/3.4/reference/datamodel.html#preparing-the-class-namespace
>
>
Sure.  Case in point: Django has been working around it since at least
python 2.4.


> > It seems to me that the "right thing" for python to do when given an
> > ordered list of key=value pairs in a function call or class
> > definition, is to retain the order.  So what's an acceptable level of
> > performance regression for the sake of doing things the "right way"
> > here?
>
> Or, rather, what is the benefit of doing things "the right way"? There
> are incredibly few cases for relying on the order of key=value pairs in
> function calls.
>

"If you build it, they will come..."

When I originally encountered the need for python to retain the order of
kwargs that my caller specified, it surprised me that there wasn't more
clamoring for kwargs being an OrderedDict.  However, since my development
timeline didn't allow for holding the project up while a patch was pushed
through python-dev and out into a real python release, I sucked it up,
forced my calling code to send in hand-crafted OrderedDicts and called it a
day.  I think most developers don't even stop to think that the language
*could* be different, they just put in the workaround and move on.

I think if python stopped dropping the order of kwargs on the floor today,
you'd see people start to rely on the order of kwargs tomorrow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130228/92d2b761/attachment.html>

From ethan at stoneleaf.us  Thu Feb 28 17:41:17 2013
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 28 Feb 2013 08:41:17 -0800
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CAMaNpgV+BXUv6W+s9NUKX9CEnpgT3CVFmGL28hUa24RoA6=9-w@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<20130228112742.09f5500b@pitrou.net>
	<CAMaNpgXSjYf778ONf5RLLTN4s1_S2719dTi4Qweq2=oiG_1G6Q@mail.gmail.com>
	<20130228164810.74f2d367@pitrou.net>
	<CAMaNpgV+BXUv6W+s9NUKX9CEnpgT3CVFmGL28hUa24RoA6=9-w@mail.gmail.com>
Message-ID: <512F88AD.30306@stoneleaf.us>

On 02/28/2013 08:14 AM, Don Spaulding wrote:
>
>
>
> On Thu, Feb 28, 2013 at 9:48 AM, Antoine Pitrou <solipsis at pitrou.net <mailto:solipsis at pitrou.net>> wrote:
>
>     Le Thu, 28 Feb 2013 09:30:50 -0600,
>     Don Spaulding <donspauldingii at gmail.com <mailto:donspauldingii at gmail.com>>
>     a ?crit :
>     >
>     > For an example of the "recommended" way to get the ordering of your
>     > class attributes:
>     >http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order
>
>     This is already possible with the __prepare__ magic method.
>     http://docs.python.org/3.4/reference/datamodel.html#preparing-the-class-namespace
>
>
> Sure.  Case in point: Django has been working around it since at least python 2.4.
>
>     > It seems to me that the "right thing" for python to do when given an
>     > ordered list of key=value pairs in a function call or class
>     > definition, is to retain the order.  So what's an acceptable level of
>     > performance regression for the sake of doing things the "right way"
>     > here?
>
>     Or, rather, what is the benefit of doing things "the right way"? There
>     are incredibly few cases for relying on the order of key=value pairs in
>     function calls.
>
>
> "If you build it, they will come..."
>
> When I originally encountered the need for python to retain the order of kwargs that my caller specified, it surprised
> me that there wasn't more clamoring for kwargs being an OrderedDict.  However, since my development timeline didn't
> allow for holding the project up while a patch was pushed through python-dev and out into a real python release, I
> sucked it up, forced my calling code to send in hand-crafted OrderedDicts and called it a day.  I think most developers
> don't even stop to think that the language *could* be different, they just put in the workaround and move on.
>
> I think if python stopped dropping the order of kwargs on the floor today, you'd see people start to rely on the order
> of kwargs tomorrow.

+1

I'd already be relying on it if it were there.

--
~Ethan~


From guido at python.org  Thu Feb 28 18:32:57 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 28 Feb 2013 09:32:57 -0800
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
Message-ID: <CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>

Hm. I write code regularly that takes a **kwargs dict and stores it
into some other longer-lasting datastructure. Having that other
datastructure suddenly receive OrderedDicts instead of plain dicts
might definitely cause some confusion and possible performance issues.
And I don't recall ever having wanted to know the order of the kwargs
in the call.

But even apart from that backwards incompatibility I think this
feature is too rarely useful to make it the default behavior -- if
anything I want **kwargs to become faster!

On Thu, Feb 28, 2013 at 2:15 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> Just wanted to put out some feelers for the feasibility of these two
> features:
>
> * have the **kwargs param be an OrderedDict rather than a dict
> * have class definitions happen relative to an OrderedDict by default rather
> than a dict, and still overridable by a metaclass's __prepare__().
>
> Both of these will need OrderedDict in C, which is getting close (issue
> #16991).
>
> -eric
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



--
--Guido van Rossum (python.org/~guido)


From ncoghlan at gmail.com  Thu Feb 28 19:02:54 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 1 Mar 2013 04:02:54 +1000
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>
Message-ID: <CADiSq7dX0mpoB3M89fJCk-vPegB9u19SQDnNpbaGP=iM-DJGew@mail.gmail.com>

On 1 Mar 2013 03:34, "Guido van Rossum" <guido at python.org> wrote:
>
> Hm. I write code regularly that takes a **kwargs dict and stores it
> into some other longer-lasting datastructure. Having that other
> datastructure suddenly receive OrderedDicts instead of plain dicts
> might definitely cause some confusion and possible performance issues.
> And I don't recall ever having wanted to know the order of the kwargs
> in the call.
>
> But even apart from that backwards incompatibility I think this
> feature is too rarely useful to make it the default behavior -- if
> anything I want **kwargs to become faster

And PEP 422 is designed to make it easier to share a common __prepare__
method with different post processing.

Cheers,
Nick.

>
> On Thu, Feb 28, 2013 at 2:15 AM, Eric Snow <ericsnowcurrently at gmail.com>
wrote:
> > Just wanted to put out some feelers for the feasibility of these two
> > features:
> >
> > * have the **kwargs param be an OrderedDict rather than a dict
> > * have class definitions happen relative to an OrderedDict by default
rather
> > than a dict, and still overridable by a metaclass's __prepare__().
> >
> > Both of these will need OrderedDict in C, which is getting close (issue
> > #16991).
> >
> > -eric
> >
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > http://mail.python.org/mailman/listinfo/python-ideas
> >
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130301/bffde7d4/attachment.html>

From guido at python.org  Thu Feb 28 19:56:51 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 28 Feb 2013 10:56:51 -0800
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CADiSq7dX0mpoB3M89fJCk-vPegB9u19SQDnNpbaGP=iM-DJGew@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>
	<CADiSq7dX0mpoB3M89fJCk-vPegB9u19SQDnNpbaGP=iM-DJGew@mail.gmail.com>
Message-ID: <CAP7+vJKmgOZUt-DAxCukfEACrSduvTdHxOJwxcBm5B5eScBXkg@mail.gmail.com>

But __prepare__ only works for classes, not for methods, right?

On Thu, Feb 28, 2013 at 10:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 1 Mar 2013 03:34, "Guido van Rossum" <guido at python.org> wrote:
>>
>> Hm. I write code regularly that takes a **kwargs dict and stores it
>> into some other longer-lasting datastructure. Having that other
>> datastructure suddenly receive OrderedDicts instead of plain dicts
>> might definitely cause some confusion and possible performance issues.
>> And I don't recall ever having wanted to know the order of the kwargs
>> in the call.
>>
>> But even apart from that backwards incompatibility I think this
>> feature is too rarely useful to make it the default behavior -- if
>> anything I want **kwargs to become faster
>
> And PEP 422 is designed to make it easier to share a common __prepare__
> method with different post processing.
>
> Cheers,
> Nick.
>
>>
>> On Thu, Feb 28, 2013 at 2:15 AM, Eric Snow <ericsnowcurrently at gmail.com>
>> wrote:
>> > Just wanted to put out some feelers for the feasibility of these two
>> > features:
>> >
>> > * have the **kwargs param be an OrderedDict rather than a dict
>> > * have class definitions happen relative to an OrderedDict by default
>> > rather
>> > than a dict, and still overridable by a metaclass's __prepare__().
>> >
>> > Both of these will need OrderedDict in C, which is getting close (issue
>> > #16991).
>> >
>> > -eric
>> >
>> >
>> > _______________________________________________
>> > Python-ideas mailing list
>> > Python-ideas at python.org
>> > http://mail.python.org/mailman/listinfo/python-ideas
>> >
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas



--
--Guido van Rossum (python.org/~guido)


From christian at python.org  Thu Feb 28 20:30:33 2013
From: christian at python.org (Christian Heimes)
Date: Thu, 28 Feb 2013 20:30:33 +0100
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
Message-ID: <kgob8l$cmo$1@ger.gmane.org>

Am 28.02.2013 11:15, schrieb Eric Snow:
> Just wanted to put out some feelers for the feasibility of these two
> features:
> 
> * have the **kwargs param be an OrderedDict rather than a dict
> * have class definitions happen relative to an OrderedDict by default
> rather than a dict, and still overridable by a metaclass's __prepare__().
> 
> Both of these will need OrderedDict in C, which is getting close (issue
> #16991).

Raymond was/is working on a modification of the dict's internal data
structure that reduces its memory consumption. IIRC the modification
adds partial ordered as a side effect. The keys are ordered in insertion
order until keys are removed.

http://mail.python.org/pipermail/python-dev/2012-December/123028.html

Christian



From ericsnowcurrently at gmail.com  Thu Feb 28 22:16:02 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 28 Feb 2013 14:16:02 -0700
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>
Message-ID: <CALFfu7Au1kDejP-qwyySRttigZcmOyV-dkx0gYm5mfXQPhUpcw@mail.gmail.com>

On Thu, Feb 28, 2013 at 10:32 AM, Guido van Rossum <guido at python.org> wrote:
> Hm. I write code regularly that takes a **kwargs dict and stores it
> into some other longer-lasting datastructure. Having that other
> datastructure suddenly receive OrderedDicts instead of plain dicts
> might definitely cause some confusion and possible performance issues.

Could you elaborate on what confusion it might cause?

As to performance relative to dict, this has definitely been my
primary concern.  I agree that the impact has to be insignificant for
the **kwargs proposal to go anywhere.  Obviously OrderedDict in C had
better be an improvement over the pure Python version or there isn't
much point.  However, it goes beyond that in the cases where we might
replace current uses of dict with OrderedDict.

My plan has been to do a bunch of performance comparison once the
implementation is complete and tune it as much as possible with an eye
toward the main potential internal use cases.  From my point of view,
those are **kwargs and class namespace.  This is in part why I've
brought those two up.

For instance, one guidepost I've used is that typically **kwargs is
going to be small.  However, even for large kwargs I don't want any
performance difference to be a problem.

> And I don't recall ever having wanted to know the order of the kwargs
> in the call.

Though it may sound a little odd, the first use case I bumped into was
with OrderedDict itself:

    OrderedDict(a=1, b=2, c=3)

There were a few other reasonable use cases mentioned in other threads
a while back.  I'll dig them up if that would help.

>
> But even apart from that backwards incompatibility I think this
> feature is too rarely useful to make it the default behavior -- if
> anything I want **kwargs to become faster!

You mean something like possibly not unpacking the **kwargs in a call
into another dict?

    def f(**kwargs):
        return kwargs
    d = {'a': 1, 'b': 2}
    assert d is f(**d)

Certainly faster, though definitely a semantic difference.

-eric


From guido at python.org  Thu Feb 28 22:28:15 2013
From: guido at python.org (Guido van Rossum)
Date: Thu, 28 Feb 2013 13:28:15 -0800
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CALFfu7Au1kDejP-qwyySRttigZcmOyV-dkx0gYm5mfXQPhUpcw@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>
	<CALFfu7Au1kDejP-qwyySRttigZcmOyV-dkx0gYm5mfXQPhUpcw@mail.gmail.com>
Message-ID: <CAP7+vJKm_XbT7=LJQyitHvLvz_o51+cmd03ufWcUWPCZHr5YPQ@mail.gmail.com>

On Thu, Feb 28, 2013 at 1:16 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Thu, Feb 28, 2013 at 10:32 AM, Guido van Rossum <guido at python.org> wrote:
>> Hm. I write code regularly that takes a **kwargs dict and stores it
>> into some other longer-lasting datastructure. Having that other
>> datastructure suddenly receive OrderedDicts instead of plain dicts
>> might definitely cause some confusion and possible performance issues.
>
> Could you elaborate on what confusion it might cause?

Well, x.__class__ is different, repr(x) is different, ...

> As to performance relative to dict, this has definitely been my
> primary concern.  I agree that the impact has to be insignificant for
> the **kwargs proposal to go anywhere.  Obviously OrderedDict in C had
> better be an improvement over the pure Python version or there isn't
> much point.  However, it goes beyond that in the cases where we might
> replace current uses of dict with OrderedDict.

What happens to the performance if I insert many thousands (or
millions) of items to an OrderedDict? What happens to the space it
uses? The thing is, what started out as OrderedDict stays one, but its
lifetime may be long and the assumptions around dict performance are
rather extreme (or we wouldn't be redoing the implementation
regularly).

> My plan has been to do a bunch of performance comparison once the
> implementation is complete and tune it as much as possible with an eye
> toward the main potential internal use cases.  From my point of view,
> those are **kwargs and class namespace.  This is in part why I've
> brought those two up.

I'm fine with doing this by default for a class namespace; the type of
cls.__dict__ is already a non-dict (it's a proxy) and it's unlikely to
have 100,000 entries.

For **kwds I'm pretty concerned; the use cases seem flimsy.

> For instance, one guidepost I've used is that typically **kwargs is
> going to be small.  However, even for large kwargs I don't want any
> performance difference to be a problem.

So, in my use case, the kwargs is small, but the object may live a
long and productive life after the function call is only a faint
memory, and it might grow dramatically.

IOW I have very high standards for backwards compatibility here.

>> And I don't recall ever having wanted to know the order of the kwargs
>> in the call.
>
> Though it may sound a little odd, the first use case I bumped into was
> with OrderedDict itself:
>
>     OrderedDict(a=1, b=2, c=3)

But because of the self-referentiality, this doesn't prove anything. :-)

> There were a few other reasonable use cases mentioned in other threads
> a while back.  I'll dig them up if that would help.

It would.

>> But even apart from that backwards incompatibility I think this
>> feature is too rarely useful to make it the default behavior -- if
>> anything I want **kwargs to become faster!
>
> You mean something like possibly not unpacking the **kwargs in a call
> into another dict?
>
>     def f(**kwargs):
>         return kwargs
>     d = {'a': 1, 'b': 2}
>     assert d is f(**d)
>
> Certainly faster, though definitely a semantic difference.

No, that would introduce nasty aliasing problems in some cases. I've
actually written code that depends on the copy being made here.
(Yesterday, actually. :-)

-- 
--Guido van Rossum (python.org/~guido)


From foogod at gmail.com  Thu Feb 28 22:32:01 2013
From: foogod at gmail.com (Alex Stewart)
Date: Thu, 28 Feb 2013 13:32:01 -0800
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <kgob8l$cmo$1@ger.gmane.org>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<kgob8l$cmo$1@ger.gmane.org>
Message-ID: <CACxLcFeCejpcrMvuv-G0xt15HkEw=OspNkwmNSzQyHSZdrnr9Q@mail.gmail.com>

On Thu, Feb 28, 2013 at 11:30 AM, Christian Heimes <christian at python.org>wrote:

> Raymond was/is working on a modification of the dict's internal data
> structure that reduces its memory consumption. IIRC the modification
> adds partial ordered as a side effect. The keys are ordered in insertion
> order until keys are removed.
>

While this is potentially convenient, in order for people to really be able
to use it for these purposes it would have to be reliable behavior going
forward.  The question is therefore: do we actually want to explicitly make
this behavior part of the dict API such that all future implementations of
dict (and all implementations in any Python, not just CPython) must be
guaranteed to also behave this way?  I suspect the answer should be "no".

On Thu, Feb 28, 2013 at 7:30 AM, Don Spaulding <donspauldingii at gmail.com>
 wrote:

> So what's an acceptable level of performance regression for the sake of
> doing things the "right way" here?
>

For class declarations, I think a little performance hit might not be too
big a deal, but those are really not the most important use case here,
since there are already ways to work around this for classes if one wants
to.

However, for function calls, which I actually believe is the more important
question, I think we would have to think very hard about anything that
introduced any real performance overhead, because functions are typically
called a _lot_.  Adding even a small extra delay to function calls could
impact the overall performance of some types of Python programs
substantially..

I also agree with Guido that there may be some unexpected consequences of
suddenly changing the type of the kwargs parameter passed to functions
which were coded assuming it would always be a plain-ol' dict..  I think it
might be doable, but we'd have to be very sure that OrderedDict is exactly
compatible in every conceivable way..

Perhaps instead we could compromise by keeping the default case as it is,
but providing some mechanism for the function declaration to specify that
it wants ordering information passed to it?  I'm not sure what a good
syntax for this would be, though ("(*args, ***okwargs)"?  "(*args,
**kwargs, *kworder)"?  Not sure I'm really big on either of those,
actually, but you get the idea..)

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130228/60075859/attachment.html>

From ericsnowcurrently at gmail.com  Thu Feb 28 22:37:45 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 28 Feb 2013 14:37:45 -0700
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CADiSq7dX0mpoB3M89fJCk-vPegB9u19SQDnNpbaGP=iM-DJGew@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>
	<CADiSq7dX0mpoB3M89fJCk-vPegB9u19SQDnNpbaGP=iM-DJGew@mail.gmail.com>
Message-ID: <CALFfu7DX3Y3VxiHKoDqO2LtSwP8AM_e_0RvARf6jDpFQnw-XHQ@mail.gmail.com>

On Thu, Feb 28, 2013 at 11:02 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> And PEP 422 is designed to make it easier to share a common __prepare__
> method with different post processing.

A major use case for __prepare__() is to have the class definition use
an OrderedDict.  It's even a point of discussion in PEP 3115.  While I
agree with the the conclusion of the PEP that using only OrderedDict
is inferior to __prepare__(), I also think defaulting to OrderedDict
is viable and useful.

Using __prepare__() necessitates the use of a metaclass, which most
people consider black magic.  Even when you can inherit from a class
that has a custom metaclass (like collections.abc.ABC), it still
necessitates inheritance and the chance for metaclass conflicts. While
I'm on board with PEP 422, I'm not clear on how it helps here.

If class namespaces were ordered by default then class decorators,
which are typically much easier to comprehend, would steal yet another
use case from metaclasses.  The recent discussions on enums got me
thinking about this.  In some cases you want you class customization
logic to be inherited.  That's where you've got to use a metaclass.
In other cases you don't.  Decorators work great for that.  If your
class decorator needs to have ordering information, then you also have
to use a metaclass.  Having OrderedDict be the default class namespace
would address that.

-eric


From ned at nedbatchelder.com  Thu Feb 28 22:51:42 2013
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 28 Feb 2013 16:51:42 -0500
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <512F88AD.30306@stoneleaf.us>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<20130228112742.09f5500b@pitrou.net>
	<CAMaNpgXSjYf778ONf5RLLTN4s1_S2719dTi4Qweq2=oiG_1G6Q@mail.gmail.com>
	<20130228164810.74f2d367@pitrou.net>
	<CAMaNpgV+BXUv6W+s9NUKX9CEnpgT3CVFmGL28hUa24RoA6=9-w@mail.gmail.com>
	<512F88AD.30306@stoneleaf.us>
Message-ID: <512FD16E.30503@nedbatchelder.com>


On 2/28/2013 11:41 AM, Ethan Furman wrote:
> On 02/28/2013 08:14 AM, Don Spaulding wrote:
>>
>>
>>
>> On Thu, Feb 28, 2013 at 9:48 AM, Antoine Pitrou <solipsis at pitrou.net 
>> <mailto:solipsis at pitrou.net>> wrote:
>>
>>     Le Thu, 28 Feb 2013 09:30:50 -0600,
>>     Don Spaulding <donspauldingii at gmail.com 
>> <mailto:donspauldingii at gmail.com>>
>>     a ?crit :
>>     >
>>     > For an example of the "recommended" way to get the ordering of 
>> your
>>     > class attributes:
>> >http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order
>>
>>     This is already possible with the __prepare__ magic method.
>> http://docs.python.org/3.4/reference/datamodel.html#preparing-the-class-namespace
>>
>>
>> Sure.  Case in point: Django has been working around it since at 
>> least python 2.4.
>>
>>     > It seems to me that the "right thing" for python to do when 
>> given an
>>     > ordered list of key=value pairs in a function call or class
>>     > definition, is to retain the order.  So what's an acceptable 
>> level of
>>     > performance regression for the sake of doing things the "right 
>> way"
>>     > here?
>>
>>     Or, rather, what is the benefit of doing things "the right way"? 
>> There
>>     are incredibly few cases for relying on the order of key=value 
>> pairs in
>>     function calls.
>>
>>
>> "If you build it, they will come..."
>>
>> When I originally encountered the need for python to retain the order 
>> of kwargs that my caller specified, it surprised
>> me that there wasn't more clamoring for kwargs being an OrderedDict.  
>> However, since my development timeline didn't
>> allow for holding the project up while a patch was pushed through 
>> python-dev and out into a real python release, I
>> sucked it up, forced my calling code to send in hand-crafted 
>> OrderedDicts and called it a day.  I think most developers
>> don't even stop to think that the language *could* be different, they 
>> just put in the workaround and move on.
>>
>> I think if python stopped dropping the order of kwargs on the floor 
>> today, you'd see people start to rely on the order
>> of kwargs tomorrow.
>
> +1
>
> I'd already be relying on it if it were there.
>

Could you advance the discussion by elaborating your use case?  I've 
never had need for ordered kwargs, so I'm having a hard time seeing how 
they would be useful.

--Ned.

> -- 
> ~Ethan~
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



From ericsnowcurrently at gmail.com  Thu Feb 28 22:53:49 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 28 Feb 2013 14:53:49 -0700
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CAP7+vJKm_XbT7=LJQyitHvLvz_o51+cmd03ufWcUWPCZHr5YPQ@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<CAP7+vJKMkmx6efW4LR-VZ+Czax=0u1AM1Sf-n2j7KxzNrsGi-w@mail.gmail.com>
	<CALFfu7Au1kDejP-qwyySRttigZcmOyV-dkx0gYm5mfXQPhUpcw@mail.gmail.com>
	<CAP7+vJKm_XbT7=LJQyitHvLvz_o51+cmd03ufWcUWPCZHr5YPQ@mail.gmail.com>
Message-ID: <CALFfu7CxNzSRCZDhGTn=Jc8a7wGOsuiRr8VERFi2DnjZo2FAzw@mail.gmail.com>

On Thu, Feb 28, 2013 at 2:28 PM, Guido van Rossum <guido at python.org> wrote:
> What happens to the performance if I insert many thousands (or
> millions) of items to an OrderedDict? What happens to the space it
> uses? The thing is, what started out as OrderedDict stays one, but its
> lifetime may be long and the assumptions around dict performance are
> rather extreme (or we wouldn't be redoing the implementation
> regularly).

<snip/>

> So, in my use case, the kwargs is small, but the object may live a
> long and productive life after the function call is only a faint
> memory, and it might grow dramatically.

Good point.  My own use of **kwargs rarely sees the object leave the
function or get very big, and this aspect of it just hadn't come up to
broaden my point of view.  I'm glad we're having this discussion.  My
intuition is that such a use case would be pretty rare, but even
then...

>
> IOW I have very high standards for backwards compatibility here.

...Python has an exceptional standard in this regard.  FWIW, I agree.
I would want to be confident about the ramifications before we made
any change like this.

>> There were a few other reasonable use cases mentioned in other threads
>> a while back.  I'll dig them up if that would help.
>
> It would.

Okay.

>
>>> But even apart from that backwards incompatibility I think this
>>> feature is too rarely useful to make it the default behavior -- if
>>> anything I want **kwargs to become faster!
>>
>> You mean something like possibly not unpacking the **kwargs in a call
>> into another dict?
>>
>>     def f(**kwargs):
>>         return kwargs
>>     d = {'a': 1, 'b': 2}
>>     assert d is f(**d)
>>
>> Certainly faster, though definitely a semantic difference.
>
> No, that would introduce nasty aliasing problems in some cases. I've
> actually written code that depends on the copy being made here.
> (Yesterday, actually. :-)

You can't blame me for trying to make **kwargs-as-OrderedDict seem
like a great idea in comparison.  <wink>

-eric


From ericsnowcurrently at gmail.com  Thu Feb 28 23:18:08 2013
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 28 Feb 2013 15:18:08 -0700
Subject: [Python-ideas] OrderedDict for kwargs and class statement
	namespace
In-Reply-To: <CACxLcFeCejpcrMvuv-G0xt15HkEw=OspNkwmNSzQyHSZdrnr9Q@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
	<kgob8l$cmo$1@ger.gmane.org>
	<CACxLcFeCejpcrMvuv-G0xt15HkEw=OspNkwmNSzQyHSZdrnr9Q@mail.gmail.com>
Message-ID: <CALFfu7Dp-yBakon4M3zNcF79YpO0ztFsJuLtfXC54Z57eYkLRA@mail.gmail.com>

On Thu, Feb 28, 2013 at 2:32 PM, Alex Stewart <foogod at gmail.com> wrote:
> such that all future implementations of
> dict (and all implementations in any Python, not just CPython) must be
> guaranteed to also behave this way?

This is also a good point applied to both **kwargs and class
namespace.  Before we did either we'd need to be clear on the impact
on other Python implementors.

> However, for function calls, which I actually believe is the more important
> question, I think we would have to think very hard about anything that
> introduced any real performance overhead, because functions are typically
> called a _lot_.  Adding even a small extra delay to function calls could
> impact the overall performance of some types of Python programs
> substantially..

**kwargs-as-OrderedDict impacts performance in two ways: the
performance of packing the "unbound" keyword arguments into the
OrderedDict and the performance of OrderedDict in normal operation
after its handed off to the function.  Otherwise you don't get a
performance impact on functions.

>
> I also agree with Guido that there may be some unexpected consequences of
> suddenly changing the type of the kwargs parameter passed to functions which
> were coded assuming it would always be a plain-ol' dict..  I think it might
> be doable, but we'd have to be very sure that OrderedDict is exactly
> compatible in every conceivable way..

For Python backward compatibility is a cornerstone.  I'm surprised
there isn't something in the Zen about it. <wink>  For **kwargs the
bar for compatibility is especially high and I agree with that.

>
> Perhaps instead we could compromise by keeping the default case as it is,
> but providing some mechanism for the function declaration to specify that it
> wants ordering information passed to it?  I'm not sure what a good syntax
> for this would be, though ("(*args, ***okwargs)"?  "(*args, **kwargs,
> *kworder)"?  Not sure I'm really big on either of those, actually, but you
> get the idea..)

**kwargs packing happens in the interpreter rather than in
relationship to functionality provided by the function, so whatever
the mechanism it would have be something the interpreter consumes,
either a new API/syntax for building functions or a new syntax like
you've mentioned.

This has come up before.  Classes have metaclasses (and __prepare__).
Modules have loaders.  Poor, poor functions.  Because of the same
concerns you've already expressed regarding the criticality of
function performance, they miss out on all sorts of fun--inside their
highly optimized box looking out at the other types showing off their
cool new features all the time. It just isn't fair. :)

-eric


From zuo at chopin.edu.pl  Thu Feb 28 23:51:21 2013
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Thu, 28 Feb 2013 23:51:21 +0100
Subject: [Python-ideas] OrderedDict for kwargs and class statement
 namespace
In-Reply-To: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
References: <CALFfu7COfqWrCgupheGG7ME--vCkaZLpA-LiV-WEBB7zVJiMKw@mail.gmail.com>
Message-ID: <32fd3c22d72c258ae8e49c7c3fa10f98@chopin.edu.pl>

28.02.2013 11:15, Eric Snow wrote:

> * have the **kwargs param be an OrderedDict rather than a dict
> * have class definitions happen relative to an OrderedDict by
> default rather than a dict, and still overridable by a metaclass's
> __prepare__().

While having class namespace ordered sounds very nice, ordered **kwargs 
sometimes may not be a desirable feature -- especially when you want to 
keep a cooperatively used interface as simple as possible (because of 
the risk that keyword argument order could then be taken into 
consideration by *some* actors while others would still operate with the 
assumption that argument order cannot be known...).

> Both of these will need OrderedDict in C, which is getting close
> (issue #16991).

Ad issue #16991: will the present semantics of inheriting from 
OrderedDict be kept untouched? (I mean: which methods call other methods 
=> which methods you need to override etc.)

If not we'll have a backward compatibility issue (maybe not very 
serious but still...).

Cheers.
*j