From martin at v.loewis.de  Sat May  1 00:00:18 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 01 May 2010 00:00:18 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <4BDB50C8.1090205@holdenweb.com>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com>
	<4BDB4D26.3040904@v.loewis.de> <4BDB50C8.1090205@holdenweb.com>
Message-ID: <4BDB52F2.6010508@v.loewis.de>

Steve Holden wrote:
> Martin v. L?wis wrote:
>>>> Without a BDFL, I think we need a committee to make decisions, e.g. by
>>>> majority vote amongst committers.
>>> Couldn't we just go with the FLUFL?
>> Not sure whether that's a serious proposal (April 1 is already some days
>> back now). As a starting point, Barry would have to indicate whether he
>> is interested in that role.
>>
> If he isn't then we can depose him and replace him with a puppet.

Ah, ok. So you didn't mean that seriously; a smiley would have helped.

Regards,
Martin

From martin at v.loewis.de  Sat May  1 00:13:23 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 01 May 2010 00:13:23 +0200
Subject: [Python-Dev] patch for review: __import__ documentation
In-Reply-To: <o2wbbaeab101004141419x794fca8csb7011418401d7739@mail.gmail.com>
References: <r2idacc8d1a1004132054w60020897n1234c03098ace0dc@mail.gmail.com>
	<v2pbbaeab101004141332u8d059eb2i6f677dca8c769da9@mail.gmail.com>
	<u2gdacc8d1a1004141341j4da1d99bv45454e9c11f23812@mail.gmail.com>
	<o2wbbaeab101004141419x794fca8csb7011418401d7739@mail.gmail.com>
Message-ID: <4BDB5603.6060907@v.loewis.de>

> I see the confusion. I think Martin meant more about open issues that
> required discussion, not simply issues that had a patch ready to go.

I actually think it is perfectly fine to point out that specific issues
are need committer action on this list. This is what the list is there for.

Waiting some time to see whether some developer reacts is certainly a
good idea: notice, however, that Chris had already waited a few days.

Regards,
Martin

From jnoller at gmail.com  Sat May  1 00:54:06 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 30 Apr 2010 18:54:06 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <4BDB34CE.9080804@v.loewis.de>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
Message-ID: <61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>



On Apr 30, 2010, at 3:51 PM, "Martin v. L?wis" <martin at v.loewis.de>  
wrote:

>> As to Guido's point about the decision making process, Nick's  
>> right.  I just
>> want to make sure we can capture the resolution in the PEP, be it  
>> by BDFL
>> pronouncement or "hey, silence is acceptance" email.
>
> I don't think "silence is acceptance" will work out in practice. For
> issues where a PEP was written in the first place, somebody will
> *always* object, and forever so, hoping that what he considers a  
> mistake
> will not be done. The advantage of Guido acting as BDFL was that
> somebody would make an decision ultimately, one which both proponents
> and opponents of the PEP would accept.
>
> Without a BDFL, I think we need a committee to make decisions, e.g. by
> majority vote amongst committers.
>
> Regards,
> Martin
>

Consider this a plaintitive -1 to any sort of rule-or-decision based  
on committee.

I'd much rather a 2x4 to the forehead.

jesse

From solipsis at pitrou.net  Sat May  1 01:00:57 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 30 Apr 2010 23:00:57 +0000 (UTC)
Subject: [Python-Dev] Two small PEP ideas
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>
Message-ID: <loom.20100501T010005-397@post.gmane.org>

Jesse Noller <jnoller <at> gmail.com> writes:
> 
> Consider this a plaintitive -1 to any sort of rule-or-decision based  
> on committee.
> 
> I'd much rather a 2x4 to the forehead.

Oops, sorry but what does "a 2x4 to the forehead" mean?
(and "plaintitive" by the way?)

Regards

Antoine.



From benjamin at python.org  Sat May  1 01:08:54 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 30 Apr 2010 18:08:54 -0500
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <loom.20100501T010005-397@post.gmane.org>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>
	<loom.20100501T010005-397@post.gmane.org>
Message-ID: <u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>

2010/4/30 Antoine Pitrou <solipsis at pitrou.net>:
> Jesse Noller <jnoller <at> gmail.com> writes:
>>
>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>> on committee.
>>
>> I'd much rather a 2x4 to the forehead.
>
> Oops, sorry but what does "a 2x4 to the forehead" mean?
> (and "plaintitive" by the way?)

The former means being hit by a board and by the latter, he probably
means "plaintive" meaning melancholy.



-- 
Regards,
Benjamin

From fijall at gmail.com  Sat May  1 01:10:22 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 30 Apr 2010 17:10:22 -0600
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> 
	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>
	<loom.20100501T010005-397@post.gmane.org> 
	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
Message-ID: <s2u693bc9ab1004301610ud8734dc6i47ea64db0049e3e@mail.gmail.com>

On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/4/30 Antoine Pitrou <solipsis at pitrou.net>:
>> Jesse Noller <jnoller <at> gmail.com> writes:
>>>
>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>> on committee.
>>>
>>> I'd much rather a 2x4 to the forehead.
>>
>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>> (and "plaintitive" by the way?)
>
> The former means being hit by a board and by the latter, he probably
> means "plaintive" meaning melancholy.
>

Does it only work for english speaking non-metric system users? (2x4
is probably something imperial)

From fuzzyman at voidspace.org.uk  Sat May  1 01:13:17 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 01 May 2010 00:13:17 +0100
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de>	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>	<loom.20100501T010005-397@post.gmane.org>
	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
Message-ID: <4BDB640D.8000000@voidspace.org.uk>

On 01/05/2010 00:08, Benjamin Peterson wrote:
> 2010/4/30 Antoine Pitrou<solipsis at pitrou.net>:
>    
>> Jesse Noller<jnoller<at>  gmail.com>  writes:
>>      
>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>> on committee.
>>>
>>> I'd much rather a 2x4 to the forehead.
>>>        
>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>> (and "plaintitive" by the way?)
>>      
> The former means being hit by a board and by the latter, he probably
> means "plaintive" meaning melancholy.
>
>    
And 2x4 refers to a board with cross-section of 4 inches x 2 inches; the 
inch being an obsolete unit of measure from the old British imperial 
system now defunct throughout the civilized world.

Michael

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From fuzzyman at voidspace.org.uk  Sat May  1 01:13:59 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 01 May 2010 00:13:59 +0100
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <s2u693bc9ab1004301610ud8734dc6i47ea64db0049e3e@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de>
	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>	<loom.20100501T010005-397@post.gmane.org>
	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
	<s2u693bc9ab1004301610ud8734dc6i47ea64db0049e3e@mail.gmail.com>
Message-ID: <4BDB6437.90001@voidspace.org.uk>

On 01/05/2010 00:10, Maciej Fijalkowski wrote:
> On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson<benjamin at python.org>  wrote:
>    
>> 2010/4/30 Antoine Pitrou<solipsis at pitrou.net>:
>>      
>>> Jesse Noller<jnoller<at>  gmail.com>  writes:
>>>        
>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>>> on committee.
>>>>
>>>> I'd much rather a 2x4 to the forehead.
>>>>          
>>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>>> (and "plaintitive" by the way?)
>>>        
>> The former means being hit by a board and by the latter, he probably
>> means "plaintive" meaning melancholy.
>>
>>      
> Does it only work for english speaking non-metric system users? (2x4
> is probably something imperial)
>    

You we would probably strike with 100x50mm boards instead.

Michael

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From benjamin at python.org  Sat May  1 01:14:28 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 30 Apr 2010 18:14:28 -0500
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <s2u693bc9ab1004301610ud8734dc6i47ea64db0049e3e@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>
	<loom.20100501T010005-397@post.gmane.org>
	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
	<s2u693bc9ab1004301610ud8734dc6i47ea64db0049e3e@mail.gmail.com>
Message-ID: <r2m1afaf6161004301614i2014a595xedbfcb0bde6285c4@mail.gmail.com>

2010/4/30 Maciej Fijalkowski <fijall at gmail.com>:
> On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2010/4/30 Antoine Pitrou <solipsis at pitrou.net>:
>>> Jesse Noller <jnoller <at> gmail.com> writes:
>>>>
>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>>> on committee.
>>>>
>>>> I'd much rather a 2x4 to the forehead.
>>>
>>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>>> (and "plaintitive" by the way?)
>>
>> The former means being hit by a board and by the latter, he probably
>> means "plaintive" meaning melancholy.
>>
>
> Does it only work for english speaking non-metric system users? (2x4
> is probably something imperial)

Yes, that's a 5,04 by 10,08 for you. :)



-- 
Regards,
Benjamin

From guido at python.org  Sat May  1 03:11:19 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 30 Apr 2010 18:11:19 -0700
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <r2m1afaf6161004301614i2014a595xedbfcb0bde6285c4@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com> 
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com> 
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> 
	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>
	<loom.20100501T010005-397@post.gmane.org> 
	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com> 
	<s2u693bc9ab1004301610ud8734dc6i47ea64db0049e3e@mail.gmail.com> 
	<r2m1afaf6161004301614i2014a595xedbfcb0bde6285c4@mail.gmail.com>
Message-ID: <r2pca471dc21004301811l1bf7585euaf1f96d8a167e825@mail.gmail.com>

On Fri, Apr 30, 2010 at 4:14 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/4/30 Maciej Fijalkowski <fijall at gmail.com>:
>> On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>> 2010/4/30 Antoine Pitrou <solipsis at pitrou.net>:
>>>> Jesse Noller <jnoller <at> gmail.com> writes:
>>>>>
>>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>>>> on committee.
>>>>>
>>>>> I'd much rather a 2x4 to the forehead.
>>>>
>>>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>>>> (and "plaintitive" by the way?)
>>>
>>> The former means being hit by a board and by the latter, he probably
>>> means "plaintive" meaning melancholy.
>>>
>>
>> Does it only work for english speaking non-metric system users? (2x4
>> is probably something imperial)
>
> Yes, that's a 5,04 by 10,08 for you. :)

Of course "2 by 4" is just the name. The actual measurements of such a
piece of lumber in the store are about 1.75 by 3.75 inch. :-)

-- 
--Guido van Rossum (python.org/~guido)

From steve at holdenweb.com  Sat May  1 04:46:19 2010
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 30 Apr 2010 22:46:19 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <4BDB640D.8000000@voidspace.org.uk>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>	<4BDB34CE.9080804@v.loewis.de>	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>	<loom.20100501T010005-397@post.gmane.org>	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
	<4BDB640D.8000000@voidspace.org.uk>
Message-ID: <hrg4m6$tlp$1@dough.gmane.org>

Michael Foord wrote:
> On 01/05/2010 00:08, Benjamin Peterson wrote:
>> 2010/4/30 Antoine Pitrou<solipsis at pitrou.net>:
>>   
>>> Jesse Noller<jnoller<at>  gmail.com>  writes:
>>>     
>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>>> on committee.
>>>>
>>>> I'd much rather a 2x4 to the forehead.
>>>>        
>>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>>> (and "plaintitive" by the way?)
>>>      
>> The former means being hit by a board and by the latter, he probably
>> means "plaintive" meaning melancholy.
>>
>>    
> And 2x4 refers to a board with cross-section of 4 inches x 2 inches; the
> inch being an obsolete unit of measure from the old British imperial
> system now defunct throughout the civilized world.
> 
The last time I was in a UK builders' yard I hear someone asking for
"two meter pieces of two by four". At the time the UK was notionally
metric (and the timber was planed to the nearest metric size) but the
old names still survived.

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See PyCon Talks from Atlanta 2010  http://pycon.blip.tv/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/


From jnoller at gmail.com  Sat May  1 05:41:14 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 30 Apr 2010 23:41:14 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <r2pca471dc21004301811l1bf7585euaf1f96d8a167e825@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>
	<loom.20100501T010005-397@post.gmane.org>
	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
	<s2u693bc9ab1004301610ud8734dc6i47ea64db0049e3e@mail.gmail.com>
	<r2m1afaf6161004301614i2014a595xedbfcb0bde6285c4@mail.gmail.com>
	<r2pca471dc21004301811l1bf7585euaf1f96d8a167e825@mail.gmail.com>
Message-ID: <x2j4222a8491004302041t87db5eccrde4aa7cd82b87f98@mail.gmail.com>

On Fri, Apr 30, 2010 at 9:11 PM, Guido van Rossum <guido at python.org> wrote:
> On Fri, Apr 30, 2010 at 4:14 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2010/4/30 Maciej Fijalkowski <fijall at gmail.com>:
>>> On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>>> 2010/4/30 Antoine Pitrou <solipsis at pitrou.net>:
>>>>> Jesse Noller <jnoller <at> gmail.com> writes:
>>>>>>
>>>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>>>>> on committee.
>>>>>>
>>>>>> I'd much rather a 2x4 to the forehead.
>>>>>
>>>>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>>>>> (and "plaintitive" by the way?)
>>>>
>>>> The former means being hit by a board and by the latter, he probably
>>>> means "plaintive" meaning melancholy.
>>>>
>>>
>>> Does it only work for english speaking non-metric system users? (2x4
>>> is probably something imperial)
>>
>> Yes, that's a 5,04 by 10,08 for you. :)
>
> Of course "2 by 4" is just the name. The actual measurements of such a
> piece of lumber in the store are about 1.75 by 3.75 inch. :-)
>

We americans round up! (all the time)

From jnoller at gmail.com  Sat May  1 05:42:14 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 30 Apr 2010 23:42:14 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>
	<loom.20100501T010005-397@post.gmane.org>
	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>
Message-ID: <r2u4222a8491004302042xcbfac0f9i4c13583ee7ec57e@mail.gmail.com>

On Fri, Apr 30, 2010 at 7:08 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/4/30 Antoine Pitrou <solipsis at pitrou.net>:
>> Jesse Noller <jnoller <at> gmail.com> writes:
>>>
>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>> on committee.
>>>
>>> I'd much rather a 2x4 to the forehead.
>>
>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>> (and "plaintitive" by the way?)
>
> The former means being hit by a board and by the latter, he probably
> means "plaintive" meaning melancholy.

Yes, well, what Benjamin said.

jesse

From ncoghlan at gmail.com  Sat May  1 06:01:25 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 01 May 2010 14:01:25 +1000
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <hrg4m6$tlp$1@dough.gmane.org>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>	<4BDB34CE.9080804@v.loewis.de>	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>	<loom.20100501T010005-397@post.gmane.org>	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>	<4BDB640D.8000000@voidspace.org.uk>
	<hrg4m6$tlp$1@dough.gmane.org>
Message-ID: <4BDBA795.7010204@gmail.com>

Steve Holden wrote:
> The last time I was in a UK builders' yard I hear someone asking for
> "two meter pieces of two by four". At the time the UK was notionally
> metric (and the timber was planed to the nearest metric size) but the
> old names still survived.

Yeah, a 2x4 is still a 2x4 here as well. It's English, even the native
speakers don't really expect it to make sense all the time ;)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From martin at v.loewis.de  Sat May  1 07:12:38 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 01 May 2010 07:12:38 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de> <4BDB3D6C.4050909@holdenweb.com>
	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
Message-ID: <4BDBB846.5020305@v.loewis.de>

> IIRC in the IETF this is done by the committee chair. I think it's a
> good idea to have this be a single person to avoid endless indecision.

It then seems that this role should go to the release manager of the
upcoming feature release. Assuming Georg can accept this additional
responsibility.

Regards,
Martin

From dirkjan at ochtman.nl  Sat May  1 08:58:03 2010
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sat, 1 May 2010 08:58:03 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <20100430150918.308984aa@heresy>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy>
Message-ID: <z2zea2499da1004302358x38760470vcfdd2a2e4b73e656@mail.gmail.com>

On Fri, Apr 30, 2010 at 21:09, Barry Warsaw <barry at python.org> wrote:
>>Though maybe it should be called Conclusion instead of Accepted and
>>used for Rejected PEPs, as well?
>
> Good point. ?What do you think about 'Resolution'?

Fine with me.

Cheers,

Dirkjan

From ziade.tarek at gmail.com  Sat May  1 11:00:10 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 1 May 2010 11:00:10 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<4BDB3D6C.4050909@holdenweb.com>
	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
Message-ID: <z2g94bdd2611005010200vb4a590c5jeb0787e9a1f5f267@mail.gmail.com>

On Fri, Apr 30, 2010 at 11:25 PM, Guido van Rossum <guido at python.org> wrote:
[..]
>>>
>>> Without a BDFL, I think we need a committee to make decisions, e.g. by
>>> majority vote amongst committers.
>>
>> Couldn't we just go with the FLUFL?
>
> IIRC in the IETF this is done by the committee chair. I think it's a
> good idea to have this be a single person to avoid endless indecision.

BPD = Benevolent Pep Dictator
BPC = Benevolent Pep Caliph   (sounds good in French, not sure in English ;) )

What about naming several BPD + and have one BPC elected each year by
all the core devs ?

== BPD ==

I am not sure if this would work for all areas in Python-core, but
looking at the maintainers.rst file, it looks like we could name for
example Brett for all the import machinery things, Marc-Andr? for all
the unicode things, I could be the one about packaging, etc.

If we could manage to split the python-core development into
categories, and add these categories in the PEP metadata, that would
define who takes the final decision in case we can't reach consensus.

== BPC ==

Of course some PEPs could concern several categories,  so we would
still need some kind of Pep dictator if there's no consensus.  So what
about electing a BPC every year ?

Tarek

-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Sat May  1 11:14:22 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 01 May 2010 11:14:22 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <z2g94bdd2611005010200vb4a590c5jeb0787e9a1f5f267@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de>	<4BDB3D6C.4050909@holdenweb.com>	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
	<z2g94bdd2611005010200vb4a590c5jeb0787e9a1f5f267@mail.gmail.com>
Message-ID: <4BDBF0EE.8040100@v.loewis.de>

> Of course some PEPs could concern several categories,  so we would
> still need some kind of Pep dictator if there's no consensus.  So what
> about electing a BPC every year ?

I think having a single body/person pronounce on all PEPs is sufficient;
as that person should certainly listen to the opinions of the respective
experts.

See also my proposal that the release manager of the next feature
release automatically assumes that role.

Regards,
Martin

From g.brandl at gmx.net  Sat May  1 11:35:04 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 01 May 2010 11:35:04 +0200
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <20100429114047.C031C20059D@kimball.webabinitio.net>
References: <4BD95C6E.9080508@gmail.com>
	<20100429114047.C031C20059D@kimball.webabinitio.net>
Message-ID: <hrgsla$nrm$1@dough.gmane.org>

Am 29.04.2010 13:40, schrieb R. David Murray:
> On Thu, 29 Apr 2010 20:16:14 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Does the online dev version of the docs build in response to docs
>> checkins, or just once a day?
> 
> I believe it does it once a day.  Georg recently changed how this is done,
> so we should get official word from him.

Yes, it builds once a day.  The build process currently takes quite long,
since PDFs are also built every time.  We could change the procedure to
rebuild the HTML more frequently or on checkin, but that would again
make the PDFs mismatch the online docs, and I would say it's not necessary.

>> (And is that written down somewhere and I've just forgotten where to
>> look...)
> 
> I don't think so, but it should be.

It is documented at <http://www.python.org/dev/doc>.  The docs themselves
bear the "last updated" date only.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From solipsis at pitrou.net  Sat May  1 12:34:21 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 1 May 2010 10:34:21 +0000 (UTC)
Subject: [Python-Dev] Two small PEP ideas
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de>	<4BDB3D6C.4050909@holdenweb.com>	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
	<z2g94bdd2611005010200vb4a590c5jeb0787e9a1f5f267@mail.gmail.com>
	<4BDBF0EE.8040100@v.loewis.de>
Message-ID: <loom.20100501T123019-167@post.gmane.org>

Martin v. L?wis <martin <at> v.loewis.de> writes:
> 
> I think having a single body/person pronounce on all PEPs is sufficient;
> as that person should certainly listen to the opinions of the respective
> experts.

The issue is more a question of personal bandwidth. Giving an informed decision
requires reading and listening to a lot of advice, material etc.

Then it's not obvious that we will have many PEPs in the future. The syntax is
pretty much frozen (even after the moratorium, it seems unlikely anything big
will happen -- except if we want to add first-class coroutines). Many
enhancements (e.g. new APIs in existing library modules) can be done, and are
done, without PEPs at all.

Regards

Antoine.



From martin at v.loewis.de  Sat May  1 13:03:16 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 01 May 2010 13:03:16 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <loom.20100501T123019-167@post.gmane.org>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>	<4BDB34CE.9080804@v.loewis.de>	<4BDB3D6C.4050909@holdenweb.com>	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>	<z2g94bdd2611005010200vb4a590c5jeb0787e9a1f5f267@mail.gmail.com>	<4BDBF0EE.8040100@v.loewis.de>
	<loom.20100501T123019-167@post.gmane.org>
Message-ID: <4BDC0A74.6090106@v.loewis.de>

> Then it's not obvious that we will have many PEPs in the future.

Given Guido's Theorem: the PEPs yet to be written will hopefully
outnumber the PEPs written so far.

Regards,
Martin

From ziade.tarek at gmail.com  Sat May  1 13:03:33 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 1 May 2010 13:03:33 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <loom.20100501T123019-167@post.gmane.org>
References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<4BDB3D6C.4050909@holdenweb.com>
	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
	<z2g94bdd2611005010200vb4a590c5jeb0787e9a1f5f267@mail.gmail.com>
	<4BDBF0EE.8040100@v.loewis.de>
	<loom.20100501T123019-167@post.gmane.org>
Message-ID: <i2l94bdd2611005010403r2ae6dae0p3bfd4c1825cb0a6c@mail.gmail.com>

On Sat, May 1, 2010 at 12:34 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Martin v. L?wis <martin <at> v.loewis.de> writes:
>>
>> I think having a single body/person pronounce on all PEPs is sufficient;
>> as that person should certainly listen to the opinions of the respective
>> experts.
>
> The issue is more a question of personal bandwidth. Giving an informed decision
> requires reading and listening to a lot of advice, material etc.

Yes that was the idea. For instance in packaging matters, I don't
expect other core devs
to have the time to learn about every aspect of a particular packaging
problem to be able to
make the best decision.

Now if the release manager takes the decision, I guess the result will
be the same at the end,
as Martin says: he'll ask for the opinion of the respective experts so
I guess it works too.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com
>



-- 
Tarek Ziad? | http://ziade.org

From rdmurray at bitdance.com  Sat May  1 16:28:57 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 01 May 2010 10:28:57 -0400
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <hrgsla$nrm$1@dough.gmane.org>
References: <4BD95C6E.9080508@gmail.com>
	<20100429114047.C031C20059D@kimball.webabinitio.net>
	<hrgsla$nrm$1@dough.gmane.org>
Message-ID: <20100501142857.3CAD51FC3F5@kimball.webabinitio.net>

On Sat, 01 May 2010 11:35:04 +0200, Georg Brandl wrote:
> Am 29.04.2010 13:40, schrieb R. David Murray:
> > On Thu, 29 Apr 2010 20:16:14 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> Does the online dev version of the docs build in response to docs
> >> checkins, or just once a day?
> > 
> > I believe it does it once a day.  Georg recently changed how this is done,
> > so we should get official word from him.
> 
> Yes, it builds once a day.  The build process currently takes quite long,
> since PDFs are also built every time.  We could change the procedure to
> rebuild the HTML more frequently or on checkin, but that would again
> make the PDFs mismatch the online docs, and I would say it's not necessary.
> 
> >> (And is that written down somewhere and I've just forgotten where to
> >> look...)
> > 
> > I don't think so, but it should be.
> 
> It is documented at <http://www.python.org/dev/doc>.  The docs themselves
> bear the "last updated" date only.

Unless I'm missing something, I don't see any docs there about the
automated build process and when and where it runs.

(Note that the old automated build process, the build.sh script, wasn't
documented in that sense anywhere that I know of, either.  And I can
never remember where its web status page is...at least for the
new docs procedure we have the last build date in an obvious place :)

--
R. David Murray                                      www.bitdance.com

From p.f.moore at gmail.com  Sat May  1 17:18:19 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 1 May 2010 16:18:19 +0100
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <20100501142857.3CAD51FC3F5@kimball.webabinitio.net>
References: <4BD95C6E.9080508@gmail.com>
	<20100429114047.C031C20059D@kimball.webabinitio.net>
	<hrgsla$nrm$1@dough.gmane.org>
	<20100501142857.3CAD51FC3F5@kimball.webabinitio.net>
Message-ID: <l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>

On 1 May 2010 15:28, R. David Murray <rdmurray at bitdance.com> wrote:
> Unless I'm missing something, I don't see any docs there about the
> automated build process and when and where it runs.

I assume Georg was referring to "Development versions of the Python
documentation are available on-line and updated daily".

Paul.

From tseaver at palladion.com  Sat May  1 19:12:12 2010
From: tseaver at palladion.com (Tres Seaver)
Date: Sat, 01 May 2010 13:12:12 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <x2j4222a8491004302041t87db5eccrde4aa7cd82b87f98@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de>	<61681639-C8FC-46F4-A02D-D40DF8733A08@gmail.com>	<loom.20100501T010005-397@post.gmane.org>	<u2j1afaf6161004301608pef79ab7fjff03080e15add2d@mail.gmail.com>	<s2u693bc9ab1004301610ud8734dc6i47ea64db0049e3e@mail.gmail.com>	<r2m1afaf6161004301614i2014a595xedbfcb0bde6285c4@mail.gmail.com>	<r2pca471dc21004301811l1bf7585euaf1f96d8a167e825@mail.gmail.com>
	<x2j4222a8491004302041t87db5eccrde4aa7cd82b87f98@mail.gmail.com>
Message-ID: <hrhndc$d0l$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesse Noller wrote:
> On Fri, Apr 30, 2010 at 9:11 PM, Guido van Rossum <guido at python.org> wrote:
>> On Fri, Apr 30, 2010 at 4:14 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>> 2010/4/30 Maciej Fijalkowski <fijall at gmail.com>:
>>>> On Fri, Apr 30, 2010 at 5:08 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>>>> 2010/4/30 Antoine Pitrou <solipsis at pitrou.net>:
>>>>>> Jesse Noller <jnoller <at> gmail.com> writes:
>>>>>>> Consider this a plaintitive -1 to any sort of rule-or-decision based
>>>>>>> on committee.
>>>>>>>
>>>>>>> I'd much rather a 2x4 to the forehead.
>>>>>> Oops, sorry but what does "a 2x4 to the forehead" mean?
>>>>>> (and "plaintitive" by the way?)
>>>>> The former means being hit by a board and by the latter, he probably
>>>>> means "plaintive" meaning melancholy.
>>>>>
>>>> Does it only work for english speaking non-metric system users? (2x4
>>>> is probably something imperial)
>>> Yes, that's a 5,04 by 10,08 for you. :)
>> Of course "2 by 4" is just the name. The actual measurements of such a
>> piece of lumber in the store are about 1.75 by 3.75 inch. :-)
>>
> 
> We americans round up! (all the time)

Lumber is, oddly, named for its "wet size" (when cut), rather than "dry
size" (when you buy it at the lumber yard after it dries).


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvcYOwACgkQ+gerLs4ltQ5SAgCdHnPJZ+UqTgjcTmW21iphpO1Y
BQ8An07mX/cy3opSpNyt0FE2jtVMtLIu
=Qqq1
-----END PGP SIGNATURE-----


From rdmurray at bitdance.com  Sun May  2 00:05:04 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 01 May 2010 18:05:04 -0400
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>
References: <4BD95C6E.9080508@gmail.com>
	<20100429114047.C031C20059D@kimball.webabinitio.net>
	<hrgsla$nrm$1@dough.gmane.org>
	<20100501142857.3CAD51FC3F5@kimball.webabinitio.net>
	<l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>
Message-ID: <20100501220504.D35AC200567@kimball.webabinitio.net>

On Sat, 01 May 2010 16:18:19 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
> On 1 May 2010 15:28, R. David Murray <rdmurray at bitdance.com> wrote:
> > Unless I'm missing something, I don't see any docs there about the
> > automated build process and when and where it runs.
> 
> I assume Georg was referring to "Development versions of the Python
> documentation are available on-line and updated daily".

Yes, most likely.  I think I am asking for more documentation than you
were :)

--
R. David Murray                                      www.bitdance.com

From martin at v.loewis.de  Sun May  2 00:44:22 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 02 May 2010 00:44:22 +0200
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <20100501220504.D35AC200567@kimball.webabinitio.net>
References: <4BD95C6E.9080508@gmail.com>	<20100429114047.C031C20059D@kimball.webabinitio.net>	<hrgsla$nrm$1@dough.gmane.org>	<20100501142857.3CAD51FC3F5@kimball.webabinitio.net>	<l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>
	<20100501220504.D35AC200567@kimball.webabinitio.net>
Message-ID: <4BDCAEC6.1020202@v.loewis.de>

R. David Murray wrote:
> On Sat, 01 May 2010 16:18:19 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 1 May 2010 15:28, R. David Murray <rdmurray at bitdance.com> wrote:
>>> Unless I'm missing something, I don't see any docs there about the
>>> automated build process and when and where it runs.
>> I assume Georg was referring to "Development versions of the Python
>> documentation are available on-line and updated daily".
> 
> Yes, most likely.  I think I am asking for more documentation than you
> were :)

Then I think you have to propose specific wording. It runs on
www.python.org (why would you expect it to run elsewhere???),
and what about "daily" do you consider imprecise? Do you really need
exact hour, minute, and second? What for?

Regards,
Martin

From ncoghlan at gmail.com  Sun May  2 05:38:02 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 02 May 2010 13:38:02 +1000
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>
References: <4BD95C6E.9080508@gmail.com>	<20100429114047.C031C20059D@kimball.webabinitio.net>	<hrgsla$nrm$1@dough.gmane.org>	<20100501142857.3CAD51FC3F5@kimball.webabinitio.net>
	<l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>
Message-ID: <4BDCF39A.7060206@gmail.com>

Paul Moore wrote:
> On 1 May 2010 15:28, R. David Murray <rdmurray at bitdance.com> wrote:
>> Unless I'm missing something, I don't see any docs there about the
>> automated build process and when and where it runs.
> 
> I assume Georg was referring to "Development versions of the Python
> documentation are available on-line and updated daily".

Thanks, that's what I was missing.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From brian at sweetapp.com  Sun May  2 09:07:25 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 2 May 2010 17:07:25 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4BA001E1.8080105@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
	<4B94DC30.2030904@gmail.com>
	<19349.25574.880099.668647@montanaro.dyndns.org>
	<4B979292.4080906@gmail.com>
	<E40B5636-E58C-4BDB-92B9-A7712DB2B531@sweetapp.com>
	<4BA001E1.8080105@gmail.com>
Message-ID: <E53BB307-F76A-4321-803A-5222B477CB5C@sweetapp.com>

I've updated the PEP to include:
- completion callbacks (for interoperability with Twisted Deferreds)
- a pointer to the discussion on stdlig-sig

See:
http://svn.python.org/view/peps/trunk/pep-3148.txt?r1=78618&r2=80679

Rejected ideas:
- Having a registration system for executors

Not yet addressed:
- where the package should live (someone in a "concurrent" package  
seems fine)
- having global executors with unbounded worker counts as a  
convenience [1]

[1] There are a few issues with global executors that need to be  
thought thought through i.e. when should workers be created and when  
should they be terminated. I'd be happy to defer this idea unless  
someone is passionate about it (in which case it would be great if  
they'd step in with concrete ideas).

Cheers,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100502/8545aca5/attachment.html>

From yaniv at aknin.name  Sun May  2 14:39:01 2010
From: yaniv at aknin.name (Yaniv Aknin)
Date: Sun, 2 May 2010 22:39:01 +1000
Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do
	triage.
In-Reply-To: <20100428112647.A2D631FF378@kimball.webabinitio.net>
References: <4BD4A69A.4080507@tummy.com> <hr4l8k$tp6$1@dough.gmane.org>
	<87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp>
	<201004281046.37939.steve@pearwood.info>
	<87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20100428112647.A2D631FF378@kimball.webabinitio.net>
Message-ID: <i2yec3cf7fd1005020539g9dc31308z72814bee0defac1d@mail.gmail.com>

>> Yes, in the last year in particular there has been some excellent effort
>> of maintaining the issue tracker content. But the question still remains
>> - who are we worried about offending?

> The people who are potential new contributors but don't currently know
> anyone in the Python community.

By definition these 'potential new and unknown contributors' are discrete
and probably can't be characterized as a whole. However, seeing myself
as one of the discrete elements in that group, I think it's worthwhile for me
to pipe in here and say that I won't be 'offended' or think of it as nepotism
if someone gets foo-privilege before I do because he happens to know core
developer (some other 'potential new contributer' lurking here feels otherwise?
- speak up!).

I don't know the community (yet) and I can't say this for sure, but my current
gut feeling about the Python community (and pretty much any OSS I can think
of) is that in the long run, I'll be judged on merit just like any other guy, no
matter who they know.

 - Yaniv

From ncoghlan at gmail.com  Sun May  2 18:55:59 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 03 May 2010 02:55:59 +1000
Subject: [Python-Dev] PEP or issue recording rationale for allowing multiple
	context managers?
Message-ID: <4BDDAE9F.2060105@gmail.com>

I was looking for a reference for the addition of multiple context
manager support to with statements in 3.1 and 2.7 and came up empty
(aside from the initial python-ideas thread [1] that I linked to from
PEP 377).

I was hoping to find something that clearly spelled out:
- the two major flaws in contextlib.nested*
- the parallel with import statements for the precise chosen syntax
- Guido's blessing to go ahead and do it

*Those flaws being
- that __init__/__new__ methods of inner context managers aren't covered
by outer context managers
- that outer context managers can't suppress exceptions from inner
__enter__ methods

Note that I'm not complaining about the decision itself (that would be
silly, since I agree with the outcome), I'm just trying to find
something to point to about it that is a little more concrete than a
python-ideas thread.

Cheers,
Nick.

[1]
http://mail.python.org/pipermail/python-ideas/2009-March/003188.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From olemis at gmail.com  Sun May  2 20:15:09 2010
From: olemis at gmail.com (Olemis Lang)
Date: Sun, 2 May 2010 13:15:09 -0500
Subject: [Python-Dev] Alias for distutils.file_util.write_file in e.g.
	shutils
Message-ID: <k2r24ea26601005021115meacf7633p90ebe2c449a9dcd9@mail.gmail.com>

Hello !

Often I have the contents to be written in a file at a given path that
I know as well. I recently tried to find a function in stdlib to do
that and to my surprise this is what I found :

  - Such function exists
  - It's `distutils.file_util.write_file`

IMO the last place where people'd look for such a function is inside
`distutils` package. Besides I reviewed modules listed under `File and
directory access` category in `Library Reference` and found nothing
even similar.

Q:
  - Is it a good idea to provide a similar function in
    e.g. shutils module ?

Probably there's already a better way to do this and my comment is
just irrelevant . Anyway is just a suggestion ;o)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Soporte para AMF (RPC) en Trac -
http://feedproxy.google.com/~r/simelo-es/~3/9dYgHeK5Be8/soporte-para-amf-rpc-en-trac.html

From ziade.tarek at gmail.com  Sun May  2 20:36:55 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 2 May 2010 20:36:55 +0200
Subject: [Python-Dev] Alias for distutils.file_util.write_file in e.g.
	shutils
In-Reply-To: <k2r24ea26601005021115meacf7633p90ebe2c449a9dcd9@mail.gmail.com>
References: <k2r24ea26601005021115meacf7633p90ebe2c449a9dcd9@mail.gmail.com>
Message-ID: <k2l94bdd2611005021136x46272daak545d4090d4f26fec@mail.gmail.com>

On Sun, May 2, 2010 at 8:15 PM, Olemis Lang <olemis at gmail.com> wrote:
> Hello !
>
> Often I have the contents to be written in a file at a given path that
> I know as well. I recently tried to find a function in stdlib to do
> that and to my surprise this is what I found :
>
> ?- Such function exists
> ?- It's `distutils.file_util.write_file`
>
> IMO the last place where people'd look for such a function is inside
> `distutils` package. Besides I reviewed modules listed under `File and
> directory access` category in `Library Reference` and found nothing
> even similar.
>
> Q:
> ?- Is it a good idea to provide a similar function in
> ? ?e.g. shutils module ?

A: Yes :)

Basically, anything useful in distutils.file_util and
distutils.dir_util can maove in Shutil.
That's why I added make_archive (and unpack_archive)

Please add an issue, I'll work on adding this function,

Thanks
Tarek

-- 
Tarek Ziad? | http://ziade.org

From rdmurray at bitdance.com  Sun May  2 21:48:22 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 02 May 2010 15:48:22 -0400
Subject: [Python-Dev] Enhanced tracker privileges for dangerjim to do
	triage.
In-Reply-To: <i2yec3cf7fd1005020539g9dc31308z72814bee0defac1d@mail.gmail.com>
References: <4BD4A69A.4080507@tummy.com> <hr4l8k$tp6$1@dough.gmane.org>
	<87fx2hk3ch.fsf@uwakimon.sk.tsukuba.ac.jp>
	<201004281046.37939.steve@pearwood.info>
	<87bpd4800f.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20100428112647.A2D631FF378@kimball.webabinitio.net>
	<i2yec3cf7fd1005020539g9dc31308z72814bee0defac1d@mail.gmail.com>
Message-ID: <20100502194823.171A8200778@kimball.webabinitio.net>

On Sun, 02 May 2010 22:39:01 +1000, Yaniv Aknin <yaniv at aknin.name> wrote:
> >> Yes, in the last year in particular there has been some excellent effort
> >> of maintaining the issue tracker content. But the question still remains
> >> - who are we worried about offending?
> 
> > The people who are potential new contributors but don't currently know
> > anyone in the Python community.
> 
> By definition these 'potential new and unknown contributors' are discrete
> and probably can't be characterized as a whole. However, seeing myself
> as one of the discrete elements in that group, I think it's worthwhile for me
> to pipe in here and say that I won't be 'offended' or think of it as nepotism
> if someone gets foo-privilege before I do because he happens to know core
> developer (some other 'potential new contributer' lurking here feels otherwise?
> - speak up!).
> 
> I don't know the community (yet) and I can't say this for sure, but my current
> gut feeling about the Python community (and pretty much any OSS I can think
> of) is that in the long run, I'll be judged on merit just like any other guy, no
> matter who they know.

Thank you very much for this feedback.

--
R. David Murray                                      www.bitdance.com

From rdmurray at bitdance.com  Sun May  2 22:21:28 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 02 May 2010 16:21:28 -0400
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <4BDCAEC6.1020202@v.loewis.de>
References: <4BD95C6E.9080508@gmail.com>
	<20100429114047.C031C20059D@kimball.webabinitio.net>
	<hrgsla$nrm$1@dough.gmane.org>
	<20100501142857.3CAD51FC3F5@kimball.webabinitio.net>
	<l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>
	<20100501220504.D35AC200567@kimball.webabinitio.net>
	<4BDCAEC6.1020202@v.loewis.de>
Message-ID: <20100502202128.CEEC3200395@kimball.webabinitio.net>

On Sun, 02 May 2010 00:44:22 +0200, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
> R. David Murray wrote:
> > On Sat, 01 May 2010 16:18:19 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
> >> On 1 May 2010 15:28, R. David Murray <rdmurray at bitdance.com> wrote:
> >>> Unless I'm missing something, I don't see any docs there about the
> >>> automated build process and when and where it runs.
> >> I assume Georg was referring to "Development versions of the Python
> >> documentation are available on-line and updated daily".
> > 
> > Yes, most likely.  I think I am asking for more documentation than you
> > were :)
> 
> Then I think you have to propose specific wording. It runs on
> www.python.org (why would you expect it to run elsewhere???),

I had in fact overlooked the indicated sentence when reading the page.

It's not that I expect it to run somewhere else, but that it did indeed
used to run somewhere else (if I'm reading the build.sh script correctly),
so I think it is worth making it explicit.  Also note that it isn't
obvious (without resolving the host names) that docs.python.org and
www.python.org are the same machine.

> and what about "daily" do you consider imprecise? Do you really need
> exact hour, minute, and second? What for?

I wasn't asking for more precision than daily (I just hadn't seen it), but
now that I think about it it would indeed be nice to know when the cron
job starts, so that we know that if the checkin didn't happen before then,
it won't show up in the online docs until the next day.  I don't think
this is particularly *important* to know, it would just be nice to know.

Is it only the development versions that get updated, or do our updates
to the next bug fix release also get posted daily (and those are the docs
web site visitors normally see, I believe)?  I was under the impression
that it was the latter, but that Georg had also changed things so that
a reference to a particular dot release got the snapshot of the docs
that were shipped with that dot release.  This impression seems to be
confirmed by examination of the various pages involved.

To answer your question about what wording I'd like, I think that it would
be worthwhile to say somewhere (not necessarily on that page...maybe in
the doc README.txt?...but it could be ont that page...) that the docs are
auto-built by a cron job on the server hosting docs.python.org running
'make dist' in the doc directory of a freshly updated checkout and
then....doing something with the resulting files?  Or maybe the apache
URLs point right at the dist directory in that autobuild checkout?
Anyway, exactly how it all works is what I would like to see documented.

Background: one of my tasks for one of my customers is helping them try to
make it so that an outsider coming in could learn everything they needed
to know to operate the system from the available docs...a goal that they
are nowhere near achieving, but which I think is a worthwhile goal for
which to strive.

--
R. David Murray                                      www.bitdance.com

From martin at v.loewis.de  Sun May  2 22:56:59 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 02 May 2010 22:56:59 +0200
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <20100502202128.CEEC3200395@kimball.webabinitio.net>
References: <4BD95C6E.9080508@gmail.com>
	<20100429114047.C031C20059D@kimball.webabinitio.net>
	<hrgsla$nrm$1@dough.gmane.org>
	<20100501142857.3CAD51FC3F5@kimball.webabinitio.net>
	<l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>
	<20100501220504.D35AC200567@kimball.webabinitio.net>
	<4BDCAEC6.1020202@v.loewis.de>
	<20100502202128.CEEC3200395@kimball.webabinitio.net>
Message-ID: <4BDDE71B.9050104@v.loewis.de>

> I wasn't asking for more precision than daily (I just hadn't seen it), but
> now that I think about it it would indeed be nice to know when the cron
> job starts, so that we know that if the checkin didn't happen before then,
> it won't show up in the online docs until the next day.  I don't think
> this is particularly *important* to know, it would just be nice to know.

That's different from asking that it be documented, though. I don't mind
you knowing (it's at 15:00 local time for the build machine, which sits
in the Europe/Amsterdam timezone). Just ask specific questions, and
people may give specific answers. Now that you know, I still don't think
it needs to be documented (else: where would that end? Would you want to
know account name and uid of the build also, and the partition of the
hard disk where the files are stored? Not even I know the rack slot in
which the machine sits).

> 
> Is it only the development versions that get updated, or do our updates
> to the next bug fix release also get posted daily (and those are the docs
> web site visitors normally see, I believe)?

That's what the documentation claims, yes. The build script currently
has these targets:

BRANCHES = [
    # checkout, target, isdev
    (BUILDROOT + '/python26', WWWROOT, False),
    (BUILDROOT + '/python31', WWWROOT + '/py3k', False),
    (BUILDROOT + '/python27', WWWROOT + '/dev', True),
    (BUILDROOT + '/python32', WWWROOT + '/dev/py3k', True),
]

> To answer your question about what wording I'd like, I think that it would
> be worthwhile to say somewhere (not necessarily on that page...maybe in
> the doc README.txt?...but it could be ont that page...) that the docs are
> auto-built by a cron job on the server hosting docs.python.org running
> 'make dist' in the doc directory of a freshly updated checkout and
> then....doing something with the resulting files?

Actually, the command is rather like this:

  'cd Doc; make autobuild-%s' % (isdev and 'dev' or 'stable')

> Background: one of my tasks for one of my customers is helping them try to
> make it so that an outsider coming in could learn everything they needed
> to know to operate the system from the available docs...a goal that they
> are nowhere near achieving, but which I think is a worthwhile goal for
> which to strive.

For this kind of work, I think looking at the actual installation is
more productive to learn how things are done (perhaps opposed to how
they were originally planned to be done). I didn't know how this all
worked myself, either, but, using the root account, it took me only a
minute to find out - much faster than finding the documentation that may
have explained it in detail.

The only starting point that you need is the machine that you know it
runs on.

Regards,
Martin

From brett at python.org  Sun May  2 23:24:47 2010
From: brett at python.org (Brett Cannon)
Date: Sun, 2 May 2010 14:24:47 -0700
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <z2g94bdd2611005010200vb4a590c5jeb0787e9a1f5f267@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> 
	<4BDB3D6C.4050909@holdenweb.com>
	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
	<z2g94bdd2611005010200vb4a590c5jeb0787e9a1f5f267@mail.gmail.com>
Message-ID: <t2vbbaeab101005021424ud8e5828ej3988521337a7fef@mail.gmail.com>

On Sat, May 1, 2010 at 02:00, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> On Fri, Apr 30, 2010 at 11:25 PM, Guido van Rossum <guido at python.org>
> wrote:
> [..]
> >>>
> >>> Without a BDFL, I think we need a committee to make decisions, e.g. by
> >>> majority vote amongst committers.
> >>
> >> Couldn't we just go with the FLUFL?
> >
> > IIRC in the IETF this is done by the committee chair. I think it's a
> > good idea to have this be a single person to avoid endless indecision.
>
> BPD = Benevolent Pep Dictator
> BPC = Benevolent Pep Caliph   (sounds good in French, not sure in English
> ;) )
>
> What about naming several BPD + and have one BPC elected each year by
> all the core devs ?
>
> == BPD ==
>
> I am not sure if this would work for all areas in Python-core, but
> looking at the maintainers.rst file, it looks like we could name for
> example Brett for all the import machinery things, Marc-Andr? for all
> the unicode things, I could be the one about packaging, etc.
>
> If we could manage to split the python-core development into
> categories, and add these categories in the PEP metadata, that would
> define who takes the final decision in case we can't reach consensus.
>

This sounds like the lieutenant setup they have for the Linux kernel. I have
no clue how that works out for them.


>
> == BPC ==
>
> Of course some PEPs could concern several categories,  so we would
> still need some kind of Pep dictator if there's no consensus.  So what
> about electing a BPC every year ?
>

So there is a "single voice" issue here (that also applies to Martin's idea
of having the release manager make the call as that is a rotating position).
One of the reasons, I think, the BDFL style of decision making has worked
out is that it lets Guido keep Python consistent; the language is always
striving to meet his mental model of the language more and more. Having this
rotate amongst us will not allow us to have this benefit. It also raises the
chance of arguing over who takes over the rotating position and that just
falls down into the hellish hole of politics that I don't think any of us
want to see happen.

But even ignoring my worry/point, what are we even discussing here? Guido
has said multiple time he is not retiring, simply scaling back his
involvement. So are we trying to figure out how to make our own decisions
about Python when Guido is not available to make one or simply doesn't care
enough to pronounce? If that's the case then we should probably choose a
vice BDFL (sort of a Benevolent Dicatator at Guido's Discretion) to keep the
voice of Python as uniform as possible. I guess this person would become the
assumed successor of the BDFL title if Guido ever does retire and the
decision is made to continue on with active development of the language
instead of just going into maintenance mode, but hopefully that problem will
be a long ways off.

If we are discussing something else, then I don't know what we are all
talking about here other than measurements of standard pieces of wood. =)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100502/ddda6332/attachment.html>

From g.brandl at gmx.net  Mon May  3 06:25:15 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 03 May 2010 06:25:15 +0200
Subject: [Python-Dev] Frequency of the dev docs autobuild
In-Reply-To: <20100502202128.CEEC3200395@kimball.webabinitio.net>
References: <4BD95C6E.9080508@gmail.com>	<20100429114047.C031C20059D@kimball.webabinitio.net>	<hrgsla$nrm$1@dough.gmane.org>	<20100501142857.3CAD51FC3F5@kimball.webabinitio.net>	<l2h79990c6b1005010818t9447726cx36009787ed014036@mail.gmail.com>	<20100501220504.D35AC200567@kimball.webabinitio.net>	<4BDCAEC6.1020202@v.loewis.de>
	<20100502202128.CEEC3200395@kimball.webabinitio.net>
Message-ID: <hrlj5m$4o2$1@dough.gmane.org>

Am 02.05.2010 22:21, schrieb R. David Murray:
> On Sun, 02 May 2010 00:44:22 +0200, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:
>> R. David Murray wrote:
>> > On Sat, 01 May 2010 16:18:19 +0100, Paul Moore <p.f.moore at gmail.com> wrote:
>> >> On 1 May 2010 15:28, R. David Murray <rdmurray at bitdance.com> wrote:
>> >>> Unless I'm missing something, I don't see any docs there about the
>> >>> automated build process and when and where it runs.
>> >> I assume Georg was referring to "Development versions of the Python
>> >> documentation are available on-line and updated daily".
>> > 
>> > Yes, most likely.  I think I am asking for more documentation than you
>> > were :)
>> 
>> Then I think you have to propose specific wording. It runs on
>> www.python.org (why would you expect it to run elsewhere???),
> 
> I had in fact overlooked the indicated sentence when reading the page.

And I realize it's less than you wanted to see :)

But I also realized it's very easy to give more detail without crowding
that page; since the script used for the daily build is in SVN under
Doc/tools/dailybuild.py; and that also contains the hostname.

I've updated the dev/doc/ page to include a link to the script text.

cheers,
Georg


From jcea at jcea.es  Mon May  3 06:57:44 2010
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 03 May 2010 06:57:44 +0200
Subject: [Python-Dev] Should we drop active support of OSF/1?
In-Reply-To: <4BD5F0D8.4020007@v.loewis.de>
References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de>
Message-ID: <4BDE57C8.5010509@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 26/04/10 22:00, "Martin v. L?wis" wrote:
>> Maybe we can drop OSF/1 safely supporting Tru64 yet, but we don't have
>> any buildbot running any of this systems...
> 
> Dropping support is fine with me, in the long term. If PEP 11 is truly
> followed, we should deprecate support in one version, and drop it in the
> next one. This would mean we add a easy-to-remove configure error in
> 3.2, and remove the support in 3.3.

Would be enough to raise an "ERROR" at configure time if OSF test is
positive?. To delete that intentional "ERROR" would be trivial.

In 3.3 I would remove the configure tests and the "#if" conditional
compilation related to them.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS95XyJlgi5GaxT1NAQLQLQP/cE2E27g8hSAbH7XzewDDYzx1AZ11ja55
xuR0PLTg/yQgE+4oifMa56Rs54YVRtRkh+i7B5yR5+lcI2DJgfqiu2Q9Of8KbwHp
SQ2BVlshUmjhImWh486Qj6aQ9sF3xMdaxOGVLG+SJOBAEVw4UWEkZYPpQOuKRTfG
K0SYYLhERoY=
=CUNo
-----END PGP SIGNATURE-----

From regebro at gmail.com  Mon May  3 08:19:54 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 3 May 2010 08:19:54 +0200
Subject: [Python-Dev] Should we drop active support of OSF/1?
In-Reply-To: <4BDE57C8.5010509@jcea.es>
References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de> 
	<4BDE57C8.5010509@jcea.es>
Message-ID: <k2u319e029f1005022319keeecae5cl359bd33b55a997f4@mail.gmail.com>

On Mon, May 3, 2010 at 06:57, Jesus Cea <jcea at jcea.es> wrote:
> Would be enough to raise an "ERROR" at configure time if OSF test is
> positive?. To delete that intentional "ERROR" would be trivial.

Oh really? I have no clue of how to do that. Doesn't like like a good
deprecation to me. :)
Is printing a warning not easily feasible in ./configure?

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From solipsis at pitrou.net  Mon May  3 13:53:29 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 May 2010 11:53:29 +0000 (UTC)
Subject: [Python-Dev] Should we drop active support of OSF/1?
References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de>
	<4BDE57C8.5010509@jcea.es>
Message-ID: <loom.20100503T135248-494@post.gmane.org>

Jesus Cea <jcea <at> jcea.es> writes:
> 
> Would be enough to raise an "ERROR" at configure time if OSF test is
> positive?. To delete that intentional "ERROR" would be trivial.

I think it's ok.  Noone will probably notice anyway.

Regards

Antoine.



From olemis at gmail.com  Mon May  3 14:47:33 2010
From: olemis at gmail.com (Olemis Lang)
Date: Mon, 3 May 2010 07:47:33 -0500
Subject: [Python-Dev] Alias for distutils.file_util.write_file in e.g.
	shutils
In-Reply-To: <k2l94bdd2611005021136x46272daak545d4090d4f26fec@mail.gmail.com>
References: <k2r24ea26601005021115meacf7633p90ebe2c449a9dcd9@mail.gmail.com>
	<k2l94bdd2611005021136x46272daak545d4090d4f26fec@mail.gmail.com>
Message-ID: <k2h24ea26601005030547l66614bf2w8f12203b9465fb2f@mail.gmail.com>

On Sun, May 2, 2010 at 1:36 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> a similar function in
>> ? ?e.g. shutils module ?
>
> A: Yes :)
>
> Basically, anything useful in distutils.file_util and
> distutils.dir_util can maove in Shutil.
> That's why I added make_archive (and unpack_archive)
>
> Please add an issue, I'll work on adding this function,
>

http://bugs.python.org/issue8604

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

From barry at python.org  Mon May  3 17:30:22 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 May 2010 11:30:22 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <z2zea2499da1004302358x38760470vcfdd2a2e4b73e656@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy>
	<z2zea2499da1004302358x38760470vcfdd2a2e4b73e656@mail.gmail.com>
Message-ID: <20100503113022.59a2e609@heresy>

On May 01, 2010, at 08:58 AM, Dirkjan Ochtman wrote:

>On Fri, Apr 30, 2010 at 21:09, Barry Warsaw <barry at python.org> wrote:
>>>Though maybe it should be called Conclusion instead of Accepted and
>>>used for Rejected PEPs, as well?
>>
>> Good point. ?What do you think about 'Resolution'?
>
>Fine with me.

I've updated PEP 1 and the supporting scripts to accept a Resolution: header.
Note that in PEP 1 I wrote:

    *Note: The Resolution header is required for Standards Track PEPs
    only.  It contains a URL that should point to an email message or
    other web resource where the pronouncement about the PEP is made.*

but in fact, the scripts make Resolution optional (it's kind of a pain to make
it required just for Standards Track PEPs - contributions welcome).

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100503/0e262342/attachment.pgp>

From fdrake at acm.org  Mon May  3 17:38:31 2010
From: fdrake at acm.org (Fred Drake)
Date: Mon, 3 May 2010 11:38:31 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <20100503113022.59a2e609@heresy>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy>
	<z2zea2499da1004302358x38760470vcfdd2a2e4b73e656@mail.gmail.com>
	<20100503113022.59a2e609@heresy>
Message-ID: <j2x9cee7ab81005030838t54377758qbb983f726681cda8@mail.gmail.com>

On Mon, May 3, 2010 at 11:30 AM, Barry Warsaw <barry at python.org> wrote:
> but in fact, the scripts make Resolution optional (it's kind of a pain to make
> it required just for Standards Track PEPs - contributions welcome).

It will also be a pain to retroactively update older PEPs with the
newly-required metadata; leaving it optional (in practice) seems
acceptable.  It should only apply to newly-resolved PEPs, which is
also likely a pain to build into the script.


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller

From barry at python.org  Mon May  3 17:44:13 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 May 2010 11:44:13 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <4BDB3D6C.4050909@holdenweb.com>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<4BDB3D6C.4050909@holdenweb.com>
Message-ID: <20100503114413.5b5e9e3e@heresy>

On Apr 30, 2010, at 04:28 PM, Steve Holden wrote:

>Martin v. L?wis wrote:
>>> As to Guido's point about the decision making process, Nick's right.  I just
>>> want to make sure we can capture the resolution in the PEP, be it by BDFL
>>> pronouncement or "hey, silence is acceptance" email.
>> 
>> I don't think "silence is acceptance" will work out in practice. For
>> issues where a PEP was written in the first place, somebody will
>> *always* object, and forever so, hoping that what he considers a mistake
>> will not be done. The advantage of Guido acting as BDFL was that
>> somebody would make an decision ultimately, one which both proponents
>> and opponents of the PEP would accept.
>> 
>> Without a BDFL, I think we need a committee to make decisions, e.g. by
>> majority vote amongst committers.
>
>Couldn't we just go with the FLUFL?

We could of course, and I would serve honorably <wink>.  But Guido's not
*that* much older than me, so you'd probably only get a few useful years out
of me before I start getting increasingly eccentric, re-opening and approving
PEPs like 313, 336, and 666.

In the meantime, let's groom Benjamin to be the Sacred Next Uncle Galvanizing
the Gamut of Language Evolution.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100503/3ab7cd92/attachment.pgp>

From barry at python.org  Mon May  3 17:57:52 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 May 2010 11:57:52 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <4BDBB846.5020305@v.loewis.de>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<4BDB3D6C.4050909@holdenweb.com>
	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
	<4BDBB846.5020305@v.loewis.de>
Message-ID: <20100503115752.4f19f2e3@heresy>

On May 01, 2010, at 07:12 AM, Martin v. L?wis wrote:

>> IIRC in the IETF this is done by the committee chair. I think it's a
>> good idea to have this be a single person to avoid endless indecision.
>
>It then seems that this role should go to the release manager of the
>upcoming feature release. Assuming Georg can accept this additional
>responsibility.

I do think it makes sense for the RM to assume these responsibilities where
Guido either can't or doesn't want to make the final decision.  I think it
will fairly substantially increase the workload on the RM, so perhaps there
are ways to off-load some of the current responsibilities (e.g. updating the
website for each release).  I also think that RMs should be term-limited so
that we can spread more experience within the community.  And past-RMs can
provide a sort of consultation group where contentious decisions can be
discussed and advice gathered.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100503/6ae05086/attachment-0001.pgp>

From guido at python.org  Mon May  3 18:40:44 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 May 2010 09:40:44 -0700
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <20100503115752.4f19f2e3@heresy>
References: <20100427134618.564e3e3b@heresy>
	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>
	<4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de> 
	<4BDB3D6C.4050909@holdenweb.com>
	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
	<4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy>
Message-ID: <h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>

On Mon, May 3, 2010 at 8:57 AM, Barry Warsaw <barry at python.org> wrote:
> On May 01, 2010, at 07:12 AM, Martin v. L?wis wrote:
>
>>> IIRC in the IETF this is done by the committee chair. I think it's a
>>> good idea to have this be a single person to avoid endless indecision.
>>
>>It then seems that this role should go to the release manager of the
>>upcoming feature release. Assuming Georg can accept this additional
>>responsibility.
>
> I do think it makes sense for the RM to assume these responsibilities where
> Guido either can't or doesn't want to make the final decision. ?I think it
> will fairly substantially increase the workload on the RM, so perhaps there
> are ways to off-load some of the current responsibilities (e.g. updating the
> website for each release). ?I also think that RMs should be term-limited so
> that we can spread more experience within the community. ?And past-RMs can
> provide a sort of consultation group where contentious decisions can be
> discussed and advice gathered.

While I certainly won't object if a release manager volunteers to also
be the final PEP arbiter, I don't want to make it a job requirement
(or even an implied expectation). The responsibility of a release
manager is very much in the here and now and the near future --
stability and consistency of the current release. Being PEP arbiter
requires a somewhat different mindset, more focused on the long-term
health of the language and its community.

-- 
--Guido van Rossum (python.org/~guido)

From martin at v.loewis.de  Mon May  3 21:40:02 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 03 May 2010 21:40:02 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <20100503114413.5b5e9e3e@heresy>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>	<4BDB34CE.9080804@v.loewis.de>	<4BDB3D6C.4050909@holdenweb.com>
	<20100503114413.5b5e9e3e@heresy>
Message-ID: <4BDF2692.3040006@v.loewis.de>

> In the meantime, let's groom Benjamin to be the Sacred Next Uncle Galvanizing
> the Gamut of Language Evolution.

I don't think anybody having such a position permanently can really work.

Regards,
Martin

From martin at v.loewis.de  Mon May  3 21:55:42 2010
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 03 May 2010 21:55:42 +0200
Subject: [Python-Dev] Should we drop active support of OSF/1?
In-Reply-To: <4BDE57C8.5010509@jcea.es>
References: <4BD5D3F5.7090503@jcea.es> <4BD5F0D8.4020007@v.loewis.de>
	<4BDE57C8.5010509@jcea.es>
Message-ID: <4BDF2A3E.2030609@v.loewis.de>

Jesus Cea wrote:
> On 26/04/10 22:00, "Martin v. L?wis" wrote:
>>> Maybe we can drop OSF/1 safely supporting Tru64 yet, but we don't have
>>> any buildbot running any of this systems...
>> Dropping support is fine with me, in the long term. If PEP 11 is truly
>> followed, we should deprecate support in one version, and drop it in the
>> next one. This would mean we add a easy-to-remove configure error in
>> 3.2, and remove the support in 3.3.
> 
> Would be enough to raise an "ERROR" at configure time if OSF test is
> positive?. To delete that intentional "ERROR" would be trivial.

That's actually what the PEP suggests (or means to suggest). Users
willing to take over maintenance would, at a minimum, know how to remove
that error. If a user doesn't know how to remove it, they can ask for
help, but more importantly, they probably consider it already
unsupported - which is exactly the desired effect.

Just to rephrase: the point of this deprecation cycle is to give users a
chance to take over maintenance, and to declare, in public, that they
are interested in that port. If nobody steps forward, we can safely
assume that nobody cares.

Regards,
Martin

From benjamin at python.org  Mon May  3 22:58:46 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 3 May 2010 15:58:46 -0500
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy> <4BD75D54.8050307@gmail.com>
	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>
	<20100430150918.308984aa@heresy> <4BDB34CE.9080804@v.loewis.de>
	<4BDB3D6C.4050909@holdenweb.com>
	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>
	<4BDBB846.5020305@v.loewis.de> <20100503115752.4f19f2e3@heresy>
	<h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>
Message-ID: <y2x1afaf6161005031358i4d7e8bd9o5586fed512c2c334@mail.gmail.com>

2010/5/3 Guido van Rossum <guido at python.org>:
> On Mon, May 3, 2010 at 8:57 AM, Barry Warsaw <barry at python.org> wrote:
>> I do think it makes sense for the RM to assume these responsibilities where
>> Guido either can't or doesn't want to make the final decision.  I think it
>> will fairly substantially increase the workload on the RM, so perhaps there
>> are ways to off-load some of the current responsibilities (e.g. updating the
>> website for each release).  I also think that RMs should be term-limited so
>> that we can spread more experience within the community.  And past-RMs can
>> provide a sort of consultation group where contentious decisions can be
>> discussed and advice gathered.

I'm very -1 on the release manager making decisions on PEPs. The RM
right now basically makes a release schedule and cuts releases. The
little power he has is mostly used for rejecting features in the beta
period. This task should not be politicized.



-- 
Regards,
Benjamin

From steve at holdenweb.com  Mon May  3 23:38:12 2010
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 03 May 2010 17:38:12 -0400
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <y2x1afaf6161005031358i4d7e8bd9o5586fed512c2c334@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>
	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de>	<4BDB3D6C.4050909@holdenweb.com>	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>	<4BDBB846.5020305@v.loewis.de>
	<20100503115752.4f19f2e3@heresy>	<h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>
	<y2x1afaf6161005031358i4d7e8bd9o5586fed512c2c334@mail.gmail.com>
Message-ID: <4BDF4244.6060605@holdenweb.com>

Benjamin Peterson wrote:
> 2010/5/3 Guido van Rossum <guido at python.org>:
>> On Mon, May 3, 2010 at 8:57 AM, Barry Warsaw <barry at python.org> wrote:
>>> I do think it makes sense for the RM to assume these responsibilities where
>>> Guido either can't or doesn't want to make the final decision.  I think it
>>> will fairly substantially increase the workload on the RM, so perhaps there
>>> are ways to off-load some of the current responsibilities (e.g. updating the
>>> website for each release).  I also think that RMs should be term-limited so
>>> that we can spread more experience within the community.  And past-RMs can
>>> provide a sort of consultation group where contentious decisions can be
>>> discussed and advice gathered.
> 
> I'm very -1 on the release manager making decisions on PEPs. The RM
> right now basically makes a release schedule and cuts releases. The
> little power he has is mostly used for rejecting features in the beta
> period. This task should not be politicized.
> 
That seems reasonable, but it also seems reasonable to try and simplify
the task of updating the web site for each release. There's plenty to do
 without having to fight that issue too.

I'm copying this message to Rich Leland, who is currently making a study
of the PSF's web requirements, to make sure this gets folded in. Many of
the tasks are essentially macrogeneration, and unless automated will
remain error-prone.

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See PyCon Talks from Atlanta 2010  http://pycon.blip.tv/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/


From jcea at jcea.es  Mon May  3 23:39:45 2010
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 03 May 2010 23:39:45 +0200
Subject: [Python-Dev] Should we drop active support of OSF/1?
In-Reply-To: <4BDF2A3E.2030609@v.loewis.de>
References: <4BD5D3F5.7090503@jcea.es>
	<4BD5F0D8.4020007@v.loewis.de>	<4BDE57C8.5010509@jcea.es>
	<4BDF2A3E.2030609@v.loewis.de>
Message-ID: <4BDF42A1.1060903@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/05/10 21:55, "Martin v. L?wis" wrote:
> Just to rephrase: the point of this deprecation cycle is to give users a
> chance to take over maintenance, and to declare, in public, that they
> are interested in that port. If nobody steps forward, we can safely
> assume that nobody cares.

http://bugs.python.org/issue8606 . I will take care of the final cleanup
in 3.3, if nobody stands up to keep OSF* alive.

I am deprecating OSF*. If it is "too much", we can change it to "OSF1".

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS99CoJlgi5GaxT1NAQLSygQAiYD0UQAKl2vv1alwmMSP02AgHxN9e3Le
bfMqysr2BE9fnG3WQlUKFn6pjbnnANFU2HRhTUJkaCNslewnUeg6pQ/s0xu321GL
HQ6v6JHSOXxRoyyZ21Wq2ECNg+tYWGkM7Eohsif8XMjTeL0lP7ddYZSFaJbK0PfM
BSK6DAV93/o=
=nlYU
-----END PGP SIGNATURE-----

From brett at python.org  Tue May  4 00:37:22 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 3 May 2010 15:37:22 -0700
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code base
Message-ID: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>

Since 2.7 is probably going to exist for a while, I am running Clang 2.7's
static analyzer (``clang --static``) over trunk. It's mostly just finding
stuff like unneeded variable initialization or variables that are never used
(compilation is picking up unused returned values, almost all from
PyObject_INIT).

When I check in these changes I will do it file by file, but my question is
how to handle Misc/NEWS. I have gone through the underscores and the 'a's in
Modules and already have six modified files, so the final count might be a
little high. Do people want individual entries per file, or simply a single
entry that lists each file modified?

We should probably go through the C code and fix the whitespace before we
hit 2.7 final (there is a ton of lines with extraneous spaces).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100503/ccc794f0/attachment.html>

From benjamin at python.org  Tue May  4 00:47:48 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 3 May 2010 17:47:48 -0500
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
Message-ID: <t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>

2010/5/3 Brett Cannon <brett at python.org>:
> Since 2.7 is probably going to exist for a while, I am running Clang 2.7's
> static analyzer (``clang --static``) over trunk. It's mostly just finding
> stuff like unneeded variable initialization or variables that are never used
> (compilation is picking up unused returned values, almost all from
> PyObject_INIT).
> When I check in these changes I will do it file by file, but my question is
> how to handle Misc/NEWS. I have gone through the underscores and the 'a's in
> Modules and already have six modified files, so the final count might be a
> little high. Do people want individual entries per file, or simply a single
> entry that lists each file modified?

Do these changes even warrant a NEWS entry? NEWS is more for user
facing changes.

> We should probably go through the C code and fix the whitespace before we
> hit 2.7 final (there is a ton of lines with extraneous spaces).

Why?

> -Brett



-- 
Regards,
Benjamin

From solipsis at pitrou.net  Tue May  4 01:00:02 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 May 2010 23:00:02 +0000 (UTC)
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the
	code	base
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
Message-ID: <hrnkhi$ldo$1@dough.gmane.org>


Benjamin Peterson <benjamin <at> python.org> writes:
> 
> 2010/5/3 Brett Cannon <brett <at> python.org>:
> > When I check in these changes I will do it file by file, but my 
question is
> > how to handle Misc/NEWS. I have gone through the underscores and the 
'a's in
> > Modules and already have six modified files, so the final count might 
be a
> > little high. Do people want individual entries per file, or simply a 
single
> > entry that lists each file modified?
> 
> Do these changes even warrant a NEWS entry? NEWS is more for user
> facing changes.

Indeed. At most a single NEWS entry sounds sufficient.

Regards

Antoine.



From brett at python.org  Tue May  4 01:11:53 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 3 May 2010 16:11:53 -0700
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
Message-ID: <y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>

I'll just do a single entry saying that the static analyzer was used and not
list the files.

And in reply to Benjamin's question about the whitespace, I guess it
actually doesn't matter. More important to clean up in py3k.

On May 3, 2010 4:00 PM, "Antoine Pitrou" <solipsis at pitrou.net> wrote:


Benjamin Peterson <benjamin <at> python.org> writes:
>
> 2010/5/3 Brett Cannon <brett <at> python.org>:

> > When I check in these changes I will do it file by file, but my
question is
> > how to handle M...
Indeed. At most a single NEWS entry sounds sufficient.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/brett%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100503/a2ba4829/attachment.html>

From solipsis at pitrou.net  Tue May  4 01:24:37 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 3 May 2010 23:24:37 +0000 (UTC)
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the
	code	base
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
Message-ID: <hrnlvl$pd3$1@dough.gmane.org>

Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit?:
> 
> And in reply to Benjamin's question about the whitespace, I guess it
> actually doesn't matter. More important to clean up in py3k.

Would it be finally time to standardize all C files on a 4-spaces 
indentation rule?

I understand the "svn annotate" argument, but we edit code far more often 
than we annotate it. Also, it seems "svn ann -x -w" would ignore 
whitespace changes.

Regards

Antoine.


From steve at holdenweb.com  Tue May  4 04:12:25 2010
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 03 May 2010 22:12:25 -0400
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <hrnlvl$pd3$1@dough.gmane.org>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>	<hrnkhi$ldo$1@dough.gmane.org>	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org>
Message-ID: <4BDF8289.2090306@holdenweb.com>

Antoine Pitrou wrote:
> Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit :
>> And in reply to Benjamin's question about the whitespace, I guess it
>> actually doesn't matter. More important to clean up in py3k.
> 
> Would it be finally time to standardize all C files on a 4-spaces 
> indentation rule?
> 
> I understand the "svn annotate" argument, but we edit code far more often 
> than we annotate it. Also, it seems "svn ann -x -w" would ignore 
> whitespace changes.
> 
Let's not forget to consider what Hg can do, since that represents the
future.

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See PyCon Talks from Atlanta 2010  http://pycon.blip.tv/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/


From barry at python.org  Tue May  4 04:34:13 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 3 May 2010 22:34:13 -0400
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
 base
In-Reply-To: <4BDF8289.2090306@holdenweb.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
Message-ID: <20100503223413.17437de7@heresy>

On May 03, 2010, at 10:12 PM, Steve Holden wrote:

>Antoine Pitrou wrote:
>> Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit :
>>> And in reply to Benjamin's question about the whitespace, I guess it
>>> actually doesn't matter. More important to clean up in py3k.
>> 
>> Would it be finally time to standardize all C files on a 4-spaces 
>> indentation rule?
>> 
>> I understand the "svn annotate" argument, but we edit code far more often 
>> than we annotate it. Also, it seems "svn ann -x -w" would ignore 
>> whitespace changes.
>> 
>Let's not forget to consider what Hg can do, since that represents the
>future.

Now would be a good time to convert the C files to 4 space indents.  We've
only been talking about it for a decade at least.  While I'm sure history can
be preserved across the svn->hg import, it's enough of a break to bite the
bullet and fix the code.  I think we only need to convert the py3k branch
though.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100503/b5697507/attachment.pgp>

From alexandre at peadrop.com  Tue May  4 06:00:47 2010
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Mon, 3 May 2010 21:00:47 -0700
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <20100503223413.17437de7@heresy>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com> 
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com> 
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com> 
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com> 
	<20100503223413.17437de7@heresy>
Message-ID: <h2vacd65fa21005032100xb810dd65pb8b808a9a17a814a@mail.gmail.com>

On Mon, May 3, 2010 at 7:34 PM, Barry Warsaw <barry at python.org> wrote:
> Now would be a good time to convert the C files to 4 space indents. ?We've
> only been talking about it for a decade at least.

Will changing the indentation of source files to 4 space indents break
patches on the bug tracker?

-- Alexandre

From martin at v.loewis.de  Tue May  4 07:12:32 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 04 May 2010 07:12:32 +0200
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
 base
In-Reply-To: <h2vacd65fa21005032100xb810dd65pb8b808a9a17a814a@mail.gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
	<20100503223413.17437de7@heresy>
	<h2vacd65fa21005032100xb810dd65pb8b808a9a17a814a@mail.gmail.com>
Message-ID: <4BDFACC0.2010808@v.loewis.de>

> Will changing the indentation of source files to 4 space indents break
> patches on the bug tracker?

Plain patch will choke, but "patch -l" might accept them. I have only
read the documentation, though, and don't know whether it does in practice.

Regards,
Martin

From martin at v.loewis.de  Tue May  4 07:16:22 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 04 May 2010 07:16:22 +0200
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
 base
In-Reply-To: <4BDF8289.2090306@holdenweb.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>	<hrnkhi$ldo$1@dough.gmane.org>	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>	<hrnlvl$pd3$1@dough.gmane.org>
	<4BDF8289.2090306@holdenweb.com>
Message-ID: <4BDFADA6.4040703@v.loewis.de>

Steve Holden wrote:
> Antoine Pitrou wrote:
>> Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit :
>>> And in reply to Benjamin's question about the whitespace, I guess it
>>> actually doesn't matter. More important to clean up in py3k.
>> Would it be finally time to standardize all C files on a 4-spaces 
>> indentation rule?
>>
>> I understand the "svn annotate" argument, but we edit code far more often 
>> than we annotate it. Also, it seems "svn ann -x -w" would ignore 
>> whitespace changes.
>>
> Let's not forget to consider what Hg can do, since that represents the
> future.

I think Mercurial chokes in the light of white space inconsistencies
just as much as Subversion. One reason for not converting in the past
was also that it would break merging, unless that whitespace
normalization is applied to all these branches.

Regards,
Martin

From martin at v.loewis.de  Tue May  4 08:50:11 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 04 May 2010 08:50:11 +0200
Subject: [Python-Dev] Daily DMGs
Message-ID: <4BDFC3A3.4000205@v.loewis.de>

David Bolen and I started producing daily OSX installers, at

http://www.python.org/dev/daily-dmg/

If you find problems with these, please report them to bugs.python.org.

Regards,
Martin

From victor.stinner at haypocalc.com  Tue May  4 10:10:33 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 4 May 2010 10:10:33 +0200
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <20100503223413.17437de7@heresy>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<4BDF8289.2090306@holdenweb.com> <20100503223413.17437de7@heresy>
Message-ID: <201005041010.33228.victor.stinner@haypocalc.com>

Le mardi 04 mai 2010 04:34:13, Barry Warsaw a ?crit :
> Now would be a good time to convert the C files to 4 space indents. (...)
> I think we only need to convert the py3k branch though.

It will make the port of patches (commits) between trunk and py3k much harder. 
Can you explain why do you want to only fix only py3k and not trunk?

-- 
Victor Stinner
http://www.haypocalc.com/

From victor.stinner at haypocalc.com  Tue May  4 10:14:19 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 4 May 2010 10:14:19 +0200
Subject: [Python-Dev]
 =?utf-8?q?Running_Clang_2=2E7=27s_static_analyzer_ov?=
 =?utf-8?q?er_the_code=09base?=
In-Reply-To: <hrnlvl$pd3$1@dough.gmane.org>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org>
Message-ID: <201005041014.19399.victor.stinner@haypocalc.com>

Le mardi 04 mai 2010 01:24:37, Antoine Pitrou a ?crit :
> Would it be finally time to standardize all C files on a 4-spaces
> indentation rule?

Indentation is painful on some files mixing tabs and spaces (sometimes in the 
same function!).

> I understand the "svn annotate" argument, but we edit code far more often
> than we annotate it. Also, it seems "svn ann -x -w" would ignore
> whitespace changes.

I use svn blame to find the commit which insered a line of code. When I get 
the commit number, I always read the commit to ensure that the patch really 
insered the line, and it's not a refactoring patch. Refactoring can be about 
the indentation, but there are a lot of other minor changes: op->ob_size => 
Py_SIZE(op), (a) => a, { with a new line => { without newline, ...

-- 
Victor Stinner
http://www.haypocalc.com/

From victor.stinner at haypocalc.com  Tue May  4 10:37:01 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 4 May 2010 10:37:01 +0200
Subject: [Python-Dev]
 =?utf-8?q?Running_Clang_2=2E7=27s_static_analyzer_ov?=
 =?utf-8?q?er_the_code=09base?=
In-Reply-To: <hrnlvl$pd3$1@dough.gmane.org>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org>
Message-ID: <201005041037.01778.victor.stinner@haypocalc.com>

Le mardi 04 mai 2010 01:24:37, Antoine Pitrou a ?crit :
> Le Mon, 03 May 2010 16:11:53 -0700, Brett Cannon a ?crit :
> > And in reply to Benjamin's question about the whitespace, I guess it
> > actually doesn't matter. More important to clean up in py3k.
> 
> Would it be finally time to standardize all C files on a 4-spaces
> indentation rule?

Oh, if we decide to reindent all C files, would it be also possible to strip 
trailing spaces?

-- 
Victor Stinner
http://www.haypocalc.com/

From barry at python.org  Tue May  4 14:23:31 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 4 May 2010 08:23:31 -0400
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
 base
In-Reply-To: <4BDFADA6.4040703@v.loewis.de>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
	<4BDFADA6.4040703@v.loewis.de>
Message-ID: <20100504082331.2c7a6062@heresy>

On May 04, 2010, at 07:16 AM, Martin v. L?wis wrote:

>I think Mercurial chokes in the light of white space inconsistencies
>just as much as Subversion. One reason for not converting in the past
>was also that it would break merging, unless that whitespace
>normalization is applied to all these branches.

Then let's do py3k and release31-maint, or whatever they will be called under
hg.  Once 2.7 is released and we're on hg, how much back porting of revisions
from Python 3 -> 2 is going to happen?

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100504/746c1ab2/attachment.pgp>

From solipsis at pitrou.net  Tue May  4 14:41:42 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 4 May 2010 12:41:42 +0000 (UTC)
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
	<4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy>
Message-ID: <loom.20100504T143837-351@post.gmane.org>

Barry Warsaw <barry <at> python.org> writes:
> 
> Then let's do py3k and release31-maint, or whatever they will be called under
> hg.  Once 2.7 is released and we're on hg, how much back porting of revisions
> from Python 3 -> 2 is going to happen?

Probably quite a bit still; all C extension bug fixes for example.

I think we should reindent all 3 branches. Most of the work can probably be
scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to
fix vertical alignments and the like.

Regards

Antoine.



From dirkjan at ochtman.nl  Tue May  4 15:09:33 2010
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Tue, 4 May 2010 15:09:33 +0200
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <loom.20100504T143837-351@post.gmane.org>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com> 
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com> 
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com> 
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com> 
	<4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> 
	<loom.20100504T143837-351@post.gmane.org>
Message-ID: <m2tea2499da1005040609sa5e620d2wf8d36366335189bc@mail.gmail.com>

On Tue, May 4, 2010 at 14:41, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I think we should reindent all 3 branches. Most of the work can probably be
> scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to
> fix vertical alignments and the like.

If the script is robust enough, I can run it as part of the
conversion, making sure that all branches get cleaned up...

Cheers,

Dirkjan

From eric at trueblade.com  Tue May  4 16:37:36 2010
From: eric at trueblade.com (Eric Smith)
Date: Tue, 04 May 2010 10:37:36 -0400
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
 base
In-Reply-To: <m2tea2499da1005040609sa5e620d2wf8d36366335189bc@mail.gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
	<4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy>
	<loom.20100504T143837-351@post.gmane.org>
	<m2tea2499da1005040609sa5e620d2wf8d36366335189bc@mail.gmail.com>
Message-ID: <4BE03130.9020700@trueblade.com>

Dirkjan Ochtman wrote:
> On Tue, May 4, 2010 at 14:41, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> I think we should reindent all 3 branches. Most of the work can probably be
>> scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to
>> fix vertical alignments and the like.
> 
> If the script is robust enough, I can run it as part of the
> conversion, making sure that all branches get cleaned up...

Could this be done as part of the conversion without affecting the history?

-- 
Eric.

From victor.stinner at haypocalc.com  Tue May  4 17:26:22 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 4 May 2010 17:26:22 +0200
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <loom.20100504T143837-351@post.gmane.org>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<20100504082331.2c7a6062@heresy>
	<loom.20100504T143837-351@post.gmane.org>
Message-ID: <201005041726.22300.victor.stinner@haypocalc.com>

Le mardi 04 mai 2010 14:41:42, Antoine Pitrou a ?crit :
> Barry Warsaw <barry <at> python.org> writes:
> > Then let's do py3k and release31-maint, or whatever they will be called
> > under hg.  Once 2.7 is released and we're on hg, how much back porting of
> > revisions from Python 3 -> 2 is going to happen?
> 
> Probably quite a bit still; all C extension bug fixes for example.

2.7 will be maintained for long time, longer than any other 2.x version.

> I think we should reindent all 3 branches. Most of the work can probably be
> scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary
>  to fix vertical alignments and the like.

We should also add pre-commit scripts to avoid the reintroduction of 
tabulations in C (and Python?) files.

-- 
Victor Stinner
http://www.haypocalc.com/

From zvezdan at zope.com  Tue May  4 17:27:58 2010
From: zvezdan at zope.com (Zvezdan Petkovic)
Date: Tue, 4 May 2010 11:27:58 -0400
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <loom.20100504T143837-351@post.gmane.org>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
	<4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy>
	<loom.20100504T143837-351@post.gmane.org>
Message-ID: <D5CD5056-5ED9-4842-9B77-12DE9306CD8B@zope.com>

On May 4, 2010, at 8:41 AM, Antoine Pitrou wrote:
> I think we should reindent all 3 branches. Most of the work can probably be scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to fix vertical alignments and the like.

> 
> I think we should reindent all 3 branches. Most of the work can probably be scripted (str.replace("\t", " " * 4)), and then a visual pass is necessary to fix vertical alignments and the like.

Is it really that simple?

People often do the following:

- use tabs inside strings ("\t" should be used instead);
- use tabs for alignment of inline comments on the right side;
- mix tabs and spaces for indentation whitespace (continuation lines);
- use tabs for alignment inside comments.

A simple replacement of a tab with 4 spaces as you propose does not work  on such a code.

To do it properly, you may end up reinventing `indent` (UNIX program).

What if there was a set of `indent` flags defined for PEP-7?
It would be a nice tool to check whether a new code complies.

However, running `indent` on an existing code would probably cause non-whitespace changes too as it rearranges the comments and code.

I'm afraid that any other kind of "script + visual pass + post-edit" would incur similar changes if done properly (only more tedious).

So, the question is what bothers developers more:

- old C files with tab indentation, or
- a lot of changes in version control to fix them?

Both?
:-)

	Zvezdan


From brett at python.org  Tue May  4 22:16:46 2010
From: brett at python.org (Brett Cannon)
Date: Tue, 4 May 2010 13:16:46 -0700
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <D5CD5056-5ED9-4842-9B77-12DE9306CD8B@zope.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com> 
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com> 
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com> 
	<4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy> 
	<loom.20100504T143837-351@post.gmane.org>
	<D5CD5056-5ED9-4842-9B77-12DE9306CD8B@zope.com>
Message-ID: <r2ubbaeab101005041316m6891d95ctba8ed850a6236638@mail.gmail.com>

On Tue, May 4, 2010 at 08:27, Zvezdan Petkovic <zvezdan at zope.com> wrote:

> On May 4, 2010, at 8:41 AM, Antoine Pitrou wrote:
> > I think we should reindent all 3 branches. Most of the work can probably
> be scripted (str.replace("\t", " " * 4)), and then a visual pass is
> necessary to fix vertical alignments and the like.
>
> >
> > I think we should reindent all 3 branches. Most of the work can probably
> be scripted (str.replace("\t", " " * 4)), and then a visual pass is
> necessary to fix vertical alignments and the like.
>
> Is it really that simple?
>
> People often do the following:
>
> - use tabs inside strings ("\t" should be used instead);
> - use tabs for alignment of inline comments on the right side;
> - mix tabs and spaces for indentation whitespace (continuation lines);
> - use tabs for alignment inside comments.
>
> A simple replacement of a tab with 4 spaces as you propose does not work
>  on such a code.
>
> To do it properly, you may end up reinventing `indent` (UNIX program).
>
> What if there was a set of `indent` flags defined for PEP-7?
> It would be a nice tool to check whether a new code complies.
>

`make patchcheck` already does this for Python code. Adding to the command
to handle C code wouldn't be hard.

-Brett



>
> However, running `indent` on an existing code would probably cause
> non-whitespace changes too as it rearranges the comments and code.
>
> I'm afraid that any other kind of "script + visual pass + post-edit" would
> incur similar changes if done properly (only more tedious).
>
> So, the question is what bothers developers more:
>
> - old C files with tab indentation, or
> - a lot of changes in version control to fix them?
>
> Both?
> :-)
>
>        Zvezdan
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100504/4bc26aca/attachment-0001.html>

From benjamin at python.org  Tue May  4 22:46:51 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 4 May 2010 15:46:51 -0500
Subject: [Python-Dev] slight release schedule change
Message-ID: <v2r1afaf6161005041346ke9a80ed1xfbb0dc717ae0af58@mail.gmail.com>

Hi,
I'm sorry I neglected to mention this before. I'm delaying all 2.7
releases a week because of beta 1's lateness. I've now updated PEP
373. So, beta 2 will be this Saturday.

-- 
Regards,
Benjamin

From solipsis at pitrou.net  Wed May  5 00:02:06 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 4 May 2010 22:02:06 +0000 (UTC)
Subject: [Python-Dev] reindenting
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
	<4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy>
	<loom.20100504T143837-351@post.gmane.org>
	<D5CD5056-5ED9-4842-9B77-12DE9306CD8B@zope.com>
Message-ID: <hrq5gu$3c5$1@dough.gmane.org>

Le Tue, 04 May 2010 11:27:58 -0400, Zvezdan Petkovic a ?crit?:
> 
> A simple replacement of a tab with 4 spaces as you propose does not work
>  on such a code.

Right, I was simplifying a bit.

I've just written a script which does a slightly more elaborate 
reindentation of C files, but still with the main objective of replacing 
tabs with 4-spaces indents. It shows good results on most of the source 
tree. In particular, it handles continuation lines and keeps vertical 
alignment correct in the majority of cases. This script could probably be 
applied with little a posteriori manual intervention.

It can be found in trunk/sandbox/untabify/untabify.py.

Regards

Antoine.


From ncoghlan at gmail.com  Wed May  5 00:32:24 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 05 May 2010 08:32:24 +1000
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
 base
In-Reply-To: <201005041726.22300.victor.stinner@haypocalc.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>	<20100504082331.2c7a6062@heresy>	<loom.20100504T143837-351@post.gmane.org>
	<201005041726.22300.victor.stinner@haypocalc.com>
Message-ID: <4BE0A078.90505@gmail.com>

Victor Stinner wrote:
> We should also add pre-commit scripts to avoid the reintroduction of 
> tabulations in C (and Python?) files.

Python and ReST files are already covered (with reindent.py and
reindent-rst.py to fix any files that get mixed up).

"make patchcheck" includes the precommit checks for both of those file
types.

It's only C files that haven't been fixed yet (because their whitespace
is such a mess overall, whereas the others were already pretty clean
even before the precommit hooks were added).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May  5 00:34:44 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 05 May 2010 08:34:44 +1000
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
 base
In-Reply-To: <D5CD5056-5ED9-4842-9B77-12DE9306CD8B@zope.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>	<hrnkhi$ldo$1@dough.gmane.org>	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>	<hrnlvl$pd3$1@dough.gmane.org>
	<4BDF8289.2090306@holdenweb.com>	<4BDFADA6.4040703@v.loewis.de>
	<20100504082331.2c7a6062@heresy>	<loom.20100504T143837-351@post.gmane.org>
	<D5CD5056-5ED9-4842-9B77-12DE9306CD8B@zope.com>
Message-ID: <4BE0A104.1020008@gmail.com>

Zvezdan Petkovic wrote:
> So, the question is what bothers developers more:
> 
> - old C files with tab indentation, or
> - a lot of changes in version control to fix them?
> 
> Both?

C files that mix tabs and spaces are actually the main source of pain
when editing :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May  5 00:41:04 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 05 May 2010 08:41:04 +1000
Subject: [Python-Dev] Standardising C file indentation (was Re: Running
 Clang 2.7's static analyzer over the code base)
In-Reply-To: <4BE03130.9020700@trueblade.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>	<hrnkhi$ldo$1@dough.gmane.org>	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>	<hrnlvl$pd3$1@dough.gmane.org>
	<4BDF8289.2090306@holdenweb.com>	<4BDFADA6.4040703@v.loewis.de>
	<20100504082331.2c7a6062@heresy>	<loom.20100504T143837-351@post.gmane.org>	<m2tea2499da1005040609sa5e620d2wf8d36366335189bc@mail.gmail.com>
	<4BE03130.9020700@trueblade.com>
Message-ID: <4BE0A280.8080903@gmail.com>

Eric Smith wrote:
> Dirkjan Ochtman wrote:
>> On Tue, May 4, 2010 at 14:41, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> I think we should reindent all 3 branches. Most of the work can
>>> probably be
>>> scripted (str.replace("\t", " " * 4)), and then a visual pass is
>>> necessary to
>>> fix vertical alignments and the like.
>>
>> If the script is robust enough, I can run it as part of the
>> conversion, making sure that all branches get cleaned up...
> 
> Could this be done as part of the conversion without affecting the history?

I don't think that would be a good idea - it's just whitespace, but I
think we risk more problems by leaving it out of the history than we do
by preserving.

Howver, if we delay fixing the C file indentation until we're already on
hg, the merge tools should offer the best chance of being able to apply
pre-fix patches and have the software figure out where the whitespace
changes need to be accounted for.

If we do it before, or don't record the change in the history at all,
then hg won't have any changeset information to work with and will
almost certainly get just as confused by the whitespace changes as the
basic "patch" command would.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Wed May  5 00:57:20 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 05 May 2010 00:57:20 +0200
Subject: [Python-Dev] Standardising C file indentation (was Re: Running
 Clang 2.7's static analyzer over the code base)
In-Reply-To: <4BE0A280.8080903@gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>
	<hrnkhi$ldo$1@dough.gmane.org>
	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>
	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
	<4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy>
	<loom.20100504T143837-351@post.gmane.org>
	<m2tea2499da1005040609sa5e620d2wf8d36366335189bc@mail.gmail.com>
	<4BE03130.9020700@trueblade.com>  <4BE0A280.8080903@gmail.com>
Message-ID: <1273013841.3688.9.camel@localhost.localdomain>

Le mercredi 05 mai 2010 ? 08:41 +1000, Nick Coghlan a ?crit :
> 
> Howver, if we delay fixing the C file indentation until we're already on
> hg, the merge tools should offer the best chance of being able to apply
> pre-fix patches and have the software figure out where the whitespace
> changes need to be accounted for.

I just tried using "patch -l" after having changed the indentation of a
C source file and it worked fine. The only issue being that the patched
lines of code have the wrong indentation (they get their indentation
from the patch, not from the input file), but it's probably unavoidable.

Regards

Antoine.



From victor.stinner at haypocalc.com  Wed May  5 01:10:19 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Wed, 5 May 2010 01:10:19 +0200
Subject: [Python-Dev] Standardising C file indentation (was Re: Running
	Clang 2.7's static analyzer over the code base)
In-Reply-To: <4BE0A280.8080903@gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<4BE03130.9020700@trueblade.com> <4BE0A280.8080903@gmail.com>
Message-ID: <201005050110.19887.victor.stinner@haypocalc.com>

Le mercredi 05 mai 2010 00:41:04, Nick Coghlan a ?crit :
> Howver, if we delay fixing the C file indentation until we're already on
> hg, the merge tools should offer the best chance of being able to apply
> pre-fix patches and have the software figure out where the whitespace
> changes need to be accounted for.

If we wait for Hg to solve all of our problems, the migration from svn to hg 
will be harder and longer.

-- 
Victor Stinner
http://www.haypocalc.com/

From g.brandl at gmx.net  Wed May  5 03:40:45 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 05 May 2010 03:40:45 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>
	<4BDB34CE.9080804@v.loewis.de>
	<4BDB3D6C.4050909@holdenweb.com>	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>	<4BDBB846.5020305@v.loewis.de>
	<20100503115752.4f19f2e3@heresy>
	<h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>
Message-ID: <hrqibu$r5m$1@dough.gmane.org>

Am 03.05.2010 18:40, schrieb Guido van Rossum:
> On Mon, May 3, 2010 at 8:57 AM, Barry Warsaw <barry at python.org> wrote:
>> On May 01, 2010, at 07:12 AM, Martin v. L?wis wrote:
>>
>>>> IIRC in the IETF this is done by the committee chair. I think it's a
>>>> good idea to have this be a single person to avoid endless indecision.
>>>
>>>It then seems that this role should go to the release manager of the
>>>upcoming feature release. Assuming Georg can accept this additional
>>>responsibility.
>>
>> I do think it makes sense for the RM to assume these responsibilities where
>> Guido either can't or doesn't want to make the final decision.  I think it
>> will fairly substantially increase the workload on the RM, so perhaps there
>> are ways to off-load some of the current responsibilities (e.g. updating the
>> website for each release).  I also think that RMs should be term-limited so
>> that we can spread more experience within the community.  And past-RMs can
>> provide a sort of consultation group where contentious decisions can be
>> discussed and advice gathered.
> 
> While I certainly won't object if a release manager volunteers to also
> be the final PEP arbiter, I don't want to make it a job requirement
> (or even an implied expectation). The responsibility of a release
> manager is very much in the here and now and the near future --
> stability and consistency of the current release. Being PEP arbiter
> requires a somewhat different mindset, more focused on the long-term
> health of the language and its community.

I agree, and I wouldn't want to make these decisions.  That person (or
group) needs to have some weight in the community, or there will be a
feeling of "... and who is he to decide anyway".  We haven't emphasized
RMship in the past; it's not a special position, except when something
about a release goes wrong and blame must be placed ;)  I think, being
Guido, you are still the best single person to do this job.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From fuzzyman at voidspace.org.uk  Wed May  5 12:43:45 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Wed, 05 May 2010 11:43:45 +0100
Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?)
Message-ID: <4BE14BE1.6090200@voidspace.org.uk>

Hello all,

It looks like the changes to the python-dev mailman archives broke some 
of the links in PEPs.

All the best,

Michael Foord

-------- Original Message --------
Subject: 	broken mailing list links in PEP(s?)
Date: 	Tue, 4 May 2010 20:22:57 -0700
From: 	Bayle Shanks <bayle.shanks at gmail.com>
To: 	webmaster at python.org



On http://www.python.org/dev/peps/pep-0225/ , in the section "Credits
and archives", there are a bunch of links like



http://mail.python.org/pipermail/python-list/2000-July/108893.html


which are broken


thanks,
   bayle

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100505/549d77ce/attachment.html>

From phd at phd.pp.ru  Wed May  5 13:09:08 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Wed, 5 May 2010 15:09:08 +0400
Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?)
In-Reply-To: <4BE14BE1.6090200@voidspace.org.uk>
References: <4BE14BE1.6090200@voidspace.org.uk>
Message-ID: <20100505110908.GA7160@phd.pp.ru>

On Wed, May 05, 2010 at 11:43:45AM +0100, Michael Foord wrote:
> http://mail.python.org/pipermail/python-list/2000-July/108893.html
>
> which are broken

   Pipermail's links aren't stable AFAIU. The numbering is changing over
time.

Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From ncoghlan at gmail.com  Wed May  5 13:15:56 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 05 May 2010 21:15:56 +1000
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <hrqibu$r5m$1@dough.gmane.org>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>	<4BDB34CE.9080804@v.loewis.de>	<4BDB3D6C.4050909@holdenweb.com>	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>	<4BDBB846.5020305@v.loewis.de>	<20100503115752.4f19f2e3@heresy>	<h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>
	<hrqibu$r5m$1@dough.gmane.org>
Message-ID: <4BE1536C.6000206@gmail.com>

Georg Brandl wrote:
> I agree, and I wouldn't want to make these decisions.  That person (or
> group) needs to have some weight in the community, or there will be a
> feeling of "... and who is he to decide anyway".  We haven't emphasized
> RMship in the past; it's not a special position, except when something
> about a release goes wrong and blame must be placed ;)  I think, being
> Guido, you are still the best single person to do this job.

Not to wish anything bad on Guido, but I think it does behoove us as
responsible stewards of the reference interpreter to at least consider
the "bus" factor (as in "hit by a...").

I suspect Martin is right that some form of committee (at least
initially) would be inevitable. While I doubt that we could find any one
person that everyone would agree to defer to, I suspect we could find a
small group of 3 or 4 people who's consensus everyone else would
respect. And if that small group couldn't agree on whether or not
something was a good idea, then the old "status quo wins a stalemate"
would kick in.

That said, this is hopefully something that we won't have to worry about
for real until some time well into the future when Guido decides he
wants to retire.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May  5 13:20:40 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 05 May 2010 21:20:40 +1000
Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?)
In-Reply-To: <20100505110908.GA7160@phd.pp.ru>
References: <4BE14BE1.6090200@voidspace.org.uk>
	<20100505110908.GA7160@phd.pp.ru>
Message-ID: <4BE15488.2090103@gmail.com>

Oleg Broytman wrote:
> On Wed, May 05, 2010 at 11:43:45AM +0100, Michael Foord wrote:
>> http://mail.python.org/pipermail/python-list/2000-July/108893.html
>>
>> which are broken
> 
>    Pipermail's links aren't stable AFAIU. The numbering is changing over
> time.

I don't think that's true in general. I do recall the archive getting
FUBARed a few months back, and one of the consequences of fixing it was
that some old messages were renumbered (since the whole archive
basically had to be reindexed or something like that).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From fuzzyman at voidspace.org.uk  Wed May  5 13:24:01 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Wed, 05 May 2010 12:24:01 +0100
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <4BE1536C.6000206@gmail.com>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>	<4BDB34CE.9080804@v.loewis.de>	<4BDB3D6C.4050909@holdenweb.com>	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>	<4BDBB846.5020305@v.loewis.de>	<20100503115752.4f19f2e3@heresy>	<h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>	<hrqibu$r5m$1@dough.gmane.org>
	<4BE1536C.6000206@gmail.com>
Message-ID: <4BE15551.4020309@voidspace.org.uk>

On 05/05/2010 12:15, Nick Coghlan wrote:
> Georg Brandl wrote:
>    
>> I agree, and I wouldn't want to make these decisions.  That person (or
>> group) needs to have some weight in the community, or there will be a
>> feeling of "... and who is he to decide anyway".  We haven't emphasized
>> RMship in the past; it's not a special position, except when something
>> about a release goes wrong and blame must be placed ;)  I think, being
>> Guido, you are still the best single person to do this job.
>>      
> Not to wish anything bad on Guido, but I think it does behoove us as
> responsible stewards of the reference interpreter to at least consider
> the "bus" factor (as in "hit by a...").
>
> I suspect Martin is right that some form of committee (at least
> initially) would be inevitable. While I doubt that we could find any one
> person that everyone would agree to defer to, I suspect we could find a
> small group of 3 or 4 people who's consensus everyone else would
> respect. And if that small group couldn't agree on whether or not
> something was a good idea, then the old "status quo wins a stalemate"
> would kick in.
>
>    

Sounds like a sensible arrangement.

> That said, this is hopefully something that we won't have to worry about
> for real until some time well into the future when Guido decides he
> wants to retire.
>    

Well, we (or more specifically Guido) may want to transition to some 
system where Guido doesn't have to make *every* decision. We only need 
this for PEPs where there isn't a community consensus.

Michael

> Regards,
> Nick.
>
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From g.brandl at gmx.net  Wed May  5 13:30:04 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 05 May 2010 13:30:04 +0200
Subject: [Python-Dev] Two small PEP ideas
In-Reply-To: <4BE15551.4020309@voidspace.org.uk>
References: <20100427134618.564e3e3b@heresy>	<v2nca471dc21004271153t3983c0b7rc630410845572d95@mail.gmail.com>	<4BD75D54.8050307@gmail.com>	<p2mea2499da1004280022j61e8922ah5b6ef6231bd5e4ec@mail.gmail.com>	<20100430150918.308984aa@heresy>	<4BDB34CE.9080804@v.loewis.de>	<4BDB3D6C.4050909@holdenweb.com>	<r2nca471dc21004301425ie3acdbe8nd83fa0ae5b2df6ab@mail.gmail.com>	<4BDBB846.5020305@v.loewis.de>	<20100503115752.4f19f2e3@heresy>	<h2ica471dc21005030940h84ff7108v9188164049abfc5c@mail.gmail.com>	<hrqibu$r5m$1@dough.gmane.org>	<4BE1536C.6000206@gmail.com>
	<4BE15551.4020309@voidspace.org.uk>
Message-ID: <hrrksu$4oj$1@dough.gmane.org>

Am 05.05.2010 13:24, schrieb Michael Foord:
> On 05/05/2010 12:15, Nick Coghlan wrote:
>> Georg Brandl wrote:
>>    
>>> I agree, and I wouldn't want to make these decisions.  That person (or
>>> group) needs to have some weight in the community, or there will be a
>>> feeling of "... and who is he to decide anyway".  We haven't emphasized
>>> RMship in the past; it's not a special position, except when something
>>> about a release goes wrong and blame must be placed ;)  I think, being
>>> Guido, you are still the best single person to do this job.
>>>      
>> Not to wish anything bad on Guido, but I think it does behoove us as
>> responsible stewards of the reference interpreter to at least consider
>> the "bus" factor (as in "hit by a...").

Sure, but I think we wouldn't have too much trouble finding a replacement
if and when that happens.  (Let's hope it doesn't.)  This is not something
that requires weeks of preparation, and even if it did, no pronouncement
on a PEP for a few weeks isn't the doom of Python.

>> I suspect Martin is right that some form of committee (at least
>> initially) would be inevitable. While I doubt that we could find any one
>> person that everyone would agree to defer to, I suspect we could find a
>> small group of 3 or 4 people who's consensus everyone else would
>> respect. And if that small group couldn't agree on whether or not
>> something was a good idea, then the old "status quo wins a stalemate"
>> would kick in.
> 
> Sounds like a sensible arrangement.

Yes, I could imagine that too.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From barry at python.org  Wed May  5 14:22:17 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 5 May 2010 08:22:17 -0400
Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?)
In-Reply-To: <20100505110908.GA7160@phd.pp.ru>
References: <4BE14BE1.6090200@voidspace.org.uk>
	<20100505110908.GA7160@phd.pp.ru>
Message-ID: <60DA94AF-6444-435A-86DD-98CAF432197A@python.org>

On May 5, 2010, at 7:09 AM, Oleg Broytman wrote:

> On Wed, May 05, 2010 at 11:43:45AM +0100, Michael Foord wrote:
>> http://mail.python.org/pipermail/python-list/2000-July/108893.html
>> 
>> which are broken
> 
>   Pipermail's links aren't stable AFAIU. The numbering is changing over
> time.

They're only unstable if you regenerate the archives and the mbox file is old enough to have been a victim of a long-fixed delimiter bug.  Which is true for python-dev.

-Barry


From solipsis at pitrou.net  Wed May  5 15:10:16 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 5 May 2010 13:10:16 +0000 (UTC)
Subject: [Python-Dev] Reindenting patches
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>	<t2h1afaf6161005031547gc40eb45bn1b0bcecb0c825422@mail.gmail.com>	<hrnkhi$ldo$1@dough.gmane.org>	<s2mbbaeab101005031610ge2315c83q82558fd99a93038a@mail.gmail.com>	<y2ibbaeab101005031611y509001eam60242e7a9ae1b2d8@mail.gmail.com>	<hrnlvl$pd3$1@dough.gmane.org>
	<4BDF8289.2090306@holdenweb.com>	<4BDFADA6.4040703@v.loewis.de>
	<20100504082331.2c7a6062@heresy>	<loom.20100504T143837-351@post.gmane.org>	<m2tea2499da1005040609sa5e620d2wf8d36366335189bc@mail.gmail.com>
	<4BE03130.9020700@trueblade.com> <4BE0A280.8080903@gmail.com>
Message-ID: <loom.20100505T150720-623@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> 
> Howver, if we delay fixing the C file indentation until we're already on
> hg, the merge tools should offer the best chance of being able to apply
> pre-fix patches and have the software figure out where the whitespace
> changes need to be accounted for.

For the record, I've added to the untabify script a patch rewriting option
("-p") which reindents all patch hunks for C files containing tabs. It should
minimize manual reformatting work with existing patches.

Regards

Antoine.



From foom at fuhm.net  Wed May  5 17:35:05 2010
From: foom at fuhm.net (James Y Knight)
Date: Wed, 5 May 2010 11:35:05 -0400
Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?)
In-Reply-To: <60DA94AF-6444-435A-86DD-98CAF432197A@python.org>
References: <4BE14BE1.6090200@voidspace.org.uk>
	<20100505110908.GA7160@phd.pp.ru>
	<60DA94AF-6444-435A-86DD-98CAF432197A@python.org>
Message-ID: <D543D7E1-CEF8-473F-B477-107A26D4EBDA@fuhm.net>


On May 5, 2010, at 8:22 AM, Barry Warsaw wrote:

> On May 5, 2010, at 7:09 AM, Oleg Broytman wrote:
>
>> On Wed, May 05, 2010 at 11:43:45AM +0100, Michael Foord wrote:
>>> http://mail.python.org/pipermail/python-list/2000-July/108893.html
>>>
>>> which are broken
>>
>>  Pipermail's links aren't stable AFAIU. The numbering is changing  
>> over
>> time.
>
> They're only unstable if you regenerate the archives and the mbox  
> file is old enough to have been a victim of a long-fixed delimiter  
> bug.  Which is true for python-dev.

And of course if you're paying attention, you can fix the mbox file  
(quoting "From" etc) such that it generates the same numbers as it did  
the first time.

James

From brett at python.org  Wed May  5 22:56:45 2010
From: brett at python.org (Brett Cannon)
Date: Wed, 5 May 2010 13:56:45 -0700
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
Message-ID: <i2ubbaeab101005051356oc69fc594we0e04fba46306896@mail.gmail.com>

I am done running the analysis over trunk. I will not svnmerge these changes
into py3k as the amount of time and effort that would take equates to
running the static analyzer again just before 3.2 is released and possibly
catching more changes (and maybe even a newer version of Clang at that
point).

On Mon, May 3, 2010 at 15:37, Brett Cannon <brett at python.org> wrote:

> Since 2.7 is probably going to exist for a while, I am running Clang 2.7's
> static analyzer (``clang --static``) over trunk. It's mostly just finding
> stuff like unneeded variable initialization or variables that are never used
> (compilation is picking up unused returned values, almost all from
> PyObject_INIT).
>
> When I check in these changes I will do it file by file, but my question is
> how to handle Misc/NEWS. I have gone through the underscores and the 'a's in
> Modules and already have six modified files, so the final count might be a
> little high. Do people want individual entries per file, or simply a single
> entry that lists each file modified?
>
> We should probably go through the C code and fix the whitespace before we
> hit 2.7 final (there is a ton of lines with extraneous spaces).
>
> -Brett
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100505/fd3f5c06/attachment.html>

From victor.stinner at haypocalc.com  Wed May  5 23:01:50 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Wed, 5 May 2010 23:01:50 +0200
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
Message-ID: <201005052301.50504.victor.stinner@haypocalc.com>

Le mardi 04 mai 2010 00:37:22, Brett Cannon a ?crit :
> Since 2.7 is probably going to exist for a while, I am running Clang 2.7's
> static analyzer (``clang --static``) over trunk. It's mostly just finding
> stuff like unneeded variable initialization or variables that are never
> used (compilation is picking up unused returned values, almost all from
> PyObject_INIT).
> 
> When I check in these changes I will do it file by file, ...

Do you plan to port the changes to py3k? and what about 2.6 and 3.1?

--

Please don't change whitespaces in the same commit. I think that we should fix 
the whitespaces of the whole project in one unique commit. Well, or least 
don't fix whitespaces file by file...

-- 
Victor Stinner
http://www.haypocalc.com/

From brett at python.org  Thu May  6 01:21:44 2010
From: brett at python.org (Brett Cannon)
Date: Wed, 5 May 2010 16:21:44 -0700
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <201005052301.50504.victor.stinner@haypocalc.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com> 
	<201005052301.50504.victor.stinner@haypocalc.com>
Message-ID: <s2rbbaeab101005051621oeb980a07m914b91a954f4c463@mail.gmail.com>

On Wed, May 5, 2010 at 14:01, Victor Stinner
<victor.stinner at haypocalc.com>wrote:

> Le mardi 04 mai 2010 00:37:22, Brett Cannon a ?crit :
> > Since 2.7 is probably going to exist for a while, I am running Clang
> 2.7's
> > static analyzer (``clang --static``) over trunk. It's mostly just finding
> > stuff like unneeded variable initialization or variables that are never
> > used (compilation is picking up unused returned values, almost all from
> > PyObject_INIT).
> >
> > When I check in these changes I will do it file by file, ...
>
> Do you plan to port the changes to py3k?


In case you didn't see my follow-up email that I sent just before this
email, I will most likely do py3k when 3.2 is closer.


> and what about 2.6 and 3.1?


Not doing 2.6 as almost all changes are too minor bother. I think I found
like two potential Py_DECREF/Py_XDECREF changes, but that's about it. And
3.1 would require py3k which I am not planning on  doing in the near future.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100505/7cc8f68c/attachment.html>

From eric at trueblade.com  Thu May  6 01:58:29 2010
From: eric at trueblade.com (Eric Smith)
Date: Wed, 05 May 2010 19:58:29 -0400
Subject: [Python-Dev] Did I miss the decision to untabify all of the C code?
Message-ID: <4BE20625.9070203@trueblade.com>

It looks like we're moving ahead with removing tabs. Was there consensus 
on this?

Last I saw Antoine had written a script that might do what we want, but 
hadn't been thoroughly tested. Now I've seen a few checkins for files 
that have been run through the script.

What gives? And why do this so close to 2.7? I don't think it will cause 
any problems, but it's hard to review commits to ensure they have no 
changes when there's a rush of large commits near a release.

Eric.


From benjamin at python.org  Thu May  6 02:48:35 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 5 May 2010 19:48:35 -0500
Subject: [Python-Dev] Did I miss the decision to untabify all of the C
	code?
In-Reply-To: <4BE20625.9070203@trueblade.com>
References: <4BE20625.9070203@trueblade.com>
Message-ID: <q2s1afaf6161005051748sfea00123l1f094e1775d08abf@mail.gmail.com>

2010/5/5 Eric Smith <eric at trueblade.com>:
> It looks like we're moving ahead with removing tabs. Was there consensus on
> this?
>
> Last I saw Antoine had written a script that might do what we want, but
> hadn't been thoroughly tested. Now I've seen a few checkins for files that
> have been run through the script.
>
> What gives? And why do this so close to 2.7? I don't think it will cause any
> problems, but it's hard to review commits to ensure they have no changes
> when there's a rush of large commits near a release.

I presume Victor was acting on an older decision that said that files
mixed with tabs and spaces can be reindented for consistency.


-- 
Regards,
Benjamin

From solipsis at pitrou.net  Thu May  6 02:51:20 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 6 May 2010 00:51:20 +0000 (UTC)
Subject: [Python-Dev] Did I miss the decision to untabify all of the C
	code?
References: <4BE20625.9070203@trueblade.com>
Message-ID: <loom.20100506T024712-79@post.gmane.org>

Eric Smith <eric <at> trueblade.com> writes:
> 
> Last I saw Antoine had written a script that might do what we want, but 
> hadn't been thoroughly tested. Now I've seen a few checkins for files 
> that have been run through the script.

As far as I'm concerned, it was a case of eating my own dog food: running the
script over a couple of files I'm interested in (_ssl.c, _fileio.c). I believe
Victor processed posixmodule.c for the same reasons.

> What gives? And why do this so close to 2.7? I don't think it will cause 
> any problems, but it's hard to review commits to ensure they have no 
> changes when there's a rush of large commits near a release.

Well, however soon or late we do this, good luck reviewing multi-thousand line
commits to check no mistake sneaked in :)
By construction, these commits only adjust whitespace in some C files, which
means the risk of breakage is very close to zero.
(I guess you could do a "svn diff -x -w" between each two revisions to expose
any potential non-whitespace changes)

Really

Antoine.



From eric at trueblade.com  Thu May  6 02:59:26 2010
From: eric at trueblade.com (Eric Smith)
Date: Wed, 05 May 2010 20:59:26 -0400
Subject: [Python-Dev] Did I miss the decision to untabify all of the C
 code?
In-Reply-To: <loom.20100506T024712-79@post.gmane.org>
References: <4BE20625.9070203@trueblade.com>
	<loom.20100506T024712-79@post.gmane.org>
Message-ID: <4BE2146E.6000702@trueblade.com>

Antoine Pitrou wrote:
> Eric Smith <eric <at> trueblade.com> writes:
>> Last I saw Antoine had written a script that might do what we want, but 
>> hadn't been thoroughly tested. Now I've seen a few checkins for files 
>> that have been run through the script.
> 
> As far as I'm concerned, it was a case of eating my own dog food: running the
> script over a couple of files I'm interested in (_ssl.c, _fileio.c). I believe
> Victor processed posixmodule.c for the same reasons.
> 
>> What gives? And why do this so close to 2.7? I don't think it will cause 
>> any problems, but it's hard to review commits to ensure they have no 
>> changes when there's a rush of large commits near a release.
> 
> Well, however soon or late we do this, good luck reviewing multi-thousand line
> commits to check no mistake sneaked in :)

That's my point. Since it's basically unreviewable, is it smart to do it 
during a beta?

I grant you that it's a largely a mechanized change (except for the "a 
posteriori manual intervention" part), but still.

Eric.

From jsbueno at python.org.br  Thu May  6 05:52:28 2010
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Thu, 6 May 2010 00:52:28 -0300
Subject: [Python-Dev] Did I miss the decision to untabify all of the C
	code?
In-Reply-To: <4BE2146E.6000702@trueblade.com>
References: <4BE20625.9070203@trueblade.com>
	<loom.20100506T024712-79@post.gmane.org>
	<4BE2146E.6000702@trueblade.com>
Message-ID: <p2m4207a7ce1005052052n89c7cf21w78103214067a22e0@mail.gmail.com>

On Wed, May 5, 2010 at 9:59 PM, Eric Smith <eric at trueblade.com> wrote:
> Antoine Pitrou wrote:
>>
>> Eric Smith <eric <at> trueblade.com> writes:
>>>
>>> Last I saw Antoine had written a script that might do what we want, but
>>> hadn't been thoroughly tested. Now I've seen a few checkins for files that
>>> have been run through the script.
>>
>> As far as I'm concerned, it was a case of eating my own dog food: running
>> the
>> script over a couple of files I'm interested in (_ssl.c, _fileio.c). I
>> believe
>> Victor processed posixmodule.c for the same reasons.
>>
>>> What gives? And why do this so close to 2.7? I don't think it will cause
>>> any problems, but it's hard to review commits to ensure they have no changes
>>> when there's a rush of large commits near a release.
>>
>> Well, however soon or late we do this, good luck reviewing multi-thousand
>> line
>> commits to check no mistake sneaked in :)
>
> That's my point. Since it's basically unreviewable, is it smart to do it
> during a beta?

Hello folks -
I don't think these modifications are that "unreviewable": the
generated binaries have to be exactly the same with the untabified
files don't they? So is a matter of stashing the binaries, applying
the patches, rebuild and check to see if the binaries match. Any
possible script defects undetected by this would be only (C code)
indentation, which could be fixed later.

Python 2.7 is in beta, but not applying such a fix now would probably
mean that python 2.x would forever remain with the mixed tabs, since
it would make much less sense for such a change in a minor revision
(although I'd favor it even there).


  js
  -><-



>
> I grant you that it's a largely a mechanized change (except for the "a
> posteriori manual intervention" part), but still.
>
> Eric.

From alexandre at peadrop.com  Thu May  6 08:17:14 2010
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Wed, 5 May 2010 23:17:14 -0700
Subject: [Python-Dev] Did I miss the decision to untabify all of the C
	code?
In-Reply-To: <p2m4207a7ce1005052052n89c7cf21w78103214067a22e0@mail.gmail.com>
References: <4BE20625.9070203@trueblade.com>
	<loom.20100506T024712-79@post.gmane.org> 
	<4BE2146E.6000702@trueblade.com>
	<p2m4207a7ce1005052052n89c7cf21w78103214067a22e0@mail.gmail.com>
Message-ID: <h2kacd65fa21005052317ib41ecfffx285207a51fd92144@mail.gmail.com>

On Wed, May 5, 2010 at 8:52 PM, Joao S. O. Bueno <jsbueno at python.org.br> wrote:
> Python 2.7 is in beta, but not applying such a fix now would probably
> mean that python 2.x would forever remain with the mixed tabs, since
> it would make much less sense for such a change in a minor revision
> (although I'd favor it even there).
>

Since 2.7 is likely the last release of the 2.x series, wouldn't it
more productive to spend time improving it instead of wasting time on
minor details like indentation?

-- Alexandre

From victor.stinner at haypocalc.com  Thu May  6 10:43:44 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Thu, 6 May 2010 10:43:44 +0200
Subject: [Python-Dev] Did I miss the decision to untabify all of the C
	code?
In-Reply-To: <h2kacd65fa21005052317ib41ecfffx285207a51fd92144@mail.gmail.com>
References: <4BE20625.9070203@trueblade.com>
	<p2m4207a7ce1005052052n89c7cf21w78103214067a22e0@mail.gmail.com>
	<h2kacd65fa21005052317ib41ecfffx285207a51fd92144@mail.gmail.com>
Message-ID: <201005061043.44725.victor.stinner@haypocalc.com>

Le jeudi 06 mai 2010 08:17:14, Alexandre Vassalotti a ?crit :
> On Wed, May 5, 2010 at 8:52 PM, Joao S. O. Bueno <jsbueno at python.org.br> 
wrote:
> > Python 2.7 is in beta, but not applying such a fix now would probably
> > mean that python 2.x would forever remain with the mixed tabs, since
> > it would make much less sense for such a change in a minor revision
> > (although I'd favor it even there).
> 
> Since 2.7 is likely the last release of the 2.x series, wouldn't it
> more productive to spend time improving it instead of wasting time on
> minor details like indentation?

Because "2.7 is likely the last release of the 2.x series", I think that it 
will maintained more longer than any other 2.x release. If we change the 
indentation in 3.x but not in 2.x, it will be harder to port the commits from 
3.x to 2.7.

I consider that the maintenance is very important for this last release, more 
important than "improving it". I prefer to work on new features on 3.x and 
keep 2.7 as stable as we can.

-- 
Victor Stinner
http://www.haypocalc.com/

From victor.stinner at haypocalc.com  Thu May  6 11:41:18 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Thu, 6 May 2010 11:41:18 +0200
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <4BDFACC0.2010808@v.loewis.de>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<h2vacd65fa21005032100xb810dd65pb8b808a9a17a814a@mail.gmail.com>
	<4BDFACC0.2010808@v.loewis.de>
Message-ID: <201005061141.19065.victor.stinner@haypocalc.com>

Le mardi 04 mai 2010 07:12:32, Martin v. L?wis a ?crit :
> > Will changing the indentation of source files to 4 space indents break
> > patches on the bug tracker?
> 
> Plain patch will choke, but "patch -l" might accept them.

Tested on posixmodule.c: it works :-)

-- 
Victor Stinner
http://www.haypocalc.com/

From solipsis at pitrou.net  Thu May  6 11:53:55 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 6 May 2010 09:53:55 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?Did_I_miss_the_decision_to_untabify_all_of?=
	=?utf-8?q?_the_C=09code=3F?=
References: <4BE20625.9070203@trueblade.com>
	<loom.20100506T024712-79@post.gmane.org>
	<4BE2146E.6000702@trueblade.com>
	<p2m4207a7ce1005052052n89c7cf21w78103214067a22e0@mail.gmail.com>
	<h2kacd65fa21005052317ib41ecfffx285207a51fd92144@mail.gmail.com>
Message-ID: <loom.20100506T114742-511@post.gmane.org>


Alexandre Vassalotti <alexandre <at> peadrop.com> writes:
> 
> Since 2.7 is likely the last release of the 2.x series, wouldn't it
> more productive to spend time improving it instead of wasting time on
> minor details like indentation?

We occasionally waste time on minor details such as code indentation,
documentation wording and punctuation, distinguishing "built-in" vs "builtin",
etc. :)
I don't think it prevents anyone from doing productive work.
(besides, my own bug queue for 2.x currently appears to be empty)

Then, as pointed out by Victor, if we want to solve the current indentation
mixup, we have to do it in all branches so as to avoid making backports more
difficult.

Regards

Antoine.



From victor.stinner at haypocalc.com  Thu May  6 10:38:01 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Thu, 6 May 2010 10:38:01 +0200
Subject: [Python-Dev] Did I miss the decision to untabify all of the C
	code?
In-Reply-To: <4BE2146E.6000702@trueblade.com>
References: <4BE20625.9070203@trueblade.com>
	<loom.20100506T024712-79@post.gmane.org>
	<4BE2146E.6000702@trueblade.com>
Message-ID: <201005061038.02285.victor.stinner@haypocalc.com>

Le jeudi 06 mai 2010 02:59:26, Eric Smith a ?crit :
> I grant you that it's a largely a mechanized change (except for the "a
> posteriori manual intervention" part), but still.

Attached patches are the "manual interventation" parts. 99% of the whole patch 
only changes the indentation. There is just *two* changes not related directly 
to indentation:
 - I moved rc and buffer variable (specific to OS/2) at the beginning of the 
function to avoid { and }.
 - I added a newline in "static PyObject * os2_error(int code)"

The manual editions is mostly to "fix" the indentation.


Diff on Python trunk (diff -uw):
---------------------------
--- Modules/posixmodule.c.r80843        2010-05-06 10:18:47.000000000 +0200
+++ Modules/posixmodule.c       2010-05-06 10:18:51.000000000 +0200
@@ -470,6 +470,10 @@
 {
        PyObject *d;
        char **e;
+#if defined(PYOS_OS2)
+    APIRET rc;
+    char   buffer[1024]; /* OS/2 Provides a Documented Max of 1024 Chars */
+#endif
        d = PyDict_New();
        if (d == NULL)
                return NULL;
@@ -505,10 +509,6 @@
                Py_DECREF(v);
        }
 #if defined(PYOS_OS2)
-    {
-        APIRET rc;
-        char   buffer[1024]; /* OS/2 Provides a Documented Max of 1024 Chars 
*/
-
         rc = DosQueryExtLIBPATH(buffer, BEGIN_LIBPATH);
        if (rc == NO_ERROR) { /* (not a type, envname is NOT 'BEGIN_LIBPATH') 
*/
             PyObject *v = PyString_FromString(buffer);
@@ -521,7 +521,6 @@
                    PyDict_SetItemString(d, "ENDLIBPATH", v);
             Py_DECREF(v);
         }
-    }
 #endif
        return d;
 }
@@ -662,7 +661,8 @@
    errors are not in a global variable e.g. 'errno' nor are
    they congruent with posix error numbers. */
 
-static PyObject * os2_error(int code)
+static PyObject *
+os2_error(int code)
 {
     char text[1024];
     PyObject *v;
---------------------------


Diff on Python py3k (diff -uw):
---------------------------
--- Modules/posixmodule.c.r80845	2010-05-06 10:36:27.000000000 +0200
+++ Modules/posixmodule.c	2010-05-06 10:36:32.000000000 +0200
@@ -445,6 +445,11 @@
 #else
 	char **e;
 #endif
+#if defined(PYOS_OS2)
+    APIRET rc;
+    char   buffer[1024]; /* OS/2 Provides a Documented Max of 1024 Chars */
+#endif
+
 	d = PyDict_New();
 	if (d == NULL)
 		return NULL;
@@ -515,10 +520,6 @@
 	}
 #endif
 #if defined(PYOS_OS2)
-    {
-        APIRET rc;
-        char   buffer[1024]; /* OS/2 Provides a Documented Max of 1024 Chars 
*/
-
         rc = DosQueryExtLIBPATH(buffer, BEGIN_LIBPATH);
 	if (rc == NO_ERROR) { /* (not a type, envname is NOT 'BEGIN_LIBPATH') */
             PyObject *v = PyBytes_FromString(buffer);
@@ -531,7 +532,6 @@
 	    PyDict_SetItemString(d, "ENDLIBPATH", v);
             Py_DECREF(v);
         }
-    }
 #endif
 	return d;
 }
@@ -672,7 +672,8 @@
    errors are not in a global variable e.g. 'errno' nor are
    they congruent with posix error numbers. */
 
-static PyObject * os2_error(int code)
+static PyObject *
+os2_error(int code)
 {
     char text[1024];
     PyObject *v;

---------------------------

-- 
Victor Stinner
http://www.haypocalc.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: posix_tabify_manual-trunk.patch
Type: text/x-patch
Size: 22334 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100506/12dd661d/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: posix_untabify_manual-py3k.patch
Type: text/x-patch
Size: 18448 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100506/12dd661d/attachment-0003.bin>

From barry at python.org  Thu May  6 14:21:38 2010
From: barry at python.org (Barry Warsaw)
Date: Thu, 6 May 2010 08:21:38 -0400
Subject: [Python-Dev] Fwd: broken mailing list links in PEP(s?)
In-Reply-To: <D543D7E1-CEF8-473F-B477-107A26D4EBDA@fuhm.net>
References: <4BE14BE1.6090200@voidspace.org.uk>
	<20100505110908.GA7160@phd.pp.ru>
	<60DA94AF-6444-435A-86DD-98CAF432197A@python.org>
	<D543D7E1-CEF8-473F-B477-107A26D4EBDA@fuhm.net>
Message-ID: <20100506082138.33d659d9@heresy>

On May 05, 2010, at 11:35 AM, James Y Knight wrote:

>And of course if you're paying attention, you can fix the mbox file  
>(quoting "From" etc) such that it generates the same numbers as it did  
>the first time.

Mailman even has a command for this (I feel like an Apple commercial).  You
should check things though because it's based on heuristics that can sometimes
be fooled.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100506/944571fe/attachment.pgp>

From catherine.devlin at gmail.com  Thu May  6 16:47:53 2010
From: catherine.devlin at gmail.com (Catherine Devlin)
Date: Thu, 6 May 2010 10:47:53 -0400
Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing
	needed
Message-ID: <l2g6523e39a1005060747oc5fc95f5wd2b8ffb880ccff2c@mail.gmail.com>

Hey, everybody... I'm Catherine, a database administrator who makes up
excuses to write Python instead.

I'm not actually here as a core developer, but as somebody who hopes to
become a developer and recruit some more, which brings me to my question:

Who lives close enough to Ohio to make it to PyOhio this summer?  I want to
use PyOhio to create new Python devs (including myself), but I need some
existing ones to seed the process.

I need a few veterans (3?) who can commit to come to PyOhio and take part as
audience/instructors in a "Teach Me [Python core / standard library]
Bugfixing" session.  (See
http://catherinedevlin.blogspot.com/2010/04/bugfixing-at-pyohio.html.)  The
PyOhio Call for Proposals is up May 10 so I'd better find you quick!

I'm pretty much ignorant enough to lead a Teach Me session.  In a Teach Me
session, the person at the projector *doesn't* know the material.  Instead,
she asks the audience questions ("How do I find a bug to work on?"), and
they talk her through it.  It's based on Teach Me Twisted, a mind-blowing
session Steve Holden led at PyCon 2008 (
http://catherinedevlin.blogspot.com/2008/03/teach-me-twisted.html).  I think
it's a fantastic way to teach, but it depends on some veterans being in the
audience.  There are folks in the greater Python community eager to get hold
of a video of such a session... if we do this well, it could become an
important tool in keeping the quality of core Python code high.

And all I need from you, my audience-instructors, is a promise to show up
(no preparation necessary).  Can you make it?  Can you pass the appeal on to
others you know of?

Thanks!  Hope to see you in July!

-- 
- Catherine
http://catherinedevlin.blogspot.com/
*** PyOhio 2010 * July 31 - Aug 1 * Columbus, OH * pyohio.org ***
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100506/5efc755a/attachment.html>

From doug.hellmann at gmail.com  Thu May  6 17:01:58 2010
From: doug.hellmann at gmail.com (Doug Hellmann)
Date: Thu, 6 May 2010 11:01:58 -0400
Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing
	needed
In-Reply-To: <l2g6523e39a1005060747oc5fc95f5wd2b8ffb880ccff2c@mail.gmail.com>
References: <l2g6523e39a1005060747oc5fc95f5wd2b8ffb880ccff2c@mail.gmail.com>
Message-ID: <8A2A6234-F4A1-4A3D-A9A8-A9BFDE0DB080@gmail.com>

What an excellent idea!  We should have these at *every* regional  
conference.

Doug

On May 6, 2010, at 10:47 AM, Catherine Devlin wrote:

> Hey, everybody... I'm Catherine, a database administrator who makes  
> up excuses to write Python instead.
>
> I'm not actually here as a core developer, but as somebody who hopes  
> to become a developer and recruit some more, which brings me to my  
> question:
>
> Who lives close enough to Ohio to make it to PyOhio this summer?  I  
> want to use PyOhio to create new Python devs (including myself), but  
> I need some existing ones to seed the process.
>
> I need a few veterans (3?) who can commit to come to PyOhio and take  
> part as audience/instructors in a "Teach Me [Python core / standard  
> library] Bugfixing" session.  (See http://catherinedevlin.blogspot.com/2010/04/bugfixing-at-pyohio.html.) 
>   The PyOhio Call for Proposals is up May 10 so I'd better find you  
> quick!
>
> I'm pretty much ignorant enough to lead a Teach Me session.  In a  
> Teach Me session, the person at the projector *doesn't* know the  
> material.  Instead, she asks the audience questions ("How do I find  
> a bug to work on?"), and they talk her through it.  It's based on  
> Teach Me Twisted, a mind-blowing session Steve Holden led at PyCon  
> 2008 (http://catherinedevlin.blogspot.com/2008/03/teach-me-twisted.html 
> ).  I think it's a fantastic way to teach, but it depends on some  
> veterans being in the audience.  There are folks in the greater  
> Python community eager to get hold of a video of such a session...  
> if we do this well, it could become an important tool in keeping the  
> quality of core Python code high.
>
> And all I need from you, my audience-instructors, is a promise to  
> show up (no preparation necessary).  Can you make it?  Can you pass  
> the appeal on to others you know of?
>
> Thanks!  Hope to see you in July!
>
> -- 
> - Catherine
> http://catherinedevlin.blogspot.com/
> *** PyOhio 2010 * July 31 - Aug 1 * Columbus, OH * pyohio.org ***
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/doug.hellmann%40gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100506/0e528baa/attachment.html>

From ronaldoussoren at mac.com  Thu May  6 17:09:11 2010
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 06 May 2010 17:09:11 +0200
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the
	code	base
In-Reply-To: <i2ubbaeab101005051356oc69fc594we0e04fba46306896@mail.gmail.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<i2ubbaeab101005051356oc69fc594we0e04fba46306896@mail.gmail.com>
Message-ID: <27829881-2894-4A95-8D01-7BCD0B06FBB1@mac.com>


On 5 May, 2010, at 22:56, Brett Cannon wrote:

> I am done running the analysis over trunk. I will not svnmerge these changes into py3k as the amount of time and effort that would take equates to running the static analyzer again just before 3.2 is released and possibly catching more changes (and maybe even a newer version of Clang at that point).

Have you looked into teaching clang's static analyser about Python's refcounting rules?  Clang's analyser can tell you about problems related to reference count management for Objective-C code and doing the same for code using the CPython API would be usefull.

Ronald
> 
> On Mon, May 3, 2010 at 15:37, Brett Cannon <brett at python.org> wrote:
> Since 2.7 is probably going to exist for a while, I am running Clang 2.7's static analyzer (``clang --static``) over trunk. It's mostly just finding stuff like unneeded variable initialization or variables that are never used (compilation is picking up unused returned values, almost all from PyObject_INIT).
> 
> When I check in these changes I will do it file by file, but my question is how to handle Misc/NEWS. I have gone through the underscores and the 'a's in Modules and already have six modified files, so the final count might be a little high. Do people want individual entries per file, or simply a single entry that lists each file modified?
> 
> We should probably go through the C code and fix the whitespace before we hit 2.7 final (there is a ton of lines with extraneous spaces).
> 
> -Brett
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100506/2ae94c8e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3567 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100506/2ae94c8e/attachment-0001.bin>

From solipsis at pitrou.net  Thu May  6 17:29:33 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 6 May 2010 15:29:33 +0000 (UTC)
Subject: [Python-Dev] audience-instructors for Teach Me Python
	Bugfixing	needed
References: <l2g6523e39a1005060747oc5fc95f5wd2b8ffb880ccff2c@mail.gmail.com>
Message-ID: <hrun8t$840$1@dough.gmane.org>


Hello,

> Who lives close enough to Ohio to make it to PyOhio this summer?  I want
> to use PyOhio to create new Python devs (including myself), but I need
> some existing ones to seed the process.

I'm not really answering your question (I'm very far from Ohio), but a 
good way to start up without physically meeting a core dev in your area 
would be to look for bugs in the issue tracker and start tackling those 
that tickle your interest. "Tackling" here can mean proposing a patch, or 
giving advice, or asking for guidance on how to produce a patch for that 
particular issue, or even reviewing an existing patch. You don't have to 
be a CPython expert to try this; just be prepared to spend some time 
exchanging with other people who will point out possible mistakes, or 
teach you some of the annoying idiosyncrasies.

Scanning through open issues will also give you a general idea of what 
kind of functionalities are looking for improvement, or need fixing.

(you can create a new issue and start tackling it yourself, too)

Also, if you have enough free time, you can hang out on #python-dev, 
which can speed up the process, but it's not required.

You can also find formal information about the development process here:
http://www.python.org/dev/faq/

Regards

Antoine.


From brett at python.org  Thu May  6 19:40:05 2010
From: brett at python.org (Brett Cannon)
Date: Thu, 6 May 2010 10:40:05 -0700
Subject: [Python-Dev] Running Clang 2.7's static analyzer over the code
	base
In-Reply-To: <27829881-2894-4A95-8D01-7BCD0B06FBB1@mac.com>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com> 
	<i2ubbaeab101005051356oc69fc594we0e04fba46306896@mail.gmail.com> 
	<27829881-2894-4A95-8D01-7BCD0B06FBB1@mac.com>
Message-ID: <i2obbaeab101005061040hb7af5ac5g71bff34eb85038a4@mail.gmail.com>

On Thu, May 6, 2010 at 08:09, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

>
> On 5 May, 2010, at 22:56, Brett Cannon wrote:
>
> I am done running the analysis over trunk. I will not svnmerge these
> changes into py3k as the amount of time and effort that would take equates
> to running the static analyzer again just before 3.2 is released and
> possibly catching more changes (and maybe even a newer version of Clang at
> that point).
>
>
> Have you looked into teaching clang's static analyser about Python's
> refcounting rules?  Clang's analyser can tell you about problems related to
> reference count management for Objective-C code and doing the same for code
> using the CPython API would be usefull.
>

That's a thought, but I have not looked into it yet. As of right now the
first thing I would do is fix its NULL de-reference analysis as it had a
bunch of false-positives on that (I don't think it handles `!ptr` as
equivalent to `ptr == NULL`).

-Brett



>
> Ronald
>
>
> On Mon, May 3, 2010 at 15:37, Brett Cannon <brett at python.org> wrote:
>
>> Since 2.7 is probably going to exist for a while, I am running Clang 2.7's
>> static analyzer (``clang --static``) over trunk. It's mostly just finding
>> stuff like unneeded variable initialization or variables that are never used
>> (compilation is picking up unused returned values, almost all from
>> PyObject_INIT).
>>
>> When I check in these changes I will do it file by file, but my question
>> is how to handle Misc/NEWS. I have gone through the underscores and the 'a's
>> in Modules and already have six modified files, so the final count might be
>> a little high. Do people want individual entries per file, or simply a
>> single entry that lists each file modified?
>>
>> We should probably go through the C code and fix the whitespace before we
>> hit 2.7 final (there is a ton of lines with extraneous spaces).
>>
>> -Brett
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100506/dc5d5f7b/attachment.html>

From amk at amk.ca  Fri May  7 03:50:35 2010
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 6 May 2010 21:50:35 -0400
Subject: [Python-Dev] What's New text on future maintenance
Message-ID: <20100507015035.GA50210@andrew-kuchlings-macbook.local>

FYI: I've just added the text below to the "What's New" document for
2.7.  I wanted to describe how 2.7 will probably be maintained, but
didn't want to write anything that sounded like an iron-clad guarantee
of a maintenance timespan.  Does this text seem like a reasonable set
of statements?

--amk

Python 2.7 is intended to be the last major release in the 2.x series.
Though more major releases have not been absolutely ruled out, the
Python maintainers are planning to focus their efforts on Python 3.x.

This means that 2.7 will remain in place for a long time, running
production systems that have not been ported to Python 3.x.
Two consequences of the long-term significance of 2.7 are:

* It's very likely the 2.7 release will have a longer period of
  maintenance compared to earlier 2.x versions.  Python 2.7 will
  continue to be maintained while the transition to 3.x is in
  progress, and that transition will itself be lengthy.  Most 2.x
  versions are maintained for about 4 years, from the first to the
  last bugfix release; patchlevel releases for Python 2.7 will
  probably be made for at least 6 years.

* Because 2.7 will be running production applications, a policy
  decision was made to silence warnings only of interest to developers
  by default.  Silencing :exc:`DeprecationWarning` and its descendants
  prevents users from seeing warnings triggered by an application.
  (Carried out in :issue:`7319`.)

  You can re-enable display of :exc:`DeprecationWarning` messages by
  running Python with the :option:`-Wdefault` (short form:
  :option:`-Wd`) switch, or you can add
  ``warnings.simplefilter('default')`` to your code.


From guido at python.org  Fri May  7 04:09:46 2010
From: guido at python.org (Guido van Rossum)
Date: Thu, 6 May 2010 19:09:46 -0700
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
Message-ID: <m2xca471dc21005061909xd173782fheef6c94b96c4940@mail.gmail.com>

On Thu, May 6, 2010 at 6:50 PM, A.M. Kuchling <amk at amk.ca> wrote:
> FYI: I've just added the text below to the "What's New" document for
> 2.7. ?I wanted to describe how 2.7 will probably be maintained, but
> didn't want to write anything that sounded like an iron-clad guarantee
> of a maintenance timespan. ?Does this text seem like a reasonable set
> of statements?
>
> --amk
>
> Python 2.7 is intended to be the last major release in the 2.x series.
> Though more major releases have not been absolutely ruled out, the

I would scrap the "Though more ... ruled out" part. That just stokes
unrealistic hopes. :-)

> Python maintainers are planning to focus their efforts on Python 3.x.
>
> This means that 2.7 will remain in place for a long time, running
> production systems that have not been ported to Python 3.x.
> Two consequences of the long-term significance of 2.7 are:
>
> * It's very likely the 2.7 release will have a longer period of
> ?maintenance compared to earlier 2.x versions. ?Python 2.7 will
> ?continue to be maintained while the transition to 3.x is in
> ?progress, and that transition will itself be lengthy. ?Most 2.x
> ?versions are maintained for about 4 years, from the first to the
> ?last bugfix release; patchlevel releases for Python 2.7 will
> ?probably be made for at least 6 years.
>
> * Because 2.7 will be running production applications, a policy
> ?decision was made to silence warnings only of interest to developers
> ?by default. ?Silencing :exc:`DeprecationWarning` and its descendants
> ?prevents users from seeing warnings triggered by an application.
> ?(Carried out in :issue:`7319`.)
>
> ?You can re-enable display of :exc:`DeprecationWarning` messages by
> ?running Python with the :option:`-Wdefault` (short form:
> ?:option:`-Wd`) switch, or you can add
> ?``warnings.simplefilter('default')`` to your code.

All this sounds fine to me. Thanks for taking the time to write this!

-- 
--Guido van Rossum (python.org/~guido)

From ben+python at benfinney.id.au  Fri May  7 04:10:45 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 07 May 2010 12:10:45 +1000
Subject: [Python-Dev] What's New text on future maintenance
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
Message-ID: <877hngfom2.fsf@benfinney.id.au>

"A.M. Kuchling" <amk at amk.ca> writes:

> FYI: I've just added the text below to the "What's New" document for
> 2.7. I wanted to describe how 2.7 will probably be maintained, but
> didn't want to write anything that sounded like an iron-clad guarantee
> of a maintenance timespan. Does this text seem like a reasonable set
> of statements?

If you give an actual time period, that's all that will actually be
communicated to most people. This text will be read by a few people, and
communicated as simply ?six years? to everyone else. It doesn't matter
how many caveats and qualifiers you surround that with; those will all
be lost in transmission from person to person.

Would it make more sense to, instead of giving a time period, rather
describe the *circumstances* that will determine when maintenance
releases will cease for 2.7?

-- 
 \         ?Religious faith is the one species of human ignorance that |
  `\     will not admit of even the *possibility* of correction.? ?Sam |
_o__)                                 Harris, _The End of Faith_, 2004 |
Ben Finney


From benjamin at python.org  Fri May  7 04:33:50 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 6 May 2010 21:33:50 -0500
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
Message-ID: <j2p1afaf6161005061933k5ff3338cs8e38cf3b1f4350e3@mail.gmail.com>

2010/5/6 A.M. Kuchling <amk at amk.ca>:
> FYI: I've just added the text below to the "What's New" document for
> 2.7. ?I wanted to describe how 2.7 will probably be maintained, but
> didn't want to write anything that sounded like an iron-clad guarantee
> of a maintenance timespan. ?Does this text seem like a reasonable set
> of statements?
>
> --amk
>
> Python 2.7 is intended to be the last major release in the 2.x series.
> Though more major releases have not been absolutely ruled out, the
> Python maintainers are planning to focus their efforts on Python 3.x.
>
> This means that 2.7 will remain in place for a long time, running
> production systems that have not been ported to Python 3.x.
> Two consequences of the long-term significance of 2.7 are:
>
> * It's very likely the 2.7 release will have a longer period of
> ?maintenance compared to earlier 2.x versions. ?Python 2.7 will
> ?continue to be maintained while the transition to 3.x is in
> ?progress, and that transition will itself be lengthy. ?Most 2.x
> ?versions are maintained for about 4 years, from the first to the
> ?last bugfix release; patchlevel releases for Python 2.7 will
> ?probably be made for at least 6 years.

I don't think there's any point in being hypothetical about. I believe
we've already said that maintence for 2.7 will last for at least 5
years, so let's proclaim it.



-- 
Regards,
Benjamin

From tjreedy at udel.edu  Fri May  7 06:43:20 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 07 May 2010 00:43:20 -0400
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
Message-ID: <hs05p7$7u2$1@dough.gmane.org>

On 5/6/2010 9:50 PM, A.M. Kuchling wrote:
> FYI: I've just added the text below to the "What's New" document for
> 2.7.  I wanted to describe how 2.7 will probably be maintained, but
> didn't want to write anything that sounded like an iron-clad guarantee
> of a maintenance timespan.  Does this text seem like a reasonable set
> of statements?
>
> --amk
>
> Python 2.7 is intended to be the last major release in the 2.x series.
> Though more major releases have not been absolutely ruled out, the
> Python maintainers are planning to focus their efforts on Python 3.x.
>
> This means that 2.7 will remain in place for a long time, running
> production systems that have not been ported to Python 3.x.
> Two consequences of the long-term significance of 2.7 are:
>
> * It's very likely the 2.7 release will have a longer period of
>    maintenance compared to earlier 2.x versions.  Python 2.7 will
>    continue to be maintained while the transition to 3.x is in
>    progress, and that transition will itself be lengthy.  Most 2.x
>    versions are maintained for about 4 years, from the first to the
>    last bugfix release; patchlevel releases for Python 2.7 will
>    probably be made for at least 6 years.

Actually, bugfix releases generally stop with the next major release, 
with about a 2 year interval. I agree with others about condensing, to 
something like:

"Python 2.7 is intended to be the last major release in the 2.x series.
Python core developers plan to focus future efforts on Python 3.x.

This means that 2.7 will remain in place for a long time, running
production systems that have not been ported to Python 3.x. Bugfix 
releases will likely continue for 5 years."

Then the warnings stuff
>
> * Because 2.7 will be running production applications, a policy

Every major version (xcept 3.0) has run production application, and 3.1 
may be and 3.2 certainly will be. So this reasoning is not clear to me.

>    decision was made to silence warnings only of interest to developers
>    by default.

I believe this is meant to say "Warnings aimed only at those porting 
code to 3.x are silenced by default."

 >    Silencing :exc:`DeprecationWarning` and its descendants
>    prevents users from seeing warnings triggered by an application.
>    (Carried out in :issue:`7319`.)
>
>    You can re-enable display of :exc:`DeprecationWarning` messages by
>    running Python with the :option:`-Wdefault` (short form:
>    :option:`-Wd`) switch, or you can add
>    ``warnings.simplefilter('default')`` to your code.

Terry Jan Reedy




From martin at v.loewis.de  Fri May  7 09:30:00 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 07 May 2010 09:30:00 +0200
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
Message-ID: <4BE3C178.2090800@v.loewis.de>

> * It's very likely the 2.7 release will have a longer period of
>   maintenance compared to earlier 2.x versions.  Python 2.7 will
>   continue to be maintained while the transition to 3.x is in
>   progress, and that transition will itself be lengthy.  Most 2.x
>   versions are maintained for about 4 years, from the first to the
>   last bugfix release; patchlevel releases for Python 2.7 will
>   probably be made for at least 6 years.

I agree with Terry: how did you arrive at the 4 years for 2.x releases?
Bug fixes releases stopped after the next feature release being made,
which gave (counting between initial release and last bug fix release):
- 2.0 9 months
- 2.1 12 months
- 2.2 17 months
- 2.3 19 months
- 2.4 22 months
- 2.5 27 months
- 2.6 < 22 months (assuming the final 2.6 release is made before August)

In addition, since 2.3, we were offering security fixes for a period of
_five_ years.

So for 2.7, the question really is how long we create bug fix releases
and Windows binaries, rather than accepting only security fixes, and
producing source-only releases.

I'd personally expect that "longer" might mean 3 years (i.e. one more
release cycle), and definitely less than 6.

Regards,
Martin

From ncoghlan at gmail.com  Fri May  7 11:52:49 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 07 May 2010 19:52:49 +1000
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <hs05p7$7u2$1@dough.gmane.org>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
	<hs05p7$7u2$1@dough.gmane.org>
Message-ID: <4BE3E2F1.4040907@gmail.com>

Terry Reedy wrote:
> Then the warnings stuff
>>
>> * Because 2.7 will be running production applications, a policy
> 
> Every major version (xcept 3.0) has run production application, and 3.1
> may be and 3.2 certainly will be. So this reasoning is not clear to me.
> 
>>    decision was made to silence warnings only of interest to developers
>>    by default.
> 
> I believe this is meant to say "Warnings aimed only at those porting
> code to 3.x are silenced by default."

Actually, the decision was indeed to make all Deprecation Warnings
silent by default. The rationale was a bit different from what AMK
currently has though (otherwise we wouldn't have made the same change in
3.x). I'll take a stab at a more accurate rationale:

"""For previous releases, it has been the policy of the CPython core
developers that :exc:`DeprecationWarning` should be enabled by default.
This provides Python developers with a clear indication when their code
has a substantial risk of breaking in the next major version of Python.
However, the nature of Python usage has changed over the years, such
that there are now a significantly larger number of Python application
users that are not directly involved in the development of those
applications. This has lead to a situation where users may be receiving
irrelevant warnings from an application that is actually working
correctly, creating unnecessary concern for affected end users and
additional overhead for the developers of these applications in
responding to these concerns.

Accordingly, starting with Python 2.7, this policy has been revised and
the CPython interpreter has been updated to silence all occurrences of
:exc:`DeprecationWarning` by default. Developers wishing to see if their
application is at significant risk of breaking on the next major Python
release can re-enable display of :exc:`DeprecationWarning` messages by
running Python with the :option:`-Wdefault` (short form: :option:`-Wd`)
switch, or you can add ``warnings.simplefilter('default')`` to your
code. (Note that the 'default' in these settings refers to the default
warning behaviour of displaying warnings once for each location where
they are encountered rather than to the overall default warning settings
used by the interpreter)"""

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From victor.stinner at haypocalc.com  Fri May  7 13:05:52 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Fri, 7 May 2010 13:05:52 +0200
Subject: [Python-Dev] Issue #8610: Set default file system encoding to ascii
	on error?
Message-ID: <201005071305.52332.victor.stinner@haypocalc.com>

Python 3.0 introduced PyUnicode_DecodeFSDefault() and 
PyUnicode_DecodeFSDefaultAndSize() functions. These functions fallback to 
UTF-8 if getting the file system encoding failed or the encoding is unknown 
(there is not nl_langinfo(CODESET) function).

UTF-8 is not a good choice for the fallback because it's incompatible with 
other encodings like Latin1. I would like to fallback to ASCII on error which 
is compatible with all encodings (thanks to surrogateescape).

I would like to ensure that sys.getfilesystemencoding() result cannot be None, 
because Python 3.2 uses it on Unix to decode os.environb and to encode 
filenames in subprocess. I can implement a fallback for os.environb and 
subprocess (and other functions calling sys.getfilesystemencoding()), but I 
prefer to have a reliable sys.getfilesystemencoding() function.

This change doesn't concern Windows and Mac OS X because the encoding is 
hardcoded (mbcs, utf-8). On Unix, I don't know in which case nl_langinfo() can 
fail. Empty environment is not an error: nl_langinfo(CODESET) returns "ascii". 
I think that few (or no) user would notice this change.

-- 
Victor Stinner
http://www.haypocalc.com/

From solipsis at pitrou.net  Fri May  7 13:24:18 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 7 May 2010 11:24:18 +0000 (UTC)
Subject: [Python-Dev] Issue #8610: Set default file system encoding to
	ascii	on error?
References: <201005071305.52332.victor.stinner@haypocalc.com>
Message-ID: <hs0t92$js4$1@dough.gmane.org>

Le Fri, 07 May 2010 13:05:52 +0200, Victor Stinner a ?crit?:
> 
> UTF-8 is not a good choice for the fallback because it's incompatible
> with other encodings like Latin1. I would like to fallback to ASCII on
> error which is compatible with all encodings (thanks to
> surrogateescape).

What do you mean with "compatible with all encodings thanks to 
surrogateescape"?

>>> "???".encode("ascii", "surrogateescape")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 
0-2: ordinal not in range(128)

> I would like to ensure that sys.getfilesystemencoding() result cannot be
> None, because Python 3.2 uses it on Unix to decode os.environb and to
> encode filenames in subprocess. I can implement a fallback for
> os.environb and subprocess (and other functions calling
> sys.getfilesystemencoding()), but I prefer to have a reliable
> sys.getfilesystemencoding() function.

Having a reliable sys.getfilesystemencoding() would be a good thing 
indeed.

> This change doesn't concern Windows and Mac OS X because the encoding is
> hardcoded (mbcs, utf-8). On Unix, I don't know in which case
> nl_langinfo() can fail. Empty environment is not an error:
> nl_langinfo(CODESET) returns "ascii". I think that few (or no) user
> would notice this change.

Ok, it sounds like a good compromise.

Regards

Antoine.


From victor.stinner at haypocalc.com  Fri May  7 13:39:39 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Fri, 7 May 2010 13:39:39 +0200
Subject: [Python-Dev]
 =?utf-8?q?Issue_=238610=3A_Set_default_file_system_e?=
 =?utf-8?q?ncoding_to_ascii=09on_error=3F?=
In-Reply-To: <hs0t92$js4$1@dough.gmane.org>
References: <201005071305.52332.victor.stinner@haypocalc.com>
	<hs0t92$js4$1@dough.gmane.org>
Message-ID: <201005071339.40070.victor.stinner@haypocalc.com>

Le vendredi 07 mai 2010 13:24:18, Antoine Pitrou a ?crit :
> > UTF-8 is not a good choice for the fallback because it's incompatible
> > with other encodings like Latin1. I would like to fallback to ASCII on
> > error which is compatible with all encodings (thanks to
> > surrogateescape).
> 
> What do you mean with "compatible with all encodings thanks to
> surrogateescape"?
> 
> >>> "???".encode("ascii", "surrogateescape")
> ...
> UnicodeEncodeError: 'ascii' codec can't encode characters ...

ascii+surrogatescape can *decode* anything:

>>> b"a\xc3\xff".decode('ascii', 'surrogateescape')
'a\udcc3\udcff'

Encode with ascii+surrogatescape raise an UnicodeEncodeError for non-ASCII 
(except for surrogates). I think it's better to raise an error than creating 
utf8 filenames on a latin1 file system.

--

I forgot to mention Marc Lemburg propositing of creating a PYTHONFSENCODING 
environment variable: #8622.

-- 
Victor Stinner
http://www.haypocalc.com/

From mal at egenix.com  Fri May  7 13:55:48 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 07 May 2010 13:55:48 +0200
Subject: [Python-Dev] Issue #8610: Set default file system encoding to
 ascii	on error?
In-Reply-To: <201005071305.52332.victor.stinner@haypocalc.com>
References: <201005071305.52332.victor.stinner@haypocalc.com>
Message-ID: <4BE3FFC4.5000409@egenix.com>

Victor Stinner wrote:
> Python 3.0 introduced PyUnicode_DecodeFSDefault() and 
> PyUnicode_DecodeFSDefaultAndSize() functions. These functions fallback to 
> UTF-8 if getting the file system encoding failed or the encoding is unknown 
> (there is not nl_langinfo(CODESET) function).
> 
> UTF-8 is not a good choice for the fallback because it's incompatible with 
> other encodings like Latin1. I would like to fallback to ASCII on error which 
> is compatible with all encodings (thanks to surrogateescape).
> 
> I would like to ensure that sys.getfilesystemencoding() result cannot be None, 
> because Python 3.2 uses it on Unix to decode os.environb and to encode 
> filenames in subprocess. I can implement a fallback for os.environb and 
> subprocess (and other functions calling sys.getfilesystemencoding()), but I 
> prefer to have a reliable sys.getfilesystemencoding() function.
> 
> This change doesn't concern Windows and Mac OS X because the encoding is 
> hardcoded (mbcs, utf-8). On Unix, I don't know in which case nl_langinfo() can 
> fail. Empty environment is not an error: nl_langinfo(CODESET) returns "ascii". 
> I think that few (or no) user would notice this change.

+1 on that change.

The UTF-8 fallback has these major problems:

 * it hides errors by always having the Unicode-bytes conversion
   succeed

 * it can cause applications to write files and create
   directories with wrongly encoded names (e.g. use UTF-8 on
   a Latin-1 file system)

Together with [issue8622] Add PYTHONFSENCODING environment variable:

    http://bugs.python.org/issue8622

which reduces the Python3 reliance on encoding guess work,
the change would make Python3 more user friendly and reduce the
number of bummers user run into when waking up in the all-new
Unicode world of Python3.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 07 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-04-23: Released mxODBC.Zope.DA 2.0.1        http://zope.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From van.lindberg at gmail.com  Fri May  7 16:37:33 2010
From: van.lindberg at gmail.com (VanL)
Date: Fri, 07 May 2010 09:37:33 -0500
Subject: [Python-Dev] Possible patch for functools partial - Interested?
Message-ID: <hs18jc$1mh$1@dough.gmane.org>

Howdy all -

I have an app where I am using functools.partial to bundle up jobs to
do, where a job is defined as a callable + args. In one case, I wanted
to keep track of whether I had previously seen a job, so I started
putting them into a set... only to find out that partials never test
equal to each other:

>>> import operator
>>> from functools import partial
>>> p1 = partial(operator.add)
>>> p2 = partial(operator.add)
>>> p1 == p2
False
>>> seen = set();seen.add(p1)
>>> p2 in seen
False

I created a subclass of functools.partial that provides appropriate
__eq__ and __hash__ methods, so that this works as expected. I called
the subclass a Job:
>>> j1 = Job(operator.add)
>>> j2 = Job(operator.add)
>>> j1 == j2
True
>>> seen = set();seen.add(j1)
>>> j2 in seen
True
>>> j1 is j2
False

While I was at it, I also added a nice repr. Would this group be
interested in a patch, or is this not interesting?

Thanks,

Van


From fuzzyman at voidspace.org.uk  Fri May  7 16:51:17 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 07 May 2010 16:51:17 +0200
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <hs18jc$1mh$1@dough.gmane.org>
References: <hs18jc$1mh$1@dough.gmane.org>
Message-ID: <4BE428E5.3050104@voidspace.org.uk>

On 07/05/2010 16:37, VanL wrote:
> Howdy all -
>
> I have an app where I am using functools.partial to bundle up jobs to
> do, where a job is defined as a callable + args. In one case, I wanted
> to keep track of whether I had previously seen a job, so I started
> putting them into a set... only to find out that partials never test
> equal to each other:
>
>    
>>>> import operator
>>>> from functools import partial
>>>> p1 = partial(operator.add)
>>>> p2 = partial(operator.add)
>>>> p1 == p2
>>>>          
> False
>    
>>>> seen = set();seen.add(p1)
>>>> p2 in seen
>>>>          
> False
>
> I created a subclass of functools.partial that provides appropriate
> __eq__ and __hash__ methods, so that this works as expected. I called
> the subclass a Job:
>    
>>>> j1 = Job(operator.add)
>>>> j2 = Job(operator.add)
>>>> j1 == j2
>>>>          
> True
>    
>>>> seen = set();seen.add(j1)
>>>> j2 in seen
>>>>          
> True
>    
>>>> j1 is j2
>>>>          
> False
>
> While I was at it, I also added a nice repr. Would this group be
> interested in a patch, or is this not interesting?
>
>    
Sounds good to me. Could you post the patch to http://bugs.python.org 
please.

Michael Foord


> Thanks,
>
> Van
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/


From solipsis at pitrou.net  Fri May  7 17:02:14 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 7 May 2010 15:02:14 +0000 (UTC)
Subject: [Python-Dev] Possible patch for functools partial - Interested?
References: <hs18jc$1mh$1@dough.gmane.org>
Message-ID: <loom.20100507T170052-252@post.gmane.org>

VanL <van.lindberg <at> gmail.com> writes:
> 
> I created a subclass of functools.partial that provides appropriate
> __eq__ and __hash__ methods, so that this works as expected. I called
> the subclass a Job:
[...]
> 
> While I was at it, I also added a nice repr. Would this group be
> interested in a patch, or is this not interesting?

It would be more useful to provide equality, hashing and repr to partial itself,
rather than a subclass. Feel free to propose a patch :)

Regards

Antoine.



From amk at amk.ca  Fri May  7 17:39:30 2010
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 7 May 2010 11:39:30 -0400
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <4BE3C178.2090800@v.loewis.de>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
	<4BE3C178.2090800@v.loewis.de>
Message-ID: <20100507153930.GA21021@amk-desktop.matrixgroup.net>

On Fri, May 07, 2010 at 09:30:00AM +0200, "Martin v. L?wis" wrote:
> I agree with Terry: how did you arrive at the 4 years for 2.x releases?
> Bug fixes releases stopped after the next feature release being made,
> which gave (counting between initial release and last bug fix release):

I used the initial release date of 2.4 and 2.5 and the dates of the
last patchlevel releases.  That may well be distorted by the recent
spasm of activity that led to 2.5.4.

--amk

From rob.cliffe at btinternet.com  Fri May  7 18:07:55 2010
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Fri, 7 May 2010 17:07:55 +0100
Subject: [Python-Dev] Possible patch for functools partial - Interested?
References: <hs18jc$1mh$1@dough.gmane.org>
Message-ID: <D30CE8E6B791483D8CF71567652FACFB@robslaptop>

Sorry to grouse, but isn't this maybe being a bit too clever?
Using your example,
    p1 = partial(operator.add)
is creating a callable, p1, i.e. a sort of function.  Yes I know technically 
it's not a function, but it behaves very much like one.

Now, if I write

    def f1(x,y): return x+y
    def f2(x,y): return x+y

I don't expect  f1==f2 to be True, even though f1 and f2 behave in exactly 
the same way,
and indeed it is not.
If I wanted it to be true, I should have written

    def f1(x): return x+y
    f2=f1

I find this behaviour natural and expected, both in my example and yours 
(although maybe I've just got used to Python's behaviour).

Similarly, if you wanted p1==p2, why not write

    p1 = partial(operator.add)
    p2 = p1

Maybe I could be persuaded otherwise by a convincing use case, but I rather 
doubt it.
Rob Cliffe

----- Original Message ----- 
From: "VanL" <van.lindberg at gmail.com>
To: <python-dev at python.org>
Sent: Friday, May 07, 2010 3:37 PM
Subject: [Python-Dev] Possible patch for functools partial - Interested?


> Howdy all -
>
> I have an app where I am using functools.partial to bundle up jobs to
> do, where a job is defined as a callable + args. In one case, I wanted
> to keep track of whether I had previously seen a job, so I started
> putting them into a set... only to find out that partials never test
> equal to each other:
>
>>>> import operator
>>>> from functools import partial
>>>> p1 = partial(operator.add)
>>>> p2 = partial(operator.add)
>>>> p1 == p2
> False
>>>> seen = set();seen.add(p1)
>>>> p2 in seen
> False
>
> I created a subclass of functools.partial that provides appropriate
> __eq__ and __hash__ methods, so that this works as expected. I called
> the subclass a Job:
>>>> j1 = Job(operator.add)
>>>> j2 = Job(operator.add)
>>>> j1 == j2
> True
>>>> seen = set();seen.add(j1)
>>>> j2 in seen
> True
>>>> j1 is j2
> False
>
> While I was at it, I also added a nice repr. Would this group be
> interested in a patch, or is this not interesting?
>
> Thanks,
>
> Van
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com 


From amk at amk.ca  Fri May  7 18:09:08 2010
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 7 May 2010 12:09:08 -0400
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <4BE3E2F1.4040907@gmail.com>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
	<hs05p7$7u2$1@dough.gmane.org> <4BE3E2F1.4040907@gmail.com>
Message-ID: <20100507160908.GB21021@amk-desktop.matrixgroup.net>

On Fri, May 07, 2010 at 07:52:49PM +1000, Nick Coghlan wrote:
> 3.x). I'll take a stab at a more accurate rationale:

Thanks!  I've applied the scalpel and reduced it to:

* A policy decision was made to silence warnings only of interest to
  developers by default.  :exc:`DeprecationWarning` and its
  descendants are now ignored unless otherwise requested, preventing
  users from seeing warnings triggered by an application.  (Carried
  out in :issue:`7319`.)

  In previous releases, :exc:`DeprecationWarning` messages were
  enabled by default, providing Python developers with a clear
  indication of where their code may break in a future major version
  of Python.

  However, there are increasingly many users of Python-based
  applications who are not directly involved in the development of
  those applications.  :exc:`DeprecationWarning` messages are
  irrelevant to such users, making them worry about an application
  that's actually working correctly and burdening the developers of
  these applications with responding to these concerns.

  You can re-enable display of :exc:`DeprecationWarning` messages by
  running Python with the :option:`-Wdefault` (short form:
  :option:`-Wd`) switch, or you can add
  ``warnings.simplefilter('default')`` to your code.

Benjamin suggested being very definite about a 5-year maintenance
period, but I don't want to write any checks our butt can't cash, so
I've left the text as "Maintenance releases for Python 2.7 will
probably be made for 5 years."  An alternative formulation might say
it will be maintained for the next two 3.x releases, not the next one
as usual.

I thought about Ben Finney's suggestion to not give a timespan and
describe the conditions for 2.x maintenance continuing, but those
conditions are complicated to describe -- if 3.x doesn't catch on?  if
the 3.x transition is slow?  if there's a significant 2.x user base
that remains?  if someone starts a 2.x maintenance team? -- and might
be a confusing tangle of what-if statements.

--amk

From status at bugs.python.org  Fri May  7 18:09:22 2010
From: status at bugs.python.org (Python tracker)
Date: Fri,  7 May 2010 18:09:22 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20100507160922.B719978141@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2010-04-30 - 2010-05-07)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2658 open (+44) / 17807 closed (+24) / 20465 total (+68)

Open issues with patches:  1081

Average duration of open issues: 724 days.
Median duration of open issues: 494 days.

Open Issues Breakdown
       open  2639 (+44)
languishing    12 ( +0)
    pending     6 ( +0)

Issues Created Or Reopened (71)
_______________________________

Bug in range() function for large values                       2010-05-01
CLOSED http://bugs.python.org/issue1533    reopened mark.dickinson                       
       patch, 26backport                                                       

configure: allow user-provided CFLAGS to override AC_PROG_CC d 2010-05-05
CLOSED http://bugs.python.org/issue8211    reopened mark.dickinson                       
       patch                                                                   

regrtest: use backslashreplace error handler for stdout        2010-05-05
       http://bugs.python.org/issue8533    reopened haypo                                
       patch                                                                   

Add missing tests for FlushKey, LoadKey, and SaveKey in winreg 2010-04-30
       http://bugs.python.org/issue8579    created  brian.curtin                         
                                                                               

Problem urllib2.URLError                                       2010-04-30
CLOSED http://bugs.python.org/issue8580    created  m_enanos                             
                                                                               

Logging handlers do not handle double-closing very well        2010-04-30
CLOSED http://bugs.python.org/issue8581    created  Jason.Baker                          
       patch                                                                   

urllib.urlretrieve fails with ValueError: Invalid format strin 2010-04-30
CLOSED http://bugs.python.org/issue8582    created  Jason.Gross                          
       patch                                                                   

Hardcoded namespace_separator in the cElementTree.XMLParser    2010-04-30
       http://bugs.python.org/issue8583    created  dmtr                                 
       patch                                                                   

test_multiprocessing skips some tests                          2010-04-30
       http://bugs.python.org/issue8584    created  pitrou                               
                                                                               

zipimporter.find_module is untested                            2010-05-01
       http://bugs.python.org/issue8585    created  alex                                 
                                                                               

test_imp.py test failures on Py3K Mac OS X                     2010-05-01
CLOSED http://bugs.python.org/issue8586    created  michael.foord                        
                                                                               

test_import.py test failures on Py3K Mac OS X                  2010-05-01
CLOSED http://bugs.python.org/issue8587    created  michael.foord                        
                                                                               

test_urllib2.py test failures on Py3K Mac OS X                 2010-05-01
       http://bugs.python.org/issue8588    created  michael.foord                        
                                                                               

test_warnings.CEnvironmentVariableTests.test_nonascii fails un 2010-05-01
       http://bugs.python.org/issue8589    created  michael.foord                        
                                                                               

test_httpservers.CGIHTTPServerTestCase failure on 3.1-maint Ma 2010-05-01
       http://bugs.python.org/issue8590    created  michael.foord                        
                                                                               

update mkpkg to latest coding standards                        2010-05-01
       http://bugs.python.org/issue8591    created  meatballhat                          
                                                                               

'y' does not check for embedded NUL bytes                      2010-05-01
       http://bugs.python.org/issue8592    created  pitrou                               
                                                                               

Improve c-api/arg.rst                                          2010-05-01
CLOSED http://bugs.python.org/issue8593    created  pitrou                               
       patch                                                                   

Add a "source_address" option to ftplib                        2010-05-01
       http://bugs.python.org/issue8594    created  giampaolo.rodola                     
       patch, needs review                                                     

Explain the default timeout in http-client-related libraries   2010-05-02
       http://bugs.python.org/issue8595    created  oddthinking                          
                                                                               

crypt blowfish 'ignores' salt                                  2010-05-02
CLOSED http://bugs.python.org/issue8596    created  pvo                                  
                                                                               

build out-of-line asm on Windows                               2010-05-02
       http://bugs.python.org/issue8597    created  zooko                                
       patch                                                                   

test/support: don't use localhost as IPv6 host name            2010-05-02
       http://bugs.python.org/issue8598    created  haypo                                
       patch                                                                   

_execvpe behaves inconsistently when PATH includes a filename  2010-05-02
CLOSED http://bugs.python.org/issue8599    created  Oren_Held                            
                                                                               

test_gdb failures                                              2010-05-03
CLOSED http://bugs.python.org/issue8600    created  pitrou                               
                                                                               

bz2.BZ2File should support "with" protocol per PEP 343         2010-05-03
CLOSED http://bugs.python.org/issue8601    created  Matt.Wartell                         
                                                                               

documentation of bz2 module mildly erroneous                   2010-05-03
       http://bugs.python.org/issue8602    created  Matt.Wartell                         
                                                                               

Create a bytes version of os.environ and getenvb()             2010-05-03
CLOSED http://bugs.python.org/issue8603    created  haypo                                
       patch                                                                   

Adding an atomic FS write API                                  2010-05-03
       http://bugs.python.org/issue8604    created  olemis                               
                                                                               

test_gdb can fail with compiler opts                           2010-05-03
       http://bugs.python.org/issue8605    created  pitrou                               
       patch                                                                   

OSF  is not supported anymore                                  2010-05-03
CLOSED http://bugs.python.org/issue8606    created  jcea                                 
                                                                               

OSX: duplicate -arch flags in CFLAGS breaks sysconfig          2010-05-04
CLOSED http://bugs.python.org/issue8607    created  robind                               
                                                                               

fix_import prefixes "." to already relative imports            2010-05-04
CLOSED http://bugs.python.org/issue8608    created  shlomme                              
       patch                                                                   

itertools: problem with nested groupby, list()                 2010-05-04
CLOSED http://bugs.python.org/issue8609    created  nicki                                
                                                                               

Python3/POSIX:  errors if file system encoding is None         2010-05-04
       http://bugs.python.org/issue8610    created  haypo                                
       patch                                                                   

Python3 doesn't support locale different than utf8 and an non- 2010-05-04
       http://bugs.python.org/issue8611    created  haypo                                
                                                                               

multiprocessing Queue module failes to send INIConfig objects  2010-05-04
       http://bugs.python.org/issue8612    created  Zaar.Hai                             
                                                                               

Decimal module flags undetermined when a signal is trapped.    2010-05-04
       http://bugs.python.org/issue8613    created  mark.dickinson                       
                                                                               

test_gzip fails on OS X                                        2010-05-04
CLOSED http://bugs.python.org/issue8614    created  mark.dickinson                       
                                                                               

turtle.py - backport of 3.1 features                           2010-05-04
CLOSED http://bugs.python.org/issue8615    created  gregorlingl                          
       patch                                                                   

Changes to content of Demo/turtle                              2010-05-04
       http://bugs.python.org/issue8616    created  gregorlingl                          
       patch                                                                   

Non-existent variables documented                              2010-05-04
       http://bugs.python.org/issue8617    created  dabrahams                            
                                                                               

test_winsound failing on Windows Server 2008                   2010-05-05
       http://bugs.python.org/issue8618    created  brian.curtin                         
                                                                               

Doc bug for urllib.request._urlopener in Python 3.1+           2010-05-05
CLOSED http://bugs.python.org/issue8619    created  sri                                  
                                                                               

wrong truncation of line in Cmd.cmd                            2010-05-05
       http://bugs.python.org/issue8620    created  omatt                                
                                                                               

uuid.uuid4() generates non-unique values on OSX                2010-05-05
CLOSED http://bugs.python.org/issue8621    created  yig                                  
       patch                                                                   

Add PYTHONFSENCODING environment variable                      2010-05-05
       http://bugs.python.org/issue8622    created  lemburg                              
                                                                               

Aliasing warnings in socketmodule.c                            2010-05-05
       http://bugs.python.org/issue8623    created  pitrou                               
                                                                               

Aliasing warnings in multiprocessing.c                         2010-05-05
       http://bugs.python.org/issue8624    created  pitrou                               
                                                                               

--with-pydebug builds now include -O2 by default               2010-05-05
CLOSED http://bugs.python.org/issue8625    created  mark.dickinson                       
       patch                                                                   

TypeError: rsplit() takes no keyword arguments                 2010-05-05
       http://bugs.python.org/issue8626    created  dabrahams                            
       easy                                                                    

Unchecked PyErr_WarnPy3k return value in Objects/typeobject.c  2010-05-05
       http://bugs.python.org/issue8627    created  mark.dickinson                       
                                                                               

Incorrect numbers.Complex.imag documentation                   2010-05-05
CLOSED http://bugs.python.org/issue8628    created  durban                               
                                                                               

Fix test_ssl failures under 2.6/3.1 with OpenSSL 1.0.0         2010-05-05
CLOSED http://bugs.python.org/issue8629    created  pitrou                               
       patch                                                                   

Keepends param in codec readline(s)                            2010-05-05
       http://bugs.python.org/issue8630    created  amccampos                            
                                                                               

subprocess.Popen.communicate(...) hangs on Windows             2010-05-05
       http://bugs.python.org/issue8631    created  Alex Quinn                           
                                                                               

subprocess doesn't handle Windows built-in commands as os.syst 2010-05-05
CLOSED http://bugs.python.org/issue8632    created  Alex Quinn                           
                                                                               

tarfile doesn't support undecodable filename in PAX format     2010-05-05
       http://bugs.python.org/issue8633    created  haypo                                
                                                                               

get method for dbm interface                                   2010-05-06
       http://bugs.python.org/issue8634    created  Kain94                               
                                                                               

enumerate() docstring doesn't cover optional start argument    2010-05-06
       http://bugs.python.org/issue8635    created  ncoghlan                             
       easy                                                                    

enumerate() test cases do not cover optional start argument    2010-05-06
       http://bugs.python.org/issue8636    created  scott.dial                           
       patch                                                                   

pydoc should respect MANPAGER over PAGER.                      2010-05-06
       http://bugs.python.org/issue8637    created  jsbronder                            
       patch                                                                   

Remove suggestion for name mangling from the tutorial          2010-05-06
       http://bugs.python.org/issue8638    created  LucianU                              
                                                                               

Allow callable objects in inspect.getargspec                   2010-05-06
       http://bugs.python.org/issue8639    created  gsakkis                              
                                                                               

subprocess: add envb argument to Popen constructor (Python3, P 2010-05-06
       http://bugs.python.org/issue8640    created  haypo                                
                                                                               

IDLE 3 doesn't highlights b"", but u""                         2010-05-07
       http://bugs.python.org/issue8641    created  puzzlet                              
                                                                               

json.loads description                                         2010-05-07
       http://bugs.python.org/issue8642    created  mft                                  
                                                                               

incorrect timedelta yielded by two on-the-fly nows subtraction 2010-05-07
CLOSED http://bugs.python.org/issue8643    created  mykhal                               
                                                                               

timedelta.total_seconds needlessly inaccurate, especially for  2010-05-07
       http://bugs.python.org/issue8644    created  mark.dickinson                       
                                                                               

PyUnicode_AsEncodedObject is undocumented                      2010-05-07
       http://bugs.python.org/issue8645    created  stutzbach                            
                                                                               

PyUnicode_EncodeDecimal is undocumented                        2010-05-07
       http://bugs.python.org/issue8646    created  stutzbach                            
                                                                               



Issues Now Closed (55)
______________________

Bug in range() function for large values                          6 days
       http://bugs.python.org/issue1533    mark.dickinson                       
       patch, 26backport                                                       

Adding start to enumerate()                                     724 days
       http://bugs.python.org/issue2831    scott.dial                           
       patch                                                                   

MacOS X framework install to non-standard directory fails       622 days
       http://bugs.python.org/issue3646    ronaldoussoren                       
       patch                                                                   

shutil.copyfile() leaks file descriptors when disk fills        546 days
       http://bugs.python.org/issue4265    haypo                                
       patch                                                                   

GC stats not accurate because of debug overhead                 501 days
       http://bugs.python.org/issue4687    pitrou                               
       patch                                                                   

Strange behavior when I logout() with IMAP4_SSL                 403 days
       http://bugs.python.org/issue5565    r.david.murray                       
                                                                               

email.encoders.encode_7or8bit(): typo "iso-2202".  "iso-2022"   146 days
       http://bugs.python.org/issue7472    r.david.murray                       
       patch, easy                                                             

random.choice should accept a set as input                      141 days
       http://bugs.python.org/issue7522    Thomas.Dybdahl.Ahle                  
                                                                               

copyright clarification for audiotest.au                        103 days
       http://bugs.python.org/issue7755    barry                                
       needs review                                                            

Minor bug in 2.6.4 related to cleanup at end of program          92 days
       http://bugs.python.org/issue7835    cburroughs                           
       patch, easy                                                             

platform module doesn't detect Windows 7                         89 days
       http://bugs.python.org/issue7863    lemburg                              
       patch, needs review                                                     

io close() swallowing exceptions                                 89 days
       http://bugs.python.org/issue7865    pakal                                
       patch                                                                   

urlparse.urlsplit mishandles novel schemes                       84 days
       http://bugs.python.org/issue7904    merwok                               
                                                                               

remove leftover macos9 support code                              83 days
       http://bugs.python.org/issue7908    ronaldoussoren                       
       patch, needs review                                                     

Allow Arbitrary OpenID providers in this bug tracker             70 days
       http://bugs.python.org/issue8008    jcea                                 
                                                                               

configure: allow user-provided CFLAGS to override AC_PROG_CC d    0 days
       http://bugs.python.org/issue8211    lemburg                              
       patch                                                                   

<unprintable AssertionError object> message in unittest traceb   30 days
       http://bugs.python.org/issue8313    haypo                                
       patch                                                                   

None shouldn't be passed to traceback.format_exception           24 days
       http://bugs.python.org/issue8388    michael.foord                        
                                                                               

tarfile: use surrogates for undecode fields                      23 days
       http://bugs.python.org/issue8390    haypo                                
       patch                                                                   

Set operations don't work for dictionary views                   22 days
       http://bugs.python.org/issue8404    rhettinger                           
       patch                                                                   

unittest Module Problem with different Kinds of Invocation       18 days
       http://bugs.python.org/issue8454    michael.foord                        
                                                                               

raise test_support.TestSkipped() is used outside main() / test   17 days
       http://bugs.python.org/issue8462    haypo                                
                                                                               

asyncore.dispatcher's __getattr__ method produces confusing ef   15 days
       http://bugs.python.org/issue8483    giampaolo.rodola                     
       patch                                                                   

regrtest: add a minimal "progress bar"                            8 days
       http://bugs.python.org/issue8560    haypo                                
       patch                                                                   

Always run regrtest.py with -bb                                   2 days
       http://bugs.python.org/issue8565    haypo                                
       patch                                                                   

2to3 should run under python 2.5                                  2 days
       http://bugs.python.org/issue8566    jyasskin                             
       patch, patch                                                            

decimal module doesn't respect precedence rules for exceptiona    5 days
       http://bugs.python.org/issue8567    mark.dickinson                       
       patch                                                                   

Buggy _strerror in asyncore                                       7 days
       http://bugs.python.org/issue8573    giampaolo.rodola                     
       easy                                                                    

Problem urllib2.URLError                                          0 days
       http://bugs.python.org/issue8580    r.david.murray                       
                                                                               

Logging handlers do not handle double-closing very well           3 days
       http://bugs.python.org/issue8581    vinay.sajip                          
       patch                                                                   

urllib.urlretrieve fails with ValueError: Invalid format strin    0 days
       http://bugs.python.org/issue8582    orsenthil                            
       patch                                                                   

test_imp.py test failures on Py3K Mac OS X                        3 days
       http://bugs.python.org/issue8586    michael.foord                        
                                                                               

test_import.py test failures on Py3K Mac OS X                     0 days
       http://bugs.python.org/issue8587    michael.foord                        
                                                                               

Improve c-api/arg.rst                                             2 days
       http://bugs.python.org/issue8593    pitrou                               
       patch                                                                   

crypt blowfish 'ignores' salt                                     1 days
       http://bugs.python.org/issue8596    mark.dickinson                       
                                                                               

_execvpe behaves inconsistently when PATH includes a filename     0 days
       http://bugs.python.org/issue8599    r.david.murray                       
                                                                               

test_gdb failures                                                 3 days
       http://bugs.python.org/issue8600    pitrou                               
                                                                               

bz2.BZ2File should support "with" protocol per PEP 343            0 days
       http://bugs.python.org/issue8601    Matt.Wartell                         
                                                                               

Create a bytes version of os.environ and getenvb()                4 days
       http://bugs.python.org/issue8603    haypo                                
       patch                                                                   

OSF  is not supported anymore                                     0 days
       http://bugs.python.org/issue8606    jcea                                 
                                                                               

OSX: duplicate -arch flags in CFLAGS breaks sysconfig             0 days
       http://bugs.python.org/issue8607    r.david.murray                       
                                                                               

fix_import prefixes "." to already relative imports               0 days
       http://bugs.python.org/issue8608    r.david.murray                       
       patch                                                                   

itertools: problem with nested groupby, list()                    0 days
       http://bugs.python.org/issue8609    nicki                                
                                                                               

test_gzip fails on OS X                                           0 days
       http://bugs.python.org/issue8614    mark.dickinson                       
                                                                               

turtle.py - backport of 3.1 features                              0 days
       http://bugs.python.org/issue8615    benjamin.peterson                    
       patch                                                                   

Doc bug for urllib.request._urlopener in Python 3.1+              1 days
       http://bugs.python.org/issue8619    r.david.murray                       
                                                                               

uuid.uuid4() generates non-unique values on OSX                   0 days
       http://bugs.python.org/issue8621    ronaldoussoren                       
       patch                                                                   

--with-pydebug builds now include -O2 by default                  0 days
       http://bugs.python.org/issue8625    haypo                                
       patch                                                                   

Incorrect numbers.Complex.imag documentation                      0 days
       http://bugs.python.org/issue8628    mark.dickinson                       
                                                                               

Fix test_ssl failures under 2.6/3.1 with OpenSSL 1.0.0            1 days
       http://bugs.python.org/issue8629    pitrou                               
       patch                                                                   

subprocess doesn't handle Windows built-in commands as os.syst    0 days
       http://bugs.python.org/issue8632    Alex Quinn                           
                                                                               

incorrect timedelta yielded by two on-the-fly nows subtraction    0 days
       http://bugs.python.org/issue8643    mark.dickinson                       
                                                                               

bdist_deb - Debian packager                                     440 days
       http://bugs.python.org/issue1054967 astraw                               
       patch                                                                   

URI parsing library                                               8 days
       http://bugs.python.org/issue1462525 orsenthil                            
       patch                                                                   

Use Py_ssize_t for rangeobject members                         1354 days
       http://bugs.python.org/issue1540617 mark.dickinson                       
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 42 Bug in range() function for large values                           6 days
closed      http://bugs.python.org/issue1533   

 31 Create a bytes version of os.environ and getenvb()                 4 days
closed      http://bugs.python.org/issue8603   

 21 test_support.find_unused_port can cause socket conflicts on Win    7 days
open        http://bugs.python.org/issue8576   

 16 test_imp.py test failures on Py3K Mac OS X                         3 days
closed      http://bugs.python.org/issue8586   

 15 Create fsencode() and fsdecode() functions in os.path             14 days
open        http://bugs.python.org/issue8514   

 13 Python3/POSIX:  errors if file system encoding is None             3 days
open        http://bugs.python.org/issue8610   

 12 Adding an atomic FS write API                                      4 days
open        http://bugs.python.org/issue8604   

 12 Add exception logging function to syslog module                   45 days
open        http://bugs.python.org/issue8214   

 11 <unprintable AssertionError object> message in unittest traceba   30 days
closed      http://bugs.python.org/issue8313   

  9 httplib getheader() throws error instead of default                8 days
open        http://bugs.python.org/issue8572   




From alexander.belopolsky at gmail.com  Fri May  7 18:46:23 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 7 May 2010 12:46:23 -0400
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <loom.20100507T170052-252@post.gmane.org>
References: <hs18jc$1mh$1@dough.gmane.org>
	<loom.20100507T170052-252@post.gmane.org>
Message-ID: <p2wd38f5331005070946pc080bc31u9d798e3aebc9a2f6@mail.gmail.com>

On Fri, May 7, 2010 at 11:02 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> VanL <van.lindberg <at> gmail.com> writes:
..
>> While I was at it, I also added a nice repr. Would this group be
>> interested in a patch, or is this not interesting?
>
> It would be more useful to provide equality, hashing and repr to partial itself,
> rather than a subclass. Feel free to propose a patch :)

While you are at it, you might want to take a look at
http://bugs.python.org/issue7830, "Flatten nested functools.partial."
For your purposes you will need to decide whether partial(partial(f,
1), 2) and partial(f, 1, 2) compare equal or not.  Most likely with
the current implementation the answer will be "no,"  which may not be
ideal.

From steve at pearwood.info  Fri May  7 19:41:58 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 8 May 2010 03:41:58 +1000
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <D30CE8E6B791483D8CF71567652FACFB@robslaptop>
References: <hs18jc$1mh$1@dough.gmane.org>
	<D30CE8E6B791483D8CF71567652FACFB@robslaptop>
Message-ID: <201005080342.00028.steve@pearwood.info>

On Sat, 8 May 2010 02:07:55 am Rob Cliffe wrote:
> Sorry to grouse, but isn't this maybe being a bit too clever?
> Using your example,
>     p1 = partial(operator.add)
> is creating a callable, p1, i.e. a sort of function.  Yes I know
> technically it's not a function, but it behaves very much like one.
>
> Now, if I write
>
>     def f1(x,y): return x+y
>     def f2(x,y): return x+y
>
> I don't expect  f1==f2 to be True, even though f1 and f2 behave in
> exactly the same way,
> and indeed it is not.

I do expect f1==f2, and I'm (mildly) disappointed that they're not.


[...]
> Similarly, if you wanted p1==p2, why not write
>
>     p1 = partial(operator.add)
>     p2 = p1

I thought the OP gave a use-case. He's generating "jobs" (partial 
applied to a callable and arguments), and wanted to avoid duplicated 
jobs.

I think it is reasonable to expect that partial(operator.add, 2) 
compares equal to partial(operator.add, 2). I don't think he's 
suggesting it should compare equal to partial(lambda x,y: x+y, 2).

+0.5 on comparing equal.
+1 on a nicer repr for partial objects.



-- 
Steven D'Aprano

From steve at holdenweb.com  Fri May  7 19:57:06 2010
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 07 May 2010 13:57:06 -0400
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <201005080342.00028.steve@pearwood.info>
References: <hs18jc$1mh$1@dough.gmane.org>	<D30CE8E6B791483D8CF71567652FACFB@robslaptop>
	<201005080342.00028.steve@pearwood.info>
Message-ID: <4BE45472.6040406@holdenweb.com>

Steven D'Aprano wrote:
> On Sat, 8 May 2010 02:07:55 am Rob Cliffe wrote:
>> Sorry to grouse, but isn't this maybe being a bit too clever?
>> Using your example,
>>     p1 = partial(operator.add)
>> is creating a callable, p1, i.e. a sort of function.  Yes I know
>> technically it's not a function, but it behaves very much like one.
>>
>> Now, if I write
>>
>>     def f1(x,y): return x+y
>>     def f2(x,y): return x+y
>>
>> I don't expect  f1==f2 to be True, even though f1 and f2 behave in
>> exactly the same way,
>> and indeed it is not.
> 
> I do expect f1==f2, and I'm (mildly) disappointed that they're not.
> 
How about

def f1(x, y): return x+y
def f2(x, y): return y+x

As you know, there are limits to everything. It seems to me that while
pure mathematics can (sometime) easily determine functional equivalence,
once you get to code it's a lot harder because there are semantic
constraints that don't apply in pure mathematics.

> 
> [...]
>> Similarly, if you wanted p1==p2, why not write
>>
>>     p1 = partial(operator.add)
>>     p2 = p1
> 
> I thought the OP gave a use-case. He's generating "jobs" (partial 
> applied to a callable and arguments), and wanted to avoid duplicated 
> jobs.
> 
> I think it is reasonable to expect that partial(operator.add, 2) 
> compares equal to partial(operator.add, 2). I don't think he's 
> suggesting it should compare equal to partial(lambda x,y: x+y, 2).
> 
Which absence, presumably, also mildly disappoints you?

regards
 Steve

> +0.5 on comparing equal.
> +1 on a nicer repr for partial objects.
> 
> 
> 


-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See PyCon Talks from Atlanta 2010  http://pycon.blip.tv/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/


From fuzzyman at voidspace.org.uk  Fri May  7 20:01:46 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 07 May 2010 20:01:46 +0200
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <4BE45472.6040406@holdenweb.com>
References: <hs18jc$1mh$1@dough.gmane.org>	<D30CE8E6B791483D8CF71567652FACFB@robslaptop>	<201005080342.00028.steve@pearwood.info>
	<4BE45472.6040406@holdenweb.com>
Message-ID: <4BE4558A.3080001@voidspace.org.uk>

On 07/05/2010 19:57, Steve Holden wrote:
> Steven D'Aprano wrote:
>    
>> On Sat, 8 May 2010 02:07:55 am Rob Cliffe wrote:
>>      
>>> Sorry to grouse, but isn't this maybe being a bit too clever?
>>> Using your example,
>>>      p1 = partial(operator.add)
>>> is creating a callable, p1, i.e. a sort of function.  Yes I know
>>> technically it's not a function, but it behaves very much like one.
>>>
>>> Now, if I write
>>>
>>>      def f1(x,y): return x+y
>>>      def f2(x,y): return x+y
>>>
>>> I don't expect  f1==f2 to be True, even though f1 and f2 behave in
>>> exactly the same way,
>>> and indeed it is not.
>>>        
>> I do expect f1==f2, and I'm (mildly) disappointed that they're not.
>>
>>      
> How about
>
> def f1(x, y): return x+y
> def f2(x, y): return y+x
>
> As you know, there are limits to everything. It seems to me that while
> pure mathematics can (sometime) easily determine functional equivalence,
> once you get to code it's a lot harder because there are semantic
> constraints that don't apply in pure mathematics.
>    
Sure, but in CPython at least you *could* detect *identical* functions 
(but not functions that do the same thing in a different way). All the 
information is exposed - it would mean comparing bytecode though (plus 
signature etc) and is not likely to be portable to other implementations.

Partials that wrap the *same* function (based on identity) comparing 
equal seems useful enough to me.

Michael

-- 
http://www.ironpythoninaction.com/


From brett at python.org  Fri May  7 20:38:07 2010
From: brett at python.org (Brett Cannon)
Date: Fri, 7 May 2010 11:38:07 -0700
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <20100507160908.GB21021@amk-desktop.matrixgroup.net>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local> 
	<hs05p7$7u2$1@dough.gmane.org> <4BE3E2F1.4040907@gmail.com> 
	<20100507160908.GB21021@amk-desktop.matrixgroup.net>
Message-ID: <w2vbbaeab101005071138s3ca0d2a6x5a91785365d3b022@mail.gmail.com>

On Fri, May 7, 2010 at 09:09, A.M. Kuchling <amk at amk.ca> wrote:

> On Fri, May 07, 2010 at 07:52:49PM +1000, Nick Coghlan wrote:
> > 3.x). I'll take a stab at a more accurate rationale:
>
> Thanks!  I've applied the scalpel and reduced it to:
>
> * A policy decision was made to silence warnings only of interest to
>  developers by default.  :exc:`DeprecationWarning` and its
>  descendants are now ignored unless otherwise requested, preventing
>   users from seeing warnings triggered by an application.  (Carried
>  out in :issue:`7319`.)
>
>   In previous releases, :exc:`DeprecationWarning` messages were
>  enabled by default, providing Python developers with a clear
>  indication of where their code may break in a future major version
>  of Python.
>
>  However, there are increasingly many users of Python-based
>  applications who are not directly involved in the development of
>  those applications.  :exc:`DeprecationWarning` messages are
>  irrelevant to such users, making them worry about an application
>  that's actually working correctly and burdening the developers of
>  these applications with responding to these concerns.
>
>  You can re-enable display of :exc:`DeprecationWarning` messages by
>   running Python with the :option:`-Wdefault` (short form:
>  :option:`-Wd`) switch, or you can add
>  ``warnings.simplefilter('default')`` to your code.
>
>
That sounds good to me.


> Benjamin suggested being very definite about a 5-year maintenance
> period, but I don't want to write any checks our butt can't cash, so
> I've left the text as "Maintenance releases for Python 2.7 will
> probably be made for 5 years."  An alternative formulation might say
> it will be maintained for the next two 3.x releases, not the next one
> as usual.
>
> I thought about Ben Finney's suggestion to not give a timespan and
> describe the conditions for 2.x maintenance continuing, but those
> conditions are complicated to describe -- if 3.x doesn't catch on?  if
> the 3.x transition is slow?  if there's a significant 2.x user base
> that remains?  if someone starts a 2.x maintenance team? -- and might
> be a confusing tangle of what-if statements.


Why can't we simply say that "we plan to support Python 2.7 beyond the
typical two years for bugfix releases"? It doesn't tie us to anything but
still lets people know our intentions. We don't have to worry about every
possible scenario now (e.g. 3.x gets no more traction or some other rare
event) and saying we plan on long term support but don't know for how long
is completely truthful; we have no timeline on how long we are willing to
keep 2.7 afloat beyond the fact that we plan to do it longer than normal.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100507/a267a833/attachment-0001.html>

From steve at pearwood.info  Fri May  7 21:13:50 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 8 May 2010 05:13:50 +1000
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <4BE45472.6040406@holdenweb.com>
References: <hs18jc$1mh$1@dough.gmane.org>
	<201005080342.00028.steve@pearwood.info>
	<4BE45472.6040406@holdenweb.com>
Message-ID: <201005080513.50741.steve@pearwood.info>

On Sat, 8 May 2010 03:57:06 am Steve Holden wrote:
> Steven D'Aprano wrote:
> > On Sat, 8 May 2010 02:07:55 am Rob Cliffe wrote:
> >> Sorry to grouse, but isn't this maybe being a bit too clever?
> >> Using your example,
> >>     p1 = partial(operator.add)
> >> is creating a callable, p1, i.e. a sort of function.  Yes I know
> >> technically it's not a function, but it behaves very much like
> >> one.
> >>
> >> Now, if I write
> >>
> >>     def f1(x,y): return x+y
> >>     def f2(x,y): return x+y
> >>
> >> I don't expect  f1==f2 to be True, even though f1 and f2 behave in
> >> exactly the same way,
> >> and indeed it is not.
> >
> > I do expect f1==f2, and I'm (mildly) disappointed that they're not.
>
> How about
>
> def f1(x, y): return x+y
> def f2(x, y): return y+x
>
> As you know, there are limits to everything. It seems to me that
> while pure mathematics can (sometime) easily determine functional
> equivalence, once you get to code it's a lot harder because there are
> semantic constraints that don't apply in pure mathematics.

This being Python, we can't assume x+y is always equal to y+x, so in 
your example I wouldn't expect them to be equal. And thus I avoid your 
trap :)

What you say is correct in general. I understand the arguments against 
making function equality more sophisticated than just identity testing: 
there aren't many use-cases for it, and it is potentially a lot of work 
depending on how clever you try to be. But if it came for free, it 
would be a sweet trick to impress your friends and scare your enemies.


[...]
> > I think it is reasonable to expect that partial(operator.add, 2)
> > compares equal to partial(operator.add, 2). I don't think he's
> > suggesting it should compare equal to partial(lambda x,y: x+y, 2).
>
> Which absence, presumably, also mildly disappoints you?

Only in the sense that in a perfect world where language features had 
benefits but no costs, I would expect nothing less.


-- 
Steven D'Aprano

From pjenvey at underboss.org  Fri May  7 21:26:10 2010
From: pjenvey at underboss.org (Philip Jenvey)
Date: Fri, 7 May 2010 12:26:10 -0700
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <20100507160908.GB21021@amk-desktop.matrixgroup.net>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
	<hs05p7$7u2$1@dough.gmane.org> <4BE3E2F1.4040907@gmail.com>
	<20100507160908.GB21021@amk-desktop.matrixgroup.net>
Message-ID: <F0EB4C7D-1CB6-4AF3-8A4B-7B448E2149D5@underboss.org>


On May 7, 2010, at 9:09 AM, A.M. Kuchling wrote:

>  You can re-enable display of :exc:`DeprecationWarning` messages by
>  running Python with the :option:`-Wdefault` (short form:
>  :option:`-Wd`) switch, or you can add
>  ``warnings.simplefilter('default')`` to your code.

The new PYTHONWARNINGS env var also needs mentioning here, how's about:

You can re-enable display of :exc:`DeprecationWarning` messages by
 either running Python with the :option:`-Wdefault` (short form:
 :option:`-Wd`) switch or by setting the :envvar:`PYTHONWARNINGS`
 environment variable to "default" (or "d") before running Python.
 Python code can also re-enable them by calling
 ``warnings.simplefilter('default')``.


--
Philip Jenvey

From martin at v.loewis.de  Fri May  7 21:45:23 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 07 May 2010 21:45:23 +0200
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <20100507153930.GA21021@amk-desktop.matrixgroup.net>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>	<4BE3C178.2090800@v.loewis.de>
	<20100507153930.GA21021@amk-desktop.matrixgroup.net>
Message-ID: <4BE46DD3.6060806@v.loewis.de>

A.M. Kuchling wrote:
> On Fri, May 07, 2010 at 09:30:00AM +0200, "Martin v. L?wis" wrote:
>> I agree with Terry: how did you arrive at the 4 years for 2.x releases?
>> Bug fixes releases stopped after the next feature release being made,
>> which gave (counting between initial release and last bug fix release):
> 
> I used the initial release date of 2.4 and 2.5 and the dates of the
> last patchlevel releases.  That may well be distorted by the recent
> spasm of activity that led to 2.5.4.

Ah. But the most recent 2.4 and 2.5 patchlevel releases are *not* bugfix
releases. They only provide security fixes, and I plan to provide them
for a period of five years (after the initial release).

Regards,
Martin

From van.lindberg at gmail.com  Fri May  7 21:53:01 2010
From: van.lindberg at gmail.com (VanL)
Date: Fri, 07 May 2010 14:53:01 -0500
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <201005080342.00028.steve@pearwood.info>
References: <hs18jc$1mh$1@dough.gmane.org>	<D30CE8E6B791483D8CF71567652FACFB@robslaptop>
	<201005080342.00028.steve@pearwood.info>
Message-ID: <hs1r2s$aje$1@dough.gmane.org>

On 5/7/2010 12:41 PM, Steven D'Aprano wrote:

>> Now, if I write
>>
>>     def f1(x,y): return x+y
>>     def f2(x,y): return x+y
>>
>> I don't expect  f1==f2 to be True, even though f1 and f2 behave in
>> exactly the same way, and indeed it is not.

This is not what I am getting after; these (IMO) should compare unequal.
I took a fairly conservative line, testing for identity of functions and
equality of arguments when converted to a normal form:

    def __eq__(self, other):
        try:
            return ((self.func == other.func) and
                        (self.args == other.args) and
                        (self.keywords == other.keywords))
        except AttributeError:
            return False

    def __hash__(self):
        if self.keywords:
            sorted_keys = sorted(self.keywords.keys())
            values = [self.keywords[k] for k in sorted_keys]
            kwhash = tuple(sorted_keys + values)
        else:
            kwhash = None
        return hash(tuple((self.func, self.args, kwhash)))

It may be slightly controversial that I read out all the keys and values
and convert them to a normal form for the hash. This is both to get
around hashing issues (keywords is a dict, even though it is read-only)
and to allow for a dict-like object to be provided as an argument (which
happens in my application sometimes).

The repr is slightly long, too, but it was designed to support (weakly)
the idea that obj == exec(repr(obj)). Its not necessary, but I always
like it when that is true.

    def __repr__(self):
        if self.args: argstr = '*%s' % repr(tuple(self.args))
        else: argstr = ''
        if self.keywords: kwstr = '**%s' % self.keywords
        else: kwstr = ''
        if argstr and kwstr: argstr += ', '
        classname = self.__class__.__name__
        funcname = self.func.__name__
        if argstr or kwstr: funcname += ', '
        return '%s(%s%s%s)' % (classname, funcname, argstr, kwstr)

I make no effort to look for functional equality between two different
functions. too many side effects could go off trying to make that work,
and it would still be only inconsistently right. (IIRC, doing it
completely is NP-hard.)

Thanks,

Van


From ijmorlan at uwaterloo.ca  Fri May  7 22:15:35 2010
From: ijmorlan at uwaterloo.ca (Isaac Morland)
Date: Fri, 7 May 2010 16:15:35 -0400 (EDT)
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <hs1r2s$aje$1@dough.gmane.org>
References: <hs18jc$1mh$1@dough.gmane.org>
	<D30CE8E6B791483D8CF71567652FACFB@robslaptop>
	<201005080342.00028.steve@pearwood.info> <hs1r2s$aje$1@dough.gmane.org>
Message-ID: <Pine.GSO.4.64.1005071613430.10010@core.cs.uwaterloo.ca>

On Fri, 7 May 2010, VanL wrote:

> I make no effort to look for functional equality between two different
> functions. too many side effects could go off trying to make that work,
> and it would still be only inconsistently right. (IIRC, doing it
> completely is NP-hard.)

Actually, doing it completely is flat-out impossible.

Godel, Halting Problem, and all that.

So you don't need to apologize for not doing it ;-)

Isaac Morland			CSCF Web Guru
DC 2554C, x36650		WWW Software Specialist

From tjreedy at udel.edu  Fri May  7 22:36:04 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 07 May 2010 16:36:04 -0400
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <hs1r2s$aje$1@dough.gmane.org>
References: <hs18jc$1mh$1@dough.gmane.org>	<D30CE8E6B791483D8CF71567652FACFB@robslaptop>	<201005080342.00028.steve@pearwood.info>
	<hs1r2s$aje$1@dough.gmane.org>
Message-ID: <hs1tjl$k0l$1@dough.gmane.org>

There is obviously interest in your proposal. I suggest you open an 
issue on the tracker. Make it a library feature request for 3.2. Upload 
the patch as a separate file. I would include the subclass form in the 
post to show the intneded effect, for discussion, and also so people can 
test and use with current installations without patching. When you have 
done so, post here in a followup with the link.

Terry Jan Reedy



From vincent at vincentdavis.net  Sat May  8 00:02:00 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Fri, 7 May 2010 16:02:00 -0600
Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing
	needed
In-Reply-To: <hrun8t$840$1@dough.gmane.org>
References: <l2g6523e39a1005060747oc5fc95f5wd2b8ffb880ccff2c@mail.gmail.com>
	<hrun8t$840$1@dough.gmane.org>
Message-ID: <w2z77e831101005071502h4e9519cfn60d73e8fbc003b7c@mail.gmail.com>

On Thu, May 6, 2010 at 9:29 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

>
> Scanning through open issues will also give you a general idea of what
> kind of functionalities are looking for improvement, or need fixing.
>
> (you can create a new issue and start tackling it yourself, too)
>
> As a wanabe Dev I think the hardest thing is to find an open issue I can
actually fix and to have a mentor to help make sure I don't miss something I
did not know about.

Please record the "Teach Me" session if it happens. (audio and/or video)


>
>
  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100507/087a839d/attachment-0001.html>

From ncoghlan at gmail.com  Sat May  8 01:50:42 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 08 May 2010 09:50:42 +1000
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <20100507160908.GB21021@amk-desktop.matrixgroup.net>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>	<hs05p7$7u2$1@dough.gmane.org>
	<4BE3E2F1.4040907@gmail.com>
	<20100507160908.GB21021@amk-desktop.matrixgroup.net>
Message-ID: <4BE4A752.2010302@gmail.com>

A.M. Kuchling wrote:
> On Fri, May 07, 2010 at 07:52:49PM +1000, Nick Coghlan wrote:
>> 3.x). I'll take a stab at a more accurate rationale:
> 
> Thanks!  I've applied the scalpel and reduced it to:

I saw this go by on the checkin list - looks good (although Philip is
right about mentioning PYTHONWARNINGS).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sat May  8 01:54:52 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 08 May 2010 09:54:52 +1000
Subject: [Python-Dev] audience-instructors for Teach Me Python Bugfixing
 needed
In-Reply-To: <w2z77e831101005071502h4e9519cfn60d73e8fbc003b7c@mail.gmail.com>
References: <l2g6523e39a1005060747oc5fc95f5wd2b8ffb880ccff2c@mail.gmail.com>	<hrun8t$840$1@dough.gmane.org>
	<w2z77e831101005071502h4e9519cfn60d73e8fbc003b7c@mail.gmail.com>
Message-ID: <4BE4A84C.9080805@gmail.com>

Vincent Davis wrote:
> As a wanabe Dev I think the hardest thing is to find an open issue I can
> actually fix and to have a mentor to help make sure I don't miss
> something I did not know about. 

Does the "easy" tag help with that at all? It's intended to mark issues
that aren't delving into deep dark corners of the interpreter or
standard library, but when you've been doing this for a long time one's
judgment of what's easy and what's difficult can get a little askew.

(e.g. I recently marked the task of fixing the enumerate docstring as
easy, but that does still require knowing how docstrings work for C code)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From dave at boostpro.com  Sat May  8 04:49:06 2010
From: dave at boostpro.com (David Abrahams)
Date: Fri, 07 May 2010 22:49:06 -0400
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
Message-ID: <m2bpcrumzh.wl%dave@boostpro.com>


This is a bug report.  bugs.python.org seems to be down.

  >>> from urlparse import *
  >>> urlunsplit(urlsplit('git+file:///foo/bar/baz'))
  git+file:/foo/bar/baz

Note the dropped slashes after the colon.

-- 
Dave Abrahams           Meet me at BoostCon: http://www.boostcon.com
BoostPro Computing
http://www.boostpro.com


From orsenthil at gmail.com  Sat May  8 06:04:04 2010
From: orsenthil at gmail.com (Senthil Kumaran)
Date: Sat, 8 May 2010 09:34:04 +0530
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <m2bpcrumzh.wl%dave@boostpro.com>
References: <m2bpcrumzh.wl%dave@boostpro.com>
Message-ID: <g2y7c42eba11005072104uaca9ac86l902d11a82a7ec77@mail.gmail.com>

On Sat, May 8, 2010 at 8:19 AM, David Abrahams <dave at boostpro.com> wrote:
>
> This is a bug report. ?bugs.python.org seems to be down.

Tracked here: http://bugs.python.org/issue8656

> ?>>> urlunsplit(urlsplit('git+file:///foo/bar/baz'))

Is 'git+file' a valid protocol? Or was it just your example?
I don't see any reason for it to be invalid but I don't find
authoritative references either.

-- 
Senthil

From ddborowitz at gmail.com  Sat May  8 06:12:28 2010
From: ddborowitz at gmail.com (David Borowitz)
Date: Fri, 7 May 2010 21:12:28 -0700
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <g2y7c42eba11005072104uaca9ac86l902d11a82a7ec77@mail.gmail.com>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<g2y7c42eba11005072104uaca9ac86l902d11a82a7ec77@mail.gmail.com>
Message-ID: <p2o70e6cf131005072112ra8693d1dne5ae75aac1ee44fc@mail.gmail.com>

On Fri, May 7, 2010 at 21:04, Senthil Kumaran <orsenthil at gmail.com> wrote:

> On Sat, May 8, 2010 at 8:19 AM, David Abrahams <dave at boostpro.com> wrote:
> >
> > This is a bug report.  bugs.python.org seems to be down.
>
> Tracked here: http://bugs.python.org/issue8656
>
> >  >>> urlunsplit(urlsplit('git+file:///foo/bar/baz'))
>
> Is 'git+file' a valid protocol? Or was it just your example?
> I don't see any reason for it to be invalid but I don't find
> authoritative references either.
>

RFC 3986 is pretty clear on allowing '+' in scheme names, in principle if
not necessarily in practice:
http://tools.ietf.org/html/rfc3986#section-3.1


> --
> Senthil
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ddborowitz%40gmail.com
>



-- 
It is better to be quotable than to be honest.
   -Tom Stoppard

Borowitz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100507/b307e57e/attachment.html>

From stephen at xemacs.org  Sat May  8 16:47:09 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 08 May 2010 23:47:09 +0900
Subject: [Python-Dev]  urlparse.urlunsplit should be smarter about +
In-Reply-To: <m2bpcrumzh.wl%dave@boostpro.com>
References: <m2bpcrumzh.wl%dave@boostpro.com>
Message-ID: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>

David Abrahams writes:
 > 
 > This is a bug report.  bugs.python.org seems to be down.
 > 
 >   >>> from urlparse import *
 >   >>> urlunsplit(urlsplit('git+file:///foo/bar/baz'))
 >   git+file:/foo/bar/baz
 > 
 > Note the dropped slashes after the colon.

That's clearly wrong, but what does "+" have to to do with it?  AFAIK,
the only thing special about + in scheme names is that it's not
allowed as the first character.

From john.arbash.meinel at gmail.com  Sat May  8 18:04:47 2010
From: john.arbash.meinel at gmail.com (John Arbash Meinel)
Date: Sat, 08 May 2010 11:04:47 -0500
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4BE58B9F.6080604@gmail.com>

Stephen J. Turnbull wrote:
> David Abrahams writes:
>  > 
>  > This is a bug report.  bugs.python.org seems to be down.
>  > 
>  >   >>> from urlparse import *
>  >   >>> urlunsplit(urlsplit('git+file:///foo/bar/baz'))
>  >   git+file:/foo/bar/baz
>  > 
>  > Note the dropped slashes after the colon.
> 
> That's clearly wrong, but what does "+" have to to do with it?  AFAIK,
> the only thing special about + in scheme names is that it's not
> allowed as the first character.

Don't you need to register the "git+file:///" url for urlparse to
properly split it?

    if protocol not in urlparse.uses_netloc:
        urlparse.uses_netloc.append(protocol)

John
=:->


From exarkun at twistedmatrix.com  Sat May  8 19:16:57 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Sat, 08 May 2010 17:16:57 -0000
Subject: [Python-Dev] Parallel test execution on buildbot
Message-ID: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain>

Hi all,

Has anyone considered using regrtest's -j option in the buildbot 
configuration to speed up the test runs?  Antoine Pitrou pointed out 
that even for single CPU slaves, this could be a win due to the number 
of tests that spend time sleeping or waiting on I/O.  And on slaves with 
multiple CPUs it would clearly be even better.  eg, I don't know what 
hardware is actually in the Solaris slave (bot loewis-sun), but if it 
has the full 4 UltraSPARCs that it could, then running with -j4 or -j5 
there might bring its runtime down from nearly 100 minutes to 20 or 25 - 
competitive with some of the more reasonable slaves.

Jean-Paul

From solipsis at pitrou.net  Sat May  8 19:48:36 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 8 May 2010 19:48:36 +0200
Subject: [Python-Dev] Parallel test execution on buildbot
References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain>
Message-ID: <20100508194836.40160211@pitrou.net>


Hi,

> Has anyone considered using regrtest's -j option in the buildbot 
> configuration to speed up the test runs?

Perhaps some buildbots are doing other useful tasks, in addition to
simply building Python. This should probably be a case by case setting.
I don't know how easy it is to add specific options to a buildslave.

One small issue would be that, currently, when a buildbot hangs, you
know which test it is hung in since it's the last one displayed.
This isn't true with -j (but perhaps we can improve the runner to fix
this).

Regards

Antoine.



From benjamin at python.org  Sat May  8 20:56:01 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 8 May 2010 13:56:01 -0500
Subject: [Python-Dev] [RELEASED] Python 2.7 beta 2
Message-ID: <x2n1afaf6161005081156qf4a673c1j88a6baac5806c401@mail.gmail.com>

On behalf of the Python development team, I'm elated to announce the second beta
release of Python 2.7.

Python 2.7 is scheduled (by Guido and Python-dev) to be the last major version
in the 2.x series.  2.7 will have an extended period of bugfix maintenance.

2.7 includes many features that were first released in Python 3.1.  The faster
io module, the new nested with statement syntax, improved float repr, set
literals, dictionary views, and the memoryview object have been backported from
3.1. Other features include an ordered dictionary implementation, unittests
improvements, a new sysconfig module, and support for ttk Tile in Tkinter.  For
a more extensive list of changes in 2.7, see
http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python
distribution.

To download Python 2.7 visit:

     http://www.python.org/download/releases/2.7/

While this is a development release and is thus not suitable for production use,
we encourage Python application and library developers to test the release with
their code and report any bugs they encounter to:

     http://bugs.python.org/

2.7 documentation can be found at:

     http://docs.python.org/2.7/


Enjoy!

--
Benjamin Peterson
2.7 Release Manager
benjamin at python.org
(on behalf of the entire python-dev team and 2.7's contributors)

From martin at v.loewis.de  Sat May  8 22:08:20 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 08 May 2010 22:08:20 +0200
Subject: [Python-Dev] Parallel test execution on buildbot
In-Reply-To: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain>
References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain>
Message-ID: <4BE5C4B4.7090703@v.loewis.de>

> Has anyone considered using regrtest's -j option in the buildbot
> configuration to speed up the test runs?

Yes, I did. I turned it off again when the tests started failing because
of it.

Regards,
Martin

From tjreedy at udel.edu  Sun May  9 00:38:21 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 08 May 2010 18:38:21 -0400
Subject: [Python-Dev] bugs.python.org intermittently not working.
Message-ID: <hs4p4u$jrf$1@dough.gmane.org>

Saturday eve (us, eastern).
Last night I had intermittent problems with bugs.python.org: issues not 
being fetched, submissions not being recorded. David Abrahams reported 
'bugs.python.org seems to be down' is his urlparse thread.

Right not, I have a fetch and submission 'loading' for over a minute, 
after fetching an issue a minute before. The new beta2 page came up with 
no problems.

The submission just stopped with
Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /issue8641.

Reason: Error reading from remote server

Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 
OpenSSL/0.9.8g Server at bugs.python.org Port 80




From ncoghlan at gmail.com  Sun May  9 03:32:42 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 09 May 2010 11:32:42 +1000
Subject: [Python-Dev] Parallel test execution on buildbot
In-Reply-To: <4BE5C4B4.7090703@v.loewis.de>
References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain>
	<4BE5C4B4.7090703@v.loewis.de>
Message-ID: <4BE610BA.7060305@gmail.com>

Martin v. L?wis wrote:
>> Has anyone considered using regrtest's -j option in the buildbot
>> configuration to speed up the test runs?
> 
> Yes, I did. I turned it off again when the tests started failing because
> of it.

Yeah, a lot of our tests weren't written with parallel execution in mind
(e.g. the existence of test_support.TESTFN, using specific ports for
test servers).

While they've been getting better (e.g. increased use of randomly
generated temporary directories over specific filenames, letting the OS
assign a free port to servers), I believe there is still a fair bit of
work to be done to make them all "parallel execution friendly" (of
course, some tests, such as those that try to trigger MemoryError, are
inherently parallel execution unfriendly).

So yes, we've thought about it, but there's still work to be done before
that option can be used without having to worry about false alarms.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From fuzzyman at voidspace.org.uk  Sun May  9 11:08:29 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sun, 09 May 2010 11:08:29 +0200
Subject: [Python-Dev] Parallel test execution on buildbot
In-Reply-To: <4BE610BA.7060305@gmail.com>
References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain>	<4BE5C4B4.7090703@v.loewis.de>
	<4BE610BA.7060305@gmail.com>
Message-ID: <4BE67B8D.9030504@voidspace.org.uk>

On 09/05/2010 03:32, Nick Coghlan wrote:
> Martin v. L?wis wrote:
>    
>>> Has anyone considered using regrtest's -j option in the buildbot
>>> configuration to speed up the test runs?
>>>        
>> Yes, I did. I turned it off again when the tests started failing because
>> of it.
>>      
> Yeah, a lot of our tests weren't written with parallel execution in mind
> (e.g. the existence of test_support.TESTFN, using specific ports for
> test servers).
>
> While they've been getting better (e.g. increased use of randomly
> generated temporary directories over specific filenames, letting the OS
> assign a free port to servers), I believe there is still a fair bit of
> work to be done to make them all "parallel execution friendly" (of
> course, some tests, such as those that try to trigger MemoryError, are
> inherently parallel execution unfriendly).
>
> So yes, we've thought about it, but there's still work to be done before
> that option can be used without having to worry about false alarms.
>    

FWIW I *usually* run the test suite with parallelization (it is just so 
much quicker) and these days *rarely* see spurious failures as a result. 
This is on Mac OS X, YMMV.

Michael Foord

> Cheers,
> Nick.
>
>    


-- 
http://www.ironpythoninaction.com/


From victor.stinner at haypocalc.com  Sun May  9 13:49:37 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sun, 9 May 2010 13:49:37 +0200
Subject: [Python-Dev] Parallel test execution on buildbot
In-Reply-To: <4BE67B8D.9030504@voidspace.org.uk>
References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain>
	<4BE610BA.7060305@gmail.com> <4BE67B8D.9030504@voidspace.org.uk>
Message-ID: <201005091349.37463.victor.stinner@haypocalc.com>

Le dimanche 09 mai 2010 11:08:29, Michael Foord a ?crit :
> FWIW I *usually* run the test suite with parallelization (it is just so
> much quicker) and these days *rarely* see spurious failures as a result.
> This is on Mac OS X, YMMV.

I use regrtest.py -j 4 on a Intel Quad Core on Linux: 4 tests are really 
running at the same time, and I only get only one spurious failure: 
test_ioctl. It should be easy to fix this test.

Or we can maybe write a blacklist of tests not compatible with multiprocessing 
mode (or the opposite, a whitelist of compatible tests).

-- 
Victor Stinner
http://www.haypocalc.com/

From martin at v.loewis.de  Sun May  9 13:50:47 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 09 May 2010 13:50:47 +0200
Subject: [Python-Dev] Parallel test execution on buildbot
In-Reply-To: <4BE67B8D.9030504@voidspace.org.uk>
References: <20100508171657.1660.1545567012.divmod.xquotient.35@localhost.localdomain>	<4BE5C4B4.7090703@v.loewis.de>
	<4BE610BA.7060305@gmail.com> <4BE67B8D.9030504@voidspace.org.uk>
Message-ID: <4BE6A197.9090808@v.loewis.de>

> FWIW I *usually* run the test suite with parallelization (it is just so
> much quicker) and these days *rarely* see spurious failures as a result.
> This is on Mac OS X, YMMV.

I may misremember the details, but IIRC, the multiprocessing tests would
fail to terminate on Solaris. This made it unsuitable for buildbot usage.

Regards,
Martin

From stephen at xemacs.org  Sun May  9 14:15:38 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 09 May 2010 21:15:38 +0900
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <4BE58B9F.6080604@gmail.com>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com>
Message-ID: <87fx21s239.fsf@uwakimon.sk.tsukuba.ac.jp>

John Arbash Meinel writes:
 > Stephen J. Turnbull wrote:
 > > David Abrahams writes:
 > >  > 
 > >  > This is a bug report.  bugs.python.org seems to be down.
 > >  > 
 > >  >   >>> from urlparse import *
 > >  >   >>> urlunsplit(urlsplit('git+file:///foo/bar/baz'))
 > >  >   git+file:/foo/bar/baz
 > >  > 
 > >  > Note the dropped slashes after the colon.
 > > 
 > > That's clearly wrong, but what does "+" have to to do with it?  AFAIK,
 > > the only thing special about + in scheme names is that it's not
 > > allowed as the first character.
 > 
 > Don't you need to register the "git+file:///" url for urlparse to
 > properly split it?
 > 
 >     if protocol not in urlparse.uses_netloc:
 >         urlparse.uses_netloc.append(protocol)

I don't know about the urlparse implementation, but from the point of
view of the RFC I think not.  Either BCP 35 or RFC 3986 (or maybe both)
makes it plain that if the scheme name is followed by "://", the
scheme is a hierarchical one.  So that URL should parse with an empty
authority, and be recomposed the same.  I would do this by parsing
'git+file:///foo/bar/baz' to ('git+file', '', '/foo/bar/baz') or
something like than, and 'git+file:/foo/bar/baz' to ('git+file', None,
'/foo/bar/baz').

I don't see any reason why implementations should abbreviate the empty
authority by removing the double slashes, unless specified in the
scheme definition.  Although my reading of RFC 3986 is that a missing
authority (no "//") *should* be dereferenced in the same way as an
empty one:

    If the URI scheme defines a default for host, then that default
    applies when the host subcomponent is undefined or when the
    registered name is empty (zero length).  (Sec. 3.2.2)

I don't see why urlparse should try to enforce that by converting from
one to the other.

From dickinsm at gmail.com  Sun May  9 15:54:06 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sun, 9 May 2010 14:54:06 +0100
Subject: [Python-Dev] Did I miss the decision to untabify all of the C
	code?
In-Reply-To: <p2m4207a7ce1005052052n89c7cf21w78103214067a22e0@mail.gmail.com>
References: <4BE20625.9070203@trueblade.com>
	<loom.20100506T024712-79@post.gmane.org>
	<4BE2146E.6000702@trueblade.com>
	<p2m4207a7ce1005052052n89c7cf21w78103214067a22e0@mail.gmail.com>
Message-ID: <i2o5c6f2a5d1005090654v1d26c8f6h6b320743eb116044@mail.gmail.com>

On Thu, May 6, 2010 at 4:52 AM, Joao S. O. Bueno <jsbueno at python.org.br> wrote:
> On Wed, May 5, 2010 at 9:59 PM, Eric Smith <eric at trueblade.com> wrote:
>> That's my point. Since it's basically unreviewable, is it smart to do it
>> during a beta?
>
> Hello folks -
> I don't think these modifications are that "unreviewable": the
> generated binaries have to be exactly the same with the untabified
> files don't they? So is a matter of stashing the binaries, applying
> the patches, rebuild and check to see if the binaries match. Any
> possible script defects undetected by this would be only (C code)
> indentation, which could be fixed later.

That's not foolproof, though:  there are lots of sections of code that
will only get compiled on certain platforms, or with certain configure
options, etc.

Mark

From solipsis at pitrou.net  Sun May  9 20:33:46 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 9 May 2010 20:33:46 +0200
Subject: [Python-Dev] PEP 7 updated
Message-ID: <20100509203346.169facf0@pitrou.net>


Hello,

The untabification of C files didn't produce any noticeable problem on
the buildbots.  I've updated PEP 7 with the mention that all C files
should be 4-space indented, and removed the obsolete wording about
some files being indented with tabs.

Regards

Antoine.



From fw at deneb.enyo.de  Sun May  9 21:51:25 2010
From: fw at deneb.enyo.de (Florian Weimer)
Date: Sun, 09 May 2010 21:51:25 +0200
Subject: [Python-Dev] [RELEASED] Python 2.7 beta 2
In-Reply-To: <x2n1afaf6161005081156qf4a673c1j88a6baac5806c401@mail.gmail.com>
	(Benjamin Peterson's message of "Sat, 8 May 2010 13:56:01 -0500")
References: <x2n1afaf6161005081156qf4a673c1j88a6baac5806c401@mail.gmail.com>
Message-ID: <87632wx39e.fsf@mid.deneb.enyo.de>

* Benjamin Peterson:

> http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python
> distribution.

Something is missing here: "* Multiple context managers in"

From dave at boostpro.com  Sun May  9 23:19:40 2010
From: dave at boostpro.com (David Abrahams)
Date: Sun, 09 May 2010 15:19:40 -0600
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <4BE58B9F.6080604@gmail.com>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com>
Message-ID: <m2eihkvklv.wl%dave@boostpro.com>

At Sat, 08 May 2010 11:04:47 -0500,
John Arbash Meinel wrote:
> 
> Stephen J. Turnbull wrote:
> > David Abrahams writes:
> >  > 
> >  > This is a bug report.  bugs.python.org seems to be down.
> >  > 
> >  >   >>> from urlparse import *
> >  >   >>> urlunsplit(urlsplit('git+file:///foo/bar/baz'))
> >  >   git+file:/foo/bar/baz
> >  > 
> >  > Note the dropped slashes after the colon.
> > 
> > That's clearly wrong, but what does "+" have to to do with it?  AFAIK,
> > the only thing special about + in scheme names is that it's not
> > allowed as the first character.
> 
> Don't you need to register the "git+file:///" url for urlparse to
> properly split it?

Yes.  But the question is whether urlparse should really be so fragile
that every hierarchical scheme needs to be explicitly registered.
Surely ending with ?+file? should be sufficient to have it recognized
as a file-based scheme

-- 
Dave Abrahams           Meet me at BoostCon: http://www.boostcon.com
BoostPro Computing
http://www.boostpro.com


From jon+python-dev at unequivocal.co.uk  Mon May 10 02:17:06 2010
From: jon+python-dev at unequivocal.co.uk (Jon Ribbens)
Date: Mon, 10 May 2010 01:17:06 +0100
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <m2eihkvklv.wl%dave@boostpro.com>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com> <m2eihkvklv.wl%dave@boostpro.com>
Message-ID: <20100510001706.GJ6976@snowy.squish.net>

On Sun, May 09, 2010 at 03:19:40PM -0600, David Abrahams wrote:
> Yes.  But the question is whether urlparse should really be so fragile
> that every hierarchical scheme needs to be explicitly registered.
> Surely ending with ?+file? should be sufficient to have it recognized
> as a file-based scheme

How do you figure?

From meadori at gmail.com  Mon May 10 03:33:14 2010
From: meadori at gmail.com (Meador Inge)
Date: Sun, 9 May 2010 20:33:14 -0500
Subject: [Python-Dev] Reindenting patches
In-Reply-To: <loom.20100505T150720-623@post.gmane.org>
References: <s2tbbaeab101005031537s21dd53f9yc0bd5dfbfd9984ba@mail.gmail.com>
	<hrnlvl$pd3$1@dough.gmane.org> <4BDF8289.2090306@holdenweb.com>
	<4BDFADA6.4040703@v.loewis.de> <20100504082331.2c7a6062@heresy>
	<loom.20100504T143837-351@post.gmane.org>
	<m2tea2499da1005040609sa5e620d2wf8d36366335189bc@mail.gmail.com>
	<4BE03130.9020700@trueblade.com> <4BE0A280.8080903@gmail.com>
	<loom.20100505T150720-623@post.gmane.org>
Message-ID: <u2q4095897c1005091833wcf6da103s3a1865973e9f0bd5@mail.gmail.com>

On Wed, May 5, 2010 at 8:10 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> For the record, I've added to the untabify script a patch rewriting option
> ("-p") which reindents all patch hunks for C files containing tabs. It
> should
> minimize manual reformatting work with existing patches.
>

I just tried '-p' with a local patch that I had created
pre-whitespace-fixes.  It worked like a charm.  Thanks Antoine.

-- Meador
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100509/e0684bf6/attachment.html>

From orsenthil at gmail.com  Mon May 10 07:38:14 2010
From: orsenthil at gmail.com (Senthil Kumaran)
Date: Mon, 10 May 2010 11:08:14 +0530
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <m2eihkvklv.wl%dave@boostpro.com>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com> <m2eihkvklv.wl%dave@boostpro.com>
Message-ID: <20100510053814.GA3608@remy>

On Sun, May 09, 2010 at 03:19:40PM -0600, David Abrahams wrote:
> John Arbash Meinel wrote:
> > Don't you need to register the "git+file:///" url for urlparse to
> > properly split it?
> 
> Yes.  But the question is whether urlparse should really be so fragile
> that every hierarchical scheme needs to be explicitly registered.
> Surely ending with ?+file? should be sufficient to have it recognized
> as a file-based scheme

Not all urls have the 'authority' component after the scheme. (sip
based urls for e.g) urlparse differentiates those by maintaining a
list of scheme names which will follow the pattern of parsing, and
joining for the urls which  have a netloc (or authority component).
This is in general according to RFC 3986 itself.

Yes,'+' is a valid char in url schemes and svn, svn+ssh will be as per
your expectations. But git and git+ssh was missing in there and I
attached a patch in issue8657 to include the same. It is rightly a bug
in the module. But for any general scheme and assuming '+file' would
follow valid authority component, is not something I am sure that
should be in urlparse's expected behavior.



-- 
Senthil

Do not seek death; death will find you.  But seek the road which makes death
a fulfillment.
		-- Dag Hammarskjold

From stephen at xemacs.org  Mon May 10 08:51:36 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 10 May 2010 15:51:36 +0900
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <m2eihkvklv.wl%dave@boostpro.com>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com> <m2eihkvklv.wl%dave@boostpro.com>
Message-ID: <878w7ss0zr.fsf@uwakimon.sk.tsukuba.ac.jp>

David Abrahams writes:
 > At Sat, 08 May 2010 11:04:47 -0500,
 > John Arbash Meinel wrote:

 > > Don't you need to register the "git+file:///" url for urlparse to
 > > properly split it?
 > 
 > Yes.  But the question is whether urlparse should really be so fragile
 > that every hierarchical scheme needs to be explicitly registered.

Exactly.  And the answer is "no".  The RFCs are quite clear that
hierarchical schemes are expected to be extremely common, and provide
several requirements for how they should be parsed, even by
nonvalidating parsers.

It's pretty clear to me that

    urlunsplit(urlsplit('git+file:///foo/bar/baz'))

should be the identity.  The remaining question is, "Should

    urlunsplit(urlsplit('git+file:/foo/bar/baz'))

be the identity?"  I would argue that if git+file is *not* registered,
it should be the identity, while there should be an optional
registry of schemes which may (or perhaps should) be canonicalized
(ie, a *missing* authority would be unsplit as an *empty* authority).

 > Surely ending with "+file" should be sufficient to have it recognized
 > as a file-based scheme

What's a "file-based scheme"?  If you mean an RFC 3986 hierarchical
scheme, that is recognized by the presence of the authority portion,
which is syntactically defined by the presence of "//" immediately
after the scheme ":" terms.  No need for any implicit magic.

In general, EIBTI applies here.  If a registry as described above is
implemented, I would argue that canonicalization should not happen
implicitly.  Nor should validation (eg, error or warning on a URI with
a scheme registered as hierarchical but lacking authority, or vice
versa).  The API should require an explicit statement from the user to
invoke those functionalities.  It might be useful for the OOWTDI API
to canonicalize/validate by default (especially given XSS attacks and
the like), but it should be simple for consenting adults to turn that
feature off.


From stephen at xemacs.org  Mon May 10 10:11:12 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 10 May 2010 17:11:12 +0900
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <20100510053814.GA3608@remy>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com> <m2eihkvklv.wl%dave@boostpro.com>
	<20100510053814.GA3608@remy>
Message-ID: <87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp>

Senthil Kumaran writes:

 > Not all urls have the 'authority' component after the scheme. (sip
 > based urls for e.g) urlparse differentiates those by maintaining a
 > list of scheme names which will follow the pattern of parsing, and
 > joining for the urls which  have a netloc (or authority component).
 > This is in general according to RFC 3986 itself.

This actually quite at variance with the RFC.  The grammar in section
3 doesn't make any reference to schemes as being significant in
parsing.  Whether an authority component is to be parsed or not is
entirely dependent on the presence or absence of the "//" delimiter
following the scheme and its colon delimiter.  AFAICS, if the "//"
delimiter is present, an authority component (possibly empty) *must*
be present in the parse.  Presumably an unparse should then include
that empty component in the generated URI (ie, a "scheme:///..." URI).

Thus, it seems that by the RFC, regardless of any registration,

    urlparse.unsplit(urlparse.split('git+file:///foo/bar'))

should produce 'git+file:///foo/bar' (or perhaps raise an error in
"validation" mode).  The only question is whether registration of
'git+file' as a use_netloc scheme should force

    urlparse.unsplit(urlparse.split('git+file:/foo/bar'))

to return 'git+file:///foo/bar', or whether 'git+file:/foo/bar' would
be acceptable (or better). 

None of what I wrote here or elsewhere takes account of backward
compatibility, it is true.  I'm only talking about the letter of the
RFC.

From orsenthil at gmail.com  Mon May 10 10:27:33 2010
From: orsenthil at gmail.com (Senthil Kumaran)
Date: Mon, 10 May 2010 13:57:33 +0530
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com> <m2eihkvklv.wl%dave@boostpro.com>
	<20100510053814.GA3608@remy>
	<87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20100510082732.GA8163@remy>

On Mon, May 10, 2010 at 05:11:12PM +0900, Stephen J. Turnbull wrote:
>  > Not all urls have the 'authority' component after the scheme. (sip
>  > based urls for e.g) urlparse differentiates those by maintaining a
>  > list of scheme names which will follow the pattern of parsing, and
>  > joining for the urls which  have a netloc (or authority component).
>  > This is in general according to RFC 3986 itself.
> 
> This actually quite at variance with the RFC.  The grammar in section

I should have said, 'treatment of urls with authority' and 'treatment
of urls without authority' in terms of parsing and joining is as per
RFC.  How it is doing practically is by maintaining a list of urls
with known scheme names which use_netloc.

-- 
Senthil

Many Myths are based on truth
		-- Spock, "The Way to Eden",  stardate 5832.3

From stephen at xemacs.org  Mon May 10 10:56:29 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 10 May 2010 17:56:29 +0900
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <20100510082732.GA8163@remy>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com> <m2eihkvklv.wl%dave@boostpro.com>
	<20100510053814.GA3608@remy>
	<87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20100510082732.GA8163@remy>
Message-ID: <874oigrv7m.fsf@uwakimon.sk.tsukuba.ac.jp>

Senthil Kumaran writes:

 > I should have said, 'treatment of urls with authority' and 'treatment
 > of urls without authority' in terms of parsing and joining is as per
 > RFC.  How it is doing practically is by maintaining a list of urls
 > with known scheme names which use_netloc.

Why do that if you can get better behavior based purely on syntactic
analysis?

From barry at python.org  Mon May 10 11:54:36 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 10 May 2010 11:54:36 +0200
Subject: [Python-Dev] What's New text on future maintenance
In-Reply-To: <j2p1afaf6161005061933k5ff3338cs8e38cf3b1f4350e3@mail.gmail.com>
References: <20100507015035.GA50210@andrew-kuchlings-macbook.local>
	<j2p1afaf6161005061933k5ff3338cs8e38cf3b1f4350e3@mail.gmail.com>
Message-ID: <20100510115436.6ae3b399@heresy>

On May 06, 2010, at 09:33 PM, Benjamin Peterson wrote:

>I don't think there's any point in being hypothetical about. I believe
>we've already said that maintence for 2.7 will last for at least 5
>years, so let's proclaim it.

+1.  If our goal is to move our community to Python 3, then I think we do them
the biggest service to be explicit about our intentions for maintenance of 2.7
and the future of Python 2.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100510/9ec89302/attachment.pgp>

From ronaldoussoren at mac.com  Mon May 10 16:09:06 2010
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 10 May 2010 16:09:06 +0200
Subject: [Python-Dev] PEP 7 updated
In-Reply-To: <20100509203346.169facf0@pitrou.net>
References: <20100509203346.169facf0@pitrou.net>
Message-ID: <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com>


On 9 May, 2010, at 20:33, Antoine Pitrou wrote:

> 
> Hello,
> 
> The untabification of C files didn't produce any noticeable problem on
> the buildbots.  I've updated PEP 7 with the mention that all C files
> should be 4-space indented, and removed the obsolete wording about
> some files being indented with tabs.

Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files? 

Ronald

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3567 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100510/4e4f6afe/attachment.bin>

From phd at phd.pp.ru  Mon May 10 18:40:00 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Mon, 10 May 2010 20:40:00 +0400
Subject: [Python-Dev] vim (was: PEP 7 updated)
In-Reply-To: <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com>
References: <20100509203346.169facf0@pitrou.net>
	<68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com>
Message-ID: <20100510164000.GA16432@phd.pp.ru>

On Mon, May 10, 2010 at 04:09:06PM +0200, Ronald Oussoren wrote:
> Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files? 

BufReadPre /usr/local/src/Python/*/*.c set sts=4 sw=4

Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From rdmurray at bitdance.com  Mon May 10 20:10:46 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 10 May 2010 14:10:46 -0400
Subject: [Python-Dev] vim (was: PEP 7 updated)
In-Reply-To: <20100510164000.GA16432@phd.pp.ru>
References: <20100509203346.169facf0@pitrou.net>
	<68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com>
	<20100510164000.GA16432@phd.pp.ru>
Message-ID: <20100510181046.2E6BF2093EE@kimball.webabinitio.net>

On Mon, 10 May 2010 20:40:00 +0400, Oleg Broytman <phd at phd.pp.ru> wrote:
> On Mon, May 10, 2010 at 04:09:06PM +0200, Ronald Oussoren wrote:
> > Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files? 
> 
> BufReadPre /usr/local/src/Python/*/*.c set sts=4 sw=4

Alternatively you could sniff the file for leading tabs and set the
"filetype" or the tab setting based on that.  I think Brett has a vimrc
somewhere that does something like that.

--David

From brett at python.org  Mon May 10 23:14:52 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 10 May 2010 14:14:52 -0700
Subject: [Python-Dev] vim (was: PEP 7 updated)
In-Reply-To: <20100510181046.2E6BF2093EE@kimball.webabinitio.net>
References: <20100509203346.169facf0@pitrou.net>
	<68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> 
	<20100510164000.GA16432@phd.pp.ru>
	<20100510181046.2E6BF2093EE@kimball.webabinitio.net>
Message-ID: <AANLkTinXkAtt4Sy57nV8WV9f8M_s0_8kKj9Jo0k_JLSc@mail.gmail.com>

On Mon, May 10, 2010 at 11:10, R. David Murray <rdmurray at bitdance.com>wrote:

> On Mon, 10 May 2010 20:40:00 +0400, Oleg Broytman <phd at phd.pp.ru> wrote:
> > On Mon, May 10, 2010 at 04:09:06PM +0200, Ronald Oussoren wrote:
> > > Does anyone know of a way to teach vim that C sources in a python
> checkout should have 4-space indents without changing the defaults for other
> C files?
> >
> > BufReadPre /usr/local/src/Python/*/*.c set sts=4 sw=4
>
> Alternatively you could sniff the file for leading tabs and set the
> "filetype" or the tab setting based on that.  I think Brett has a vimrc
> somewhere that does something like that.
>

Misc/Vim/python.vim does what you need to do.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100510/a106989d/attachment.html>

From brett at python.org  Mon May 10 23:14:52 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 10 May 2010 14:14:52 -0700
Subject: [Python-Dev] vim (was: PEP 7 updated)
In-Reply-To: <20100510181046.2E6BF2093EE@kimball.webabinitio.net>
References: <20100509203346.169facf0@pitrou.net>
	<68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> 
	<20100510164000.GA16432@phd.pp.ru>
	<20100510181046.2E6BF2093EE@kimball.webabinitio.net>
Message-ID: <AANLkTinXkAtt4Sy57nV8WV9f8M_s0_8kKj9Jo0k_JLSc@mail.gmail.com>

On Mon, May 10, 2010 at 11:10, R. David Murray <rdmurray at bitdance.com>wrote:

> On Mon, 10 May 2010 20:40:00 +0400, Oleg Broytman <phd at phd.pp.ru> wrote:
> > On Mon, May 10, 2010 at 04:09:06PM +0200, Ronald Oussoren wrote:
> > > Does anyone know of a way to teach vim that C sources in a python
> checkout should have 4-space indents without changing the defaults for other
> C files?
> >
> > BufReadPre /usr/local/src/Python/*/*.c set sts=4 sw=4
>
> Alternatively you could sniff the file for leading tabs and set the
> "filetype" or the tab setting based on that.  I think Brett has a vimrc
> somewhere that does something like that.
>

Misc/Vim/python.vim does what you need to do.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100510/a106989d/attachment-0001.html>

From borowitz at stanford.edu  Mon May 10 23:34:23 2010
From: borowitz at stanford.edu (David Borowitz)
Date: Mon, 10 May 2010 14:34:23 -0700
Subject: [Python-Dev] PEP 7 updated
In-Reply-To: <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com>
References: <20100509203346.169facf0@pitrou.net>
	<68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com>
Message-ID: <AANLkTilNeOmF5htadA4fylVCzOcgjvx3_Su5pYoifBiB@mail.gmail.com>

On Mon, May 10, 2010 at 07:09, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

>
> On 9 May, 2010, at 20:33, Antoine Pitrou wrote:
>
> >
> > Hello,
> >
> > The untabification of C files didn't produce any noticeable problem on
> > the buildbots.  I've updated PEP 7 with the mention that all C files
> > should be 4-space indented, and removed the obsolete wording about
> > some files being indented with tabs.
>
> Does anyone know of a way to teach vim that C sources in a python checkout
> should have 4-space indents without changing the defaults for other C files?
>

:help autocmd-patterns has this example, which should be easy to adapt:

:autocmd BufRead /vim/src/*.c set cindent
Set the 'cindent' option for C files in the /vim/src directory.




>  Ronald
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ddborowitz%40gmail.com
>
>


-- 
It is better to be quotable than to be honest.
   -Tom Stoppard

Borowitz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100510/894840d7/attachment-0001.html>

From orsenthil at gmail.com  Tue May 11 06:55:30 2010
From: orsenthil at gmail.com (Senthil Kumaran)
Date: Tue, 11 May 2010 10:25:30 +0530
Subject: [Python-Dev] urlparse.urlunsplit should be smarter about +
In-Reply-To: <874oigrv7m.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <m2bpcrumzh.wl%dave@boostpro.com>
	<87hbmisb6a.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BE58B9F.6080604@gmail.com> <m2eihkvklv.wl%dave@boostpro.com>
	<20100510053814.GA3608@remy>
	<87632wrxb3.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20100510082732.GA8163@remy>
	<874oigrv7m.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20100511045530.GA6340@remy>

On Mon, May 10, 2010 at 05:56:29PM +0900, Stephen J. Turnbull wrote:
> Senthil Kumaran writes:
> 
>  > I should have said, 'treatment of urls with authority' and 'treatment
>  > of urls without authority' in terms of parsing and joining is as per
>  > RFC.  How it is doing practically is by maintaining a list of urls
>  > with known scheme names which use_netloc.
> 
> Why do that if you can get better behavior based purely on syntactic
> analysis?

For the cases for just parsing and splitting, the syntactic behaviours
are fine enough. I agree with your comments and reinstatement of RFC
rules in the previous emails. 

The problem as we know off, comes while unparsing and joining, ( also
I have not yet looked at the relative url joining behaviour where
redundant /'s can be ignored).

As you may already know, when the data is

ParseResult(scheme='file', netloc='', path='/tmp/junk.txt', params='',
query='', fragment='')

You might expect the output to be file:///tmp/junk.txt
Original might be same too.

But for:
ParseResult(scheme='x', netloc='', path='/y', params='', query='',
fragment='')

One can expect a valid output to be: x:/y

Your suggestion of netloc/authority being differentiate by '' and None
seems a good one to analyze.

Also, by keeping a registry of valid schemes, are you not proposing
something very similar to uses_netloc? But with a different API to
handle parsing based on registry values. Is my understanding of your
proposal correct?

FWIW, I looked at the history of uses_netloc list and it seems that it
been there from the first version when urlparse module followed
different rfc specs for different protocols (telnet, sip etc), so any
changes should be carefully incorporated as not to break the existing
solutions.

-- 
Senthil



From solipsis at pitrou.net  Wed May 12 01:16:15 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 12 May 2010 01:16:15 +0200
Subject: [Python-Dev] bugs.python.org down?
Message-ID: <20100512011615.59ee8c7f@pitrou.net>


Apparently the tracker has been unresponding for some time now,
although port 80 still accepts connections.

Regards

Antoine.



From sridharr at activestate.com  Wed May 12 01:44:36 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Tue, 11 May 2010 16:44:36 -0700
Subject: [Python-Dev] PEP 7 updated
In-Reply-To: <20100509203346.169facf0@pitrou.net>
References: <20100509203346.169facf0@pitrou.net>
Message-ID: <4B06CB5F-6304-42B1-8B2D-F386E74DB63D@activestate.com>

Nor did it break any of our ActivePython 2.7 (Python trunk) builds ... though I had to hand-edit the patches to use 4 spaces now. Will this untabification change be made to the `release2.6-maint` branch too?

-srid

On 2010-05-09, at 11:33 AM, Antoine Pitrou wrote:

> 
> Hello,
> 
> The untabification of C files didn't produce any noticeable problem on
> the buildbots.  I've updated PEP 7 with the mention that all C files
> should be 4-space indented, and removed the obsolete wording about
> some files being indented with tabs.
> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com


From solipsis at pitrou.net  Wed May 12 01:51:12 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 12 May 2010 01:51:12 +0200
Subject: [Python-Dev] PEP 7 updated
In-Reply-To: <4B06CB5F-6304-42B1-8B2D-F386E74DB63D@activestate.com>
References: <20100509203346.169facf0@pitrou.net>
	<4B06CB5F-6304-42B1-8B2D-F386E74DB63D@activestate.com>
Message-ID: <1273621872.3248.2.camel@localhost.localdomain>

Le mardi 11 mai 2010 ? 16:44 -0700, Sridhar Ratnakumar a ?crit :
> Nor did it break any of our ActivePython 2.7 (Python trunk) builds ...
> though I had to hand-edit the patches to use 4 spaces now. Will this
> untabification change be made to the `release2.6-maint` branch too?

It has already been made to the 2.6 branch.
By the way, you don't have to untabify patches by hand, you can just use
"untabify.py -p".

Regards

Antoine.



From cs at zip.com.au  Wed May 12 06:54:20 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 12 May 2010 14:54:20 +1000
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <hs1r2s$aje$1@dough.gmane.org>
References: <hs1r2s$aje$1@dough.gmane.org>
Message-ID: <20100512045420.GA27885@cskk.homeip.net>

On 07May2010 14:53, VanL <van.lindberg at gmail.com> wrote:
| On 5/7/2010 12:41 PM, Steven D'Aprano wrote:
| >> Now, if I write
| >>
| >>     def f1(x,y): return x+y
| >>     def f2(x,y): return x+y
| >>
| >> I don't expect  f1==f2 to be True, even though f1 and f2 behave in
| >> exactly the same way, and indeed it is not.
| 
| This is not what I am getting after; these (IMO) should compare unequal.
| I took a fairly conservative line, testing for identity of functions and
| equality of arguments when converted to a normal form:
| 
|     def __eq__(self, other):
|         try:
|             return ((self.func == other.func) and
|                         (self.args == other.args) and
|                         (self.keywords == other.keywords))
[...]

I think that if you're going to say "identity" above you should have:

    self.func is other.func

in your code.

If you want "==" in your code (and I think you do, since you're implementing
__eq__) you should say "equality" instead of "identity".

I know for functions "==" and "is" currently are equivalent, but we should be
really finicky here about intent, especially since a few messages in the
thread is contemplate testing function for equivalence to one degree or
other. At which point "==" and "is" aren't the same any more.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Careful and correct use of language is a powerful aid to straight thinking,
for putting into words precisely what we mean necessitates getting our own
minds quite clear on what we mean.      - W.I.B. Beveridge

From lie.1296 at gmail.com  Wed May 12 09:49:48 2010
From: lie.1296 at gmail.com (Lie Ryan)
Date: Wed, 12 May 2010 17:49:48 +1000
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <4BE45472.6040406@holdenweb.com>
References: <hs18jc$1mh$1@dough.gmane.org>	<D30CE8E6B791483D8CF71567652FACFB@robslaptop>	<201005080342.00028.steve@pearwood.info>
	<4BE45472.6040406@holdenweb.com>
Message-ID: <hsdmlr$d1c$1@dough.gmane.org>

On 05/08/10 03:57, Steve Holden wrote:
> Steven D'Aprano wrote:
>>
>> [...]
>>> Similarly, if you wanted p1==p2, why not write
>>>
>>>     p1 = partial(operator.add)
>>>     p2 = p1
>>
>> I thought the OP gave a use-case. He's generating "jobs" (partial 
>> applied to a callable and arguments), and wanted to avoid duplicated 
>> jobs.
>>
>> I think it is reasonable to expect that partial(operator.add, 2) 
>> compares equal to partial(operator.add, 2). I don't think he's 
>> suggesting it should compare equal to partial(lambda x,y: x+y, 2).
>>
> Which absence, presumably, also mildly disappoints you?
> 

it disappoints me this does not compare equal:

add3 = lambda a, b, c: a + b + c
a = partial(partial(add3, 1), 2)
b = partial(partial(add3, 2), 1)
print a == b

:-)



From stephen at xemacs.org  Wed May 12 10:34:50 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 12 May 2010 17:34:50 +0900
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <hsdmlr$d1c$1@dough.gmane.org>
References: <hs18jc$1mh$1@dough.gmane.org>
	<D30CE8E6B791483D8CF71567652FACFB@robslaptop>
	<201005080342.00028.steve@pearwood.info>
	<4BE45472.6040406@holdenweb.com> <hsdmlr$d1c$1@dough.gmane.org>
Message-ID: <8739xxr00l.fsf@uwakimon.sk.tsukuba.ac.jp>

Lie Ryan writes:
 > it disappoints me this does not compare equal:
 > 
 > add3 = lambda a, b, c: a + b + c
 > a = partial(partial(add3, 1), 2)
 > b = partial(partial(add3, 2), 1)
 > print a == b
 > 
 > :-)

But it's not even true for floating point.<wink>

From steve at pearwood.info  Wed May 12 12:37:51 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 12 May 2010 20:37:51 +1000
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <8739xxr00l.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <hs18jc$1mh$1@dough.gmane.org> <hsdmlr$d1c$1@dough.gmane.org>
	<8739xxr00l.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <201005122037.51976.steve@pearwood.info>

On Wed, 12 May 2010 06:34:50 pm Stephen J. Turnbull wrote:
> Lie Ryan writes:
>  > it disappoints me this does not compare equal:
>  >
>  > add3 = lambda a, b, c: a + b + c
>  > a = partial(partial(add3, 1), 2)
>  > b = partial(partial(add3, 2), 1)
>  > print a == b
>  >
>  > :-)
>
> But it's not even true for floating point.<wink>


All humour aside, let's not ruin the chances of this patch being 
accepted by overloading it. I think there is a good use-case for 
partial objects to compare equal if they were constructed with 
arguments that compare equal. That's a nice, conservative change that 
is unlikely to lead to bugs, unlike some of the more "clever" proposals 
that rely on mathematical equivalences that don't hold for Python 
objects.



-- 
Steven D'Aprano

From van.lindberg at gmail.com  Wed May 12 14:33:58 2010
From: van.lindberg at gmail.com (VanL)
Date: Wed, 12 May 2010 07:33:58 -0500
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <20100512045420.GA27885@cskk.homeip.net>
References: <hs1r2s$aje$1@dough.gmane.org>
	<20100512045420.GA27885@cskk.homeip.net>
Message-ID: <4BEAA036.3070800@gmail.com>

On 5/11/2010 11:54 PM, Cameron Simpson wrote:
> I know for functions "==" and "is" currently are equivalent, but we should be
> really finicky here about intent, especially since a few messages in the
> thread is contemplate testing function for equivalence to one degree or
> other. At which point "==" and "is" aren't the same any more.
>   

As I stated above, I make no effort to address equivalence of the
functions, nor of the arguments. I am addressing identity of the partial
object, which I am defining as identity of the functions + identity of
the args + equivalence of the keyword arguments after a deterministic
process has been applied.



From jcea at jcea.es  Wed May 12 15:13:21 2010
From: jcea at jcea.es (Jesus Cea)
Date: Wed, 12 May 2010 15:13:21 +0200
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
Message-ID: <4BEAA971.3040001@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[Zlib-devel] HEADS UP: Apparent bad compilation under (just released)
GCC 4.5.0
<http://mail.madler.net/pipermail/zlib-devel_madler.net/2010-May/002267.html>

GCC Bugzilla Bug 40838
gcc shouldn't assume that the stack is aligned
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838


Short history: new GCC 4.5.0 (released a month ago), when compiling with
- -O3, is adding MMX/SSE instructions that requires stack aligned to 16
byte. This is wrong, since x86 ABI only requires stack aligned to 4 bytes.

If you compile EVERYTHING with GCC 4.5.0, you are safe (I guess!), but
if your environment has mixed compiled code (for instance, the OS
libraries), you can possibly "core dump". If you have an old compiled
Python and you update libs compiled with GCC 4.5.0, you can crash in the
process.

Psyco is showing the issue, but it is not the culprit.  It only leaves
- -correctly- the stack in not 16-byte alignment. But there are plenty of
examples of crashes not related to python+psyco.

Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2.
Warm 2.3/2.4/2.5/2.6/3.0/3.1 users. Or warm users compiling with GCC 4.5.0.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS+qpcZlgi5GaxT1NAQLAjgP/anZ5C2lED190++ffcluApF3ASF20iSnH
t21ynHxrz2fgkPeOxeGRAqGEGCc3jZ0UuXchECcLLzFhI8WQS8K766EOgOKdwZLV
WCt0ohWZ0FfsJZX4J5Y73x39uhjShbnl6SSek2uEvPi/tua/R4W/kVXrAZ0ZZR6/
vRqSUXpdolM=
=5K+S
-----END PGP SIGNATURE-----

From ncoghlan at gmail.com  Wed May 12 15:32:55 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 12 May 2010 23:32:55 +1000
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
In-Reply-To: <4BEAA971.3040001@jcea.es>
References: <4BEAA971.3040001@jcea.es>
Message-ID: <4BEAAE07.90903@gmail.com>

Jesus Cea wrote:
> Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2.

Will this actually help? Won't there still be a problem if any extension
module is compiled with GCC 4.5.0 without that option, regardless of the
options used to build Python itself?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From foom at fuhm.net  Wed May 12 15:39:56 2010
From: foom at fuhm.net (James Y Knight)
Date: Wed, 12 May 2010 09:39:56 -0400
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
In-Reply-To: <4BEAA971.3040001@jcea.es>
References: <4BEAA971.3040001@jcea.es>
Message-ID: <D0FCA42C-276C-4707-AC13-1851F5AAC18E@fuhm.net>


On May 12, 2010, at 9:13 AM, Jesus Cea wrote:
> Short history: new GCC 4.5.0 (released a month ago), when compiling  
> with
> - -O3, is adding MMX/SSE instructions that requires stack aligned to  
> 16
> byte. This is wrong, since x86 ABI only requires stack aligned to 4  
> bytes.
>
> If you compile EVERYTHING with GCC 4.5.0, you are safe (I guess!), but
> if your environment has mixed compiled code (for instance, the OS
> libraries), you can possibly "core dump". If you have an old compiled
> Python and you update libs compiled with GCC 4.5.0, you can crash in  
> the
> process.
>
> Psyco is showing the issue, but it is not the culprit.  It only leaves
> - -correctly- the stack in not 16-byte alignment. But there are  
> plenty of
> examples of crashes not related to python+psyco.
>
> Proposal: add "-fno-tree-vectorize" to compilation options for  
> 2.7/3.2.
> Warm 2.3/2.4/2.5/2.6/3.0/3.1 users. Or warm users compiling with GCC  
> 4.5.0.


While assuming the stack is 16byte aligned is undeniably an ABI- 
violation in GCC, at this point, it's surely simpler to just go along:  
the new unofficial ABI for x86 is that the stack must always be left  
in 16-byte alignment...

So, just change psyco to always use 16-byte-aligned stackframes. GCC  
has used 16byte-aligned stackframes for a heck of a long time now (so  
if the stack starts 16byte aligned on entry to a function it will stay  
that way on calls). So usually the only way people run into unaligned  
stacks is via hand-written assembly code or JIT compilers.

I think you'll be a lot happier just modifying Psyco than making  
everyone else in the world change their compiler flags.

James

From jcea at jcea.es  Wed May 12 15:56:48 2010
From: jcea at jcea.es (Jesus Cea)
Date: Wed, 12 May 2010 15:56:48 +0200
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
In-Reply-To: <4BEAAE07.90903@gmail.com>
References: <4BEAA971.3040001@jcea.es> <4BEAAE07.90903@gmail.com>
Message-ID: <4BEAB3A0.9010302@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/05/10 15:32, Nick Coghlan wrote:
> Jesus Cea wrote:
>> Proposal: add "-fno-tree-vectorize" to compilation options for 2.7/3.2.
> 
> Will this actually help? Won't there still be a problem if any extension
> module is compiled with GCC 4.5.0 without that option, regardless of the
> options used to build Python itself?

When I do a "python setup.py install", the options used to compile the
module are the same that used to compile python itself.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS+qzn5lgi5GaxT1NAQJ5BgQAimvcMssT57iuuLpaI9P9GXKGZf9VUAzj
F4XJM/lT4Qtzu22jecIvEej0MyiMR/4qHvns8lgqLXn/30pkYrkmYcjxFigpM7Bl
rOKMYeAr3ki8dN87APkoiMKOJHByXlEUZu+BowCOXcUCtGYcENLwQ+STr1NyCsUB
IINfh1pJ2Nc=
=GiUG
-----END PGP SIGNATURE-----

From jcea at jcea.es  Wed May 12 16:01:19 2010
From: jcea at jcea.es (Jesus Cea)
Date: Wed, 12 May 2010 16:01:19 +0200
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
In-Reply-To: <D0FCA42C-276C-4707-AC13-1851F5AAC18E@fuhm.net>
References: <4BEAA971.3040001@jcea.es>
	<D0FCA42C-276C-4707-AC13-1851F5AAC18E@fuhm.net>
Message-ID: <4BEAB4AF.80908@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/05/10 15:39, James Y Knight wrote:
> While assuming the stack is 16byte aligned is undeniably an
> ABI-violation in GCC, at this point, it's surely simpler to just go
> along: the new unofficial ABI for x86 is that the stack must always be
> left in 16-byte alignment...

You can not rule out other software embedding python inside, or
callbacks from foreign code. For instance, Berkeley DB library can do
callbacks to Python code.

> So, just change psyco to always use 16-byte-aligned stackframes. GCC has
> used 16byte-aligned stackframes for a heck of a long time now (so if the
> stack starts 16byte aligned on entry to a function it will stay that way
> on calls). So usually the only way people run into unaligned stacks is
> via hand-written assembly code or JIT compilers.

Not all the universe is GCC based. For instance, Solaris system
libraries are not compiled using GCC. The world is bigger that Linux/GCC.

> I think you'll be a lot happier just modifying Psyco than making
> everyone else in the world change their compiler flags.

Would be nice if GCC 4.5.1 would solve this :). They are objectivelly
breaking the x86 ABI.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS+q0rplgi5GaxT1NAQK98wP+NJoqNCpvjemP4Gv7y1G/iPkQgjuidslT
uiPxDcN9Eprcluc+mGTBu6N+fCTj09xYhUCD1wWhoJq2dRyoA8b+XC1fCSyL4VXc
mzsy0rGmKeQh4lyAw+7agFCqryd6n+/oyl+9aOT6YkzyLFjQd4KDEcGNZ0h+6PAf
4jtx1+p3k0c=
=TzTY
-----END PGP SIGNATURE-----

From foom at fuhm.net  Wed May 12 16:27:09 2010
From: foom at fuhm.net (James Y Knight)
Date: Wed, 12 May 2010 10:27:09 -0400
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
In-Reply-To: <4BEAB4AF.80908@jcea.es>
References: <4BEAA971.3040001@jcea.es>
	<D0FCA42C-276C-4707-AC13-1851F5AAC18E@fuhm.net>
	<4BEAB4AF.80908@jcea.es>
Message-ID: <E72BF78E-F821-4844-82F1-61A3BC13DED8@fuhm.net>


On May 12, 2010, at 10:01 AM, Jesus Cea wrote:
> On 12/05/10 15:39, James Y Knight wrote:
>> While assuming the stack is 16byte aligned is undeniably an
>> ABI-violation in GCC, at this point, it's surely simpler to just go
>> along: the new unofficial ABI for x86 is that the stack must always  
>> be
>> left in 16-byte alignment...
>
> You can not rule out other software embedding python inside, or
> callbacks from foreign code. For instance, Berkeley DB library can do
> callbacks to Python code.

So? When calling callback functions, the Berkeley DB library won't  
un-16byte-align the stack, will it? (Assuming it's been compiled with  
gcc in the last 10 years)

> Not all the universe is GCC based. For instance, Solaris system
> libraries are not compiled using GCC. The world is bigger that Linux/ 
> GCC.

If the Solaris compilers don't use 16byte-aligned stackframes, and GCC  
on Solaris/x86 also assumes 16byte-aligned stacks, I guess GCC on  
Solaris/x86 is pretty broken indeed. But for Linux/x86, stacks have  
been de-facto 16byte aligned for so long, you can *almost* excuse the  
ABI violation as unimportant.

But anyways, psyco should keep the stackframes 16byte aligned  
regardless, for performance reasons: even when accessing datatypes for  
which unaligned access doesn't crash, it's faster when it's aligned.

James

From martin at v.loewis.de  Wed May 12 17:14:25 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 12 May 2010 17:14:25 +0200
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
In-Reply-To: <4BEAA971.3040001@jcea.es>
References: <4BEAA971.3040001@jcea.es>
Message-ID: <4BEAC5D1.2050906@v.loewis.de>

> Short history: new GCC 4.5.0 (released a month ago), when compiling with
> -O3, is adding MMX/SSE instructions that requires stack aligned to 16
> byte. This is wrong, since x86 ABI only requires stack aligned to 4 bytes.

I think this is debatable. It depends on the operating system also;
ultimately, it is the OS vendor who specifies the C ABI for their
systems. On Linux, in absence of a vendor, the ABI is what the kernel
and gcc define it to be.

Regards,
Martin

From urban.dani at gmail.com  Wed May 12 17:11:21 2010
From: urban.dani at gmail.com (Daniel Urban)
Date: Wed, 12 May 2010 17:11:21 +0200
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <loom.20100507T170052-252@post.gmane.org>
References: <hs18jc$1mh$1@dough.gmane.org>
	<loom.20100507T170052-252@post.gmane.org>
Message-ID: <AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com>

On Fri, May 7, 2010 at 17:02, Antoine Pitrou <solipsis at pitrou.net> wrote:
> It would be more useful to provide equality, hashing and repr to partial itself,
> rather than a subclass. Feel free to propose a patch :)

Hi!

I've done that.
I've opened a feature request: http://bugs.python.org/issue8699
The patch is also at Rietveld: http://codereview.appspot.com/1179044

I'm a beginner, so my patch is probably far from perfect, but I'd
appreciate any help, and will try to correct my mistakes.

Thanks,
Daniel Urban

From jyasskin at gmail.com  Wed May 12 17:58:19 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Wed, 12 May 2010 08:58:19 -0700
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
In-Reply-To: <D0FCA42C-276C-4707-AC13-1851F5AAC18E@fuhm.net>
References: <4BEAA971.3040001@jcea.es>
	<D0FCA42C-276C-4707-AC13-1851F5AAC18E@fuhm.net>
Message-ID: <AANLkTimrb1UnMrW5ryn8_E53dy2vAhmNOvKUO1yR_C3L@mail.gmail.com>

On Wed, May 12, 2010 at 6:39 AM, James Y Knight <foom at fuhm.net> wrote:
> I think you'll be a lot happier just modifying Psyco than making everyone
> else in the world change their compiler flags.

Aye, there's the rub. Nobody's happier modifying Psyco. :)  But that
just means people will gradually have to stop using psyco in favor of
maintainable JITs. Gcc's not going to change its stack requirements
just because some people think they know what the ABI should be better
than the people defining the ABI. Btw, this has been a problem since
at least gcc-4.4.

From tjreedy at udel.edu  Wed May 12 19:09:11 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 12 May 2010 13:09:11 -0400
Subject: [Python-Dev] bugs.python.org down?
In-Reply-To: <20100512011615.59ee8c7f@pitrou.net>
References: <20100512011615.59ee8c7f@pitrou.net>
Message-ID: <hsenbk$e07$1@dough.gmane.org>

On 5/11/2010 7:16 PM, Antoine Pitrou wrote:
>
> Apparently the tracker has been unresponding for some time now,
> although port 80 still accepts connections.

As I reported before, there have been on and off problems for days.
Messages like (minutes ago) upon trying to login:

"Service Temporarily Unavailable

The server is temporarily unable to service your request due to 
maintenance downtime or capacity problems. Please try again later.
Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 
OpenSSL/0.9.8g Server at bugs.python.org Port 80"

Now I retry and it works.


From rdmurray at bitdance.com  Wed May 12 20:09:22 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 12 May 2010 14:09:22 -0400
Subject: [Python-Dev] bugs.python.org down?
In-Reply-To: <hsenbk$e07$1@dough.gmane.org>
References: <20100512011615.59ee8c7f@pitrou.net> <hsenbk$e07$1@dough.gmane.org>
Message-ID: <20100512180923.00A20200560@kimball.webabinitio.net>

On Wed, 12 May 2010 13:09:11 -0400, Terry Reedy <tjreedy at udel.edu> wrote:
> On 5/11/2010 7:16 PM, Antoine Pitrou wrote:
> >
> > Apparently the tracker has been unresponding for some time now,
> > although port 80 still accepts connections.
> 
> As I reported before, there have been on and off problems for days.
> Messages like (minutes ago) upon trying to login:
> 
> "Service Temporarily Unavailable
> 
> The server is temporarily unable to service your request due to 
> maintenance downtime or capacity problems. Please try again later.
> Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 
> OpenSSL/0.9.8g Server at bugs.python.org Port 80"
> 
> Now I retry and it works.

That last was us restarting roundup, which cured the problem at least
temporarily.  We've added a robots.txt file to hopefully slow down the
bot hits (which are the bulk of the traffic on the tracker) once they
bother to notice it.  I'm not convinced that hit count load is the
problem, but it might be.

--
R. David Murray                                      www.bitdance.com

From janssen at parc.com  Thu May 13 05:17:59 2010
From: janssen at parc.com (Bill Janssen)
Date: Wed, 12 May 2010 20:17:59 PDT
Subject: [Python-Dev] configuring the buildbot to skip some tests?
Message-ID: <84766.1273720679@parc.com>

I've got parc-tiger-1 up and running again.  It's failing on test_tk,
which makes sense, because it's running as a background twisted process,
and thus can't access the window server.  I should configure that out.

I'm looking for documentation on how to configure the build slave so
that it skips this test.

Bill

From trent at snakebite.org  Thu May 13 05:38:48 2010
From: trent at snakebite.org (Trent Nelson)
Date: Wed, 12 May 2010 23:38:48 -0400
Subject: [Python-Dev] Python on Windows with CoApp
Message-ID: <4BEB7448.8080204@snakebite.org>

Howdy folks,

Quick e-mail at 34,000ft (aren't wifi-enabled flights great?) to mention 
a new initiative that's been started by Microsoft called CoApp (Common 
Opensource Application Publishing Platform).  The aim is simple: make 
open source software rock on Windows ;-)

It's probably easiest to think of it as a 
Microsoft-endorsed-but-community-run open source distribution for 
Windows, akin to all the various package managers for Linux 
distributions and ports/packages for the *BSDs.  There are specific user 
and developer experiences we'll be addressing -- like making it easy to 
install and use open source software, or use it within your own project 
(open source or not).

CoApp will affect Python in one of two ways.  Once there's a clear-cut 
specification for open source projects to follow, Python can either 
decide to follow it, or not.  The same applies to all open source 
packages, actually.  For those that follow it, great!  If not, no 
problem -- the plan is to shallow-fork such projects via launchpad and 
the CoApp community will take responsibility for getting releases of 
open source projects into CoApp shape.

It's in its infancy at the moment -- it took the chap (Garrett Serack) 
who's spearheading it at Microsoft about six months to get it all signed 
off by the lawyers and platform/server VPs.

So, for those of you out there who are Windows-inclined, now's a perfect 
time to get involved to help shape the direction of CoApp going forward.

The website/wiki is http://coapp.org/ and the launchpad project site is 
http://launchpad.net/coapp (which is where the mailing list is hosted).

We're actually having a 'CoApp Development Summit' tomorrow and Friday 
in Seattle (that Microsoft's graciously sponsored).  The event will be 
accessible via Live Meeting for those that are interested: 
http://coapp.org/Project_Planning/CoApp_Design_and_Development_Summit

Regards,

	Trent.

From trent at snakebite.org  Thu May 13 05:47:59 2010
From: trent at snakebite.org (Trent Nelson)
Date: Wed, 12 May 2010 23:47:59 -0400
Subject: [Python-Dev] PEP 7 updated
In-Reply-To: <68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com>
References: <20100509203346.169facf0@pitrou.net>
	<68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com>
Message-ID: <4BEB766F.7070100@snakebite.org>


> Does anyone know of a way to teach vim that C sources in a python checkout should have 4-space indents without changing the defaults for other C files?

I use this in my vimrc:

"----------------------------------------------------------------------------"
" indentation: use detectindent plugin if possible
"----------------------------------------------------------------------------"
set autoindent
set smartindent
try
     let g:detectindent_preferred_expandtab = 1
     let g:detectindent_preferred_tabsize = 8
     let g:detectindent_preferred_indent = 4

     source $VIMRUNTIME/plugin/detectindent.vim
     au BufNewFile,BufRead * .* DetectIndent
catch
     set smarttab
     set expandtab
     set tabstop=8
     set shiftwidth=4
     set softtabstop=4
     set textwidth=80
endtry

*** And this is plugin/detectindent.vim:

" Name:          detectindent (global plugin)
" Version:       1.0
" Author:        Ciaran McCreesh <ciaranm at gentoo.org>
" Updates:       http://dev.gentoo.org/~ciaranm/vim/
" Purpose:       Detect file indent settings
"
" License:       You may redistribute this plugin under the same terms 
as Vim
"                itself.
"
" Usage:         :DetectIndent
"
"                " to prefer expandtab to noexpandtab when detection is
"                " impossible:
"                :let g:detectindent_preferred_expandtab = 1
"
"                " to set a preferred indent level when detection is
"                " impossible:
"                :let g:detectindent_preferred_indent = 4
"
" Requirements:  Untested on Vim versions below 6.2

fun! <SID>IsCommentStart(line)
     " &comments isn't reliable
     if &ft == "c" || &ft == "cpp"
         return -1 != match(a:line, '/\*')
     else
         return 0
     endif
endfun

fun! <SID>IsCommentEnd(line)
     if &ft == "c" || &ft == "cpp"
         return -1 != match(a:line, '\*/')
     else
         return 0
     endif
endfun

fun! <SID>DetectIndent()
     let l:has_leading_tabs            = 0
     let l:has_leading_spaces          = 0
     let l:shortest_leading_spaces_run = 0
     let l:longest_leading_spaces_run  = 0

     let l:idx_end = line("$")
     let l:idx = 1
     while l:idx <= l:idx_end
         let l:line = getline(l:idx)

         " try to skip over comment blocks, they can give really screwy 
indent
         " settings in c/c++ files especially
         if <SID>IsCommentStart(l:line)
             while l:idx <= l:idx_end && ! <SID>IsCommentEnd(l:line)
                 let l:line = getline(l:idx)
                 let l:idx = l:idx + 1
             endwhile
             let l:idx = l:idx + 1
             continue
         endif

         let l:leading_char = strpart(l:line, 0, 1)

         if l:leading_char == "\t"
             let l:has_leading_tabs = 1

         elseif l:leading_char == " "
             " only interested if we don't have a run of spaces followed 
by a
             " tab.
             if -1 == match(l:line, '^ \+\t')
                 let l:has_leading_spaces = 1
                 let l:spaces = strlen(matchstr(l:line, '^ \+'))
                 if l:shortest_leading_spaces_run == 0 ||
                             \ l:spaces < l:shortest_leading_spaces_run
                     let l:shortest_leading_spaces_run = l:spaces
                 endif
                 if l:spaces > l:longest_leading_spaces_run
                     let l:longest_leading_spaces_run = l:spaces
                 endif
             endif

         endif

         let l:idx = l:idx + 1
     endwhile

     if l:has_leading_tabs && ! l:has_leading_spaces
         " tabs only, no spaces
         set noexpandtab
         if exists("g:detectindent_preferred_tabsize")
             let &shiftwidth  = g:detectindent_preferred_indent
             let &tabstop     = g:detectindent_preferred_indent
         endif

     elseif l:has_leading_spaces && ! l:has_leading_tabs
         " spaces only, no tabs
         set expandtab
         let &shiftwidth  = l:shortest_leading_spaces_run

     elseif l:has_leading_spaces && l:has_leading_tabs
         " spaces and tabs
         set noexpandtab
         let &shiftwidth = l:shortest_leading_spaces_run

         " mmmm, time to guess how big tabs are
         if l:longest_leading_spaces_run < 2
             let &tabstop = 2
         elseif l:longest_leading_spaces_run < 4
             let &tabstop = 4
         else
             let &tabstop = 8
         endif

     else
         " no spaces, no tabs
         if exists("g:detectindent_preferred_tabsize")
             let &shiftwidth  = g:detectindent_preferred_indent
             let &tabstop     = g:detectindent_preferred_indent
         endif
         if exists("g:detectindent_preferred_expandtab")
             set expandtab
         endif

     endif
endfun

command! -nargs=0 DetectIndent call <SID>DetectIndent()


From martin at v.loewis.de  Thu May 13 09:20:07 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 13 May 2010 09:20:07 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <84766.1273720679@parc.com>
References: <84766.1273720679@parc.com>
Message-ID: <4BEBA827.5040000@v.loewis.de>

Bill Janssen wrote:
> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
> which makes sense, because it's running as a background twisted process,
> and thus can't access the window server.

It doesn't really make sense. It should skip the test, instead of
failing it. I.e. aborting the Python process is definitely not a good
response.

Regards,
Martin

From yaniv at aknin.name  Thu May 13 10:50:02 2010
From: yaniv at aknin.name (Yaniv Aknin)
Date: Thu, 13 May 2010 18:50:02 +1000
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com>
References: <hs18jc$1mh$1@dough.gmane.org>
	<loom.20100507T170052-252@post.gmane.org>
	<AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com>
Message-ID: <AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com>

I'm never certain where to reply in such a case, on the list or on the
issue, but since no one is nosy yet to Daniel's patch, I thought I'd ask
here.

While a partial object should reasonably never change, you could change it:
>>> from functools import partial
>>> p = partial(lambda *a, **kw: kw, 1, 2, spam='eggs')
>>> p()
{'spam': 'eggs'}
>>> p.keywords['spam'] = 'bacon'
>>> p()
{'spam': 'bacon'}
>>>

I realize touching p.keywords voids your warranty, but if we can stop people
from doing it, maybe we should (or at least put a warning in the
documentation, no?). So I'm thinking either we make an immutable/hashable
dict while we're at it, or store the keyword arguments as a tuple (which
guarantees immutability), and only convert them back to a dict when you want
to call the partial object (simpler, slower).

Your thoughts? Should we continue this discussion at issue8699?

 - Yaniv

On Thu, May 13, 2010 at 1:11 AM, Daniel Urban <urban.dani at gmail.com> wrote:

> On Fri, May 7, 2010 at 17:02, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > It would be more useful to provide equality, hashing and repr to partial
> itself,
> > rather than a subclass. Feel free to propose a patch :)
>
> Hi!
>
> I've done that.
> I've opened a feature request: http://bugs.python.org/issue8699
> The patch is also at Rietveld: http://codereview.appspot.com/1179044
>
> I'm a beginner, so my patch is probably far from perfect, but I'd
> appreciate any help, and will try to correct my mistakes.
>
> Thanks,
> Daniel Urban
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/yaniv%40aknin.name
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100513/f1ec8640/attachment.html>

From ncoghlan at gmail.com  Thu May 13 12:39:13 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 May 2010 20:39:13 +1000
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <84766.1273720679@parc.com>
References: <84766.1273720679@parc.com>
Message-ID: <4BEBD6D1.8020702@gmail.com>

Bill Janssen wrote:
> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
> which makes sense, because it's running as a background twisted process,
> and thus can't access the window server.  I should configure that out.
> 
> I'm looking for documentation on how to configure the build slave so
> that it skips this test.

It may be better to try to detect the "no window server" case and skip
it in the test itself rather than in the build slave configuration.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From steve at pearwood.info  Thu May 13 13:30:29 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 13 May 2010 21:30:29 +1000
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com>
References: <hs18jc$1mh$1@dough.gmane.org>
	<AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com>
	<AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com>
Message-ID: <201005132130.30619.steve@pearwood.info>

On Thu, 13 May 2010 06:50:02 pm Yaniv Aknin wrote:
> I'm never certain where to reply in such a case, on the list or on
> the issue, but since no one is nosy yet to Daniel's patch, I thought
> I'd ask here.
>
> While a partial object should reasonably never change, you could 
change it:
[...]
> I realize touching p.keywords voids your warranty, but if we can stop
> people from doing it, maybe we should (or at least put a warning in
> the documentation, no?). 

Modifying partial.keywords will almost certainly effect hashing, so I 
think this is relevant to the patch.



> So I'm thinking either we make an 
> immutable/hashable dict while we're at it, or store the keyword
> arguments as a tuple (which guarantees immutability), and only
> convert them back to a dict when you want to call the partial object
> (simpler, slower).

I'd support an immutable dict. partial objects already impose a 
significant (~ 30%) performance penalty:

>>> from timeit import Timer
>>> min(Timer('f(5)', 'f = lambda x: x').repeat())
0.93580079078674316
>>> min(Timer('p(5)', 'from functools import partial; p = partial(lambda 
x: x)').repeat())
1.2715129852294922

No need to make that worse if that can be avoided.



-- 
Steven D'Aprano

From urban.dani at gmail.com  Thu May 13 11:51:23 2010
From: urban.dani at gmail.com (Daniel Urban)
Date: Thu, 13 May 2010 11:51:23 +0200
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com>
References: <hs18jc$1mh$1@dough.gmane.org>
	<loom.20100507T170052-252@post.gmane.org> 
	<AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com> 
	<AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com>
Message-ID: <AANLkTimvjHKNN01dfQYIlE8rfbTaXh7CXjapkrnLiC7x@mail.gmail.com>

> While a partial object should reasonably never change, you could change it:
>>>> from functools import partial
>>>> p = partial(lambda *a, **kw: kw, 1, 2, spam='eggs')
>>>> p()
> {'spam': 'eggs'}
>>>> p.keywords['spam'] = 'bacon'
>>>> p()
> {'spam': 'bacon'}
>>>>
> I realize touching p.keywords voids your warranty, but if we can stop people
> from doing it, maybe we should (or at least put a warning in the
> documentation, no?). So I'm thinking either we make an immutable/hashable
> dict while we're at it, or?store the keyword arguments as a tuple (which
> guarantees immutability), and only convert them back to a dict when you want
> to call the partial object (simpler, slower).

You're right. I think it is possible to stop people from modifying
p.keywords. If p.keywords wouldn't be a dict, but a read-only proxy
for that dict, that may solve this problem. I think that is possible,
with PyDictProxy_New [1]. But I'm not certain if that is a good idea,
it may have side effects I'm not aware of. Any thoughts about this?

> Your thoughts? Should we continue this discussion at issue8699?
I don't know, I'm new here...

[1] http://docs.python.org/py3k/c-api/dict.html#PyDictProxy_New

Thanks,
Daniel Urban

From exarkun at twistedmatrix.com  Thu May 13 15:41:36 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Thu, 13 May 2010 13:41:36 -0000
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <84766.1273720679@parc.com>
References: <84766.1273720679@parc.com>
Message-ID: <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>

On 03:17 am, janssen at parc.com wrote:
>I've got parc-tiger-1 up and running again.  It's failing on test_tk,
>which makes sense, because it's running as a background twisted 
>process,
>and thus can't access the window server.  I should configure that out.

You can run it in an xvfb.

Jean-Paul

From martin at v.loewis.de  Thu May 13 16:42:18 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 13 May 2010 16:42:18 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <4BEBD6D1.8020702@gmail.com>
References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com>
Message-ID: <4BEC0FCA.40601@v.loewis.de>

>> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
>> which makes sense, because it's running as a background twisted process,
>> and thus can't access the window server.  I should configure that out.
>>
>> I'm looking for documentation on how to configure the build slave so
>> that it skips this test.
> 
> It may be better to try to detect the "no window server" case and skip
> it in the test itself rather than in the build slave configuration.

Even better would be if Python wouldn't crash when you try to run Tk
commands without a window server. Instead of aborting Python, that
should raise an exception (which can then be detected as a test skip).

Regards,
Martin

From martin at v.loewis.de  Thu May 13 16:43:49 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 13 May 2010 16:43:49 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>
References: <84766.1273720679@parc.com>
	<20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>
Message-ID: <4BEC1025.2050205@v.loewis.de>

exarkun at twistedmatrix.com wrote:
> On 03:17 am, janssen at parc.com wrote:
>> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
>> which makes sense, because it's running as a background twisted process,
>> and thus can't access the window server.  I should configure that out.
> 
> You can run it in an xvfb.

But that's beside the point! The slave configuration detected a bug in
Python, so rather than working around the bug by modifying the slave
configuration, the bug should get fixed.

Of course, the slave is then useless until somebody contributes such a fix.

Regards,
Martin

From p.f.moore at gmail.com  Thu May 13 17:49:48 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 13 May 2010 16:49:48 +0100
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <4BEC1025.2050205@v.loewis.de>
References: <84766.1273720679@parc.com>
	<20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>
	<4BEC1025.2050205@v.loewis.de>
Message-ID: <AANLkTilnm4EYinwTXdityHfO-I-G1Ql_AEaVAdOT9dUU@mail.gmail.com>

On 13 May 2010 15:43, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Of course, the slave is then useless until somebody contributes such a fix.

That's the sad part. If there was a means of temporarily marking the
test on a particular slave as a known issue, it would avoid a single
bug rendering a buildslave useless...

(Having said that, a similar situation with my buildslave prompted me
to spend the time fixing the bug so I didn't have to keep restarting
the slave, so maybe it's a good thing after all :-))

Paul.

From martin at v.loewis.de  Thu May 13 17:57:45 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 13 May 2010 17:57:45 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <AANLkTilnm4EYinwTXdityHfO-I-G1Ql_AEaVAdOT9dUU@mail.gmail.com>
References: <84766.1273720679@parc.com>	
	<20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>	
	<4BEC1025.2050205@v.loewis.de>
	<AANLkTilnm4EYinwTXdityHfO-I-G1Ql_AEaVAdOT9dUU@mail.gmail.com>
Message-ID: <4BEC2179.9070507@v.loewis.de>

> (Having said that, a similar situation with my buildslave prompted me
> to spend the time fixing the bug so I didn't have to keep restarting
> the slave, so maybe it's a good thing after all :-))

Indeed. More generally, I'd question the point of automated testing if
people try to work around serious problems rather than fixing them. And
an interpreter crash in the test suite *is* a serious problem, IMO.

Of course, it may turn out to be an unfixable Tcl or Apple bug, in which
case working around would become more interesting.

Regards,
Martin

From janssen at parc.com  Thu May 13 19:42:23 2010
From: janssen at parc.com (Bill Janssen)
Date: Thu, 13 May 2010 10:42:23 PDT
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <4BEBA827.5040000@v.loewis.de>
References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de>
Message-ID: <81551.1273772543@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> Bill Janssen wrote:
> > I've got parc-tiger-1 up and running again.  It's failing on test_tk,
> > which makes sense, because it's running as a background twisted process,
> > and thus can't access the window server.
> 
> It doesn't really make sense. It should skip the test, instead of
> failing it. I.e. aborting the Python process is definitely not a good
> response.

Yes, you're right.  It's a bug in the test.

Bill

From glyph at twistedmatrix.com  Thu May 13 19:45:36 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Thu, 13 May 2010 13:45:36 -0400
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>
References: <84766.1273720679@parc.com>
	<20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>
Message-ID: <A3D46154-9982-4DE1-90A5-CB1F8F4850EF@twistedmatrix.com>


On May 13, 2010, at 9:41 AM, exarkun at twistedmatrix.com wrote:

> On 03:17 am, janssen at parc.com wrote:
>> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
>> which makes sense, because it's running as a background twisted process,
>> and thus can't access the window server.  I should configure that out.
> 
> You can run it in an xvfb.

See <http://www.python.org/dev/buildbot/builders/PPC%20Tiger%20trunk/builds/7/steps/test/logs/stdio>: this isn't an X server that he's talking about, it's "WindowServer", the OS X windowing system, so Xvfb won't help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100513/70decf26/attachment.html>

From janssen at parc.com  Thu May 13 19:47:56 2010
From: janssen at parc.com (Bill Janssen)
Date: Thu, 13 May 2010 10:47:56 PDT
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>
References: <84766.1273720679@parc.com>
	<20100513134136.1694.1471493852.divmod.xquotient.0@localhost.localdomain>
Message-ID: <81726.1273772876@parc.com>

exarkun at twistedmatrix.com wrote:

> On 03:17 am, janssen at parc.com wrote:
> >I've got parc-tiger-1 up and running again.  It's failing on test_tk,
> > which makes sense, because it's running as a background twisted
> > process,
> >and thus can't access the window server.  I should configure that out.
> 
> You can run it in an xvfb.

I don't think that would work -- it's a Mac.

Bill

From brett at python.org  Thu May 13 20:39:23 2010
From: brett at python.org (Brett Cannon)
Date: Thu, 13 May 2010 11:39:23 -0700
Subject: [Python-Dev] PEP 7 updated
In-Reply-To: <4BEB766F.7070100@snakebite.org>
References: <20100509203346.169facf0@pitrou.net>
	<68F4207C-FFD0-4FED-8D23-68C18760BFE9@mac.com> 
	<4BEB766F.7070100@snakebite.org>
Message-ID: <AANLkTilB8nJxgMXyTOl3oifDV0IlOz5hQRXNMDGVic6L@mail.gmail.com>

Feel free to look at Misc/Vim/python.vim and see if this works better
than what is already there.

On Wed, May 12, 2010 at 20:47, Trent Nelson <trent at snakebite.org> wrote:
>
>> Does anyone know of a way to teach vim that C sources in a python checkout
>> should have 4-space indents without changing the defaults for other C files?
>
> I use this in my vimrc:
>
> "----------------------------------------------------------------------------"
> " indentation: use detectindent plugin if possible
> "----------------------------------------------------------------------------"
> set autoindent
> set smartindent
> try
> ? ?let g:detectindent_preferred_expandtab = 1
> ? ?let g:detectindent_preferred_tabsize = 8
> ? ?let g:detectindent_preferred_indent = 4
>
> ? ?source $VIMRUNTIME/plugin/detectindent.vim
> ? ?au BufNewFile,BufRead * .* DetectIndent
> catch
> ? ?set smarttab
> ? ?set expandtab
> ? ?set tabstop=8
> ? ?set shiftwidth=4
> ? ?set softtabstop=4
> ? ?set textwidth=80
> endtry
>
> *** And this is plugin/detectindent.vim:
>
> " Name: ? ? ? ? ?detectindent (global plugin)
> " Version: ? ? ? 1.0
> " Author: ? ? ? ?Ciaran McCreesh <ciaranm at gentoo.org>
> " Updates: ? ? ? http://dev.gentoo.org/~ciaranm/vim/
> " Purpose: ? ? ? Detect file indent settings
> "
> " License: ? ? ? You may redistribute this plugin under the same terms as
> Vim
> " ? ? ? ? ? ? ? ?itself.
> "
> " Usage: ? ? ? ? :DetectIndent
> "
> " ? ? ? ? ? ? ? ?" to prefer expandtab to noexpandtab when detection is
> " ? ? ? ? ? ? ? ?" impossible:
> " ? ? ? ? ? ? ? ?:let g:detectindent_preferred_expandtab = 1
> "
> " ? ? ? ? ? ? ? ?" to set a preferred indent level when detection is
> " ? ? ? ? ? ? ? ?" impossible:
> " ? ? ? ? ? ? ? ?:let g:detectindent_preferred_indent = 4
> "
> " Requirements: ?Untested on Vim versions below 6.2
>
> fun! <SID>IsCommentStart(line)
> ? ?" &comments isn't reliable
> ? ?if &ft == "c" || &ft == "cpp"
> ? ? ? ?return -1 != match(a:line, '/\*')
> ? ?else
> ? ? ? ?return 0
> ? ?endif
> endfun
>
> fun! <SID>IsCommentEnd(line)
> ? ?if &ft == "c" || &ft == "cpp"
> ? ? ? ?return -1 != match(a:line, '\*/')
> ? ?else
> ? ? ? ?return 0
> ? ?endif
> endfun
>
> fun! <SID>DetectIndent()
> ? ?let l:has_leading_tabs ? ? ? ? ? ?= 0
> ? ?let l:has_leading_spaces ? ? ? ? ?= 0
> ? ?let l:shortest_leading_spaces_run = 0
> ? ?let l:longest_leading_spaces_run ?= 0
>
> ? ?let l:idx_end = line("$")
> ? ?let l:idx = 1
> ? ?while l:idx <= l:idx_end
> ? ? ? ?let l:line = getline(l:idx)
>
> ? ? ? ?" try to skip over comment blocks, they can give really screwy indent
> ? ? ? ?" settings in c/c++ files especially
> ? ? ? ?if <SID>IsCommentStart(l:line)
> ? ? ? ? ? ?while l:idx <= l:idx_end && ! <SID>IsCommentEnd(l:line)
> ? ? ? ? ? ? ? ?let l:line = getline(l:idx)
> ? ? ? ? ? ? ? ?let l:idx = l:idx + 1
> ? ? ? ? ? ?endwhile
> ? ? ? ? ? ?let l:idx = l:idx + 1
> ? ? ? ? ? ?continue
> ? ? ? ?endif
>
> ? ? ? ?let l:leading_char = strpart(l:line, 0, 1)
>
> ? ? ? ?if l:leading_char == "\t"
> ? ? ? ? ? ?let l:has_leading_tabs = 1
>
> ? ? ? ?elseif l:leading_char == " "
> ? ? ? ? ? ?" only interested if we don't have a run of spaces followed by a
> ? ? ? ? ? ?" tab.
> ? ? ? ? ? ?if -1 == match(l:line, '^ \+\t')
> ? ? ? ? ? ? ? ?let l:has_leading_spaces = 1
> ? ? ? ? ? ? ? ?let l:spaces = strlen(matchstr(l:line, '^ \+'))
> ? ? ? ? ? ? ? ?if l:shortest_leading_spaces_run == 0 ||
> ? ? ? ? ? ? ? ? ? ? ? ? ? ?\ l:spaces < l:shortest_leading_spaces_run
> ? ? ? ? ? ? ? ? ? ?let l:shortest_leading_spaces_run = l:spaces
> ? ? ? ? ? ? ? ?endif
> ? ? ? ? ? ? ? ?if l:spaces > l:longest_leading_spaces_run
> ? ? ? ? ? ? ? ? ? ?let l:longest_leading_spaces_run = l:spaces
> ? ? ? ? ? ? ? ?endif
> ? ? ? ? ? ?endif
>
> ? ? ? ?endif
>
> ? ? ? ?let l:idx = l:idx + 1
> ? ?endwhile
>
> ? ?if l:has_leading_tabs && ! l:has_leading_spaces
> ? ? ? ?" tabs only, no spaces
> ? ? ? ?set noexpandtab
> ? ? ? ?if exists("g:detectindent_preferred_tabsize")
> ? ? ? ? ? ?let &shiftwidth ?= g:detectindent_preferred_indent
> ? ? ? ? ? ?let &tabstop ? ? = g:detectindent_preferred_indent
> ? ? ? ?endif
>
> ? ?elseif l:has_leading_spaces && ! l:has_leading_tabs
> ? ? ? ?" spaces only, no tabs
> ? ? ? ?set expandtab
> ? ? ? ?let &shiftwidth ?= l:shortest_leading_spaces_run
>
> ? ?elseif l:has_leading_spaces && l:has_leading_tabs
> ? ? ? ?" spaces and tabs
> ? ? ? ?set noexpandtab
> ? ? ? ?let &shiftwidth = l:shortest_leading_spaces_run
>
> ? ? ? ?" mmmm, time to guess how big tabs are
> ? ? ? ?if l:longest_leading_spaces_run < 2
> ? ? ? ? ? ?let &tabstop = 2
> ? ? ? ?elseif l:longest_leading_spaces_run < 4
> ? ? ? ? ? ?let &tabstop = 4
> ? ? ? ?else
> ? ? ? ? ? ?let &tabstop = 8
> ? ? ? ?endif
>
> ? ?else
> ? ? ? ?" no spaces, no tabs
> ? ? ? ?if exists("g:detectindent_preferred_tabsize")
> ? ? ? ? ? ?let &shiftwidth ?= g:detectindent_preferred_indent
> ? ? ? ? ? ?let &tabstop ? ? = g:detectindent_preferred_indent
> ? ? ? ?endif
> ? ? ? ?if exists("g:detectindent_preferred_expandtab")
> ? ? ? ? ? ?set expandtab
> ? ? ? ?endif
>
> ? ?endif
> endfun
>
> command! -nargs=0 DetectIndent call <SID>DetectIndent()
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>

From martin at v.loewis.de  Thu May 13 20:41:45 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Thu, 13 May 2010 20:41:45 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <81551.1273772543@parc.com>
References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de>
	<81551.1273772543@parc.com>
Message-ID: <4BEC47E9.20804@v.loewis.de>

Bill Janssen wrote:
> Martin v. L?wis <martin at v.loewis.de> wrote:
> 
>> Bill Janssen wrote:
>>> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
>>> which makes sense, because it's running as a background twisted process,
>>> and thus can't access the window server.
>> It doesn't really make sense. It should skip the test, instead of
>> failing it. I.e. aborting the Python process is definitely not a good
>> response.
> 
> Yes, you're right.  It's a bug in the test.

No, I'd say it's even deeper, in the Tcl integration.

There shouldn't be a way to cause an interpreter abort, except by
calling os.abort().

Regards,
Martin

From ronaldoussoren at mac.com  Thu May 13 20:50:24 2010
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 13 May 2010 20:50:24 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <4BEC47E9.20804@v.loewis.de>
References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de>
	<81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de>
Message-ID: <05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com>


On 13 May, 2010, at 20:41, Martin v. L?wis wrote:

> Bill Janssen wrote:
>> Martin v. L?wis <martin at v.loewis.de> wrote:
>> 
>>> Bill Janssen wrote:
>>>> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
>>>> which makes sense, because it's running as a background twisted process,
>>>> and thus can't access the window server.
>>> It doesn't really make sense. It should skip the test, instead of
>>> failing it. I.e. aborting the Python process is definitely not a good
>>> response.
>> 
>> Yes, you're right.  It's a bug in the test.
> 
> No, I'd say it's even deeper, in the Tcl integration.
> 
> There shouldn't be a way to cause an interpreter abort, except by
> calling os.abort().

This is a bug in Tk: 

>>> root = Tkinter.Tk()
Thu May 13 20:45:13 Rivendell.local python[84887] <Error>: kCGErrorFailure: Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are logged.
_RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>>> 2010-05-13 20:45:16.751 Python[84887:d07] An uncaught exception was raised
2010-05-13 20:45:16.762 Python[84887:d07] Error (1002) creating CGSWindow
2010-05-13 20:45:16.955 Python[84887:d07] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Error (1002) creating CGSWindow'
*** Call stack at first throw:
(
	0   CoreFoundation                      0x00007fff85e31d24 __exceptionPreprocess + 180
	1   libobjc.A.dylib                     0x00007fff860000f3 objc_exception_throw + 45
	2   CoreFoundation                      0x00007fff85e31b47 +[NSException raise:format:arguments:] + 103
	3   CoreFoundation                      0x00007fff85e31ad4 +[NSException raise:format:] + 148
	4   AppKit                              0x00007fff84614aba _NSCreateWindowWithOpaqueShape2 + 473
	5   AppKit                              0x00007fff845a9055 -[NSWindow _commonAwake] + 1214
	6   AppKit                              0x00007fff845c6d3d -[NSWindow _makeKeyRegardlessOfVisibility] + 96
	7   AppKit                              0x00007fff845c6cb2 -[NSWindow makeKeyAndOrderFront:] + 24
	8   Tk                                  0x000000010075b86c XMapWindow + 155
	9   Tk                                  0x00000001006ca6d0 Tk_MapWindow + 89
	10  Tk                                  0x00000001006d35e6 TkToplevelWindowForCommand + 2658
	11  Tcl                                 0x00000001006300d3 TclServiceIdle + 76
	12  Tcl                                 0x00000001006162ce Tcl_DoOneEvent + 329
	13  _tkinter.so                         0x0000000100595683 Tkapp_CallDeallocArgs + 277
	14  readline.so                         0x00000001001f1f9a initreadline + 1280
	15  Python                              0x00000001000076a1 PyOS_Readline + 239
	16  Python                              0x0000000100008a57 PyTokenizer_FromString + 1322
	17  Python                              0x00000001000090a0 PyTokenizer_Get + 154
	18  Python                              0x0000000100005698 PyParser_AddToken + 1018
	19  Python                              0x00000001000a2320 PyParser_ASTFromFile + 146
	20  Python                              0x00000001000a443f PyRun_InteractiveOneFlags + 345
	21  Python                              0x00000001000a4615 PyRun_InteractiveLoopFlags + 206
	22  Python                              0x00000001000a4685 PyRun_AnyFileExFlags + 76
	23  Python                              0x00000001000b0286 Py_Main + 2718
	24  Python                              0x0000000100000e6c start + 52
	25  ???                                 0x0000000000000001 0x0 + 1
)
terminate called after throwing an instance of 'NSException'
Abort trap


This is running /usr/bin/python in a session as a user that doesn't have access to the GUI.  The text above says that there is an uncaught ObjC exception, caused by the lack of a connection to the window server. Tk should have converted that to its own style of errors but didn't.

Bill: could you please file an issue for this in the python tracker, it should be possible to add a workaround for this to the Tkinter extension.

Ronald

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3567 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100513/a4698956/attachment-0001.bin>

From martin at v.loewis.de  Thu May 13 21:10:00 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 13 May 2010 21:10:00 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com>
References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de>
	<81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de>
	<05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com>
Message-ID: <4BEC4E88.6070705@v.loewis.de>


> This is running /usr/bin/python in a session as a user that doesn't
> have access to the GUI.  The text above says that there is an
> uncaught ObjC exception, caused by the lack of a connection to the
> window server. Tk should have converted that to its own style of
> errors but didn't.

That makes sense to me; thanks for investigating it.

If we are kind, we could also file a Tk bug, then.

Regards,
Martin

From janssen at parc.com  Thu May 13 22:44:11 2010
From: janssen at parc.com (Bill Janssen)
Date: Thu, 13 May 2010 13:44:11 PDT
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <4BEC47E9.20804@v.loewis.de>
References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de>
	<81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de>
Message-ID: <83788.1273783451@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> Bill Janssen wrote:
> > Martin v. L?wis <martin at v.loewis.de> wrote:
> > 
> >> Bill Janssen wrote:
> >>> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
> >>> which makes sense, because it's running as a background twisted process,
> >>> and thus can't access the window server.
> >> It doesn't really make sense. It should skip the test, instead of
> >> failing it. I.e. aborting the Python process is definitely not a good
> >> response.
> > 
> > Yes, you're right.  It's a bug in the test.
> 
> No, I'd say it's even deeper, in the Tcl integration.
> 
> There shouldn't be a way to cause an interpreter abort, except by
> calling os.abort().

There are some cases where the OS X security mechanism interprets an
invalid attempt to connect to the window server as a privilege
violation, and just terminates the process.  I've seen that before.

This is basically a resource issue in the testing framework, too, like
having network access.  The problem is, for lot of external libraries,
it's not clear just what resources they assume are available.  If we
believe that Tk doesn't intend to require access to the window server,
it's a Tk bug.  If we know that it does, it's a resource issue.

Bill

From janssen at parc.com  Thu May 13 23:01:15 2010
From: janssen at parc.com (Bill Janssen)
Date: Thu, 13 May 2010 14:01:15 PDT
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <4BEC47E9.20804@v.loewis.de>
References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de>
	<81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de>
Message-ID: <83959.1273784475@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> Bill Janssen wrote:
> > Martin v. L?wis <martin at v.loewis.de> wrote:
> > 
> >> Bill Janssen wrote:
> >>> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
> >>> which makes sense, because it's running as a background twisted process,
> >>> and thus can't access the window server.
> >> It doesn't really make sense. It should skip the test, instead of
> >> failing it. I.e. aborting the Python process is definitely not a good
> >> response.
> > 
> > Yes, you're right.  It's a bug in the test.
> 
> No, I'd say it's even deeper, in the Tcl integration.
> 
> There shouldn't be a way to cause an interpreter abort, except by
> calling os.abort().

Yes, I agree.  It's an undesirable design bug in the Apple OS, IMO.
When some Apple libraries ask for some things that they're not allowed
to have, the library calls abort() instead of signalling an error.

Google for "FAILED TO GET ASN FROM CORESERVICES" sometime.

Bill

From ncoghlan at gmail.com  Thu May 13 23:44:07 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 14 May 2010 07:44:07 +1000
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <4BEC0FCA.40601@v.loewis.de>
References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com>
	<4BEC0FCA.40601@v.loewis.de>
Message-ID: <4BEC72A7.2020107@gmail.com>

Martin v. L?wis wrote:
>>> I've got parc-tiger-1 up and running again.  It's failing on test_tk,
>>> which makes sense, because it's running as a background twisted process,
>>> and thus can't access the window server.  I should configure that out.
>>>
>>> I'm looking for documentation on how to configure the build slave so
>>> that it skips this test.
>> It may be better to try to detect the "no window server" case and skip
>> it in the test itself rather than in the build slave configuration.
> 
> Even better would be if Python wouldn't crash when you try to run Tk
> commands without a window server. Instead of aborting Python, that
> should raise an exception (which can then be detected as a test skip).

Yes, when I commented I didn't realise that "failing" in this case
actually meant "crashing" :P

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From urban.dani+py at gmail.com  Fri May 14 06:39:12 2010
From: urban.dani+py at gmail.com (Daniel Urban)
Date: Fri, 14 May 2010 06:39:12 +0200
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <AANLkTikYXw5k67-HO4J-K0ylgrsK7U2ATRELMeG6luEV@mail.gmail.com>
References: <hs18jc$1mh$1@dough.gmane.org>
	<AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com> 
	<AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com> 
	<201005132130.30619.steve@pearwood.info>
	<AANLkTikYXw5k67-HO4J-K0ylgrsK7U2ATRELMeG6luEV@mail.gmail.com>
Message-ID: <AANLkTilzNIu-8h4xbtS4BpsgIqlGFdkPGrgD0pQ6ao9n@mail.gmail.com>

I wrote an e-mail yesterday, but it seems, it didn't reach python-dev.
Here it is again:

On Thu, May 13, 2010 at 13:30, Steven D'Aprano <steve at pearwood.info> wrote:
> I'd support an immutable dict. partial objects already impose a
> significant (~ 30%) performance penalty:
>
>>>> from timeit import Timer
>>>> min(Timer('f(5)', 'f = lambda x: x').repeat())
> 0.93580079078674316
>>>> min(Timer('p(5)', 'from functools import partial; p = partial(lambda
> x: x)').repeat())
> 1.2715129852294922
>
> No need to make that worse if that can be avoided.

I've made a new patch, in which the keywords attribute is a read-only
proxy of the dictionary. I've used your benchmark, and I haven't found
any significant difference in execution times.
The patch is in the tracker (http://bugs.python.org/issue8699) and
Rietveld (http://codereview.appspot.com/1179044).


Daniel Urban

From vincent at vincentdavis.net  Fri May 14 06:47:18 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Thu, 13 May 2010 22:47:18 -0600
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <4BEC72A7.2020107@gmail.com>
References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com>
	<4BEC0FCA.40601@v.loewis.de> <4BEC72A7.2020107@gmail.com>
Message-ID: <AANLkTikYiCMPAdxadB87vkYprYJnjt29On9RmjQB5pvi@mail.gmail.com>

Not to interrupt you  you conversation but I am interested in setting
up a buildbot on one of my Macs. Is there any documentations or advise
that is different from that of a linux machine?  Any advise would be
appreciated.
Thanks
Vincent


On Thu, May 13, 2010 at 3:44 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Martin v. L?wis wrote:
>>>> I've got parc-tiger-1 up and running again. ?It's failing on test_tk,
>>>> which makes sense, because it's running as a background twisted process,
>>>> and thus can't access the window server. ?I should configure that out.
>>>>
>>>> I'm looking for documentation on how to configure the build slave so
>>>> that it skips this test.
>>> It may be better to try to detect the "no window server" case and skip
>>> it in the test itself rather than in the build slave configuration.
>>
>> Even better would be if Python wouldn't crash when you try to run Tk
>> commands without a window server. Instead of aborting Python, that
>> should raise an exception (which can then be detected as a test skip).
>
> Yes, when I commented I didn't realise that "failing" in this case
> actually meant "crashing" :P
>
> Regards,
> Nick.
>
> --
> Nick Coghlan ? | ? ncoghlan at gmail.com ? | ? Brisbane, Australia
> ---------------------------------------------------------------
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/vincent%40vincentdavis.net
>

From martin at v.loewis.de  Fri May 14 09:28:29 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 14 May 2010 09:28:29 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <AANLkTikYiCMPAdxadB87vkYprYJnjt29On9RmjQB5pvi@mail.gmail.com>
References: <84766.1273720679@parc.com>
	<4BEBD6D1.8020702@gmail.com>	<4BEC0FCA.40601@v.loewis.de>
	<4BEC72A7.2020107@gmail.com>
	<AANLkTikYiCMPAdxadB87vkYprYJnjt29On9RmjQB5pvi@mail.gmail.com>
Message-ID: <4BECFB9D.8060502@v.loewis.de>

Vincent Davis wrote:
> Not to interrupt you  you conversation but I am interested in setting
> up a buildbot on one of my Macs. Is there any documentations or advise
> that is different from that of a linux machine?  Any advise would be
> appreciated.

This is a little bit out of context: what exactly do you want to set up?
A buildbot master, or a buildbot slave? For running what tests?

Buildbot is available from http://buildbot.net/trac

Regards,
Martin

From status at bugs.python.org  Fri May 14 18:07:37 2010
From: status at bugs.python.org (Python tracker)
Date: Fri, 14 May 2010 18:07:37 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20100514160737.2E53C7814D@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2010-05-07 - 2010-05-14)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2678 open (+43) / 17853 closed (+23) / 20531 total (+66)

Open issues with patches:  1090

Average duration of open issues: 722 days.
Median duration of open issues: 495 days.

Open Issues Breakdown
       open  2658 (+43)
languishing    12 ( +0)
    pending     7 ( +0)

Issues Created Or Reopened (72)
_______________________________

__import__ with fromlist=                                      2010-05-08
       http://bugs.python.org/issue2090    reopened brett.cannon                         
       patch                                                                   

SSL handshake fails after TCP connection in getpeername()      2010-05-07
CLOSED http://bugs.python.org/issue4171    reopened ddvoinikov                           
                                                                               

A number tests "crash" if python is compiled --without-threads 2010-05-13
       http://bugs.python.org/issue7449    reopened skrah                                
       patch, easy, needs review                                               

pep-0370 on osx duplicates existing functionality              2010-05-08
       http://bugs.python.org/issue8084    reopened mark.dickinson                       
       patch, needs review, buildbot                                           

tiger buildbot: test_abspath_issue3426 failure (test_genericpa 2010-05-13
CLOSED http://bugs.python.org/issue8422    reopened flox                                 
       buildbot                                                                

PyUnicode_GetMax is undocumented                               2010-05-07
       http://bugs.python.org/issue8647    created  stutzbach                            
                                                                               

The UTF-7 codec functions are undocumented                     2010-05-07
       http://bugs.python.org/issue8648    created  stutzbach                            
                                                                               

Py_UNICODE_* functions are undocumented                        2010-05-07
       http://bugs.python.org/issue8649    created  stutzbach                            
                                                                               

zlibmodule.c isn't 64-bit clean                                2010-05-07
       http://bugs.python.org/issue8650    created  pitrou                               
                                                                               

"s#" and friends can silently truncate buffer length           2010-05-07
       http://bugs.python.org/issue8651    created  pitrou                               
                                                                               

Minor improvements to the "Handling Exceptions" part of the tu 2010-05-07
       http://bugs.python.org/issue8652    created  marienz                              
                                                                               

urlparse.urlparse/urlsplit doc missing                         2010-05-07
       http://bugs.python.org/issue8653    created  dabrahams                            
                                                                               

Improve ABI compatibility between UCS2 and UCS4 builds         2010-05-07
       http://bugs.python.org/issue8654    created  stutzbach                            
       patch                                                                   

Problem with getpeercert in the ssl module when retrieving cli 2010-05-07
       http://bugs.python.org/issue8655    created  Westly.Ward                          
                                                                               

urllib2 mangles file://-scheme URLs                            2010-05-08
CLOSED http://bugs.python.org/issue8656    created  dabrahams                            
                                                                               

urlparse.urlunsplit should be smarter about +                  2010-05-08
CLOSED http://bugs.python.org/issue8657    created  dabrahams                            
       patch                                                                   

urlparse.urlunsplit should be smarter about + and file urls    2010-05-08
CLOSED http://bugs.python.org/issue8658    created  orsenthil                            
       patch                                                                   

ABS(x) == 0 can be simplified to x == 0                        2010-05-08
CLOSED http://bugs.python.org/issue8659    created  stutzbach                            
       patch, needs review                                                     

py3k weakref documentation mentions the long built-in type     2010-05-08
CLOSED http://bugs.python.org/issue8660    created  durban                               
                                                                               

FAQ item 2.25 is unclear                                       2010-05-08
       http://bugs.python.org/issue8661    created  exarkun                              
                                                                               

bytes join cannot join bytes                                   2010-05-08
CLOSED http://bugs.python.org/issue8662    created  perey                                
                                                                               

Failed compile in a non-ASCII path                             2010-05-08
       http://bugs.python.org/issue8663    created  pitrou                               
       patch                                                                   

py_compile.compile() should consistently create parent directo 2010-05-08
CLOSED http://bugs.python.org/issue8664    created  Arfrever                             
       patch                                                                   

"make pycremoval" fails                                        2010-05-08
       http://bugs.python.org/issue8665    created  pitrou                               
                                                                               

Allow ConfigParser.get*() to take a default value              2010-05-08
       http://bugs.python.org/issue8666    created  Gumnos                               
       patch                                                                   

Link to PEP 3147 from importlib docs                           2010-05-08
       http://bugs.python.org/issue8667    created  brett.cannon                         
       easy                                                                    

add a 'develop' command                                        2010-05-09
       http://bugs.python.org/issue8668    created  meatballhat                          
                                                                               

lack of bdist_rpm module raises error on 'setup.py --help-comm 2010-05-09
       http://bugs.python.org/issue8669    created  meatballhat                          
                                                                               

c_types.c_wchar should not assume that sizeof(wchar_t) == size 2010-05-09
       http://bugs.python.org/issue8670    created  stutzbach                            
       patch                                                                   

A small erorr on http://docs.python.org/library/re.html        2010-05-09
CLOSED http://bugs.python.org/issue8671    created  bones7456                            
                                                                               

Error decompressing valid zlib data                            2010-05-09
CLOSED http://bugs.python.org/issue8672    created  matthew.brett                        
       patch                                                                   

configure script doesn't recognize 10.5 SDK correctly          2010-05-10
CLOSED http://bugs.python.org/issue8673    created  ronaldoussoren                       
                                                                               

audioop: incorrect integer overflow checks                     2010-05-10
CLOSED http://bugs.python.org/issue8674    created  thoger                               
       patch                                                                   

audioop module needs an int -> Py_ssize_t upgrade              2010-05-10
CLOSED http://bugs.python.org/issue8675    created  mark.dickinson                       
       patch, easy                                                             

Py3k Built-in Exceptions documentation mentions the raise stat 2010-05-10
CLOSED http://bugs.python.org/issue8676    created  durban                               
                                                                               

Modules needing PY_SSIZE_T_CLEAN                               2010-05-10
       http://bugs.python.org/issue8677    created  pitrou                               
                                                                               

crashers in rgbimg                                             2010-05-10
       http://bugs.python.org/issue8678    created  brett.cannon                         
                                                                               

write a distutils to distutils2 converter                      2010-05-10
       http://bugs.python.org/issue8679    created  tarek                                
                                                                               

Add a sandbox in Distutils2                                    2010-05-10
       http://bugs.python.org/issue8680    created  tarek                                
                                                                               

Make the zlib module emit more detailed error messages         2010-05-10
CLOSED http://bugs.python.org/issue8681    created  pitrou                               
       patch                                                                   

_ssl.c uses PyWeakref_GetObject but doesn't incref result      2010-05-10
       http://bugs.python.org/issue8682    created  pitrou                               
                                                                               

HPUX Segmentation fault in Modules/gcmodule.c -- if (!gc_list_ 2010-05-11
CLOSED http://bugs.python.org/issue8683    created  srid                                 
                                                                               

improvements to sched.py                                       2010-05-11
       http://bugs.python.org/issue8684    created  josiahcarlson                        
       patch, patch, needs review                                              

set(range(100000)).difference(set()) is slow                   2010-05-11
       http://bugs.python.org/issue8685    created  spiv                                 
       patch                                                                   

"This isn't defined beyond that" phrase is not friendly to non 2010-05-11
       http://bugs.python.org/issue8686    created  naoki                                
                                                                               

sched.py module doesn't have a test suite                      2010-05-11
       http://bugs.python.org/issue8687    created  giampaolo.rodola                     
       patch                                                                   

distutils sdist is too laze w.r.t. recalculating MANIFEST      2010-05-11
       http://bugs.python.org/issue8688    created  ronaldoussoren                       
                                                                               

sqlite3 parameter substitution breaks with multiple parameters 2010-05-11
       http://bugs.python.org/issue8689    created  azriphale                            
                                                                               

multiprocessing.dummy.Queue does not expose same interface as  2010-05-11
       http://bugs.python.org/issue8690    created  Jason.Baker                          
                                                                               

Doc: left alignment is not the default for numbers             2010-05-11
       http://bugs.python.org/issue8691    created  tjreedy                              
                                                                               

Use divide-and-conquer for faster factorials                   2010-05-11
       http://bugs.python.org/issue8692    created  stutzbach                            
       patch, patch, needs review                                              

Py_Initialize: can't initialize sys standard streams           2010-05-12
CLOSED http://bugs.python.org/issue8693    reopened mbnoimi                              
                                                                               

python3 FAQ mentions unicode()                                 2010-05-12
       http://bugs.python.org/issue8694    created  ohnobinki                            
                                                                               

Issue while installing Python 2.6.5 in IBM AIX 6.1             2010-05-12
       http://bugs.python.org/issue8695    created  senthil_l                            
                                                                               

PEP 391: Adding documentation of logging.config.dictConfig     2010-05-12
CLOSED http://bugs.python.org/issue8696    created  akuchling                            
       patch                                                                   

PEP 391: Adding documentation of logging.config.dictConfig     2010-05-12
CLOSED http://bugs.python.org/issue8697    created  akuchling                            
       patch, needs review                                                     

PEP 391: Adding documentation of logging.config.dictConfig     2010-05-12
CLOSED http://bugs.python.org/issue8698    created  akuchling                            
       patch, needs review                                                     

Equality and hashing for functools.partial                     2010-05-12
       http://bugs.python.org/issue8699    created  durban                               
       patch                                                                   

strip() is removing an extra character if the strip pattern co 2010-05-12
CLOSED http://bugs.python.org/issue8700    created  Balachander.Ganesan                  
                                                                               

tarfile: first character of member names doubled               2010-05-12
CLOSED http://bugs.python.org/issue8701    created  pooryorick                           
                                                                               

difflib: unified_diff produces wrong patches (again)           2010-05-13
CLOSED http://bugs.python.org/issue8702    created  techtonik                            
       easy                                                                    

Py3k PyList_Type documentation mentions types.ListType         2010-05-13
CLOSED http://bugs.python.org/issue8703    created  durban                               
                                                                               

cgitb sends a bogus HTTP header if the app crashes before fini 2010-05-13
       http://bugs.python.org/issue8704    created  stutzbach                            
       easy                                                                    

shutil.rmtree with empty filepath                              2010-05-13
       http://bugs.python.org/issue8705    created  dbkoch                               
                                                                               

accept keyword arguments on all base type methods and builtins 2010-05-13
       http://bugs.python.org/issue8706    created  gregory.p.smith                      
                                                                               

Duplicated document in telnetlib.                              2010-05-13
       http://bugs.python.org/issue8707    created  naoki                                
                                                                               

OpenID blunder                                                 2010-05-14
       http://bugs.python.org/issue8708    created  Jack.Diederich                       
                                                                               

mention explicitly Windows support for os.devnull              2010-05-14
       http://bugs.python.org/issue8709    created  akira                                
       patch                                                                   

test_xpickle: compat tests: workaround for missing test_suppor 2010-05-14
       http://bugs.python.org/issue8710    created  skrah                                
                                                                               

Document	PyUnicode_DecodeFSDefault()	and	PyUnicode_DecodeFSDef 2010-05-14
       http://bugs.python.org/issue8711    created  haypo                                
       patch                                                                   

Skip libpthread related test failures on OpenBSD               2010-05-14
       http://bugs.python.org/issue8712    created  skrah                                
       patch, needs review                                                     

bdist_deb - Debian packager                                    2010-05-07
CLOSED http://bugs.python.org/issue1054967 reopened merwok                               
       patch                                                                   



Issues Now Closed (54)
______________________

cmath is numerically unsound                                    920 days
       http://bugs.python.org/issue1381    bins                                 
       patch                                                                   

test_uuid is warning about unreliable functions                 904 days
       http://bugs.python.org/issue1481    skrah                                
       patch                                                                   

Add a factorial function                                        786 days
       http://bugs.python.org/issue2138    mark.dickinson                       
                                                                               

Py_FatalError causes infinite loop                              632 days
       http://bugs.python.org/issue3605    jyasskin                             
       needs review                                                            

SSL handshake fails after TCP connection in getpeername()         1 days
       http://bugs.python.org/issue4171    pitrou                               
                                                                               

{context,unified}_diff add spurious trailing whitespace if fro  490 days
       http://bugs.python.org/issue4898    r.david.murray                       
       patch                                                                   

Implement PEP 376                                               484 days
       http://bugs.python.org/issue4908    tarek                                
       patch                                                                   

subprocess.Popen.__del__ causes AttributeError (os module == N  469 days
       http://bugs.python.org/issue5099    brett.cannon                         
       patch, needs review                                                     

test_smtplib fails on os x 10.6                                 220 days
       http://bugs.python.org/issue7040    giampaolo.rodola                     
                                                                               

difflib should separate filename from timestamp with tab        137 days
       http://bugs.python.org/issue7585    techtonik                            
       patch                                                                   

unittest: allow an 'import_path' as an alternative to 'top_lev  103 days
       http://bugs.python.org/issue7780    michael.foord                        
                                                                               

Provide unittest.TestCase.assertNotRegexpMatches                 67 days
       http://bugs.python.org/issue8038    michael.foord                        
       patch                                                                   

Python 3.1.2rc1 doesn't compile using the 10.4 sdk on a	10.6 M   62 days
       http://bugs.python.org/issue8126    ronaldoussoren                       
                                                                               

Putting a function in a unittest.TestSuite doesn't work          35 days
       http://bugs.python.org/issue8301    michael.foord                        
                                                                               

python -m unittest -h and python -m unittest discover -h messa   37 days
       http://bugs.python.org/issue8303    michael.foord                        
                                                                               

siginterrupt with flag=False is reset when signal received       30 days
       http://bugs.python.org/issue8354    exarkun                              
       patch, needs review                                                     

tiger buildbot: test_abspath_issue3426 failure (test_genericpa    0 days
       http://bugs.python.org/issue8422    haypo                                
       buildbot                                                                

openssl version detection doesn't work properly when using OSX   20 days
       http://bugs.python.org/issue8444    ronaldoussoren                       
                                                                               

asyncore test suite                                              19 days
       http://bugs.python.org/issue8490    giampaolo.rodola                     
       patch, patch                                                            

update to autoconf2.65                                           18 days
       http://bugs.python.org/issue8510    mark.dickinson                       
       patch, needs review                                                     

Add fsencode() functions to os module                            15 days
       http://bugs.python.org/issue8514    haypo                                
       patch                                                                   

unittest test discovery can fail when package under test is al   11 days
       http://bugs.python.org/issue8547    michael.foord                        
                                                                               

zlib causes a SystemError when decompressing a chunk >1GB         8 days
       http://bugs.python.org/issue8571    pitrou                               
       patch                                                                   

Update/reorganize _winreg documentation                          12 days
       http://bugs.python.org/issue8575    brian.curtin                         
       patch, patch, needs review                                              

test_urllib2.py test failures on Py3K Mac OS X                    8 days
       http://bugs.python.org/issue8588    mark.dickinson                       
                                                                               

enumerate() test cases do not cover optional start argument       2 days
       http://bugs.python.org/issue8636    benjamin.peterson                    
       patch                                                                   

json.loads description                                            3 days
       http://bugs.python.org/issue8642    georg.brandl                         
       patch                                                                   

timedelta.total_seconds needlessly inaccurate, especially for     2 days
       http://bugs.python.org/issue8644    mark.dickinson                       
       patch                                                                   

urllib2 mangles file://-scheme URLs                               0 days
       http://bugs.python.org/issue8656    orsenthil                            
                                                                               

urlparse.urlunsplit should be smarter about +                     5 days
       http://bugs.python.org/issue8657    orsenthil                            
       patch                                                                   

urlparse.urlunsplit should be smarter about + and file urls       0 days
       http://bugs.python.org/issue8658    orsenthil                            
       patch                                                                   

ABS(x) == 0 can be simplified to x == 0                           0 days
       http://bugs.python.org/issue8659    mark.dickinson                       
       patch, needs review                                                     

py3k weakref documentation mentions the long built-in type        0 days
       http://bugs.python.org/issue8660    benjamin.peterson                    
                                                                               

bytes join cannot join bytes                                      0 days
       http://bugs.python.org/issue8662    perey                                
                                                                               

py_compile.compile() should consistently create parent directo    0 days
       http://bugs.python.org/issue8664    benjamin.peterson                    
       patch                                                                   

A small erorr on http://docs.python.org/library/re.html           0 days
       http://bugs.python.org/issue8671    eric.smith                           
                                                                               

Error decompressing valid zlib data                               2 days
       http://bugs.python.org/issue8672    pitrou                               
       patch                                                                   

configure script doesn't recognize 10.5 SDK correctly             3 days
       http://bugs.python.org/issue8673    ronaldoussoren                       
                                                                               

audioop: incorrect integer overflow checks                        1 days
       http://bugs.python.org/issue8674    mark.dickinson                       
       patch                                                                   

audioop module needs an int -> Py_ssize_t upgrade                 1 days
       http://bugs.python.org/issue8675    mark.dickinson                       
       patch, easy                                                             

Py3k Built-in Exceptions documentation mentions the raise stat    0 days
       http://bugs.python.org/issue8676    benjamin.peterson                    
                                                                               

Make the zlib module emit more detailed error messages            1 days
       http://bugs.python.org/issue8681    pitrou                               
       patch                                                                   

HPUX Segmentation fault in Modules/gcmodule.c -- if (!gc_list_    1 days
       http://bugs.python.org/issue8683    pitrou                               
                                                                               

Py_Initialize: can't initialize sys standard streams              0 days
       http://bugs.python.org/issue8693    mbnoimi                              
                                                                               

PEP 391: Adding documentation of logging.config.dictConfig        0 days
       http://bugs.python.org/issue8696    akuchling                            
       patch                                                                   

PEP 391: Adding documentation of logging.config.dictConfig        0 days
       http://bugs.python.org/issue8697    akuchling                            
       patch, needs review                                                     

PEP 391: Adding documentation of logging.config.dictConfig        1 days
       http://bugs.python.org/issue8698    meatballhat                          
       patch, needs review                                                     

strip() is removing an extra character if the strip pattern co    0 days
       http://bugs.python.org/issue8700    r.david.murray                       
                                                                               

tarfile: first character of member names doubled                  1 days
       http://bugs.python.org/issue8701    pooryorick                           
                                                                               

difflib: unified_diff produces wrong patches (again)              0 days
       http://bugs.python.org/issue8702    mark.dickinson                       
       easy                                                                    

Py3k PyList_Type documentation mentions types.ListType            0 days
       http://bugs.python.org/issue8703    benjamin.peterson                    
                                                                               

Carbon Event ReceiveNextEvent                                  2479 days
       http://bugs.python.org/issue779285  tjreedy                              
                                                                               

bdist_deb - Debian packager                                       0 days
       http://bugs.python.org/issue1054967 tarek                                
       patch                                                                   

update urlparse to RFC 3986                                    1280 days
       http://bugs.python.org/issue1591035 orsenthil                            
       easy                                                                    



Top Issues Most Discussed (10)
______________________________

 45 Use divide-and-conquer for faster factorials                       3 days
open        http://bugs.python.org/issue8692   

 25 Improve ABI compatibility between UCS2 and UCS4 builds             7 days
open        http://bugs.python.org/issue8654   

 14 Failed compile in a non-ASCII path                                 6 days
open        http://bugs.python.org/issue8663   

 11 let's equip ftplib.FTP with __enter__ and __exit__               482 days
open        http://bugs.python.org/issue4972   

 10 add a distutils test command                                      39 days
open        http://bugs.python.org/issue8324   

 10 cookielib doesn't handle URLs with / in parameters               625 days
open        http://bugs.python.org/issue3704   

  9 Py_Initialize: can't initialize sys standard streams               0 days
closed      http://bugs.python.org/issue8693   

  9 timedelta.total_seconds needlessly inaccurate, especially for n    2 days
closed      http://bugs.python.org/issue8644   

  9 update to autoconf2.65                                            18 days
closed      http://bugs.python.org/issue8510   

  9 input() doesn't catch _PyUnicode_AsString() exception; io.Strin   47 days
open        http://bugs.python.org/issue8256   




From janssen at parc.com  Fri May 14 18:15:06 2010
From: janssen at parc.com (Bill Janssen)
Date: Fri, 14 May 2010 09:15:06 PDT
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <AANLkTikYiCMPAdxadB87vkYprYJnjt29On9RmjQB5pvi@mail.gmail.com>
References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com>
	<4BEC0FCA.40601@v.loewis.de> <4BEC72A7.2020107@gmail.com>
	<AANLkTikYiCMPAdxadB87vkYprYJnjt29On9RmjQB5pvi@mail.gmail.com>
Message-ID: <83407.1273853706@parc.com>

Vincent Davis <vincent at vincentdavis.net> wrote:

> Not to interrupt you  you conversation but I am interested in setting
> up a buildbot on one of my Macs. Is there any documentations or advise
> that is different from that of a linux machine?  Any advise would be
> appreciated.

Assuming you mean "set up a build slave for Python testing"...

Here's what I did.  I was installing on a Tiger machine, which comes
with Python 2.3.5, so the first thing I did was to install Xcode 2.5,
the latest release for Tiger.  I also needed Subversion, so I downloaded
and installed that.  Later versions of OS X Xcode include Subversion, so
you won't need to do it separately if you're using those.

I then installed Python 2.6.5 from python.org, which winds up in
/usr/local/bin, and modified my PATH to put /usr/local/bin on it.  I
downloaded the source bundles for Twisted, zope.interface, and buildbot,
and installed them in that order -- "sudo python setup.py install" works
fine for each of them.

After that, I followed the instructions on the Wiki at
http://wiki.python.org/moin/BuildBot.  Create a buildbot account, make
sure /usr/local/bin is on the PATH of the buildbot account, and issue
the commands shown there.

Bill

From vincent at vincentdavis.net  Fri May 14 18:28:12 2010
From: vincent at vincentdavis.net (Vincent Davis)
Date: Fri, 14 May 2010 10:28:12 -0600
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <83407.1273853706@parc.com>
References: <84766.1273720679@parc.com> <4BEBD6D1.8020702@gmail.com>
	<4BEC0FCA.40601@v.loewis.de> <4BEC72A7.2020107@gmail.com>
	<AANLkTikYiCMPAdxadB87vkYprYJnjt29On9RmjQB5pvi@mail.gmail.com>
	<83407.1273853706@parc.com>
Message-ID: <AANLkTimLHsmeOzXO-eoioq9wyPsPkwhK0J65ZUTg-YEC@mail.gmail.com>

On Fri, May 14, 2010 at 10:15 AM, Bill Janssen <janssen at parc.com> wrote:

> Vincent Davis <vincent at vincentdavis.net> wrote:
>
> > Not to interrupt you  you conversation but I am interested in setting
> > up a buildbot on one of my Macs. Is there any documentations or advise
> > that is different from that of a linux machine?  Any advise would be
> > appreciated.
>
> Assuming you mean "set up a build slave for Python testing"...
>
> Here's what I did.  I was installing on a Tiger machine, which comes
> with Python 2.3.5, so the first thing I did was to install Xcode 2.5,
> the latest release for Tiger.  I also needed Subversion, so I downloaded
> and installed that.  Later versions of OS X Xcode include Subversion, so
> you won't need to do it separately if you're using those.
>
> I then installed Python 2.6.5 from python.org, which winds up in
> /usr/local/bin, and modified my PATH to put /usr/local/bin on it.  I
> downloaded the source bundles for Twisted, zope.interface, and buildbot,
> and installed them in that order -- "sudo python setup.py install" works
> fine for each of them.
>
> After that, I followed the instructions on the Wiki at
> http://wiki.python.org/moin/BuildBot.  Create a buildbot account, make
> sure /usr/local/bin is on the PATH of the buildbot account, and issue
> the commands shown there.


Yes thanks this is what I was thinking "set up a build slave for Python
testing", No reason this would not work on a leopard 10.6 machine?

Thanks

*Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100514/f501ef1d/attachment.html>

From fijall at gmail.com  Fri May 14 19:03:49 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 14 May 2010 11:03:49 -0600
Subject: [Python-Dev] HEADS UP: Compilation risk with new GCC 4.5.0
In-Reply-To: <AANLkTimrb1UnMrW5ryn8_E53dy2vAhmNOvKUO1yR_C3L@mail.gmail.com>
References: <4BEAA971.3040001@jcea.es>
	<D0FCA42C-276C-4707-AC13-1851F5AAC18E@fuhm.net> 
	<AANLkTimrb1UnMrW5ryn8_E53dy2vAhmNOvKUO1yR_C3L@mail.gmail.com>
Message-ID: <AANLkTil8-etNmuGBZWbo-zG63Bn0-1CkGp9j1Q33pogT@mail.gmail.com>

On Wed, May 12, 2010 at 9:58 AM, Jeffrey Yasskin <jyasskin at gmail.com> wrote:
> On Wed, May 12, 2010 at 6:39 AM, James Y Knight <foom at fuhm.net> wrote:
>> I think you'll be a lot happier just modifying Psyco than making everyone
>> else in the world change their compiler flags.
>
> Aye, there's the rub. Nobody's happier modifying Psyco. :) ?But that
> just means people will gradually have to stop using psyco in favor of
> maintainable JITs. Gcc's not going to change its stack requirements
> just because some people think they know what the ABI should be better
> than the people defining the ABI. Btw, this has been a problem since
> at least gcc-4.4.

Heh.

psyco already does it for Mac OS X (which defines 16 byte stack
alignment as ABI), so should be super trivial. Good rant Jeffrey
though.

Cheers,
fijal

From martin at v.loewis.de  Fri May 14 19:56:29 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 14 May 2010 19:56:29 +0200
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <AANLkTimLHsmeOzXO-eoioq9wyPsPkwhK0J65ZUTg-YEC@mail.gmail.com>
References: <84766.1273720679@parc.com>
	<4BEBD6D1.8020702@gmail.com>	<4BEC0FCA.40601@v.loewis.de>
	<4BEC72A7.2020107@gmail.com>	<AANLkTikYiCMPAdxadB87vkYprYJnjt29On9RmjQB5pvi@mail.gmail.com>	<83407.1273853706@parc.com>
	<AANLkTimLHsmeOzXO-eoioq9wyPsPkwhK0J65ZUTg-YEC@mail.gmail.com>
Message-ID: <4BED8ECD.5000800@v.loewis.de>

> Yes thanks this is what I was thinking "set up a build slave for Python
> testing", No reason this would not work on a leopard 10.6 machine?

In my experience, it is mandatory that the slave admin has really good
understanding of Python, and of the operating system that the slave runs
on. Otherwise, the slave will be down most of the time, and just not
function correctly; it was then a waste of time to set it up in the
first place.

Regards,
Martin

From janssen at parc.com  Fri May 14 20:23:32 2010
From: janssen at parc.com (Bill Janssen)
Date: Fri, 14 May 2010 11:23:32 PDT
Subject: [Python-Dev] configuring the buildbot to skip some tests?
In-Reply-To: <05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com>
References: <84766.1273720679@parc.com> <4BEBA827.5040000@v.loewis.de>
	<81551.1273772543@parc.com> <4BEC47E9.20804@v.loewis.de>
	<05A4049A-BBDA-4149-AB65-6DD77716B8DA@mac.com>
Message-ID: <85682.1273861412@parc.com>

Ronald Oussoren <ronaldoussoren at mac.com> wrote:

> Bill: could you please file an issue for this in the python tracker,

http://bugs.python.org/issue8716

> it should be possible to add a workaround for this to the Tkinter
> extension.

That would be good.

Bill

From urban.dani at gmail.com  Thu May 13 18:48:15 2010
From: urban.dani at gmail.com (Daniel Urban)
Date: Thu, 13 May 2010 18:48:15 +0200
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <201005132130.30619.steve@pearwood.info>
References: <hs18jc$1mh$1@dough.gmane.org>
	<AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com> 
	<AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com> 
	<201005132130.30619.steve@pearwood.info>
Message-ID: <AANLkTikYXw5k67-HO4J-K0ylgrsK7U2ATRELMeG6luEV@mail.gmail.com>

On Thu, May 13, 2010 at 13:30, Steven D'Aprano <steve at pearwood.info> wrote:
> I'd support an immutable dict. partial objects already impose a
> significant (~ 30%) performance penalty:
>
>>>> from timeit import Timer
>>>> min(Timer('f(5)', 'f = lambda x: x').repeat())
> 0.93580079078674316
>>>> min(Timer('p(5)', 'from functools import partial; p = partial(lambda
> x: x)').repeat())
> 1.2715129852294922
>
> No need to make that worse if that can be avoided.

I've made a new patch, in which the keywords attribute is a read-only
proxy of the dictionary. I've used your benchmark, and I haven't found
any significant difference in execution times.
The patch is in the tracker (http://bugs.python.org/issue8699) and
Rietveld (http://codereview.appspot.com/1179044).


Daniel Urban

From kiko at meebo-inc.com  Fri May 14 01:26:17 2010
From: kiko at meebo-inc.com (Kiko Griffin)
Date: Thu, 13 May 2010 16:26:17 -0700
Subject: [Python-Dev] Python Web Developer @ Meebo (Perm in Mountain View,
	CA.)
Message-ID: <AANLkTildlaGvP_p8JEJwQAzFAAni3JztqJg0fquiXzQh@mail.gmail.com>

Hello All,

Meebo is looking for a bright, fun, dedicated software engineer who's
interested in supporting a growing revenue team, loves a good challenge, and
wants to learn about how advertising operations works from the inside out.
In this role, you will be responsible for implementing and maintaining
critical tools that support the ongoing adops infrastructure, ranging from
content acquisition to ad inventory management. As a developer on the ads
team, you'll be exposed to many business functions and you'll support and
develop tools and systems to streamline the entire process. Our team is fun,
friendly, and willing to work really hard to meet our goals. We love a great
user experience, internal and external.

*Responsibilities:*

   - Work with ads team to create and maintain mission critical support
   tools
   - Work closely with Sales and Marketing teams to elicit feedback from
   partners
   - Design and implement partner facing tools
   - Participate in product meetings to determine necessary systems
   - Scale and optimize with a growing revenue stream
   - Interface with both server and client side teams to leverage existing
   tools

*What you will need:*

   - 3-5+ years of software engineering experience in Linux/Unix
   - BS, MS in Computer Science (or equivalent)
   - Strong expertise in Python, Perl, php, or other scripting languages
   - Knowledge of Django, PIG a plus
   - Database experience a plus
   - Ability to work with technical and non-technical teams, in an extremely
   fast-paced environment
   - Strong UI design sense with the ability to adapt, take critical
   feedback, and execute quickly on tasks

*Big pluses:*

   - Experience working with large scale systems
   - Experience working with ad ops industry tools such as: Doubleclick,
   Atlas, 24/7, Salesforce, Dart Sales Manager, Solbright, Operative


 If at all interested OR know someone who might be interested please reply
here Kiko at meebo-inc.com.

Thanks,

Kiko Griffin
kiko at meebo-inc.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100513/bf7dbdad/attachment.html>

From tjreedy at udel.edu  Sat May 15 16:42:12 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 15 May 2010 10:42:12 -0400
Subject: [Python-Dev] Python Web Developer @ Meebo (Perm in Mountain
	View, CA.)
In-Reply-To: <AANLkTildlaGvP_p8JEJwQAzFAAni3JztqJg0fquiXzQh@mail.gmail.com>
References: <AANLkTildlaGvP_p8JEJwQAzFAAni3JztqJg0fquiXzQh@mail.gmail.com>
Message-ID: <hsmbs5$cqv$1@dough.gmane.org>

On 5/13/2010 7:26 PM, Kiko Griffin wrote:

<off-topic job spam deleted>

Dear Kiko - The python-dev mailing list and the gmane.comp.python.devel 
mirror are for development of Python and CPython. Job announcements are 
considered off-topic spam and constitute a dis-promotion for your firm. 
Please do not repeat.

Please redirect your efforts to the python.org job board where you can 
post such announcements.
http://www.python.org/community/jobs/


From solipsis at pitrou.net  Sat May 15 19:48:39 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 May 2010 19:48:39 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
Message-ID: <20100515194839.22b71aa6@pitrou.net>


Hello,

The buildbots are sometimes subject to a flood of "svn exception"
errors. It has been conjectured that these errors are caused by Web
crawlers pressing "force build" buttons without filling any of the
fields (of course, the fact that we get such ugly errors in the
buildbot results, rather than a clean error message when pressing
the button, is a buildbot bug in itself). Couldn't we simply exclude all
crawlers from the buildbot Web pages?

Regards

Antoine.



From exarkun at twistedmatrix.com  Sat May 15 19:57:28 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Sat, 15 May 2010 17:57:28 -0000
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <20100515194839.22b71aa6@pitrou.net>
References: <20100515194839.22b71aa6@pitrou.net>
Message-ID: <20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain>

On 05:48 pm, solipsis at pitrou.net wrote:
>
>Hello,
>
>The buildbots are sometimes subject to a flood of "svn exception"
>errors. It has been conjectured that these errors are caused by Web
>crawlers pressing "force build" buttons without filling any of the
>fields (of course, the fact that we get such ugly errors in the
>buildbot results, rather than a clean error message when pressing
>the button, is a buildbot bug in itself). Couldn't we simply exclude 
>all
>crawlers from the buildbot Web pages?

Most (all?) legitimate crawlers won't submit forms.  Do you think 
there's a non-form link to the force build URL (which _will_ accept a 
GET request to mean the same thing as a POST)?

One thing I have noticed is that spammers find these forms and submit 
them with garbage.  We can probably suppose that such people are going 
to ignore a robots.txt file.

Jean-Paul

From solipsis at pitrou.net  Sat May 15 20:07:09 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 May 2010 20:07:09 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
References: <20100515194839.22b71aa6@pitrou.net>
	<20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain>
Message-ID: <20100515200709.64be2103@pitrou.net>

On Sat, 15 May 2010 17:57:28 -0000
exarkun at twistedmatrix.com wrote:
> 
> One thing I have noticed is that spammers find these forms and submit 
> them with garbage.  We can probably suppose that such people are going 
> to ignore a robots.txt file.

So we could "just" fix the buggy buildbot code.
Not that I want to do it myself :S



From fijall at gmail.com  Sat May 15 21:06:15 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 15 May 2010 13:06:15 -0600
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <20100515200709.64be2103@pitrou.net>
References: <20100515194839.22b71aa6@pitrou.net>
	<20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain>
	<20100515200709.64be2103@pitrou.net>
Message-ID: <AANLkTikAlFWxr_H0ecripSsDp5iP-ICrrQ4WqJ3Pa5E8@mail.gmail.com>

On Sat, May 15, 2010 at 12:07 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 15 May 2010 17:57:28 -0000
> exarkun at twistedmatrix.com wrote:
>>
>> One thing I have noticed is that spammers find these forms and submit
>> them with garbage. ?We can probably suppose that such people are going
>> to ignore a robots.txt file.
>
> So we could "just" fix the buggy buildbot code.
> Not that I want to do it myself :S
>

I can help modify buildbot if you want, but I suppose I need a
specification what precisely is a bug here. Not accepting forms with
garbage? By default buildbot "force build" does not require forms to
be filled and that's on purpose.

From solipsis at pitrou.net  Sat May 15 21:12:14 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 May 2010 21:12:14 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <AANLkTikAlFWxr_H0ecripSsDp5iP-ICrrQ4WqJ3Pa5E8@mail.gmail.com>
References: <20100515194839.22b71aa6@pitrou.net>
	<20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain>
	<20100515200709.64be2103@pitrou.net>
	<AANLkTikAlFWxr_H0ecripSsDp5iP-ICrrQ4WqJ3Pa5E8@mail.gmail.com>
Message-ID: <20100515211214.0d001162@pitrou.net>


Hi,

> I can help modify buildbot if you want, but I suppose I need a
> specification what precisely is a bug here. Not accepting forms with
> garbage? By default buildbot "force build" does not require forms to
> be filled and that's on purpose.

Well, the "fix" would be to forbid an empty "Branch to build" since it
doesn't point to anything buildable (even worse, the fact that the
branch then ends up as None rather than an empty string produces an
exception in buildbot code).

I'm not sure what the process to get the fix in would be, but it
probably involves discussing it with Martin :)

Regards

Antoine.

From martin at v.loewis.de  Sat May 15 21:49:07 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 15 May 2010 21:49:07 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <20100515194839.22b71aa6@pitrou.net>
References: <20100515194839.22b71aa6@pitrou.net>
Message-ID: <4BEEFAB3.4010907@v.loewis.de>

> The buildbots are sometimes subject to a flood of "svn exception"
> errors. It has been conjectured that these errors are caused by Web
> crawlers pressing "force build" buttons without filling any of the
> fields (of course, the fact that we get such ugly errors in the
> buildbot results, rather than a clean error message when pressing
> the button, is a buildbot bug in itself). Couldn't we simply exclude all
> crawlers from the buildbot Web pages?

Hmm. Before doing any modifications, I'd rather have a definite analysis
on this. Are you absolutely certain that, when that happened, the
individual builds that caused this svn exception where actually
triggered over the web, rather than by checkin?

When it happens next, please report exact date and time, and the build
log URL. Due to log rotation, it would then be necessary to investigate
that in a timely manner.

Without any reference to the specific case, I'd guess that a flood of
svn exceptions is caused due to an svn outage, which in turn might be
caused when a build is triggered while the daily Apache restart happens
(i.e. around 6:30 UTC+2).

That said: /dev/buildbot has been disallowed for all robots for quite
some time now:

http://www.python.org/robots.txt

There is really no point robots crawling the build logs, as they don't
contain much useful information for a search engine.

Regards,
Martin

From solipsis at pitrou.net  Sat May 15 21:55:13 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 May 2010 21:55:13 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <4BEEFAB3.4010907@v.loewis.de>
References: <20100515194839.22b71aa6@pitrou.net> <4BEEFAB3.4010907@v.loewis.de>
Message-ID: <20100515215513.24991b4d@pitrou.net>

On Sat, 15 May 2010 21:49:07 +0200
"Martin v. L?wis" <martin at v.loewis.de> wrote:
> 
> Hmm. Before doing any modifications, I'd rather have a definite analysis
> on this. Are you absolutely certain that, when that happened, the
> individual builds that caused this svn exception where actually
> triggered over the web, rather than by checkin?

How can I be "absolutely certain"? As I said, it's a conjecture, and
the suggested fix is just that: a suggestion.

> When it happens next, please report exact date and time, and the build
> log URL. Due to log rotation, it would then be necessary to investigate
> that in a timely manner.

Please take a look at http://www.python.org/dev/buildbot/trunk/. There
are still a bunch of violet buildbots there.
For example:
http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/175
http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/176

Regards

Antoine.

From fijall at gmail.com  Sat May 15 22:01:26 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 15 May 2010 14:01:26 -0600
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <20100515215513.24991b4d@pitrou.net>
References: <20100515194839.22b71aa6@pitrou.net>
	<4BEEFAB3.4010907@v.loewis.de> 
	<20100515215513.24991b4d@pitrou.net>
Message-ID: <AANLkTimg2HIdNCN0vo5yNqgPIGD9Iv8PAWb0GAxEsvKV@mail.gmail.com>

For us at least no branch specified builds the default branch (trunk)
and does not end up with exception in buildbot code. How about
specifying the default branch in config file?

On Sat, May 15, 2010 at 1:55 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 15 May 2010 21:49:07 +0200
> "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>
>> Hmm. Before doing any modifications, I'd rather have a definite analysis
>> on this. Are you absolutely certain that, when that happened, the
>> individual builds that caused this svn exception where actually
>> triggered over the web, rather than by checkin?
>
> How can I be "absolutely certain"? As I said, it's a conjecture, and
> the suggested fix is just that: a suggestion.
>
>> When it happens next, please report exact date and time, and the build
>> log URL. Due to log rotation, it would then be necessary to investigate
>> that in a timely manner.
>
> Please take a look at http://www.python.org/dev/buildbot/trunk/. There
> are still a bunch of violet buildbots there.
> For example:
> http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/175
> http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/176
>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

From janssen at parc.com  Sat May 15 22:01:46 2010
From: janssen at parc.com (Bill Janssen)
Date: Sat, 15 May 2010 13:01:46 PDT
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <20100515194839.22b71aa6@pitrou.net>
References: <20100515194839.22b71aa6@pitrou.net>
Message-ID: <829.1273953706@parc.com>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> The buildbots are sometimes subject to a flood of "svn exception"
> errors. It has been conjectured that these errors are caused by Web
> crawlers pressing "force build" buttons without filling any of the
> fields (of course, the fact that we get such ugly errors in the
> buildbot results, rather than a clean error message when pressing
> the button, is a buildbot bug in itself). Couldn't we simply exclude all
> crawlers from the buildbot Web pages?

I caused a few of those myself yesterday updating my PPC buildbots.

Apologies!

Bill

From janssen at parc.com  Sat May 15 22:03:59 2010
From: janssen at parc.com (Bill Janssen)
Date: Sat, 15 May 2010 13:03:59 PDT
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <AANLkTikAlFWxr_H0ecripSsDp5iP-ICrrQ4WqJ3Pa5E8@mail.gmail.com>
References: <20100515194839.22b71aa6@pitrou.net>
	<20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain>
	<20100515200709.64be2103@pitrou.net>
	<AANLkTikAlFWxr_H0ecripSsDp5iP-ICrrQ4WqJ3Pa5E8@mail.gmail.com>
Message-ID: <848.1273953839@parc.com>

Maciej Fijalkowski <fijall at gmail.com> wrote:

> On Sat, May 15, 2010 at 12:07 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sat, 15 May 2010 17:57:28 -0000
> > exarkun at twistedmatrix.com wrote:
> >>
> >> One thing I have noticed is that spammers find these forms and submit
> >> them with garbage. ?We can probably suppose that such people are going
> >> to ignore a robots.txt file.
> >
> > So we could "just" fix the buggy buildbot code.
> > Not that I want to do it myself :S
> >
> 
> I can help modify buildbot if you want, but I suppose I need a
> specification what precisely is a bug here. Not accepting forms with
> garbage? By default buildbot "force build" does not require forms to
> be filled and that's on purpose.

I'd find it useful if the "branch" field was a choice pull-down listing
valid branches, rather than a plain text field, and if the "revision"
field always defaulted to "HEAD".  Seems to me that since the form is
coming from the buildmaster, that should be possible.

Bill

From janssen at parc.com  Sat May 15 22:09:12 2010
From: janssen at parc.com (Bill Janssen)
Date: Sat, 15 May 2010 13:09:12 PDT
Subject: [Python-Dev] buildbot svn exceptions again...
Message-ID: <898.1273954152@parc.com>

parc-leopard-1 (and most of the other builders) are failing the svn
checkout with the following error:

svn: PROPFIND of '/projects/python/trunk': Could not resolve hostname `svn.python.org': Temporary failure in name resolution (http://svn.python.org)

When I log into that machine as "buildbot" and do the svn checkout
manually using the following command

/usr/bin/svn checkout --non-interactive --no-auth-cache --revision HEAD http://svn.python.org/projects/python/trunk build

it works fine.

Bill

From solipsis at pitrou.net  Sat May 15 22:17:15 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 15 May 2010 22:17:15 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <848.1273953839@parc.com>
References: <20100515194839.22b71aa6@pitrou.net>
	<20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain>
	<20100515200709.64be2103@pitrou.net>
	<AANLkTikAlFWxr_H0ecripSsDp5iP-ICrrQ4WqJ3Pa5E8@mail.gmail.com>
	<848.1273953839@parc.com>
Message-ID: <20100515221715.5b76caaa@pitrou.net>

On Sat, 15 May 2010 13:03:59 PDT
Bill Janssen <janssen at parc.com> wrote:
> I'd find it useful if the "branch" field was a choice pull-down listing
> valid branches, rather than a plain text field, and if the "revision"
> field always defaulted to "HEAD".  Seems to me that since the form is
> coming from the buildmaster, that should be possible.

Actually, I think that it does already default to HEAD if you leave it
empty.

Regards

Antoine.

From martin at v.loewis.de  Sat May 15 22:28:23 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 15 May 2010 22:28:23 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <20100515215513.24991b4d@pitrou.net>
References: <20100515194839.22b71aa6@pitrou.net> <4BEEFAB3.4010907@v.loewis.de>
	<20100515215513.24991b4d@pitrou.net>
Message-ID: <4BEF03E7.9000400@v.loewis.de>


>> Hmm. Before doing any modifications, I'd rather have a definite analysis
>> on this. Are you absolutely certain that, when that happened, the
>> individual builds that caused this svn exception where actually
>> triggered over the web, rather than by checkin?
> 
> How can I be "absolutely certain"?

In the example you gave, the build log says

"The web-page 'force build' button was pressed by '<unknown>': <no
reason specified> "

So ISTM that it's indeed certain that the build was triggered over the
web, rather than by a checkin.

> http://www.python.org/dev/buildbot/builders/sparc%20Ubuntu%20trunk/builds/175

AFAICT from the twistd logs, the user agent triggering this build was
"Mozilla/4.7 [ja] (Win98; I)". It still may have been a bot; it was
using a GET request, even though the form asks for a POST.

The IP address points to some Japanese dialup network (reverse lookup
reports address.dy.bbexcite.jp.)

The bot probably has malicious intent: it has been using about 10
different user-agent strings, on various parts of the site. I have
now blackholed this IP address (although it stopped contacting
python.org around 8 hours ago, anyway).

If desired, we could password-protect the "force build" forms. If that
is to be done, some help from a buildbot expert on what to change
specifically would be appreciated.

Regards,
Martin

From martin at v.loewis.de  Sat May 15 22:31:21 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 15 May 2010 22:31:21 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <829.1273953706@parc.com>
References: <20100515194839.22b71aa6@pitrou.net> <829.1273953706@parc.com>
Message-ID: <4BEF0499.2020905@v.loewis.de>

> I caused a few of those myself yesterday updating my PPC buildbots.
> 
> Apologies!

No need to apologize! these are not the ones Antoine is talking about.
By convention, filling out the "Your name" field in a web build is
recommended, so people know that this was an intentional build. I
usually also fill out a reason.

Regards,
Martin

From martin at v.loewis.de  Sat May 15 22:32:54 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 15 May 2010 22:32:54 +0200
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <848.1273953839@parc.com>
References: <20100515194839.22b71aa6@pitrou.net>	<20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain>	<20100515200709.64be2103@pitrou.net>	<AANLkTikAlFWxr_H0ecripSsDp5iP-ICrrQ4WqJ3Pa5E8@mail.gmail.com>
	<848.1273953839@parc.com>
Message-ID: <4BEF04F6.3090308@v.loewis.de>

> I'd find it useful if the "branch" field was a choice pull-down listing
> valid branches, rather than a plain text field, and if the "revision"
> field always defaulted to "HEAD".  Seems to me that since the form is
> coming from the buildmaster, that should be possible.

Unfortunately, these forms are deeply hidden in the buildbot code. So
I'd rather avoid editing them, or else upgrading to the next buildbot
version becomes even more tedious.

Regards,
Martin

From martin at v.loewis.de  Sat May 15 22:45:35 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 15 May 2010 22:45:35 +0200
Subject: [Python-Dev] buildbot svn exceptions again...
In-Reply-To: <898.1273954152@parc.com>
References: <898.1273954152@parc.com>
Message-ID: <4BEF07EF.8070900@v.loewis.de>

Bill Janssen wrote:
> parc-leopard-1 (and most of the other builders) are failing the svn 
> checkout with the following error:
> 
> svn: PROPFIND of '/projects/python/trunk': Could not resolve hostname
> `svn.python.org': Temporary failure in name resolution
> (http://svn.python.org)
> 
> When I log into that machine as "buildbot" and do the svn checkout 
> manually using the following command
> 
> /usr/bin/svn checkout --non-interactive --no-auth-cache --revision
> HEAD http://svn.python.org/projects/python/trunk build
> 
> it works fine.

That's a common OSX problem/bug. Processes occasionally lose the ability
to resolve host names. Various theories float around what's causing this
(most commonly, people expect that a "controlling terminal" must be
present); my theory is this:

There is a Mach port in the system that lets a process talk to the local
resolver (lookupd, or whatever Apple's naming infrastructure is called
today). For some reason, this port gets closed in a "background"
process, perhaps because the parent process closed it when you logged
out. Or the Kerberos credentials expired, or some such...

In short, if some OS X expert could shed some light on this, it would be
appreciated. It would then be good to add this to the wiki. See also

http://buildbot.net/trac/wiki/UsingLaunchd
http://developer.apple.com/mac/library/technotes/tn2005/tn2083.html

Regards,
Martin



From exarkun at twistedmatrix.com  Sun May 16 01:35:31 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Sat, 15 May 2010 23:35:31 -0000
Subject: [Python-Dev] robots exclusion file on the buildbot pages?
In-Reply-To: <4BEF04F6.3090308@v.loewis.de>
References: <20100515194839.22b71aa6@pitrou.net>
	<20100515175728.1632.1573034403.divmod.xquotient.3@localhost.localdomain>
	<20100515200709.64be2103@pitrou.net>
	<AANLkTikAlFWxr_H0ecripSsDp5iP-ICrrQ4WqJ3Pa5E8@mail.gmail.com>
	<848.1273953839@parc.com> <4BEF04F6.3090308@v.loewis.de>
Message-ID: <20100515233531.1632.315497902.divmod.xquotient.5@localhost.localdomain>

On 08:32 pm, martin at v.loewis.de wrote:
>>I'd find it useful if the "branch" field was a choice pull-down 
>>listing
>>valid branches, rather than a plain text field, and if the "revision"
>>field always defaulted to "HEAD".  Seems to me that since the form is
>>coming from the buildmaster, that should be possible.
>
>Unfortunately, these forms are deeply hidden in the buildbot code. So
>I'd rather avoid editing them, or else upgrading to the next buildbot
>version becomes even more tedious.

Someone sufficiently interested in this feature could work with buildbot 
upstream to get the feature added to an upcoming release, though 
(obviously).

Jean-Paul

From yaniv at aknin.name  Sun May 16 03:55:14 2010
From: yaniv at aknin.name (Yaniv Aknin)
Date: Sun, 16 May 2010 11:55:14 +1000
Subject: [Python-Dev] frozendict (was: Possible patch for functools
	partial...)
Message-ID: <AANLkTimhoG74udFneCTYirUMgEZGFwrUFTad2u2soKGc@mail.gmail.com>

>
> > So I'm thinking either we make an
> > immutable/hashable dict while we're at it, or store the keyword
> > arguments as a tuple (which guarantees immutability), and only
> > convert them back to a dict when you want to call the partial object
> > (simpler, slower).
>
> I'd support an immutable dict. [...]


I've set out to implement a frozendict (like frozenset) for use to store
functools.keywords' attribute, and quickly realized I didn't think of an
obvious flaw in that idea. A frozenset will only accept hashable members,
but a frozendict can't afford this luxury for its values.

I'm not sure how should I go about handling that, if at all. Should I
implement a frozendict which will remain unhashable but support equality
testing like a regular dict? A frozendict that is only hashable if all its
values are hashable, like a tuple? Is the whole notion of a frozendict
worthy, given this limitation?

I'd be happy to hear python-dev's guidance on this.

Cheers,
 - Yaniv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100516/c8fd32eb/attachment.html>

From solipsis at pitrou.net  Sun May 16 12:53:58 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 16 May 2010 12:53:58 +0200
Subject: [Python-Dev] frozendict (was: Possible patch for functools
	partial...)
References: <AANLkTimhoG74udFneCTYirUMgEZGFwrUFTad2u2soKGc@mail.gmail.com>
Message-ID: <20100516125358.2f08e90f@pitrou.net>

On Sun, 16 May 2010 11:55:14 +1000
Yaniv Aknin <yaniv at aknin.name> wrote:
> 
> Is the whole notion of a frozendict
> worthy, given this limitation?

I don't think so. Of course we how have a more general problem:
- if we choose to implement only equality testing for partials and
  leave hashing as is, then hashing isn't consistent with equality
  anymore -- which is unacceptable
- if we choose to implement only equality testing for partials and make
  them unhashable, we are breaking programs which store partials in a
  set or a dict

So we are left with the following choice:
- implement hashing consistently with equality testing, even though the
  keyword arguments can be changed later. I think this is acceptable
  from a practicality standpoint
- abandon the whole idea of improving equality testing

Regards

Antoine.



From ncoghlan at gmail.com  Sun May 16 13:10:02 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 16 May 2010 21:10:02 +1000
Subject: [Python-Dev] frozendict (was: Possible patch for
	functools	partial...)
In-Reply-To: <20100516125358.2f08e90f@pitrou.net>
References: <AANLkTimhoG74udFneCTYirUMgEZGFwrUFTad2u2soKGc@mail.gmail.com>
	<20100516125358.2f08e90f@pitrou.net>
Message-ID: <4BEFD28A.70209@gmail.com>

Antoine Pitrou wrote:
> On Sun, 16 May 2010 11:55:14 +1000
> Yaniv Aknin <yaniv at aknin.name> wrote:
>> Is the whole notion of a frozendict
>> worthy, given this limitation?
> 
> I don't think so. Of course we how have a more general problem:
> - if we choose to implement only equality testing for partials and
>   leave hashing as is, then hashing isn't consistent with equality
>   anymore -- which is unacceptable
> - if we choose to implement only equality testing for partials and make
>   them unhashable, we are breaking programs which store partials in a
>   set or a dict
> 
> So we are left with the following choice:
> - implement hashing consistently with equality testing, even though the
>   keyword arguments can be changed later. I think this is acceptable
>   from a practicality standpoint

Another option is to compare the keywords by identity rather than value.
This is likely to be more correct anyway, since items may compare equal
without giving the same result for all functions:

>>> dict(a=1) == dict(a=1.0)
True
>>> partial(type, 1)()
<type 'int'>
>>> partial(type, 1.0)()
<type 'float'>

Once you're comparing the supplied arguments by identity rather than
value, the question of hashability goes away.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From janssen at parc.com  Sun May 16 19:53:23 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 10:53:23 PDT
Subject: [Python-Dev] buildbot svn exceptions again...
In-Reply-To: <4BEF07EF.8070900@v.loewis.de>
References: <898.1273954152@parc.com> <4BEF07EF.8070900@v.loewis.de>
Message-ID: <99371.1274032403@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> Bill Janssen wrote:
> > parc-leopard-1 (and most of the other builders) are failing the svn 
> > checkout with the following error:
> > 
> > svn: PROPFIND of '/projects/python/trunk': Could not resolve hostname
> > `svn.python.org': Temporary failure in name resolution
> > (http://svn.python.org)
> > 
> > When I log into that machine as "buildbot" and do the svn checkout 
> > manually using the following command
> > 
> > /usr/bin/svn checkout --non-interactive --no-auth-cache --revision
> > HEAD http://svn.python.org/projects/python/trunk build
> > 
> > it works fine.
> 
> That's a common OSX problem/bug. Processes occasionally lose the ability
> to resolve host names.

A workaround is to just put them in the /etc/hosts file.

Bill

From janssen at parc.com  Sun May 16 19:54:11 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 10:54:11 PDT
Subject: [Python-Dev] buildbot svn exceptions again...
In-Reply-To: <4BEF07EF.8070900@v.loewis.de>
References: <898.1273954152@parc.com> <4BEF07EF.8070900@v.loewis.de>
Message-ID: <99382.1274032451@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> That's a common OSX problem/bug. Processes occasionally lose the ability
> to resolve host names. Various theories float around what's causing this
> (most commonly, people expect that a "controlling terminal" must be
> present); my theory is this:
> 
> There is a Mach port in the system that lets a process talk to the local
> resolver (lookupd, or whatever Apple's naming infrastructure is called
> today). For some reason, this port gets closed in a "background"
> process, perhaps because the parent process closed it when you logged
> out. Or the Kerberos credentials expired, or some such...
> 
> In short, if some OS X expert could shed some light on this, it would be
> appreciated. It would then be good to add this to the wiki. See also

This is as close to an explanation as I've been able to get from Apple:

http://lists.apple.com/archives/Darwin-kernel/2009/Apr/msg00016.html

> 
> http://buildbot.net/trac/wiki/UsingLaunchd
> http://developer.apple.com/mac/library/technotes/tn2005/tn2083.html
> 
> Regards,
> Martin
> 
> 

From martin at v.loewis.de  Sun May 16 20:27:23 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 16 May 2010 20:27:23 +0200
Subject: [Python-Dev] buildbot svn exceptions again...
In-Reply-To: <99371.1274032403@parc.com>
References: <898.1273954152@parc.com> <4BEF07EF.8070900@v.loewis.de>
	<99371.1274032403@parc.com>
Message-ID: <4BF0390B.9@v.loewis.de>

> A workaround is to just put them in the /etc/hosts file.

That doesn't really help: the test suite also relies on a number of host
names.

Regards,
Martin

From janssen at parc.com  Sun May 16 21:33:20 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 12:33:20 PDT
Subject: [Python-Dev] OS X buildbots and launchd
Message-ID: <566.1274038400@parc.com>

I can find no evidence that the buildbot installation process given on
the wiki will cause the buildbot slave to be restarted after a reboot of
the machine.  To accomplish this, you should also undertake the work
described in 

  http://buildbot.net/trac/wiki/UsingLaunchd

On my Leopard slave, I created the file
/Library/LaunchAgents/org.python.buildbot.slave.plist, and put in it
this XML:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>StandardOutPath</key>
	<string>twistd.log</string>
	<key>StandardErrorPath</key>
	<string>twistd-err.log</string>
	<key>EnvironmentVariables</key>
	<dict>
		<key>PATH</key>
		<string>/usr/local/bin:/sbin:/usr/sbin:/bin:/usr/bin</string>
	</dict>
	<key>GroupName</key>
	<string>daemon</string>
	<key>KeepAlive</key>
	<dict>
		<key>SuccessfulExit</key>
		<false/>
	</dict>
	<key>Label</key>
	<string>org.python.buildbot.slave</string>
	<key>ProgramArguments</key>
	<array>
		<string>/usr/bin/twistd</string>
		<string>-no</string>
		<string>-y</string>
		<string>/Users/buildbot/buildarea/buildbot.tac</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
	<key>UserName</key>
	<string>buildbot</string>
	<key>WorkingDirectory</key>
	<string>/Users/buildbot/buildarea</string>
</dict>
</plist>

Note that I am using the system Python 2.5, and that I have configured
the buildbot slave machine to login the "buildbot" account automatically
upon boot, which login will cause this daemon to start, and will provide
access to the GUI context for testing tk.  I've configured it with a
locking screensaver that starts immediately; let's see if that works OK.

I've also put "dinsdale.python.org" and "svn.python.org" in /etc/hosts.

Bill

From janssen at parc.com  Sun May 16 22:02:07 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 13:02:07 PDT
Subject: [Python-Dev] what ports are needed for a buildbot?
Message-ID: <1132.1274040127@parc.com>

I'm running my buildbots in a captive firewalled subnet, and I have to
enumerate all the ports that the buildbot is allowed to open to the
outside world.  Right now, I'm getting this error:

ERROR: test_connect (test.test_smtpnet.SmtpSSLTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/test/test_smtpnet.py", line 15, in test_connect
    server = smtplib.SMTP_SSL(self.testServer, self.remotePort)
  File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 757, in __init__
    SMTP.__init__(self, host, port, local_hostname, timeout)
  File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 239, in __init__
    (code, msg) = self.connect(host, port)
  File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 295, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 762, in _get_socket
    new_socket = socket.create_connection((host, port), timeout)
  File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/socket.py", line 321, in create_connection
    raise error(msg)
socket.error: [Errno 61] Connection refused

This make me think that the list of ports enumerated at
http://wiki.python.org/moin/BuildBot is incomplete.  We need a complete
list.  If you know of additional ports, please add them to the wiki page.

Bill

From nir at winpdb.org  Sun May 16 22:07:06 2010
From: nir at winpdb.org (Nir Aides)
Date: Sun, 16 May 2010 23:07:06 +0300
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Message-ID: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>

Hi all,

Here is a second (last?) attempt at getting traction on fixing the GIL (is
it broken?) with BFS (WTF?).
So don't be shy (don't be too rude either) since ignoring counts as down
voting.

Relevant Python issue: http://bugs.python.org/issue7946


*Bottom line first*

I submitted an implementation of BFS (
http://ck.kolivas.org/patches/bfs/sched-BFS.txt) as a patch to the GIL,
which to the extent I have tested it, behaves nicely on Windows XP, Windows
7, GNU/Linux with either CFS or O(1) schedulers, 1/2/4 cores, laptop,
desktop and VirtualBox VM guest (some data below).

The patch is still work in progress and requires work in terms of style,
moving code where it belongs, test code, etc... nevertheless, Python core
developers recommended I already (re)post to python-dev for discussion.


*So is the GIL broken?*

There seems to be some disagreement on that question among Python core
developers (unless you all agree it is not broken :) ). Some developers
maintain the effects described by David Beazley do not affect real world
systems. Even I took the role of a devil's advocate in a previous
discussion, but in fact I think that Python, being a general purpose
language, is similar to the OS in that regard. It is used across many
application domains, platforms, and development paradigms, just as OS
schedulers
are, and therefore accepting thread scheduling with such properties as a
fact of life is not a good idea.

I was first bitten by the original GIL last year while testing a system, and
found David's research while looking for answers, and later had to work
around that problem in another system. Here are other real world cases:

1) Zope people hit this back in 2002 and documented the problem with
interesting insight:
http://www.zope.org/Members/glpb/solaris/multiproc
"I have directly observed a 30% penalty under MP constraints when the
sys.setcheckinterval value was too low (and there was too much GIL
thrashing)."

http://www.zope.org/Members/glpb/solaris/report_ps
"A machine that's going full-throttle isn't as bad, curiously enough --
because the other CPU's are busy doing real work, the GIL doesn't have as
much opportunity to get shuffled between CPUs.  On a MP box it's very
important to set sys.setcheckinterval() up to a fairly large number, I
recommend pystones / 50 or so."

2) Python mailing list - 2005
http://mail.python.org/pipermail/python-list/2005-August/336286.html
"The app suffers from serious performance degradation (compared to
pure c/C++) and high context switches that I suspect the GIL unlocking
may be aggravating ?"

3) Python mailing list - 2008
http://mail.python.org/pipermail/python-list/2008-June/1143217.html
"When I start the server, it sometimes eats up 100% of the CPU for a good
minute or so... though none of the  threads are CPU-intensive"

4) Twisted
http://twistedmatrix.com/pipermail/twisted-python/2005-July/011048.html
"When I run a CPU intensive method via threads.deferToThread it takes all
the CPU away and renders the twisted process unresponsive."

Admittedly, it is not easy to dig reports up in Google.

Finally, I think David explained the relevance of this problem quite nicely:
http://mail.python.org/pipermail/python-dev/2010-March/098416.html


*What about the new GIL?*

There is no real world experience with the new GIL since it is under
development. What we have is David's analysis and a few benchmarks from the
bug report.


*Evolving the GIL into a scheduler*

The problem addressed by the GIL has always been *scheduling* threads to the
interpreter, not just controlling access to it. The patches by Antoine and
David essentially evolve the GIL into a scheduler, however both cause thread
starvation or high rate of context switching in some scenarios (see data
below).


*BFS*

Enter BFS, a new scheduler designed by Con Kolivas, a Linux kernel hacker
who is an expert in this field:
http://ck.kolivas.org/patches/bfs/sched-BFS.txt
"The goal of the Brain Fuck Scheduler, referred to as BFS from here on, is
to completely do away with the complex designs of the past for the cpu
process scheduler and instead implement one that is very simple in basic
design. The main focus of BFS is to achieve excellent desktop interactivity
and responsiveness without heuristics and tuning knobs that are difficult
to understand, impossible to model and predict the effect of, and when tuned
to one workload cause massive detriment to another."

I submitted an implementation of BFS (bfs.patch) which on my machines gives
comparable performance to gilinter2.patch (Antoine's) and seems to schedule
threads more fairly, predictably, and with lower rate of context switching
(see data below).

There are however, some issues in bfs.patch:

1) It works on top of the OS scheduler, which means (for all GIL patches!):
    a) It does not control and is not aware of information such as OS thread
preemption, CPU core to run on, etc...
    b) There may be hard to predict interaction between BFS and the
underlying OS scheduler, which needs to be tested on each target platform.

2) It works best when TSC (http://en.wikipedia.org/wiki/Time_Stamp_Counter)
is available and otherwise falls back to gettimeofday(). I expect the
scheduler to misbehave to some degree or affect performance when TSC is not
available and either of the following is true:
    a) if gettimeofday() is very expensive to read (impacts release/acquire
overhead).
    b) if gettimeofday() has very low precision ~10ms.

By design of BFS, once CPU load crosses a given threshold (about 8 CPU bound
tasks which need the CPU at once), the scheduler falls back to FIFO behavior
and latency goes up sharply.

I have no data on how bfs.patch behaves on ARM, AMD, old CPU models, OSX,
FreeBSD, Solaris, or mobile. The patch may require some tuning to work
properly on those systems, so data is welcome (make sure TSC code in
Include/cycle.h works on those systems before benching).

All that said, to the extent I have tested it, bfs.patch behaves nicely on
Windows XP, Windows 7, GNU/Linux with either CFS or O(1) schedulers, 1/2/4
cores, laptop, desktop and VirtualBox VM guest.


*Data*

Comparison of proposed patches running ccbench on Windows XP:
http://bugs.python.org/issue7946#msg104899

Comparison of proposed patches running Florent's write.py test on Ubuntu
Karmic:
http://bugs.python.org/issue7946#msg105687

Comparison of old GIL, new GIL and BFS running ccbench on Ubuntu Karmic:
http://bugs.python.org/issue7946#msg105874

Last comparison includes a run of old GIL with sys.setcheckinterval(2500) as
Zope people do. IO latency shoots up to ~1000ms as result.


*What can be done with it?*

Here are some options:
1) Abandon it - no one is interested, yawn.
 2) Take ideas and workarounds from its code and apply to other patches.
3) Include it in the interpreter as an auxiliary (turn on with a runtime
switch) scheduler.
4) Adopt it as the Python scheduler.


*Opinion?*

Your opinion is needed (however, please submit code review comments which
are not likely to interest other people, e.g. "why did you use volatile for
X?", at the issue page: http://bugs.python.org/issue7946).


Thanks,
Nir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100516/845a35b2/attachment.html>

From janssen at parc.com  Sun May 16 22:32:19 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 13:32:19 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
Message-ID: <1743.1274041939@parc.com>

I'm interested in having *something*, but I'm not particularly
interested in the 3.x branch.  I expect it to be at least 5 years before
the various Python subsystems I depend on are all ported to 3.x, and I
expect it to be at least that long before I can move onto a version of
OS X that ships with Python 3 as the default system Python.

Right now, I'm faced with the prospect of Apple's next OS including
Python 2.7, and being saddled with the current Python 2.x terrible
multicore behavior for the next 5 years or so.  We badly need some kind
of patch for this in the 2.x branch.

Bill

From janssen at parc.com  Sun May 16 22:35:49 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 13:35:49 PDT
Subject: [Python-Dev] OS X buildbots and launchd
In-Reply-To: <566.1274038400@parc.com>
References: <566.1274038400@parc.com>
Message-ID: <2100.1274042149@parc.com>

Bill Janssen <janssen at parc.com> wrote:

> I can find no evidence that the buildbot installation process given on
> the wiki will cause the buildbot slave to be restarted after a reboot of
> the machine.  To accomplish this, you should also undertake the work
> described in 
> 
>   http://buildbot.net/trac/wiki/UsingLaunchd
> 
> On my Leopard slave, I created the file
> /Library/LaunchAgents/org.python.buildbot.slave.plist, and put in it
> this XML:
>
> [...]
>
> Note that I am using the system Python 2.5, and that I have configured
> the buildbot slave machine to login the "buildbot" account automatically
> upon boot, which login will cause this daemon to start, and will provide
> access to the GUI context for testing tk.  I've configured it with a
> locking screensaver that starts immediately; let's see if that works OK.

This seems to work.  test_tk now passes.

Bill

From ncoghlan at gmail.com  Sun May 16 22:45:12 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 17 May 2010 06:45:12 +1000
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <1743.1274041939@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com>
Message-ID: <4BF05958.7080206@gmail.com>

Bill Janssen wrote:
> Right now, I'm faced with the prospect of Apple's next OS including
> Python 2.7, and being saddled with the current Python 2.x terrible
> multicore behavior for the next 5 years or so.  We badly need some kind
> of patch for this in the 2.x branch.

The matter of the GIL seems far less urgent to those of us that don't
see threading as a particularly good way to exploit multiple cores.

Either way, with the first 2.7 release candidate out soon, it's already
too late to contemplate significant changes to the GIL for that release.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From victor.stinner at haypocalc.com  Sun May 16 22:52:59 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sun, 16 May 2010 22:52:59 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
Message-ID: <201005162253.00229.victor.stinner@haypocalc.com>

Hi,

Le dimanche 16 mai 2010 22:07:06, Nir Aides a ?crit :
> *Evolving the GIL into a scheduler*
> 
> The problem addressed by the GIL has always been *scheduling* threads to
> the interpreter, not just controlling access to it. The patches by Antoine
> and David essentially evolve the GIL into a scheduler, however both cause
> thread starvation or high rate of context switching in some scenarios (see
> data below).

I didn't followed last development around the GIL. Can you explain me why 
Python should have its own scheduler whereas each OS has already its own 
scheduler? The OS has useful informations to help scheduling that Python 
doesn't have. Linux and FreeBSD schedulers are faster each year since... 5 
years?, especially with multiple CPU/cores.

-- 
Victor Stinner
http://www.haypocalc.com/

From janssen at parc.com  Sun May 16 22:57:06 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 13:57:06 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4BF05958.7080206@gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
Message-ID: <3430.1274043426@parc.com>

Nick Coghlan <ncoghlan at gmail.com> wrote:

> Bill Janssen wrote:
> > Right now, I'm faced with the prospect of Apple's next OS including
> > Python 2.7, and being saddled with the current Python 2.x terrible
> > multicore behavior for the next 5 years or so.  We badly need some kind
> > of patch for this in the 2.x branch.
> 
> The matter of the GIL seems far less urgent to those of us that don't
> see threading as a particularly good way to exploit multiple cores.

Nick, this isn't about exploiting cores.  This is about Python programs
that used to work fine on single-core machines suddenly becoming
horrible resource hogs when moved to a more modern machine with a more
modern version of Python.  As far as I'm concerned, just tying all of
the program's threads to a single core would be fine, though I imagine
others would differ.

> Either way, with the first 2.7 release candidate out soon, it's already
> too late to contemplate significant changes to the GIL for that release.

The release schedule, and labelling things as "release candidates" or
not, are all under our control.  Nothing is "too late".  And there's
always Python 2.8 :-).  But I'd consider this a bug in the threading
library, not some unmotivated blue-sky change to the GIL.

Bill

From ncoghlan at gmail.com  Sun May 16 23:12:14 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 17 May 2010 07:12:14 +1000
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <3430.1274043426@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com>
Message-ID: <4BF05FAE.7020809@gmail.com>

Bill Janssen wrote:
> As far as I'm concerned, just tying all of
> the program's threads to a single core would be fine, though I imagine
> others would differ.

Which can be done through the OS tools for setting an application's CPU
affinity.

Fixing the Python thread scheduling is only necessary if we want to be
able to exploit the extra power of those cores rather than forcing
reversion to single core behaviour.

Note that I'm not *opposed* to fixing it, and the discussion in the
tracker issue over Nir and Dave's different approaches to the problem
looks interesting.

> The release schedule, and labelling things as "release candidates" or
> not, are all under our control.  Nothing is "too late".  And there's
> always Python 2.8  :-) .  But I'd consider this a bug in the threading
> library, not some unmotivated blue-sky change to the GIL.

Yes, but if we never said "too late" we'd never ship anything :)

And you do have a reasonable case for considering this a bug, but it
wouldn't be the first time we've escalated bug fixes to "new feature"
level simply because they had a relatively high impact on core parts of
the code.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From janssen at parc.com  Sun May 16 23:27:09 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 14:27:09 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4BF05FAE.7020809@gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
Message-ID: <4328.1274045229@parc.com>

Nick Coghlan <ncoghlan at gmail.com> wrote:

> Bill Janssen wrote:
> > As far as I'm concerned, just tying all of
> > the program's threads to a single core would be fine, though I imagine
> > others would differ.
> 
> Which can be done through the OS tools for setting an application's CPU
> affinity.

Yes, as I say, if the initialization of the threading module called
those tools appropriately and automatically, I'd be happy.  Well, less
unhappy :-).

I have to admit, I don't know how to do this on OS X.  What's the
tool and the process, if we're not getting too far afield?

Bill

From martin at v.loewis.de  Sun May 16 23:38:35 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 16 May 2010 23:38:35 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4328.1274045229@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>	<1743.1274041939@parc.com>
	<4BF05958.7080206@gmail.com>	<3430.1274043426@parc.com>
	<4BF05FAE.7020809@gmail.com> <4328.1274045229@parc.com>
Message-ID: <4BF065DB.2090703@v.loewis.de>

Bill Janssen wrote:
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
>> Bill Janssen wrote:
>>> As far as I'm concerned, just tying all of
>>> the program's threads to a single core would be fine, though I imagine
>>> others would differ.
>> Which can be done through the OS tools for setting an application's CPU
>> affinity.
> 
> Yes, as I say, if the initialization of the threading module called
> those tools appropriately and automatically, I'd be happy.  Well, less
> unhappy :-).
> 
> I have to admit, I don't know how to do this on OS X.  What's the
> tool and the process, if we're not getting too far afield?

OSX doesn't really support thread affinity. The affinity API that they
have is described on

http://developer.apple.com/mac/library/releasenotes/Performance/RN-AffinityAPI/index.html

You can't bind a thread to specific core with it, though, but you can
requests that multiple threads all run on the same core (leaving the
choice "which core" to the system).

IIUC, an affinity preference does not survive exec(2), so you can't
write a tool that binds all threads of its child processes to a single
core (such a tool is available on many unices, though).

Regards,
Martin

From janssen at parc.com  Mon May 17 00:13:44 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 15:13:44 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4BF065DB.2090703@v.loewis.de>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
Message-ID: <5806.1274048024@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> OSX doesn't really support thread affinity. The affinity API that they
> have is described on
> 
> http://developer.apple.com/mac/library/releasenotes/Performance/RN-AffinityAPI/index.html
> 
> You can't bind a thread to specific core with it, though, but you can
> requests that multiple threads all run on the same core (leaving the
> choice "which core" to the system).

I believe that would be sufficient to fix the problem with Python,
though I wonder about the effect on JCC-generated modules like pylucene,
where the threads are really Java threads as well as Python threads.

So the patch to the threading code would presumably, for those OSs where
the capability exists, try to put all created threads in the same
affinity set.  Presumably there would also be a way to clear that
binding, for folks who know what they're doing.

Bill

From janssen at parc.com  Mon May 17 01:46:23 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 16 May 2010 16:46:23 PDT
Subject: [Python-Dev] OS X buildbots and launchd
In-Reply-To: <566.1274038400@parc.com>
References: <566.1274038400@parc.com>
Message-ID: <8192.1274053583@parc.com>

I've repeated this setup with OS X 10.4 (Tiger).  Some changes to the
XML file were necessary, because I'd installed a non-system Python,
twisted, and zope.interface on this machine, and because on Tiger,
launchd daemon were not allowed to set their group ID.  Here's the Tiger
version, which again goes in
/Library/LaunchAgents/org.python.buildbot.slave.plist.

Bill

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>StandardOutPath</key>
	<string>twistd.log</string>
	<key>StandardErrorPath</key>
	<string>twistd-err.log</string>
	<key>EnvironmentVariables</key>
	<dict>
		<key>PATH</key>
		<string>/Library/Frameworks/Python.framework/Versions/2.6/bin:/sbin:/usr/sbin:/bin:/usr/bin</string>
		<key>PYTHONPATH</key>
		<string>/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages</string>
	</dict>
	<key>KeepAlive</key>
	<dict>
		<key>SuccessfulExit</key>
		<false/>
	</dict>
	<key>Label</key>
	<string>org.python.buildbot.slave</string>
	<key>ProgramArguments</key>
	<array>
		<string>/Library/Frameworks/Python.framework/Versions/2.6/bin/twistd</string>
		<string>-no</string>
		<string>-y</string>
		<string>/Users/buildbot/buildarea/buildbot.tac</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
	<key>UserName</key>
	<string>buildbot</string>
	<key>WorkingDirectory</key>
	<string>/Users/buildbot/buildarea</string>
</dict>
</plist>

From regebro at gmail.com  Mon May 17 08:28:08 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 17 May 2010 08:28:08 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <201005162253.00229.victor.stinner@haypocalc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com> 
	<201005162253.00229.victor.stinner@haypocalc.com>
Message-ID: <AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com>

On Sun, May 16, 2010 at 22:52, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> I didn't followed last development around the GIL. Can you explain me why
> Python should have its own scheduler whereas each OS has already its own
> scheduler?

Because the GIL locks and unlocks threads, in practice, it already
have. But the scheduler is so simplistic it ends up fighting with the
OS scheduler, and a large amount of CPU time is used up switching
instead of executing.
Having a proper scheduler fixes this.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From hrvoje.niksic at avl.com  Mon May 17 09:47:10 2010
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Mon, 17 May 2010 09:47:10 +0200
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <AANLkTilzNIu-8h4xbtS4BpsgIqlGFdkPGrgD0pQ6ao9n@mail.gmail.com>
References: <hs18jc$1mh$1@dough.gmane.org>	<AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com>
	<AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com>
	<201005132130.30619.steve@pearwood.info>	<AANLkTikYXw5k67-HO4J-K0ylgrsK7U2ATRELMeG6luEV@mail.gmail.com>
	<AANLkTilzNIu-8h4xbtS4BpsgIqlGFdkPGrgD0pQ6ao9n@mail.gmail.com>
Message-ID: <4BF0F47E.6060604@avl.com>

On 05/14/2010 06:39 AM, Daniel Urban wrote:
> I've made a new patch, in which the keywords attribute is a read-only
> proxy of the dictionary.

What about backward compatibility?  This looks like an incompatible change.

From urban.dani+py at gmail.com  Mon May 17 13:51:40 2010
From: urban.dani+py at gmail.com (Daniel Urban)
Date: Mon, 17 May 2010 13:51:40 +0200
Subject: [Python-Dev] Possible patch for functools partial - Interested?
In-Reply-To: <4BF0F47E.6060604@avl.com>
References: <hs18jc$1mh$1@dough.gmane.org>
	<AANLkTinG585U_YIiAg4kPlnSQpTN_T2RprWm59IBoJDW@mail.gmail.com> 
	<AANLkTikupcucJNq03WCLYFwakyngzgl-MRD7pgtEZkNP@mail.gmail.com> 
	<201005132130.30619.steve@pearwood.info>
	<AANLkTikYXw5k67-HO4J-K0ylgrsK7U2ATRELMeG6luEV@mail.gmail.com> 
	<AANLkTilzNIu-8h4xbtS4BpsgIqlGFdkPGrgD0pQ6ao9n@mail.gmail.com> 
	<4BF0F47E.6060604@avl.com>
Message-ID: <AANLkTilQmj31fo2T4_Amh7GQtoeHO4qLm-8Uqj3G81QO@mail.gmail.com>

On Mon, May 17, 2010 at 09:47, Hrvoje Niksic <hrvoje.niksic at avl.com> wrote:
> On 05/14/2010 06:39 AM, Daniel Urban wrote:
>>
>> I've made a new patch, in which the keywords attribute is a read-only
>> proxy of the dictionary.
>
> What about backward compatibility? ?This looks like an incompatible change.

You're probably right. I'd be happy to make patch which doesn't
replace the dict with the proxy. But that would mean, that the hash
value isn't really immutable.

Daniel Urban

From solipsis at pitrou.net  Mon May 17 14:12:29 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 17 May 2010 14:12:29 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com>
Message-ID: <20100517141229.13a2d66c@pitrou.net>

On Mon, 17 May 2010 08:28:08 +0200
Lennart Regebro <regebro at gmail.com> wrote:
> But the scheduler is so simplistic it ends up fighting with the
> OS scheduler, and a large amount of CPU time is used up switching
> instead of executing.

This is already fixed with py3k.

Antoine.



From solipsis at pitrou.net  Mon May 17 14:14:55 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 17 May 2010 14:14:55 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com>
Message-ID: <20100517141455.6e1b6f64@pitrou.net>

On Sun, 16 May 2010 15:13:44 PDT
Bill Janssen <janssen at parc.com> wrote:
> 
> So the patch to the threading code would presumably, for those OSs where
> the capability exists, try to put all created threads in the same
> affinity set.

This is not really a good idea. There's some code which releases the
GIL, precisely so that you can run several threads (computations) at
once.

If you aren't too concerned with interactivity, you can increase
sys.setcheckinterval() to alleviate the problem on 2.x.

Regards

Antoine.



From regebro at gmail.com  Mon May 17 14:47:25 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 17 May 2010 14:47:25 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100517141229.13a2d66c@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com> 
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com> 
	<20100517141229.13a2d66c@pitrou.net>
Message-ID: <AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>

On Mon, May 17, 2010 at 14:12, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 17 May 2010 08:28:08 +0200
> Lennart Regebro <regebro at gmail.com> wrote:
>> But the scheduler is so simplistic it ends up fighting with the
>> OS scheduler, and a large amount of CPU time is used up switching
>> instead of executing.
>
> This is already fixed with py3k.

Are you referring to the "New GIL"?

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From solipsis at pitrou.net  Mon May 17 15:05:26 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 17 May 2010 15:05:26 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com>
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>
Message-ID: <20100517150526.0295af9e@pitrou.net>

On Mon, 17 May 2010 14:47:25 +0200
Lennart Regebro <regebro at gmail.com> wrote:
> On Mon, May 17, 2010 at 14:12, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Mon, 17 May 2010 08:28:08 +0200
> > Lennart Regebro <regebro at gmail.com> wrote:
> >> But the scheduler is so simplistic it ends up fighting with the
> >> OS scheduler, and a large amount of CPU time is used up switching
> >> instead of executing.
> >
> > This is already fixed with py3k.
> 
> Are you referring to the "New GIL"?

Yes.

From janssen at parc.com  Mon May 17 18:05:03 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 17 May 2010 09:05:03 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100517141455.6e1b6f64@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
Message-ID: <6887.1274112303@parc.com>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sun, 16 May 2010 15:13:44 PDT
> Bill Janssen <janssen at parc.com> wrote:
> > 
> > So the patch to the threading code would presumably, for those OSs where
> > the capability exists, try to put all created threads in the same
> > affinity set.
> 
> This is not really a good idea.

Yes, fixing the GIL for multicore in 2.7 would be a better idea, IMO.
But that might be too large a change.

> There's some code which releases the GIL, precisely so that you can
> run several threads (computations) at once.

If they can get hold of the GIL in the first place!  Yes, you'd want to
be able to "unbind" threads if you knew what you were doing, so that
they could run on other cores, and you'd want a switch to disable the
affinity mechanism entirely.  But I'd be happy to have things in the
naive case run as well as they do on single-core machines, and let
experts do optimizations.

> If you aren't too concerned with interactivity, you can increase
> sys.setcheckinterval() to alleviate the problem on 2.x.

Unfortunately, many use cases might well be concerned with
interactivity.  Things like event-processing loops.

Bill

From solipsis at pitrou.net  Mon May 17 18:16:52 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 17 May 2010 18:16:52 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <6887.1274112303@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
Message-ID: <1274113012.3050.8.camel@localhost.localdomain>

Le lundi 17 mai 2010 ? 09:05 -0700, Bill Janssen a ?crit :
> Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> > On Sun, 16 May 2010 15:13:44 PDT
> > Bill Janssen <janssen at parc.com> wrote:
> > > 
> > > So the patch to the threading code would presumably, for those OSs where
> > > the capability exists, try to put all created threads in the same
> > > affinity set.
> > 
> > This is not really a good idea.
> 
> Yes, fixing the GIL for multicore in 2.7 would be a better idea, IMO.
> But that might be too large a change.

Well, 2.7rc1 is scheduled in less than three weeks now. IMO any patch
changing fundamental threading properties is a no-no (even the processor
affinity proposal).

Someone had tried to backport the new GIL to 2.7 (there's a tracker
issue for it, I'm too lazy to search right now), but it was concluded
that compatibility requirements for Python 2.x (compatibility with
various legacy threading libraries) made things too complicated and
tedious.

> > There's some code which releases the GIL, precisely so that you can
> > run several threads (computations) at once.
> 
> If they can get hold of the GIL in the first place!  Yes, you'd want to
> be able to "unbind" threads if you knew what you were doing, so that
> they could run on other cores, and you'd want a switch to disable the
> affinity mechanism entirely.  But I'd be happy to have things in the
> naive case run as well as they do on single-core machines, and let
> experts do optimizations.

"Letting experts do optimizations" is a regression, though, because
right now you don't have to be an expert to take advantage of such
optimizations (just run a separate thread doing e.g. some zlib
compression).

Regards

Antoine.



From janssen at parc.com  Mon May 17 19:11:03 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 17 May 2010 10:11:03 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <1274113012.3050.8.camel@localhost.localdomain>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
	<1274113012.3050.8.camel@localhost.localdomain>
Message-ID: <9017.1274116263@parc.com>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> Well, 2.7rc1 is scheduled in less than three weeks now. IMO any patch
> changing fundamental threading properties is a no-no (even the processor
> affinity proposal).

Unfortunately, our "fundamental threading properties" are broken for
multicore machines.  And multicore seems to be the wave of the future.

Not fixing this is a big problem.  It relegates the only Python which
will be usable by many many people for many years, 2.x, to a toy
language status on modern machines.  It will have threads, but the use
of them, either directly or indirectly by modules you may import, may
cause unpredictable performance problems.

I'd hate to let a fundamental flaw like this go through simply because
someone somewhere somewhen set a completely synthetic deadline.

> Someone had tried to backport the new GIL to 2.7 (there's a tracker
> issue for it, I'm too lazy to search right now), but it was concluded
> that compatibility requirements for Python 2.x (compatibility with
> various legacy threading libraries) made things too complicated and
> tedious.

http://bugs.python.org/issue7753, from January.

I've read through that issue (several times), and I have to say that I
wind up gnashing my teeth each time.  Here's a fix that's rejected
because it "only" supports NT and POSIX threads.  What percentage of
Python use cases do those two threading systems cover?  Do we know?

If by "compatibility requirements" you mean that new releases of Python
should run on antique systems just as the older releases did, that's
satisfied by the issue's current state, you're right.  On the other
hand, to me that seems an odd goal to prioritize.  After all, the older
releases are still there, for that antique system.

Nor do I see an answer to Russ' final question:

``What if, as you proposed earlier, the patch were to leave the old
behavior if the threading model on the given platform were not
supported?''

> > > There's some code which releanses the GIL, precisely so that you can
> > > run several threads (computations) at once.
> > 
> > If they can get hold of the GIL in the first place!  Yes, you'd want to
> > be able to "unbind" threads if you knew what you were doing, so that
> > they could run on other cores, and you'd want a switch to disable the
> > affinity mechanism entirely.  But I'd be happy to have things in the
> > naive case run as well as they do on single-core machines, and let
> > experts do optimizations.
> 
> "Letting experts do optimizations" is a regression, though, because
> right now you don't have to be an expert to take advantage of such
> optimizations (just run a separate thread doing e.g. some zlib
> compression).

If threading performance wasn't broken on multicore, I'd agree with you.
But right now, *everyone* has to be an expert just to use Python 2.x
effectively on modern multicore hardware -- you have to find the right
patch in issue 7753, apply it to the sources, build a custom python, and
use it.  Whether or not use explicitly know you are using threads
(because some other package may be using them under the covers).

Bill

From solipsis at pitrou.net  Mon May 17 19:39:52 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 17 May 2010 19:39:52 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <9017.1274116263@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
	<1274113012.3050.8.camel@localhost.localdomain>
	<9017.1274116263@parc.com>
Message-ID: <20100517193952.48875717@pitrou.net>

On Mon, 17 May 2010 10:11:03 PDT
Bill Janssen <janssen at parc.com> wrote:
> 
> I'd hate to let a fundamental flaw like this go through simply because
> someone somewhere somewhen set a completely synthetic deadline.
[...]
> I've read through that issue (several times), and I have to say that I
> wind up gnashing my teeth each time.  Here's a fix that's rejected
> because it "only" supports NT and POSIX threads.  What percentage of
> Python use cases do those two threading systems cover?  Do we know?

Well, if instead of gnashing your teeth, you had contributed to the
issue, perhaps a patch would have been committed by now (or perhaps
not, but who knows?). If you stay silent, you cannot assume that
someone else will stand up for *your* opinion (and the fact that nobody
did could indicate that not many people care about the issue, actually).

> But right now, *everyone* has to be an expert just to use Python 2.x
> effectively on modern multicore hardware

Python works reasonably well on multicore hardware, especially if you
don't run spinning loops and if you're not on Mac OS X. It may lose *at
most* 10-20% performance compared to a single-core run but that's hardly
the end of the world. And some workloads won't suffer any degradation.

Besides, today's multicore CPUs have far better single-threaded
performance than yesteryear's single-core CPUs, which makes the
performance regression issue more theoretical than practical.
In real life, you have very little risk of witnessing a performance
regression when switching your Python from a single-core to a multicore
machine.

Regards

Antoine.

From janssen at parc.com  Mon May 17 19:44:19 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 17 May 2010 10:44:19 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <9017.1274116263@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
	<1274113012.3050.8.camel@localhost.localdomain>
	<9017.1274116263@parc.com>
Message-ID: <10221.1274118259@parc.com>

Bill Janssen <janssen at parc.com> wrote:

> use it.  Whether or not use explicitly know you are using threads
> (because some other package may be using them under the covers).

Of course, I meant to say, "Whether or not *youse* explicitly know you
are using threads (because some other package may be using them under
the covers)."  :-).

Bill

From janssen at parc.com  Mon May 17 20:15:49 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 17 May 2010 11:15:49 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100517193952.48875717@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
	<1274113012.3050.8.camel@localhost.localdomain>
	<9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net>
Message-ID: <10810.1274120149@parc.com>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> Well, if instead of gnashing your teeth, you had contributed to the
> issue, perhaps a patch would have been committed by now (or perhaps
> not, but who knows?). If you stay silent, you cannot assume that
> someone else will stand up for *your* opinion (and the fact that nobody
> did could indicate that not many people care about the issue, actually).

Unfortunately, I somehow did not even *know* about the issue until
February, after the issue had been closed.  What I did know was that
some of our big complicated Python multi-threaded daemons had shown
puzzling resource hogging when moved from small Macs to large 8-core
machines with hardware RAID and lots of memory.

But, simpleton that I am, I'd presumed that threading in Python wasn't
broken, and was looking elsewhere for the cause.

> Python works reasonably well on multicore hardware, especially if you
> don't run spinning loops and if you're not on Mac OS X.

I'm not sure what you mean by "spinning loops".

But I *am* on Mac OS X, along with an increasing percentage of the
world.  And I'm dismayed that there's no momentum to fix this problem.
Not a good sign.

Bill

From solipsis at pitrou.net  Mon May 17 20:28:47 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 17 May 2010 20:28:47 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <10810.1274120149@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
	<1274113012.3050.8.camel@localhost.localdomain>
	<9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net>
	<10810.1274120149@parc.com>
Message-ID: <20100517202847.6d9aa8d3@pitrou.net>

On Mon, 17 May 2010 11:15:49 PDT
Bill Janssen <janssen at parc.com> wrote:
> 
> What I did know was that
> some of our big complicated Python multi-threaded daemons had shown
> puzzling resource hogging when moved from small Macs to large 8-core
> machines with hardware RAID and lots of memory.

Could you give detailed information about this?

Since you're talking about a "big complicated Python multi-threaded
daemon", I presume you can't port it to Python 3 very quickly, but it
would be nice to know if the problem disappears with 3.2.

> I'm not sure what you mean by "spinning loops".

It was an allusion to Dave Beazley's first benchmarks, which merely ran
a spinning loop over several threads, and showed catastrophic
degradation under OS X.

> But I *am* on Mac OS X, along with an increasing percentage of the
> world.  And I'm dismayed that there's no momentum to fix this problem.

There /has/ been momentum in fixing it. In py3k.

Regards

Antoine.

From florent.xicluna at gmail.com  Mon May 17 20:30:27 2010
From: florent.xicluna at gmail.com (Florent Xicluna)
Date: Mon, 17 May 2010 20:30:27 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
Message-ID: <AANLkTik4Rn0yrRdP3uzZdVXihVm6IuFv35ts1FCg7nW8@mail.gmail.com>

2010/5/16 Nir Aides <nir at winpdb.org>

>
> *What can be done with it?*
>
> Here are some options:
> 1) Abandon it - no one is interested, yawn.
>  2) Take ideas and workarounds from its code and apply to other patches.
> 3) Include it in the interpreter as an auxiliary (turn on with a runtime
> switch) scheduler.
> 4) Adopt it as the Python scheduler.
>
>
> *Opinion?*
>
>
I would like to have the possibility to "./configure --without-broken-GIL"
or "./configure --with-bfs-scheduler" in Python 2.7  like we "./configure
--with-computed-gotos" in 3.2.

It will let the opportunity for the experienced user (or the distribution
packager) to enable the threading optimizations on its platform, while it
preserves the current behaviour by default.

It will give more chance that people test this experimental configure option
and report any issue they find, so it can be fixed and improved in the next
bugfix version (2.7.1).
Since the 2.7 branch will have long term support, it makes sense to support
multi-core platforms.

And more users of the fixed GIL means more bugfixes (even for 3.x branch).

-- 
Florent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100517/de963b29/attachment-0001.html>

From janssen at parc.com  Mon May 17 20:59:16 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 17 May 2010 11:59:16 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100517202847.6d9aa8d3@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
	<1274113012.3050.8.camel@localhost.localdomain>
	<9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net>
	<10810.1274120149@parc.com> <20100517202847.6d9aa8d3@pitrou.net>
Message-ID: <12368.1274122756@parc.com>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 17 May 2010 11:15:49 PDT
> Bill Janssen <janssen at parc.com> wrote:
> > 
> > What I did know was that
> > some of our big complicated Python multi-threaded daemons had shown
> > puzzling resource hogging when moved from small Macs to large 8-core
> > machines with hardware RAID and lots of memory.
> 
> Could you give detailed information about this?

Probably not detailed enough.  IP issues.  It's a version of UpLib.

> Since you're talking about a "big complicated Python multi-threaded
> daemon", I presume you can't port it to Python 3 very quickly, but it
> would be nice to know if the problem disappears with 3.2.

Yes, it would.  As soon as I have working 3.x versions of BeautifulSoup,
PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser,
dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and
vobject, I'll let you know. :-)

> There /has/ been momentum in fixing it. In py3k.

Yes, I specifically meant in the 2.x branch.  I'm guessing I'll have to
stay on 2.x for at least 5 more years, due to the other package
dependencies.

Bill

From tjreedy at udel.edu  Mon May 17 21:39:36 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 17 May 2010 15:39:36 -0400
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <12368.1274122756@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>	<1743.1274041939@parc.com>
	<4BF05958.7080206@gmail.com>	<3430.1274043426@parc.com>
	<4BF05FAE.7020809@gmail.com>	<4328.1274045229@parc.com>
	<4BF065DB.2090703@v.loewis.de>	<5806.1274048024@parc.com>
	<20100517141455.6e1b6f64@pitrou.net>	<6887.1274112303@parc.com>	<1274113012.3050.8.camel@localhost.localdomain>	<9017.1274116263@parc.com>
	<20100517193952.48875717@pitrou.net>	<10810.1274120149@parc.com>
	<20100517202847.6d9aa8d3@pitrou.net> <12368.1274122756@parc.com>
Message-ID: <hss61o$j7q$1@dough.gmane.org>

On 5/17/2010 2:59 PM, Bill Janssen wrote:
> Yes, it would.  As soon as I have working 3.x versions of BeautifulSoup,
> PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser,
> dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and
> vobject, I'll let you know. :-)
>
>> There /has/ been momentum in fixing it. In py3k.
>
> Yes, I specifically meant in the 2.x branch.  I'm guessing I'll have to
> stay on 2.x for at least 5 more years, due to the other package
> dependencies.

I suspect it will be sooner than that, especially if users like you 
ask/beg/plead with the maintainers of libraries like those you listed to 
make them work with 3.2. Give your particular reason, that Python3 will 
work increasingly well with multicore machines. I an sure a couple of 
such people have posted that they see no reason to upgrade until users 
start requesting them to.

Terry Jan Reedy


From martin at v.loewis.de  Mon May 17 22:20:14 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 May 2010 22:20:14 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <9017.1274116263@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>	<1743.1274041939@parc.com>
	<4BF05958.7080206@gmail.com>	<3430.1274043426@parc.com>
	<4BF05FAE.7020809@gmail.com>	<4328.1274045229@parc.com>
	<4BF065DB.2090703@v.loewis.de>	<5806.1274048024@parc.com>
	<20100517141455.6e1b6f64@pitrou.net>	<6887.1274112303@parc.com>	<1274113012.3050.8.camel@localhost.localdomain>
	<9017.1274116263@parc.com>
Message-ID: <4BF1A4FE.3030505@v.loewis.de>

> Not fixing this is a big problem.  It relegates the only Python which
> will be usable by many many people for many years, 2.x, to a toy
> language status on modern machines.  It will have threads, but the use
> of them, either directly or indirectly by modules you may import, may
> cause unpredictable performance problems.

People may disagree with this characterization, but if we take that
for granted, then, yes, we are willing to accept that as the state
of things for the coming years.

People running into these problems will have a number of choices to
them: switch operating systems (i.e. drop OSX for something that
actually works), switch programming languages (i.e. drop Python for
something that actually works), switch application architectures (i.e.
drop threads for something that actually works), switch to 3.x, or
just accept the problem, and hope that the system will find something
else to do while switching Python threads.

> I'd hate to let a fundamental flaw like this go through simply because
> someone somewhere somewhen set a completely synthetic deadline.

No, it's not like that. We set the deadline so that we are able to
cancel discussions like this one. It would be possible to change the
schedule, if we would agree that it was for a good cause - which we don't.

> If threading performance wasn't broken on multicore, I'd agree with you.
> But right now, *everyone* has to be an expert just to use Python 2.x
> effectively on modern multicore hardware

Not at all. Just use the multiprocessing module instead, and be done.
It's really easy to use if you already managed to understand threads.

Regards,
Martin

From janssen at parc.com  Mon May 17 23:09:07 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 17 May 2010 14:09:07 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4BF1A4FE.3030505@v.loewis.de>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
	<1274113012.3050.8.camel@localhost.localdomain>
	<9017.1274116263@parc.com> <4BF1A4FE.3030505@v.loewis.de>
Message-ID: <14331.1274130547@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> > I'd hate to let a fundamental flaw like this go through simply because
> > someone somewhere somewhen set a completely synthetic deadline.
> 
> No, it's not like that. We set the deadline so that we are able to
> cancel discussions like this one. It would be possible to change the
> schedule, if we would agree that it was for a good cause - which we don't.

I do appreciate that, and also what you and Antoine are saying.

> > If threading performance wasn't broken on multicore, I'd agree with you.
> > But right now, *everyone* has to be an expert just to use Python 2.x
> > effectively on modern multicore hardware
> 
> Not at all. Just use the multiprocessing module instead, and be done.
> It's really easy to use if you already managed to understand threads.

But that's just the problem.  Most folks don't use "threads", they use a
higher-level abstraction like the nltk library.  Does it use threads?
Has its owner ported it to py3k?  Has its owner ported it to the
multiprocessing module?  I have to be an expert to know.

I'll stop talking about this now...  At least, here.  Apparently we only
need to fix this for OS X.

Bill

From ncoghlan at gmail.com  Mon May 17 23:27:59 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 18 May 2010 07:27:59 +1000
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4BF1A4FE.3030505@v.loewis.de>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>	<1743.1274041939@parc.com>	<4BF05958.7080206@gmail.com>	<3430.1274043426@parc.com>	<4BF05FAE.7020809@gmail.com>	<4328.1274045229@parc.com>	<4BF065DB.2090703@v.loewis.de>	<5806.1274048024@parc.com>	<20100517141455.6e1b6f64@pitrou.net>	<6887.1274112303@parc.com>	<1274113012.3050.8.camel@localhost.localdomain>	<9017.1274116263@parc.com>
	<4BF1A4FE.3030505@v.loewis.de>
Message-ID: <4BF1B4DF.9010908@gmail.com>

Martin v. L?wis wrote:
> People running into these problems will have a number of choices to
> them: switch operating systems (i.e. drop OSX for something that
> actually works), switch programming languages (i.e. drop Python for
> something that actually works), switch application architectures (i.e.
> drop threads for something that actually works), switch to 3.x, or
> just accept the problem, and hope that the system will find something
> else to do while switching Python threads.

There's even another option: if the new-GIL backport patch in
http://bugs.python.org/issue7753 works for you, apply it and run with it
(and advocate for a Python 2.8 release to make it more widely available,
possibly even contributing the fallback behaviour discussed in the issue
to make that situation more likely).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From janssen at parc.com  Mon May 17 23:28:22 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 17 May 2010 14:28:22 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <hss61o$j7q$1@dough.gmane.org>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<6887.1274112303@parc.com>
	<1274113012.3050.8.camel@localhost.localdomain>
	<9017.1274116263@parc.com> <20100517193952.48875717@pitrou.net>
	<10810.1274120149@parc.com> <20100517202847.6d9aa8d3@pitrou.net>
	<12368.1274122756@parc.com> <hss61o$j7q$1@dough.gmane.org>
Message-ID: <14980.1274131702@parc.com>

Terry Reedy <tjreedy at udel.edu> wrote:

> On 5/17/2010 2:59 PM, Bill Janssen wrote:
> > Yes, it would.  As soon as I have working 3.x versions of BeautifulSoup,
> > PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser,
> > dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and
> > vobject, I'll let you know. :-)
> >
> >> There /has/ been momentum in fixing it. In py3k.
> >
> > Yes, I specifically meant in the 2.x branch.  I'm guessing I'll have to
> > stay on 2.x for at least 5 more years, due to the other package
> > dependencies.
> 
> I suspect it will be sooner than that, especially if users like you
> ask/beg/plead with the maintainers of libraries like those you listed
> to make them work with 3.2.

Oh, that's the way I like to spend my day (and, as you can tell from
this conversation, I'm really good at it :-).  Though I will of course
do that.

But some of these, like JCC+pylucene, nltk, and vobject, were developed
with idiosyncratic funding resources which no longer exist.  Others,
like pyglet, were developed for a completely different purpose, and I
doubt the developers care what I want.  So, realistically, I doubt it
will be less than five years.

Bill

From ncoghlan at gmail.com  Mon May 17 23:31:22 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 18 May 2010 07:31:22 +1000
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100517141455.6e1b6f64@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>	<1743.1274041939@parc.com>
	<4BF05958.7080206@gmail.com>	<3430.1274043426@parc.com>
	<4BF05FAE.7020809@gmail.com>	<4328.1274045229@parc.com>
	<4BF065DB.2090703@v.loewis.de>	<5806.1274048024@parc.com>
	<20100517141455.6e1b6f64@pitrou.net>
Message-ID: <4BF1B5AA.8060208@gmail.com>

Antoine Pitrou wrote:
> On Sun, 16 May 2010 15:13:44 PDT
> Bill Janssen <janssen at parc.com> wrote:
>> So the patch to the threading code would presumably, for those OSs where
>> the capability exists, try to put all created threads in the same
>> affinity set.
> 
> This is not really a good idea. There's some code which releases the
> GIL, precisely so that you can run several threads (computations) at
> once.

Somewhat irrelevant given the rest of this thread, but you could
potentially deal with that by playing CPU affinity games in the
BEGIN/END_ALLOW_THREADS macros.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon May 17 23:33:37 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 18 May 2010 07:33:37 +1000
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTik4Rn0yrRdP3uzZdVXihVm6IuFv35ts1FCg7nW8@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<AANLkTik4Rn0yrRdP3uzZdVXihVm6IuFv35ts1FCg7nW8@mail.gmail.com>
Message-ID: <4BF1B631.1040503@gmail.com>

Florent Xicluna wrote:
> I would like to have the possibility to "./configure
> --without-broken-GIL" or "./configure --with-bfs-scheduler" in Python
> 2.7  like we "./configure --with-computed-gotos" in 3.2.
> 
> It will let the opportunity for the experienced user (or the
> distribution packager) to enable the threading optimizations on its
> platform, while it preserves the current behaviour by default.
> 
> It will give more chance that people test this experimental configure
> option and report any issue they find, so it can be fixed and improved
> in the next bugfix version (2.7.1).
> Since the 2.7 branch will have long term support, it makes sense to
> support multi-core platforms.
> 
> And more users of the fixed GIL means more bugfixes (even for 3.x branch).

Would you suggest extending this as far as providing "new GIL" binaries
for Windows and Mac OS X?

(If that was the case, it begins to sound a lot like
http://bugs.python.org/issue7753 with reversion to the old GIL on
non-NT/POSIX platforms)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From janssen at parc.com  Mon May 17 23:36:44 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 17 May 2010 14:36:44 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100517141455.6e1b6f64@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
Message-ID: <15678.1274132204@parc.com>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sun, 16 May 2010 15:13:44 PDT
> Bill Janssen <janssen at parc.com> wrote:
> > 
> > So the patch to the threading code would presumably, for those OSs where
> > the capability exists, try to put all created threads in the same
> > affinity set.
> 
> This is not really a good idea. There's some code which releases the
> GIL, precisely so that you can run several threads (computations) at
> once.

Could the macro that releases the GIL also release the thread affinity?
And the macro that acquires it again set the affinity tag back again?

Bill

From solipsis at pitrou.net  Mon May 17 23:44:59 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 17 May 2010 23:44:59 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <15678.1274132204@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<1743.1274041939@parc.com> <4BF05958.7080206@gmail.com>
	<3430.1274043426@parc.com> <4BF05FAE.7020809@gmail.com>
	<4328.1274045229@parc.com> <4BF065DB.2090703@v.loewis.de>
	<5806.1274048024@parc.com> <20100517141455.6e1b6f64@pitrou.net>
	<15678.1274132204@parc.com>
Message-ID: <1274132699.3054.3.camel@localhost.localdomain>

Le lundi 17 mai 2010 ? 14:36 -0700, Bill Janssen a ?crit :
> 
> Could the macro that releases the GIL also release the thread affinity?

We release the GIL in a variety of situations which don't necessarily
involve heavy computations (such as: waiting for IO or sleeping).
Furthermore, having several threads taking / dropping the GIL would make
the affinity setting / unsetting pattern quite chaotic.

Really, I think the processor affinity solution can only be
application-specific. There doesn't seem to be an easy, generic way of
guessing whether some kind of processor affinity should be requested or
not.

Regards

Antoine.



From martin at v.loewis.de  Mon May 17 23:48:55 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 May 2010 23:48:55 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <14980.1274131702@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>	<1743.1274041939@parc.com>
	<4BF05958.7080206@gmail.com>	<3430.1274043426@parc.com>
	<4BF05FAE.7020809@gmail.com>	<4328.1274045229@parc.com>
	<4BF065DB.2090703@v.loewis.de>	<5806.1274048024@parc.com>
	<20100517141455.6e1b6f64@pitrou.net>	<6887.1274112303@parc.com>	<1274113012.3050.8.camel@localhost.localdomain>	<9017.1274116263@parc.com>
	<20100517193952.48875717@pitrou.net>	<10810.1274120149@parc.com>
	<20100517202847.6d9aa8d3@pitrou.net>	<12368.1274122756@parc.com>
	<hss61o$j7q$1@dough.gmane.org> <14980.1274131702@parc.com>
Message-ID: <4BF1B9C7.9030200@v.loewis.de>

> But some of these, like JCC+pylucene, nltk, and vobject, were
> developed with idiosyncratic funding resources which no longer exist.
> Others, like pyglet, were developed for a completely different
> purpose, and I doubt the developers care what I want.  So,
> realistically, I doubt it will be less than five years.

Make it a GSoC project next summer, and you have them ported next fall.

Regards,
Martin

From nir at winpdb.org  Tue May 18 01:01:32 2010
From: nir at winpdb.org (Nir Aides)
Date: Tue, 18 May 2010 02:01:32 +0300
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
Message-ID: <AANLkTik83UiUwK33KTPCXCs1_ZeJvGMhx7rJXl8UPb8C@mail.gmail.com>

I would like to restart this thread with 2 notes from the lively discussion:

a) Issue 7946 (and this discussion?) concerns Python 3.2
b) The GIL problems are not specific to OSX. The old and new GIL misbehave
on GNU/Linux and Windows too.

[Putting on anti-frying-pan helmet]

Nir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100518/9b85aae2/attachment.html>

From regebro at gmail.com  Tue May 18 08:45:41 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 18 May 2010 08:45:41 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100517150526.0295af9e@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com> 
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com> 
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com> 
	<20100517150526.0295af9e@pitrou.net>
Message-ID: <AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com>

On Mon, May 17, 2010 at 15:05, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 17 May 2010 14:47:25 +0200
> Lennart Regebro <regebro at gmail.com> wrote:
>> On Mon, May 17, 2010 at 14:12, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> > On Mon, 17 May 2010 08:28:08 +0200
>> > Lennart Regebro <regebro at gmail.com> wrote:
>> >> But the scheduler is so simplistic it ends up fighting with the
>> >> OS scheduler, and a large amount of CPU time is used up switching
>> >> instead of executing.
>> >
>> > This is already fixed with py3k.
>>
>> Are you referring to the "New GIL"?
>
> Yes.

At has been shown, it also in certain cases will race with the OS
scheduler, so this is not already fixed, although apparently improved,
if I understand correctly.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From solipsis at pitrou.net  Tue May 18 12:53:12 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 18 May 2010 12:53:12 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com>
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>
	<20100517150526.0295af9e@pitrou.net>
	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com>
Message-ID: <20100518125312.404b0206@pitrou.net>

On Tue, 18 May 2010 08:45:41 +0200
Lennart Regebro <regebro at gmail.com> wrote:
> >>
> >> Are you referring to the "New GIL"?
> >
> > Yes.
> 
> At has been shown, it also in certain cases will race with the OS
> scheduler, so this is not already fixed, although apparently improved,
> if I understand correctly.

"Race" is a strange term here and I'm not sure what you mean.
The issue found out by Dave Beazley can't be reasonably described by
this word, I think.

Please read and understand the issue report mentioned by Nir before
trying to make statements based on rumours heard here and there.

Antoine.

From ncoghlan at gmail.com  Tue May 18 13:22:55 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 18 May 2010 21:22:55 +1000
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTik83UiUwK33KTPCXCs1_ZeJvGMhx7rJXl8UPb8C@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<AANLkTik83UiUwK33KTPCXCs1_ZeJvGMhx7rJXl8UPb8C@mail.gmail.com>
Message-ID: <4BF2788F.901@gmail.com>

Nir Aides wrote:
> I would like to restart this thread with 2 notes from the lively discussion:
> 
> a) Issue 7946 (and this discussion?) concerns Python 3.2
> b) The GIL problems are not specific to OSX. The old and new GIL
> misbehave on GNU/Linux and Windows too.

I think Antoine and Bill went off an a bit of a tangent that *is*
specific to Mac OS X and the old GIL (where a Python application not
only fails to take advantage of additional cores, but actually runs
slower than it does on a less powerful single core machine).

The convoying problem identified in issue 7946 does indeed apply to the
new GIL on multiple platforms. Without reviewing either proposed patch
in detail, I personally am slightly inclined to favour David's suggested
solution, both because I better understand the explanation of how it
works (and simplicity is a virtue from a maintainability point of view),
and because the BFS approach appears to run into trouble when it comes
to identifying a suitable cross platform time reference.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From stefan_ml at behnel.de  Tue May 18 13:38:05 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 18 May 2010 13:38:05 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <14331.1274130547@parc.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>	<1743.1274041939@parc.com>
	<4BF05958.7080206@gmail.com>	<3430.1274043426@parc.com>
	<4BF05FAE.7020809@gmail.com>	<4328.1274045229@parc.com>
	<4BF065DB.2090703@v.loewis.de>	<5806.1274048024@parc.com>
	<20100517141455.6e1b6f64@pitrou.net>	<6887.1274112303@parc.com>	<1274113012.3050.8.camel@localhost.localdomain>	<9017.1274116263@parc.com>
	<4BF1A4FE.3030505@v.loewis.de> <14331.1274130547@parc.com>
Message-ID: <hstu6s$4j1$1@dough.gmane.org>

Bill Janssen, 17.05.2010 23:09:
> Most folks don't use "threads"

Seems like a somewhat reasonable assumption to me.


> they use a higher-level abstraction like the nltk library.

I have my doubts that this applies to "most folks" - likely not even to 
most of those who use threads.

Stefan


From regebro at gmail.com  Tue May 18 14:16:37 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 18 May 2010 14:16:37 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100518125312.404b0206@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com> 
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com> 
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com> 
	<20100517150526.0295af9e@pitrou.net>
	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com> 
	<20100518125312.404b0206@pitrou.net>
Message-ID: <AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com>

On Tue, May 18, 2010 at 12:53, Antoine Pitrou <solipsis at pitrou.net> wrote:
> "Race" is a strange term here and I'm not sure what you mean.
> The issue found out by Dave Beazley can't be reasonably described by
> this word, I think.

OK, maybe "race" is the wrong word. But that doesn't mean the issue
doesn't exist.

> Please read and understand the issue report mentioned by Nir before
> trying to make statements based on rumours heard here and there.

Oh, so Dave Beazleys reports is a rumour now.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From solipsis at pitrou.net  Tue May 18 14:52:32 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 18 May 2010 14:52:32 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com>
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>
	<20100517150526.0295af9e@pitrou.net>
	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com>
	<20100518125312.404b0206@pitrou.net>
	<AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com>
Message-ID: <1274187152.3083.13.camel@localhost.localdomain>


Le mardi 18 mai 2010 ? 14:16 +0200, Lennart Regebro a ?crit :
> > Please read and understand the issue report mentioned by Nir before
> > trying to make statements based on rumours heard here and there.
> 
> Oh, so Dave Beazleys reports is a rumour now.

Your and other people's grandiloquent interpretation of them is.

Now let me ask you a question: did you witness some of the effects
mentioned here? Did it disturb the proper functioning one of your
applications, programs or services?
If yes, please be so kind as to explain how; it will be an useful
datapoint. Bonus points if the issue affects Python 3.2, since that's
really what Nir is talking about.

If not, then do you have any valuable information to contribute to this
discussion?



From regebro at gmail.com  Tue May 18 18:10:24 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 18 May 2010 18:10:24 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <1274187152.3083.13.camel@localhost.localdomain>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com> 
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com> 
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com> 
	<20100517150526.0295af9e@pitrou.net>
	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com> 
	<20100518125312.404b0206@pitrou.net>
	<AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com> 
	<1274187152.3083.13.camel@localhost.localdomain>
Message-ID: <AANLkTilH_eakAGXv9PgWH0IVRSNdp2sjjui_GOAsTI1D@mail.gmail.com>

On Tue, May 18, 2010 at 14:52, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> Le mardi 18 mai 2010 ? 14:16 +0200, Lennart Regebro a ?crit :
>> > Please read and understand the issue report mentioned by Nir before
>> > trying to make statements based on rumours heard here and there.
>>
>> Oh, so Dave Beazleys reports is a rumour now.
>
> Your and other people's grandiloquent interpretation of them is.
>
> Now let me ask you a question: did you witness some of the effects
> mentioned here? Did it disturb the proper functioning one of your
> applications, programs or services?
> If yes, please be so kind as to explain how; it will be an useful
> datapoint. Bonus points if the issue affects Python 3.2, since that's
> really what Nir is talking about.
>
> If not, then do you have any valuable information to contribute to this
> discussion?

I doubt anything I say can be less constructive than your rude comments.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From regebro at gmail.com  Tue May 18 18:10:24 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 18 May 2010 18:10:24 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <1274187152.3083.13.camel@localhost.localdomain>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com> 
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com> 
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com> 
	<20100517150526.0295af9e@pitrou.net>
	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com> 
	<20100518125312.404b0206@pitrou.net>
	<AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com> 
	<1274187152.3083.13.camel@localhost.localdomain>
Message-ID: <AANLkTilH_eakAGXv9PgWH0IVRSNdp2sjjui_GOAsTI1D@mail.gmail.com>

On Tue, May 18, 2010 at 14:52, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> Le mardi 18 mai 2010 ? 14:16 +0200, Lennart Regebro a ?crit :
>> > Please read and understand the issue report mentioned by Nir before
>> > trying to make statements based on rumours heard here and there.
>>
>> Oh, so Dave Beazleys reports is a rumour now.
>
> Your and other people's grandiloquent interpretation of them is.
>
> Now let me ask you a question: did you witness some of the effects
> mentioned here? Did it disturb the proper functioning one of your
> applications, programs or services?
> If yes, please be so kind as to explain how; it will be an useful
> datapoint. Bonus points if the issue affects Python 3.2, since that's
> really what Nir is talking about.
>
> If not, then do you have any valuable information to contribute to this
> discussion?

I doubt anything I say can be less constructive than your rude comments.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From martin at v.loewis.de  Tue May 18 21:43:30 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 18 May 2010 21:43:30 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTilH_eakAGXv9PgWH0IVRSNdp2sjjui_GOAsTI1D@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<201005162253.00229.victor.stinner@haypocalc.com>	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com>
	<20100517141229.13a2d66c@pitrou.net>	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>
	<20100517150526.0295af9e@pitrou.net>	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com>
	<20100518125312.404b0206@pitrou.net>	<AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com>
	<1274187152.3083.13.camel@localhost.localdomain>
	<AANLkTilH_eakAGXv9PgWH0IVRSNdp2sjjui_GOAsTI1D@mail.gmail.com>
Message-ID: <4BF2EDE2.8090906@v.loewis.de>

Lennart Regebro wrote:
> On Tue, May 18, 2010 at 14:52, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> Le mardi 18 mai 2010 ? 14:16 +0200, Lennart Regebro a ?crit :
>>>> Please read and understand the issue report mentioned by Nir before
>>>> trying to make statements based on rumours heard here and there.
>>> Oh, so Dave Beazleys reports is a rumour now.
>> Your and other people's grandiloquent interpretation of them is.
>>
>> Now let me ask you a question: did you witness some of the effects
>> mentioned here? Did it disturb the proper functioning one of your
>> applications, programs or services?
>> If yes, please be so kind as to explain how; it will be an useful
>> datapoint. Bonus points if the issue affects Python 3.2, since that's
>> really what Nir is talking about.
>>
>> If not, then do you have any valuable information to contribute to this
>> discussion?
> 
> I doubt anything I say can be less constructive than your rude comments.

I can understand why Antoine is being offended: it's his implementation
that you attacked. You literally said "At has been shown, it also in
certain cases will race with the OS scheduler, so this is not already
fixed", claiming that it is not fixed

I believe Antoine does consider it fixed, on the grounds that all
counter-examples provided so far are made-up toy examples, rather than
actual applications that still suffer from the original problems.

So please join us in considering the issue fixed unless you can provide
a really world example that demonstrates the contrary.

Regards,
Martin

From digitalxero at gmail.com  Tue May 18 22:26:19 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Tue, 18 May 2010 16:26:19 -0400
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4BF2EDE2.8090906@v.loewis.de>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com> 
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com> 
	<20100517150526.0295af9e@pitrou.net>
	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com> 
	<20100518125312.404b0206@pitrou.net>
	<AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com> 
	<1274187152.3083.13.camel@localhost.localdomain>
	<AANLkTilH_eakAGXv9PgWH0IVRSNdp2sjjui_GOAsTI1D@mail.gmail.com> 
	<4BF2EDE2.8090906@v.loewis.de>
Message-ID: <AANLkTikO7U6phTSlLR4t8VXAeoeaph9JsIYscxjMqfYe@mail.gmail.com>

On Tue, May 18, 2010 at 3:43 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> So please join us in considering the issue fixed unless you can provide
> a really world example that demonstrates the contrary.

The server software I maintain (openrpg) experiences this issue with
when I tried porting the server code to 3.2. Granted it was only a 5
to 7% speed drop over single core, though with the old GIL (py2.5)
there was a 25% to 30% speed drop (It can be running upto 300 IO bound
threads & 30 CPU bound threads) so a net improvement of 20% so I am
more than happy with the new GIL.

I think the new GIL should be given a year or so in the wild before
you start trying to optimize theoretical issues you may run into. If
in a year people come back and have some examples of where a proper
scheduler would help improve speed on multi-core systems even more,
then we can address the issue at that time.

From mike.klaas at gmail.com  Tue May 18 23:39:43 2010
From: mike.klaas at gmail.com (Mike Klaas)
Date: Tue, 18 May 2010 14:39:43 -0700
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
Message-ID: <AANLkTinW_JB2MMeO3BPgrRd8hIPy2dQLKdLDctiaE4co@mail.gmail.com>

On Sun, May 16, 2010 at 1:07 PM, Nir Aides <nir at winpdb.org> wrote:

> Relevant Python issue:?http://bugs.python.org/issue7946

Is there any chance Antoine's "gilinter" patch from that issue might
be applied to python 2.7?  I have been experiencing rare long delays
in simple io operations in multithreaded python applications, and I
suspect that they might be related to this issue.

-Mike

From solipsis at pitrou.net  Tue May 18 23:50:26 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 18 May 2010 23:50:26 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<AANLkTinW_JB2MMeO3BPgrRd8hIPy2dQLKdLDctiaE4co@mail.gmail.com>
Message-ID: <20100518235026.03cad25d@pitrou.net>

On Tue, 18 May 2010 14:39:43 -0700
Mike Klaas <mike.klaas at gmail.com> wrote:
> On Sun, May 16, 2010 at 1:07 PM, Nir Aides <nir at winpdb.org> wrote:
> 
> > Relevant Python issue:?http://bugs.python.org/issue7946
> 
> Is there any chance Antoine's "gilinter" patch from that issue might
> be applied to python 2.7?  I have been experiencing rare long delays
> in simple io operations in multithreaded python applications, and I
> suspect that they might be related to this issue.

There's no chance for this since the patch relies on the new GIL.
(that's unless there's a rush to backport the new GIL in 2.7, of course)

I think your "rare long delays" might be related to the old GIL's own
problems, though. How long are they?

Regards

Antoine.



From barry at python.org  Wed May 19 00:07:16 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 18 May 2010 18:07:16 -0400
Subject: [Python-Dev] Python versions for Ubuntu 10.10 (Maverick Meerkat)
Message-ID: <20100518180716.0bba61df@heresy>

I just wanted to let the python-dev community know about some tracks we had at
the recently concluded Ubuntu Developer Summit in Brussels.  Among the several
Python-related discussions, we talked about what versions of Python will be
supported and default in the next version of Ubuntu (10.10, code name Maverick
Meerkat, to be released in October).

If you're interested in following and participating in this discussion, I've
started a wiki page as the central place to collect information:

https://wiki.ubuntu.com/MaverickMeerkat/TechnicalOverview/Python

While we don't have consensus yet, and we have a lot of footwork to do before
we can make a decision, my goal is to include Python 2.6, 2.7, and 3.2 (beta)
in Ubuntu 10.10.  I'd like to make Python 2.7 the default if possible, and we
will need to get pre-approval to do stable release upgrades to Python 3.2 to
track the move to its final release, which will happen after Ubuntu 10.10. is
released.

It seems that most discussions are happening on the debian-python mailing list
right now.

they-don't-call-it-maverick-for-nuthin'-ly y'rs,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100518/2beab8fa/attachment.pgp>

From me at gustavonarea.net  Wed May 19 00:00:20 2010
From: me at gustavonarea.net (Gustavo Narea)
Date: Tue, 18 May 2010 23:00:20 +0100
Subject: [Python-Dev] Incorrect length of collections.Counter objects /
	Multiplicity function
Message-ID: <201005182300.21079.me@gustavonarea.net>

Hello, everyone.

I've checked the new collections.Counter class and I think I've found a bug:

> >>> from collections import Counter
> >>> c1 = Counter([1, 2, 1, 3, 2])
> >>> c2 = Counter([1, 1, 2, 2, 3])
> >>> c3 = Counter([1, 1, 2, 3])
> >>> c1 == c2 and c3 not in (c1, c2)
> True
> >>> # Perfect, so far. But... There's always a "but":
> ...
> >>> len(c1)
> 3

The length of a Counter is the amount of unique elements. But the length must 
be the cardinality, and the cardinality of a multiset is the total number of 
elements (including duplicates) [1] [2]. The source code mentions that the 
recipe on ActiveState [3] was one of the references, but that recipe has this 
right.

Also, why is it indexed? The indexes of a multiset call to mind the position 
of its elements, but there's no such thing in sets. I think this is 
inconsistent with the built-in set. I would have implemented the multiplicity 
function as a method instead of the indexes:
    c1.get_multiplicity(element)
    # instead of
    c1[element]

Is this the intended behavior? If so, I'd like to propose a proper multiset 
implementation for the standard library (preferably called "Multiset"; should 
I create a PEP?). If not, I can write a patch to fix it, although I'm afraid 
it'd be a backwards incompatible change.

Cheers,

[1] http://en.wikipedia.org/wiki/Multiset#Overview
[2] http://preview.tinyurl.com/smalltalk-bag
[3] http://code.activestate.com/recipes/259174/
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

From me at gustavonarea.net  Wed May 19 00:13:42 2010
From: me at gustavonarea.net (Gustavo Narea)
Date: Tue, 18 May 2010 23:13:42 +0100
Subject: [Python-Dev] Unordered tuples/lists
Message-ID: <201005182313.42935.me@gustavonarea.net>

Hello, everybody.

I've been searching for a data structure like a tuple/list *but* unordered -- 
like a set, but duplicated elements shouldn't be removed. I have not even 
found a recipe, so I'd like to write an implementation and contribute it to 
the "collections" module in the standard library.

This is the situation I have: I have a data structure that represents an 
arithmetic/boolean operation. Operations can be commutative, which means that 
the order of their operands don't change the result of the operation. This is, 
the following operations are equivalent:
    operation(a, b, c) == operation(c, b, a) == operation(b, a, c)
    operation(a, b, a) == operation(a, a, b) == operation(b, a, a)
    operation(a, a) == operation(a, a)

So, I need a type to store the arguments/operands so that if two of these 
collections have the same elements with the same multiplicity, they are 
equivalent, regardless of the order.

A multiset is not exactly what I need: I still need to use the elements in the 
order they were given. For example, the logical conjuction (aka "and" 
operator) has a left and right operands; I need to evaluate the first/left one 
and, if it returned True, then call the second/right one. They must not be 
evaluated in a random order.

To sum up, it would behave like a tuple or a list, except when it's compared 
with another object: They would be equivalent if they're both unordered 
tuples/lists, and have the same elements. There can be mutable and immutable 
editions (UnorderedList and UnorderedTuple, respectively).

I will write a PEP to elaborate on this if you think it'd be nice to have. Or, 
should I have written the PEP first?

Cheers,
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

From solipsis at pitrou.net  Wed May 19 01:10:24 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 19 May 2010 01:10:24 +0200
Subject: [Python-Dev] Summing up
In-Reply-To: <4BF2EDE2.8090906@v.loewis.de>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<201005162253.00229.victor.stinner@haypocalc.com>
	<AANLkTil_XUeycS3Fih5Km8wxvXM9W3FAc5d2ofmSzedb@mail.gmail.com>
	<20100517141229.13a2d66c@pitrou.net>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>
	<20100517150526.0295af9e@pitrou.net>
	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com>
	<20100518125312.404b0206@pitrou.net>
	<AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com>
	<1274187152.3083.13.camel@localhost.localdomain>
	<AANLkTilH_eakAGXv9PgWH0IVRSNdp2sjjui_GOAsTI1D@mail.gmail.com>
	<4BF2EDE2.8090906@v.loewis.de>
Message-ID: <20100519011024.52deb1da@pitrou.net>

On Tue, 18 May 2010 21:43:30 +0200
"Martin v. L?wis" <martin at v.loewis.de> wrote:
> 
> I can understand why Antoine is being offended: it's his implementation
> that you attacked. You literally said "At has been shown, it also in
> certain cases will race with the OS scheduler, so this is not already
> fixed", claiming that it is not fixed
> 
> I believe Antoine does consider it fixed, on the grounds that all
> counter-examples provided so far are made-up toy examples, rather than
> actual applications that still suffer from the original problems.

Ok, this is a good opportunity to try to sum up, from my point of view.

The main problem of the old GIL, which was evidenced in Dave's original
study (not this year's, but the previous one) *is* fixed unless someone
demonstrates otherwise.

It should be noted that witnessing a slight performance degradation on
a multi-core machine is not enough to demonstrate such a thing. The
degradation could be caused by other factors, such as thread migration,
bad OS behaviour, or even locking peculiarities in your own
application, which are not related to the GIL. A good test is whether
performance improves if you play with sys.setswitchinterval().


Dave's newer study regards another issue, which I must stress is also
present in the old GIL algorithm, and therefore must have affected, if
it is serious, real-world applications in 2.x. And indeed, the test I
recently added to ccbench evidences the huge drop in socket I/Os per
second when there's a background CPU thread; this test exercises the
same situation as Dave's demos, only with a less trivial CPU workload:

== CPython 2.7b2+.0 (trunk:81274M) ==
== x86_64 Linux on 'x86_64' ==

--- I/O bandwidth ---

Background CPU task: Pi calculation (Python)

CPU threads=0: 23034.5 packets/s.
CPU threads=1: 6.4 ( 0 %)
CPU threads=2: 15.7 ( 0 %)
CPU threads=3: 13.9 ( 0 %)
CPU threads=4: 20.8 ( 0 %)

(note: I've just changed my desktop machine, so these figures are
different from what I've posted weeks or months ago)


Regardless of the fact that apparently noone reported it in real-world
conditions, we *could* decide that the issue needs fixing. If we
decide so, Nir's approach is the most rigorous one: it tries to fix
the problem thoroughly, rather than graft an additional heuristic. Nir
also has tested his patch on a variety of machines, more so than Dave
and I did with our own patches; he is obviously willing to go forward.

Right now, there are two problems with Nir's proposal:

- first, what Nick said: the difficulty of having reliable
  high-precision cross-platform time sources, which are necessary for
  the BFS algorithm. Ironically, timestamp counters have their own
  problems on multi-core machines (they can go out of sync between
  CPUs). gettimeofday() and clock_gettime() may be precise enough on
  most Unices, though.

- second, the BFS algorithm is not that well-studied, since AFAIK it
  was refused for inclusion in the Linux kernel; someone in the
  python-dev community would therefore have to make sense of, and
  evaluate, its heuristic.

I also don't consider my own patch a very satisfactory "solution",
although it has the reassuring quality of being simple and short (and
easy to revert!).


That said, most of us are programmers and we love to invent ways
of fixing technical issues. It sometimes leads us to consider some
things issues even when they are mostly theoretical. This is why I
am lukewarm on this. I think interested people should focus on
real-world testing (rather than Dave and I's synthetic tests) of the new
GIL, with or without the various patches, and share the results.

Otherwise, Dj Gilcrease's suggestion of waiting for third-party reports
is also a very good one.

Regards

Antoine.

From guido at python.org  Wed May 19 01:11:58 2010
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 May 2010 16:11:58 -0700
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <201005182313.42935.me@gustavonarea.net>
References: <201005182313.42935.me@gustavonarea.net>
Message-ID: <AANLkTinPopVWOT7K-AK5qESo4G51IHUdN89tohuTGB2P@mail.gmail.com>

This is typically called a "bag". Maybe searching for that will help
you find a recipe?

On Tue, May 18, 2010 at 3:13 PM, Gustavo Narea <me at gustavonarea.net> wrote:
> Hello, everybody.
>
> I've been searching for a data structure like a tuple/list *but* unordered --
> like a set, but duplicated elements shouldn't be removed. I have not even
> found a recipe, so I'd like to write an implementation and contribute it to
> the "collections" module in the standard library.
>
> This is the situation I have: I have a data structure that represents an
> arithmetic/boolean operation. Operations can be commutative, which means that
> the order of their operands don't change the result of the operation. This is,
> the following operations are equivalent:
> ? ?operation(a, b, c) == operation(c, b, a) == operation(b, a, c)
> ? ?operation(a, b, a) == operation(a, a, b) == operation(b, a, a)
> ? ?operation(a, a) == operation(a, a)
>
> So, I need a type to store the arguments/operands so that if two of these
> collections have the same elements with the same multiplicity, they are
> equivalent, regardless of the order.
>
> A multiset is not exactly what I need: I still need to use the elements in the
> order they were given. For example, the logical conjuction (aka "and"
> operator) has a left and right operands; I need to evaluate the first/left one
> and, if it returned True, then call the second/right one. They must not be
> evaluated in a random order.
>
> To sum up, it would behave like a tuple or a list, except when it's compared
> with another object: They would be equivalent if they're both unordered
> tuples/lists, and have the same elements. There can be mutable and immutable
> editions (UnorderedList and UnorderedTuple, respectively).
>
> I will write a PEP to elaborate on this if you think it'd be nice to have. Or,
> should I have written the PEP first?
>
> Cheers,
> --
> Gustavo Narea <xri://=Gustavo>.
> | Tech blog: =Gustavo/(+blog)/tech ?~ ?About me: =Gustavo/about |
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From benjamin at python.org  Wed May 19 01:22:49 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 18 May 2010 19:22:49 -0400
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <AANLkTinPopVWOT7K-AK5qESo4G51IHUdN89tohuTGB2P@mail.gmail.com>
References: <201005182313.42935.me@gustavonarea.net>
	<AANLkTinPopVWOT7K-AK5qESo4G51IHUdN89tohuTGB2P@mail.gmail.com>
Message-ID: <AANLkTinrcKaN2_CACYB9OC_-HtBJK01nfl5fnWcl5JeI@mail.gmail.com>

2010/5/18 Guido van Rossum <guido at python.org>:
> This is typically called a "bag". Maybe searching for that will help
> you find a recipe?

Yes, and we have one in Python 2.7+ called collections.Counter.


-- 
Regards,
Benjamin

From steve at pearwood.info  Wed May 19 01:28:29 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 19 May 2010 09:28:29 +1000
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <201005182313.42935.me@gustavonarea.net>
References: <201005182313.42935.me@gustavonarea.net>
Message-ID: <201005190928.30079.steve@pearwood.info>

On Wed, 19 May 2010 08:13:42 am Gustavo Narea wrote:
> Hello, everybody.
>
> I've been searching for a data structure like a tuple/list *but*
> unordered -- like a set, but duplicated elements shouldn't be
> removed. I have not even found a recipe, so I'd like to write an
> implementation and contribute it to the "collections" module in the
> standard library.
[...]
> So, I need a type to store the arguments/operands so that if two of
> these collections have the same elements with the same multiplicity,
> they are equivalent, regardless of the order.

If I've understood your requirements, this should probably do it:

# Untested.
class MyCounter(collections.Counter):
    def __eq__(self, other):
        try:
            other = other.items()
        except AttributeError:
            return False
        return sorted(self.items()) == sorted(other)
    def __ne__(self, other):
        return not self == other


> A multiset is not exactly what I need: I still need to use the
> elements in the order they were given. For example, the logical
> conjuction (aka "and" operator) has a left and right operands; I need
> to evaluate the first/left one and, if it returned True, then call
> the second/right one. They must not be evaluated in a random order.

These requirements seem specialised enough to me that I expect it would 
be wasteful to add your class to the standard library. It's not quite 
an OrderedMultiSet (perhaps built on an OrderedDict), but a MultiSet 
that sometimes behaves as ordered and sometimes doesn't.


> To sum up, it would behave like a tuple or a list, except when it's
> compared with another object: They would be equivalent if they're
> both unordered tuples/lists, and have the same elements. There can be
> mutable and immutable editions (UnorderedList and UnorderedTuple,
> respectively).
>
> I will write a PEP to elaborate on this if you think it'd be nice to
> have. Or, should I have written the PEP first?

Neither. I think you will need to demonstrate the need for these first. 
The Python standard library doesn't generally add data types on the 
basis of theoretical nice-to-have, or even for a single person's 
use-case (unless that person is Guido *wink*). Your first step should 
be to publish the classes somewhere else. If they are small enough, the 
Python recipes on ActiveState would be good, otherwise PyPI. 

If there is a demonstrated need for these classes, and you (or somebody 
else) is willing to maintain them, then they may be added to the 
collections module (perhaps for Python 3.3, since I think 3.2 and 2.7 
are in feature-freeze).

I suggest you also take this idea to python-list at python.org or 
comp.lang.python first, to see whether there is any enthusiasm from the 
wider Python community.



-- 
Steven D'Aprano

From phd at phd.pp.ru  Wed May 19 01:53:33 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Wed, 19 May 2010 03:53:33 +0400
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <201005182313.42935.me@gustavonarea.net>
References: <201005182313.42935.me@gustavonarea.net>
Message-ID: <20100518235333.GA31031@phd.pp.ru>

On Tue, May 18, 2010 at 11:13:42PM +0100, Gustavo Narea wrote:
> To sum up, it would behave like a tuple or a list, except when it's compared 
> with another object: They would be equivalent if they're both unordered 
> tuples/lists, and have the same elements. There can be mutable and immutable 
> editions (UnorderedList and UnorderedTuple, respectively).

class UnorderedList(list):
    def __eq__(self, other):
        if not isinstance(other, UnorderedList):
            return False
        return sorted(self) == sorted(other)

    def __ne__(self, other):
        return not self.__eq__(other)

   Do you need more than that?

Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From ben+python at benfinney.id.au  Wed May 19 02:19:07 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 19 May 2010 10:19:07 +1000
Subject: [Python-Dev] Unordered tuples/lists
References: <201005182313.42935.me@gustavonarea.net>
Message-ID: <87y6fgiw04.fsf@benfinney.id.au>

Gustavo Narea <me at gustavonarea.net> writes:

> I've been searching for a data structure like a tuple/list *but*
> unordered -- like a set, but duplicated elements shouldn't be removed.

By that description, you're looking for the ?Bag? pattern.

[?]
> A multiset is not exactly what I need: I still need to use the
> elements in the order they were given.

This appears to contradict your explicit request that the collection
type be unordered.

[?]
> To sum up, it would behave like a tuple or a list, except when it's
> compared with another object: They would be equivalent if they're both
> unordered tuples/lists, and have the same elements. There can be
> mutable and immutable editions (UnorderedList and UnorderedTuple,
> respectively).

Okay, so you want something rather special; it would be misleading to
call this an unordered type, since you *do* want ordering preserved.
It's only the comparison operators that would ignore ordering.

AFAIK you'll need to design and implement this yourself.

> I will write a PEP to elaborate on this if you think it'd be nice to
> have. Or, should I have written the PEP first?

Either way; but I think this discussion belongs on ?python-ideas? while
the behaviour is still being designed.

-- 
 \      ?On the internet you simply can't outsource parenting.? ?Eliza |
  `\      Cussen, _Top 10 Internet Filter Lies_, The Punch, 2010-03-25 |
_o__)                                                                  |
Ben Finney


From mike.klaas at gmail.com  Wed May 19 02:26:44 2010
From: mike.klaas at gmail.com (Mike Klaas)
Date: Tue, 18 May 2010 17:26:44 -0700
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <20100518235026.03cad25d@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<AANLkTinW_JB2MMeO3BPgrRd8hIPy2dQLKdLDctiaE4co@mail.gmail.com>
	<20100518235026.03cad25d@pitrou.net>
Message-ID: <AANLkTinLasNZBUuSDONGg1AP2YdTEyEsqXw1q8Joh7Qy@mail.gmail.com>

On Tue, May 18, 2010 at 2:50 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> There's no chance for this since the patch relies on the new GIL.
> (that's unless there's a rush to backport the new GIL in 2.7, of course)

Thanks I missed that detail.

> I think your "rare long delays" might be related to the old GIL's own
> problems, though. How long are they?

Typically between 20 and 60s.  This is the time it takes to send and
receive a single small packet on an already-active tcp connection to
ensure it is still alive.   Most of the time it is < 1ms.  I don't
have strong evidence that GIL issues are causing the problem, because
I can't reliably reproduce the issue.  But the general setup is
similar (one thread doing light io experiencing odd delays in a
process with multiple threads that are often cpu-bound, on a
multi-core machine)

thanks,
-Mike

From dave at dabeaz.com  Wed May 19 02:35:48 2010
From: dave at dabeaz.com (David Beazley)
Date: Tue, 18 May 2010 19:35:48 -0500
Subject: [Python-Dev] Summing up
In-Reply-To: <mailman.1263.1274226817.32708.python-dev@python.org>
References: <mailman.1263.1274226817.32708.python-dev@python.org>
Message-ID: <FC982303-36A8-4A5F-B503-B886DF024E57@dabeaz.com>

Antoine,

This is a pretty good summary that mirrors my thoughts on the GIL matter as well.   In the big picture, I do think it's desirable for Python to address the multicore performance issue--namely to not have the performance needlessly thrashed in that environment.   The original new GIL addressed this.

The I/O convoy effect problem is more subtle.   Personally, I think it's an issue that at least merits further study because trying to overlap I/O with computation is a known programming technique that  might be useful for people using Python to do message passing, distributed computation, etc.   As an example, the multiprocessing module uses threads as part of its queue implementation.  Is it impacted by convoying?  I honestly don't know.  I agree that getting some more real-world experience would be useful.

Cheers,
Dave


> From: Antoine Pitrou <solipsis at pitrou.net>
> 
> Ok, this is a good opportunity to try to sum up, from my point of view.
> 
> The main problem of the old GIL, which was evidenced in Dave's original
> study (not this year's, but the previous one) *is* fixed unless someone
> demonstrates otherwise.
> 
> It should be noted that witnessing a slight performance degradation on
> a multi-core machine is not enough to demonstrate such a thing. The
> degradation could be caused by other factors, such as thread migration,
> bad OS behaviour, or even locking peculiarities in your own
> application, which are not related to the GIL. A good test is whether
> performance improves if you play with sys.setswitchinterval().
> 
> 
> Dave's newer study regards another issue, which I must stress is also
> present in the old GIL algorithm, and therefore must have affected, if
> it is serious, real-world applications in 2.x. And indeed, the test I
> recently added to ccbench evidences the huge drop in socket I/Os per
> second when there's a background CPU thread; this test exercises the
> same situation as Dave's demos, only with a less trivial CPU workload:
> 
> == CPython 2.7b2+.0 (trunk:81274M) ==
> == x86_64 Linux on 'x86_64' ==
> 
> --- I/O bandwidth ---
> 
> Background CPU task: Pi calculation (Python)
> 
> CPU threads=0: 23034.5 packets/s.
> CPU threads=1: 6.4 ( 0 %)
> CPU threads=2: 15.7 ( 0 %)
> CPU threads=3: 13.9 ( 0 %)
> CPU threads=4: 20.8 ( 0 %)
> 
> (note: I've just changed my desktop machine, so these figures are
> different from what I've posted weeks or months ago)
> 
> 
> Regardless of the fact that apparently noone reported it in real-world
> conditions, we *could* decide that the issue needs fixing. If we
> decide so, Nir's approach is the most rigorous one: it tries to fix
> the problem thoroughly, rather than graft an additional heuristic. Nir
> also has tested his patch on a variety of machines, more so than Dave
> and I did with our own patches; he is obviously willing to go forward.
> 
> Right now, there are two problems with Nir's proposal:
> 
> - first, what Nick said: the difficulty of having reliable
>  high-precision cross-platform time sources, which are necessary for
>  the BFS algorithm. Ironically, timestamp counters have their own
>  problems on multi-core machines (they can go out of sync between
>  CPUs). gettimeofday() and clock_gettime() may be precise enough on
>  most Unices, though.
> 
> - second, the BFS algorithm is not that well-studied, since AFAIK it
>  was refused for inclusion in the Linux kernel; someone in the
>  python-dev community would therefore have to make sense of, and
>  evaluate, its heuristic.
> 
> I also don't consider my own patch a very satisfactory "solution",
> although it has the reassuring quality of being simple and short (and
> easy to revert!).
> 
> 
> That said, most of us are programmers and we love to invent ways
> of fixing technical issues. It sometimes leads us to consider some
> things issues even when they are mostly theoretical. This is why I
> am lukewarm on this. I think interested people should focus on
> real-world testing (rather than Dave and I's synthetic tests) of the new
> GIL, with or without the various patches, and share the results.
> 
> Otherwise, Dj Gilcrease's suggestion of waiting for third-party reports
> is also a very good one.
> 
> Regards
> 
> Antoine.

From solipsis at pitrou.net  Wed May 19 02:43:15 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 19 May 2010 02:43:15 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTinLasNZBUuSDONGg1AP2YdTEyEsqXw1q8Joh7Qy@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<AANLkTinW_JB2MMeO3BPgrRd8hIPy2dQLKdLDctiaE4co@mail.gmail.com>
	<20100518235026.03cad25d@pitrou.net>
	<AANLkTinLasNZBUuSDONGg1AP2YdTEyEsqXw1q8Joh7Qy@mail.gmail.com>
Message-ID: <20100519024315.28f6f19c@pitrou.net>

On Tue, 18 May 2010 17:26:44 -0700
Mike Klaas <mike.klaas at gmail.com> wrote:
> 
> > I think your "rare long delays" might be related to the old GIL's own
> > problems, though. How long are they?
> 
> Typically between 20 and 60s.

You mean milliseconds I suppose?
If it's the case, then you may simply be witnessing garbage collection
runs. I've measured garbage collection runs of about 50 ms each on a
Web application, with the full framework loaded and a bunch of objects
in memory.

If you really meant seconds, it looks a bit too high to be GIL-related.
What kind of things are the CPU threads doing?

Regards

Antoine.

From ncoghlan at gmail.com  Wed May 19 05:20:36 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 19 May 2010 13:20:36 +1000
Subject: [Python-Dev] Summing up
In-Reply-To: <FC982303-36A8-4A5F-B503-B886DF024E57@dabeaz.com>
References: <mailman.1263.1274226817.32708.python-dev@python.org>
	<FC982303-36A8-4A5F-B503-B886DF024E57@dabeaz.com>
Message-ID: <4BF35904.50503@gmail.com>

On 19/05/10 10:35, David Beazley wrote:
> Antoine,
>
> This is a pretty good summary that mirrors my thoughts on the GIL
> matter as well.   In the big picture, I do think it's desirable for
> Python to address the multicore performance issue--namely to not have
> the performance needlessly thrashed in that environment.   The
> original new GIL addressed this.
>
> The I/O convoy effect problem is more subtle.   Personally, I think
> it's an issue that at least merits further study because trying to
> overlap I/O with computation is a known programming technique that
> might be useful for people using Python to do message passing,
> distributed computation, etc.   As an example, the multiprocessing
> module uses threads as part of its queue implementation.  Is it
> impacted by convoying?  I honestly don't know.  I agree that getting
> some more real-world experience would be useful.

My takeaway from this discussion is that:

A. we should leave the new GIL in 3.2 in its current (relatively) simple 
form for now, keeping the various patches in issue 7946 in our back 
pocket if someone finds real world examples of the convoying effect 
discussed there. The idea here being that we shouldn't complicate the 
implementation without some solid evidence that doing so is actually 
necessary for real world workloads.

B. some more thought should be given to incorporating the new GIL into 
2.7. However, this requires two things:
  - an update to the patch in 7753 to either retain the old GIL for 
platforms not supported by the new GIL or else to make the new GIL a 
configure option
  - Benjamin accepting that patch (as it would likely mean adding 
another beta release to the cycle)

In the absence of an updated version of the 7753 patch, backporting the 
new GIL to 2.7 isn't really a serious option.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From martin at v.loewis.de  Wed May 19 08:32:03 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 May 2010 08:32:03 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <AANLkTikO7U6phTSlLR4t8VXAeoeaph9JsIYscxjMqfYe@mail.gmail.com>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<20100517141229.13a2d66c@pitrou.net>	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>
	<20100517150526.0295af9e@pitrou.net>	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com>
	<20100518125312.404b0206@pitrou.net>	<AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com>
	<1274187152.3083.13.camel@localhost.localdomain>	<AANLkTilH_eakAGXv9PgWH0IVRSNdp2sjjui_GOAsTI1D@mail.gmail.com>
	<4BF2EDE2.8090906@v.loewis.de>
	<AANLkTikO7U6phTSlLR4t8VXAeoeaph9JsIYscxjMqfYe@mail.gmail.com>
Message-ID: <4BF385E3.9030903@v.loewis.de>

> I think the new GIL should be given a year or so in the wild before
> you start trying to optimize theoretical issues you may run into. If
> in a year people come back and have some examples of where a proper
> scheduler would help improve speed on multi-core systems even more,
> then we can address the issue at that time.

Exactly my feelings.

Regards,
Martin

From dave at dabeaz.com  Wed May 19 13:00:04 2010
From: dave at dabeaz.com (David Beazley)
Date: Wed, 19 May 2010 06:00:04 -0500
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <mailman.49.1274263209.17637.python-dev@python.org>
References: <mailman.49.1274263209.17637.python-dev@python.org>
Message-ID: <E9A81720-26FD-4984-BF6D-66A9950CB383@dabeaz.com>

> From: "Martin v. L?wis" <martin at v.loewis.de>
> To: Dj Gilcrease <digitalxero at gmail.com>
> Cc: python-dev at python.org
> Subject: Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
> Message-ID: <4BF385E3.9030903 at v.loewis.de>
> Content-Type: text/plain; charset=ISO-8859-1
> 
>> I think the new GIL should be given a year or so in the wild before
>> you start trying to optimize theoretical issues you may run into. If
>> in a year people come back and have some examples of where a proper
>> scheduler would help improve speed on multi-core systems even more,
>> then we can address the issue at that time.
> 
> Exactly my feelings.
> 

Although I don't agree that the problem of I/O convoying is merely some "theoretical issue", I would agree with a go-slow approach---after all, the new GIL hasn't even appeared in any actual release yet.    It might be a good idea to prominently document the fact that the new GIL has some known performance issues (e.g., possible I/O convoying), but that feedback concerning the performance of real-world applications is desired.

Cheers,
Dave


From janssen at parc.com  Wed May 19 18:04:26 2010
From: janssen at parc.com (Bill Janssen)
Date: Wed, 19 May 2010 09:04:26 PDT
Subject: [Python-Dev] Summing up
In-Reply-To: <4BF35904.50503@gmail.com>
References: <mailman.1263.1274226817.32708.python-dev@python.org>
	<FC982303-36A8-4A5F-B503-B886DF024E57@dabeaz.com>
	<4BF35904.50503@gmail.com>
Message-ID: <38562.1274285066@parc.com>

Nick Coghlan <ncoghlan at gmail.com> wrote:

> B. some more thought should be given to incorporating the new GIL into
> 2.7. However, this requires two things:
>  - an update to the patch in 7753 to either retain the old GIL for
> platforms not supported by the new GIL or else to make the new GIL a
> configure option
>  - Benjamin accepting that patch (as it would likely mean adding
> another beta release to the cycle)

I think I agree with that, at least for 2.7.0.  My fallback plan is to
write an extension that does the thread affinity hack, monkeypatching
threading.py, only for OS X.  People like me who are really annoyed with
the continued presence of the multicore bug can just add that to site.py
or sitecustomize.py (does that still work?).  I'd like to find out first
whether you can actually change the thread affinity on OS X after the
thread has been started.

> In the absence of an updated version of the 7753 patch, backporting
> the new GIL to 2.7 isn't really a serious option.

Right.  At least, not for this revision level.

Bill

From janssen at parc.com  Wed May 19 18:09:16 2010
From: janssen at parc.com (Bill Janssen)
Date: Wed, 19 May 2010 09:09:16 PDT
Subject: [Python-Dev] pybuildbot.identify?
Message-ID: <38589.1274285356@parc.com>

The PPC buildbots are running pretty well, now that I've opened a few
more ports, but I'd like to find this script "pybuildbot.identify" that
they keep complaining about, and install it.  I've poked around the
Python sources, but haven't found it.

Anyone know where to get it from?

Thanks.

Bill

From debatem1 at gmail.com  Wed May 19 18:27:36 2010
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 19 May 2010 09:27:36 -0700
Subject: [Python-Dev] Summing up
In-Reply-To: <20100519011024.52deb1da@pitrou.net>
References: <AANLkTikeehdcl42hXyUuVTZUNhgNTnhu-PHrWHYvvTLO@mail.gmail.com>
	<AANLkTil5njaq6cJiwOr65_0bkQQZFtnI7V6_9YVCxW1G@mail.gmail.com>
	<20100517150526.0295af9e@pitrou.net>
	<AANLkTilDPJ954NarpZH5j0fF0N-Sxm7wFWqHGZgeiJaV@mail.gmail.com>
	<20100518125312.404b0206@pitrou.net>
	<AANLkTim_IOhfE7tAwETWyA2n2SAgyBKkb4yL2hqRoTOT@mail.gmail.com>
	<1274187152.3083.13.camel@localhost.localdomain>
	<AANLkTilH_eakAGXv9PgWH0IVRSNdp2sjjui_GOAsTI1D@mail.gmail.com>
	<4BF2EDE2.8090906@v.loewis.de> <20100519011024.52deb1da@pitrou.net>
Message-ID: <AANLkTikVKKDwaBcWpn4F7sKf1julKlXTJoGlhj6DEP4o@mail.gmail.com>

On Tue, May 18, 2010 at 4:10 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 18 May 2010 21:43:30 +0200

<snip>

> Regardless of the fact that apparently noone reported it in real-world
> conditions, we *could* decide that the issue needs fixing. If we
> decide so, Nir's approach is the most rigorous one: it tries to fix
> the problem thoroughly, rather than graft an additional heuristic. Nir
> also has tested his patch on a variety of machines, more so than Dave
> and I did with our own patches; he is obviously willing to go forward.
>
> Right now, there are two problems with Nir's proposal:
>
> - first, what Nick said: the difficulty of having reliable
> ?high-precision cross-platform time sources, which are necessary for
> ?the BFS algorithm. Ironically, timestamp counters have their own
> ?problems on multi-core machines (they can go out of sync between
> ?CPUs). gettimeofday() and clock_gettime() may be precise enough on
> ?most Unices, though.
>
> - second, the BFS algorithm is not that well-studied, since AFAIK it
> ?was refused for inclusion in the Linux kernel; someone in the
> ?python-dev community would therefore have to make sense of, and
> ?evaluate, its heuristic.

I don't have the expertise to do this, but I'll be playing with the
patch over the next few weeks, so if there's a specific piece of
data you want, let me know.

Geremy Condra

From doko at ubuntu.com  Wed May 19 18:27:49 2010
From: doko at ubuntu.com (Matthias Klose)
Date: Wed, 19 May 2010 18:27:49 +0200
Subject: [Python-Dev] pybuildbot.identify?
In-Reply-To: <38589.1274285356@parc.com>
References: <38589.1274285356@parc.com>
Message-ID: <4BF41185.3090600@ubuntu.com>

On 19.05.2010 18:09, Bill Janssen wrote:
> The PPC buildbots are running pretty well, now that I've opened a few
> more ports, but I'd like to find this script "pybuildbot.identify" that
> they keep complaining about, and install it.  I've poked around the
> Python sources, but haven't found it.
>
> Anyone know where to get it from?

it's just a custom script, currently very linux specific. attached here. maybe 
it's worth to check it in and extend it for other configurations.

   Matthias
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pybuildbot.identify
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100519/bd2a0d0c/attachment.ksh>

From martin at v.loewis.de  Wed May 19 19:03:13 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 May 2010 19:03:13 +0200
Subject: [Python-Dev] pybuildbot.identify?
In-Reply-To: <38589.1274285356@parc.com>
References: <38589.1274285356@parc.com>
Message-ID: <4BF419D1.3060000@v.loewis.de>

Bill Janssen wrote:
> The PPC buildbots are running pretty well, now that I've opened a few
> more ports, but I'd like to find this script "pybuildbot.identify" that
> they keep complaining about, and install it.  I've poked around the
> Python sources, but haven't found it.

You don't need to bother about it - it's fine that it's missing.

Regards,
Martin

From janssen at parc.com  Wed May 19 19:58:44 2010
From: janssen at parc.com (Bill Janssen)
Date: Wed, 19 May 2010 10:58:44 PDT
Subject: [Python-Dev] pybuildbot.identify?
In-Reply-To: <4BF41185.3090600@ubuntu.com>
References: <38589.1274285356@parc.com> <4BF41185.3090600@ubuntu.com>
Message-ID: <43886.1274291924@parc.com>

Matthias Klose <doko at ubuntu.com> wrote:

> On 19.05.2010 18:09, Bill Janssen wrote:
> > The PPC buildbots are running pretty well, now that I've opened a few
> > more ports, but I'd like to find this script "pybuildbot.identify" that
> > they keep complaining about, and install it.  I've poked around the
> > Python sources, but haven't found it.
> >
> > Anyone know where to get it from?
> 
> it's just a custom script, currently very linux specific. attached
> here. maybe it's worth to check it in and extend it for other
> configurations.
> 
>   Matthias

Well, please check it in somewhere, at least.  Thanks.

Bill


From martin at v.loewis.de  Wed May 19 21:45:39 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 May 2010 21:45:39 +0200
Subject: [Python-Dev] pybuildbot.identify?
In-Reply-To: <43886.1274291924@parc.com>
References: <38589.1274285356@parc.com> <4BF41185.3090600@ubuntu.com>
	<43886.1274291924@parc.com>
Message-ID: <4BF43FE3.8040803@v.loewis.de>

Bill Janssen wrote:
> Matthias Klose <doko at ubuntu.com> wrote:
> 
>> On 19.05.2010 18:09, Bill Janssen wrote:
>>> The PPC buildbots are running pretty well, now that I've opened a few
>>> more ports, but I'd like to find this script "pybuildbot.identify" that
>>> they keep complaining about, and install it.  I've poked around the
>>> Python sources, but haven't found it.
>>>
>>> Anyone know where to get it from?
>> it's just a custom script, currently very linux specific. attached
>> here. maybe it's worth to check it in and extend it for other
>> configurations.
>>
>>   Matthias
> 
> Well, please check it in somewhere, at least.  Thanks.

Can you please investigate why it was causing an error message in the
first place? The which test should have determined that the script
doesn't exist, and then it shouldn't have been invoked. So I'm puzzled
why there is an error message at all.

I slightly object to checking it in. It's *not* a mandatory part of a
buildbot slave setup, and if you don't know what it is, you don't need
to know.

Regards,
Martin

From peter.a.portante at gmail.com  Wed May 19 22:17:22 2010
From: peter.a.portante at gmail.com (Peter Portante)
Date: Wed, 19 May 2010 16:17:22 -0400
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <E9A81720-26FD-4984-BF6D-66A9950CB383@dabeaz.com>
Message-ID: <C819BF92.3B2D8%peter.a.portante@gmail.com>

Does anybody think that by having problems with the new GIL that it might
further weaken the adoption rate for 3k? -peter


On 5/19/10 7:00 AM, "David Beazley" <dave at dabeaz.com> wrote:

>> From: "Martin v. L?wis" <martin at v.loewis.de>
>> To: Dj Gilcrease <digitalxero at gmail.com>
>> Cc: python-dev at python.org
>> Subject: Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
>> Message-ID: <4BF385E3.9030903 at v.loewis.de>
>> Content-Type: text/plain; charset=ISO-8859-1
>> 
>>> I think the new GIL should be given a year or so in the wild before
>>> you start trying to optimize theoretical issues you may run into. If
>>> in a year people come back and have some examples of where a proper
>>> scheduler would help improve speed on multi-core systems even more,
>>> then we can address the issue at that time.
>> 
>> Exactly my feelings.
>> 
> 
> Although I don't agree that the problem of I/O convoying is merely some
> "theoretical issue", I would agree with a go-slow approach---after all, the
> new GIL hasn't even appeared in any actual release yet.    It might be a good
> idea to prominently document the fact that the new GIL has some known
> performance issues (e.g., possible I/O convoying), but that feedback
> concerning the performance of real-world applications is desired.
> 
> Cheers,
> Dave
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com



From martin at v.loewis.de  Wed May 19 22:34:50 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 May 2010 22:34:50 +0200
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <C819BF92.3B2D8%peter.a.portante@gmail.com>
References: <C819BF92.3B2D8%peter.a.portante@gmail.com>
Message-ID: <4BF44B6A.5010805@v.loewis.de>

Peter Portante wrote:
> Does anybody think that by having problems with the new GIL that it might
> further weaken the adoption rate for 3k? -peter

No, to the contrary. By having the new GIL being superior to the old
implementation, the adoption rate for 3k will increase.

Regards,
Martin

From digitalxero at gmail.com  Wed May 19 22:39:14 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Wed, 19 May 2010 16:39:14 -0400
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <C819BF92.3B2D8%peter.a.portante@gmail.com>
References: <E9A81720-26FD-4984-BF6D-66A9950CB383@dabeaz.com> 
	<C819BF92.3B2D8%peter.a.portante@gmail.com>
Message-ID: <AANLkTiloPSS_oAp6amZ23qL0DLMxPMOa3Pb7Xz9cutAT@mail.gmail.com>

On Wed, May 19, 2010 at 4:17 PM, Peter Portante
<peter.a.portante at gmail.com> wrote:
> Does anybody think that by having problems with the new GIL that it might
> further weaken the adoption rate for 3k? -peter

Nope, because the remaining issues with the new GIL affect the old GIL
as well, and have yet to be proven to affect real world apps. My
server is the only real world example I have seen and the modest 5%
slow down it experiences on the new GIL & multi-core vs new GIL &
single core can potentially be explained by other issues. The old GIL
has a 25 to 30% slowdown on multi vs single core so over all the new
GIL gives my server a 20% speed boost.

Incremental upgrades to core functionality like the GIL is the better
way to go then to try an optimize every facet of it before hand,
because some of those optimizations may have unintended side effects
in real world apps and if you attempt all the optimizations at once it
makes it harder to figure out which is the cause of the unintended
side effects.

From dave at dabeaz.com  Wed May 19 22:49:02 2010
From: dave at dabeaz.com (David Beazley)
Date: Wed, 19 May 2010 15:49:02 -0500
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4BF44B6A.5010805@v.loewis.de>
References: <C819BF92.3B2D8%peter.a.portante@gmail.com>
	<4BF44B6A.5010805@v.loewis.de>
Message-ID: <44A67E74-8D06-4A04-972C-B36EDCC1447E@dabeaz.com>

Yes, but the million dollar question is whether or not it really is superior with the I/O convoying problem in the current implementation (an effect that is substantially worse with the new GIL than with the old one by the way).  Personally, I think the convoying issue is something that will have to be fixed eventually.   Although, I also agree that it merits more study--especially with real-world applications.

Cheers,
Dave

On May 19, 2010, at 3:34 PM, Martin v. L?wis wrote:

> Peter Portante wrote:
>> Does anybody think that by having problems with the new GIL that it might
>> further weaken the adoption rate for 3k? -peter
> 
> No, to the contrary. By having the new GIL being superior to the old
> implementation, the adoption rate for 3k will increase.
> 
> Regards,
> Martin


From janssen at parc.com  Wed May 19 23:27:27 2010
From: janssen at parc.com (Bill Janssen)
Date: Wed, 19 May 2010 14:27:27 PDT
Subject: [Python-Dev] pybuildbot.identify?
In-Reply-To: <4BF43FE3.8040803@v.loewis.de>
References: <38589.1274285356@parc.com> <4BF41185.3090600@ubuntu.com>
	<43886.1274291924@parc.com> <4BF43FE3.8040803@v.loewis.de>
Message-ID: <46492.1274304447@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> Bill Janssen wrote:
> > Matthias Klose <doko at ubuntu.com> wrote:
> > 
> >> On 19.05.2010 18:09, Bill Janssen wrote:
> >>> The PPC buildbots are running pretty well, now that I've opened a few
> >>> more ports, but I'd like to find this script "pybuildbot.identify" that
> >>> they keep complaining about, and install it.  I've poked around the
> >>> Python sources, but haven't found it.
> >>>
> >>> Anyone know where to get it from?
> >> it's just a custom script, currently very linux specific. attached
> >> here. maybe it's worth to check it in and extend it for other
> >> configurations.
> >>
> >>   Matthias
> > 
> > Well, please check it in somewhere, at least.  Thanks.
> 
> Can you please investigate why it was causing an error message in the
> first place? The which test should have determined that the script
> doesn't exist, and then it shouldn't have been invoked. So I'm puzzled
> why there is an error message at all.

I agree.  Must be something in the buildbot code.

> I slightly object to checking it in. It's *not* a mandatory part of a
> buildbot slave setup, and if you don't know what it is, you don't need
> to know.

Well, if it failed silently, I would agree.  As it is, I created a Darwin
version and sent it off to Matthias.

Bill

From janssen at parc.com  Wed May 19 23:35:25 2010
From: janssen at parc.com (Bill Janssen)
Date: Wed, 19 May 2010 14:35:25 PDT
Subject: [Python-Dev] Fixing the GIL (with a BFS scheduler)
In-Reply-To: <4BF44B6A.5010805@v.loewis.de>
References: <C819BF92.3B2D8%peter.a.portante@gmail.com>
	<4BF44B6A.5010805@v.loewis.de>
Message-ID: <46623.1274304925@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> Peter Portante wrote:
> > Does anybody think that by having problems with the new GIL that it might
> > further weaken the adoption rate for 3k? -peter
> 
> No, to the contrary. By having the new GIL being superior to the old
> implementation, the adoption rate for 3k will increase.

Well, if that's a strategy, there's still time to make 2+2 equal 5 in
the "old" implementation.

Bill

From yaniv at aknin.name  Thu May 20 00:13:03 2010
From: yaniv at aknin.name (Yaniv Aknin)
Date: Thu, 20 May 2010 08:13:03 +1000
Subject: [Python-Dev] Documenting [C]Python's Internals
Message-ID: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>

Hi,

I wanted to let python-dev know about a series of articles about CPython's
internals I'm publishing under the collective title "Guido's Python"* (
http://tech.blog.aknin.name/tag/guidos-python/). Three articles already were
published already, more are planned (mainly focused on CPython/py3k, but
comparisons with other implementations may also be covered; we'll see). So
far I've done an introduction/whirlwind tour of Py_Main and a two-article
in-depth review of the (new-style) object system.

I'm sharing this with you (and hope you care) due to three reasons, probably
in escalating importance:
(a) Maybe some of python-dev's readers would be interested (possibly the
newer and more silent members).

(b) Maybe my scales are wrong, but I was a bit surprised by the number of
readers (>20,000 in the past two weeks); I wouldn't want to mislead such a
reader base and would be happy if a veteran here would be interested in
aiding by technically proofing the material (shan't be too hard I hope, feel
free to contact me directly if qualified and interested).

(c) While the content is currently geared to be blog-oriented, if it's found
worthy by the group I'd be delighted to formulate it into something more
'reference-material-ish' and give it back to the community. I found no
centrally organized CPython-internals material other than bits and pieces
(descrintro, eclectic blog posts, lectures, C-API reference, etc), and I
hope maybe something like this could be featured more officially on
python.org, with the relevant 'this is subject to change' disclaimers (can
be a document for new contributors, for pure Python programmers who're just
interested, or for whatever we decide).

Questions? Comments?
 - Yaniv

* think "Tim Berners-Lee's Web" or "Keanu Reeves' Green Gibberish", see the
first post for details
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100520/baa67ef2/attachment-0001.html>

From fuzzyman at voidspace.org.uk  Thu May 20 00:35:07 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Wed, 19 May 2010 23:35:07 +0100
Subject: [Python-Dev] Documenting [C]Python's Internals
In-Reply-To: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
References: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
Message-ID: <4BF4679B.3000809@voidspace.org.uk>

On 19/05/2010 23:13, Yaniv Aknin wrote:
> Hi,
>
> I wanted to let python-dev know about a series of articles about 
> CPython's internals I'm publishing under the collective title "Guido's 
> Python"* (http://tech.blog.aknin.name/tag/guidos-python/). Three 
> articles already were published already, more are planned (mainly 
> focused on CPython/py3k, but comparisons with other implementations 
> may also be covered; we'll see). So far I've done an 
> introduction/whirlwind tour of Py_Main and a two-article in-depth 
> review of the (new-style) object system.

Whether or not they become part of the Python documentation I have very 
much enjoyed and appreciated this series of blog entries. I still covet 
the ability to contribute to Python in C and these articles are a great 
introduction to the underlying Python interpreter and object system.

Please continue!

All the best,

Michael Foord

>
> I'm sharing this with you (and hope you care) due to three reasons, 
> probably in escalating importance:
> (a) Maybe some of python-dev's readers would be interested (possibly 
> the newer and more silent members).
>
> (b) Maybe my scales are wrong, but I was a bit surprised by the number 
> of readers (>20,000 in the past two weeks); I wouldn't want to mislead 
> such a reader base and would be happy if a veteran here would be 
> interested in aiding by technically proofing the material (shan't be 
> too hard I hope, feel free to contact me directly if qualified and 
> interested).
>
> (c) While the content is currently geared to be blog-oriented, if it's 
> found worthy by the group I'd be delighted to formulate it into 
> something more 'reference-material-ish' and give it back to the 
> community. I found no centrally organized CPython-internals material 
> other than bits and pieces (descrintro, eclectic blog posts, lectures, 
> C-API reference, etc), and I hope maybe something like this could be 
> featured more officially on python.org <http://python.org>, with the 
> relevant 'this is subject to change' disclaimers (can be a document 
> for new contributors, for pure Python programmers who're just 
> interested, or for whatever we decide).
>
> Questions? Comments?
>  - Yaniv
>
> * think "Tim Berners-Lee's Web" or "Keanu Reeves' Green Gibberish", 
> see the first post for details
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100519/4e67f571/attachment.html>

From g.rodola at gmail.com  Thu May 20 01:08:22 2010
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Thu, 20 May 2010 01:08:22 +0200
Subject: [Python-Dev] Documenting [C]Python's Internals
In-Reply-To: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
References: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
Message-ID: <AANLkTilUUpbYYh2vAw3HxUMqh7M9TnDQYqCtWsh25kFL@mail.gmail.com>

2010/5/20 Yaniv Aknin <yaniv at aknin.name>:
> Hi,
> I wanted to let python-dev know about a series of articles about CPython's
> internals I'm publishing under the collective title "Guido's Python"*
> (http://tech.blog.aknin.name/tag/guidos-python/). Three articles already
> were published already, more are planned (mainly focused on CPython/py3k,
> but comparisons with other implementations may also be covered; we'll
> see).?So far I've done an introduction/whirlwind?tour of Py_Main and a
> two-article in-depth review of the (new-style) object system.
> I'm sharing this with you (and hope you care) due to three reasons, probably
> in escalating importance:
> (a) Maybe some of python-dev's readers would be interested (possibly the
> newer and more silent members).
> (b) Maybe my scales are wrong, but I was a bit surprised by the number of
> readers (>20,000 in the past two weeks); I wouldn't want to mislead such a
> reader base and would be happy if a veteran here would be interested in
> aiding by technically proofing the material (shan't be too hard I hope, feel
> free to contact me directly if qualified and interested).
> (c) While the content is currently geared to be blog-oriented, if it's found
> worthy by the group I'd be delighted to formulate it into something more
> 'reference-material-ish' and give it back to the community. I found no
> centrally organized CPython-internals material other than bits and pieces
> (descrintro, eclectic blog posts, lectures, C-API reference, etc), and I
> hope maybe something like this could be featured more officially
> on?python.org, with the relevant 'this is subject to change' disclaimers
> (can be a document for new contributors, for pure Python programmers who're
> just interested, or?for whatever we decide).
> Questions? Comments?
> ?- Yaniv
> *?think "Tim Berners-Lee's Web" or "Keanu Reeves' Green Gibberish", see the
> first post for details
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com

Great!
This can be *extremely* useful for new developers like me who still
haven't took a look at cPython internals.
Thanks for the effort.


--- Giampaolo
http://code.google.com/p/pyftpdlib
http://code.google.com/p/psutil

From g.rodola at gmail.com  Thu May 20 01:42:08 2010
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Thu, 20 May 2010 01:42:08 +0200
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the wrong number of arguments to a method
Message-ID: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>

>>> class A:
...     def echo(self, x):
...             return x
...
>>> a = A()
>>> a.echo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: echo() takes exactly 2 arguments (1 given)
>>>

I bet my last 2 cents this has already been raised in past but I want
to give it a try and revamp the subject anyway.
Is there a reason why the error shouldn't be adjusted to state that
*1* argument is actually required instead of 2?


--- Giampaolo
http://code.google.com/p/pyftpdlib
http://code.google.com/p/psutil

From fuzzyman at voidspace.org.uk  Thu May 20 01:47:33 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 20 May 2010 00:47:33 +0100
Subject: [Python-Dev] Reasons behind misleading TypeError message when
 passing the wrong number of arguments to a method
In-Reply-To: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
Message-ID: <4BF47895.3000700@voidspace.org.uk>

On 20/05/2010 00:42, Giampaolo Rodol? wrote:
>>>> class A:
>>>>          
> ...     def echo(self, x):
> ...             return x
> ...
>    
>>>> a = A()
>>>> a.echo()
>>>>          
> Traceback (most recent call last):
>    File "<stdin>", line 1, in<module>
> TypeError: echo() takes exactly 2 arguments (1 given)
>    
>>>>          
> I bet my last 2 cents this has already been raised in past but I want
> to give it a try and revamp the subject anyway.
> Is there a reason why the error shouldn't be adjusted to state that
> *1* argument is actually required instead of 2?
>
>    

+1 - I've seen many newbies confused by this error and I sometimes do a 
double-take myself.

All the best,

Michael

> --- Giampaolo
> http://code.google.com/p/pyftpdlib
> http://code.google.com/p/psutil
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From john.arbash.meinel at gmail.com  Thu May 20 01:53:03 2010
From: john.arbash.meinel at gmail.com (John Arbash Meinel)
Date: Wed, 19 May 2010 18:53:03 -0500
Subject: [Python-Dev] Reasons behind misleading TypeError message when
 passing the wrong number of arguments to a method
In-Reply-To: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
Message-ID: <4BF479DF.4020102@gmail.com>

Giampaolo Rodol? wrote:
>>>> class A:
> ...     def echo(self, x):
> ...             return x
> ...
>>>> a = A()
>>>> a.echo()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: echo() takes exactly 2 arguments (1 given)
> 
> I bet my last 2 cents this has already been raised in past but I want
> to give it a try and revamp the subject anyway.
> Is there a reason why the error shouldn't be adjusted to state that
> *1* argument is actually required instead of 2?
> 

Because you wouldn't want to have

A.echo()

Say that it takes 1 argument and (-1 given) ?

John
=:->


From debatem1 at gmail.com  Thu May 20 02:15:43 2010
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 19 May 2010 17:15:43 -0700
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the wrong number of arguments to a method
In-Reply-To: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
Message-ID: <AANLkTiklX_Cw0NnawiQUP5ZeyIqz-dHkkn_NRuTJ8Z-X@mail.gmail.com>

On Wed, May 19, 2010 at 4:42 PM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
>>>> class A:
> ... ? ? def echo(self, x):
> ... ? ? ? ? ? ? return x
> ...
>>>> a = A()
>>>> a.echo()
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> TypeError: echo() takes exactly 2 arguments (1 given)
>>>>
>
> I bet my last 2 cents this has already been raised in past but I want
> to give it a try and revamp the subject anyway.
> Is there a reason why the error shouldn't be adjusted to state that
> *1* argument is actually required instead of 2?
>
>
> --- Giampaolo

Because it actually does take two arguments (self and x) and it
only got one (self).

I understand the confusion (and was bitten by it myself when I
was a newbie) but the interpreter is only telling you the truth.

Geremy Condra

From rdmurray at bitdance.com  Thu May 20 02:20:04 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 19 May 2010 20:20:04 -0400
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the wrong number of arguments to a method
In-Reply-To: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
Message-ID: <20100520002004.9A3C1209B76@kimball.webabinitio.net>

On Thu, 20 May 2010 01:42:08 +0200, =?ISO-8859-1?Q?Giampaolo_Rodol=E0?= <g.rodola at gmail.com> wrote:
> >>> class A:
> ...     def echo(self, x):
> ...             return x
> ...
> >>> a = A()
> >>> a.echo()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: echo() takes exactly 2 arguments (1 given)
> >>>
> 
> I bet my last 2 cents this has already been raised in past but I want
> to give it a try and revamp the subject anyway.
> Is there a reason why the error shouldn't be adjusted to state that
> *1* argument is actually required instead of 2?

You can contribute to issue 2516 if you like :)

http://bugs.python.org/issue2516

--
R. David Murray                                      www.bitdance.com

From stephen at xemacs.org  Thu May 20 04:55:02 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 20 May 2010 11:55:02 +0900
Subject: [Python-Dev] Reasons behind misleading TypeError message
	when	passing the wrong number of arguments to a method
In-Reply-To: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
Message-ID: <87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp>

Giampaolo Rodol? writes:
 > >>> class A:
 > ...     def echo(self, x):
 > ...             return x
 > ...
 > >>> a = A()
 > >>> a.echo()
 > Traceback (most recent call last):
 >   File "<stdin>", line 1, in <module>
 > TypeError: echo() takes exactly 2 arguments (1 given)
 > >>>
 > 
 > I bet my last 2 cents this has already been raised in past but I want
 > to give it a try and revamp the subject anyway.
 > Is there a reason why the error shouldn't be adjusted to state that
 > *1* argument is actually required instead of 2?

As a function, it does take two arguments, and can be called
explicitly that way, no?  Adjustment is not enough, the message needs
to be substantially rewritten.  Something like

TypeError: invoked as a method, echo() takes exactly 1 argument (0 given)

captures the semantics, but is perhaps too verbose.

From tjreedy at udel.edu  Thu May 20 06:01:16 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 20 May 2010 00:01:16 -0400
Subject: [Python-Dev] Documenting [C]Python's Internals
In-Reply-To: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
References: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
Message-ID: <ht2c6b$1o4$1@dough.gmane.org>

On 5/19/2010 6:13 PM, Yaniv Aknin wrote:
> Hi,
>
> I wanted to let python-dev know about a series of articles about
> CPython's internals I'm publishing under the collective title "Guido's
> Python"* (http://tech.blog.aknin.name/tag/guidos-python/).

This link has all post concatenated together in reverse order of how 
they should be read. The tags link returns the same page. Does your blog 
software allow you to make a master post and update with new links as 
available?


  Three
> articles already were published already, more are planned (mainly
> focused on CPython/py3k, but comparisons with other implementations may
> also be covered; we'll see). So far I've done an
> introduction/whirlwind tour of Py_Main and a two-article in-depth review
> of the (new-style) object system.
>
> I'm sharing this with you (and hope you care) due to three reasons,
> probably in escalating importance:
> (a) Maybe some of python-dev's readers would be interested (possibly the
> newer and more silent members).

> (b) Maybe my scales are wrong, but I was a bit surprised by the number
> of readers (>20,000 in the past two weeks); I wouldn't want to mislead
> such a reader base and would be happy if a veteran here would be
> interested in aiding by technically proofing the material (shan't be too
> hard I hope, feel free to contact me directly if qualified and interested).

I would if I were qualified, but I an mot. One way to get people to help 
with details is to publish mistakes. This happens all the time on 
python-list ;-). Pre-review would be nice though.

> (c) While the content is currently geared to be blog-oriented, if it's
> found worthy by the group I'd be delighted to formulate it into
> something more 'reference-material-ish' and give it back to the
> community. I found no centrally organized CPython-internals material
> other than bits and pieces (descrintro, eclectic blog posts, lectures,
> C-API reference, etc), and I hope maybe something like this could be
> featured more officially on python.org <http://python.org>, with the
> relevant 'this is subject to change' disclaimers (can be a document for
> new contributors, for pure Python programmers who're just interested,
> or for whatever we decide).

People have asked for an internals-doc since I started over a decade 
ago. A coherent CPython3.2 Internals would be nice to have with the 3.2 
release next December or so, whether or not it was made 'official'.

Terry Jan Reedy


From greg.ewing at canterbury.ac.nz  Thu May 20 09:26:22 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 20 May 2010 19:26:22 +1200
Subject: [Python-Dev] Reasons behind misleading TypeError message when
 passing the wrong number of arguments to a method
In-Reply-To: <4BF479DF.4020102@gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<4BF479DF.4020102@gmail.com>
Message-ID: <4BF4E41E.3050800@canterbury.ac.nz>

John Arbash Meinel wrote:

> Because you wouldn't want to have
> 
> A.echo()
> 
> Say that it takes 1 argument and (-1 given) ?

Something like "1 argument in addition to 'self'" would be
reasonably clear and would cover both situations.

+1 on fixing this from me, too. Even when you understand
exactly what's going on, it's annoying having to make the
mental adjustment every time.

-- 
Greg

From ben+python at benfinney.id.au  Thu May 20 09:46:54 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 20 May 2010 17:46:54 +1000
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the wrong number of arguments to a method
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<4BF479DF.4020102@gmail.com> <4BF4E41E.3050800@canterbury.ac.nz>
Message-ID: <87iq6jgglt.fsf@benfinney.id.au>

Greg Ewing <greg.ewing at canterbury.ac.nz> writes:

> John Arbash Meinel wrote:
>
> > Because you wouldn't want to have
> >
> > A.echo()
> >
> > Say that it takes 1 argument and (-1 given) ?
>
> Something like "1 argument in addition to 'self'" would be reasonably
> clear and would cover both situations.

Except that there's nothing special to the syntax or parser about the
name ?self?.

> +1 on fixing this from me, too. Even when you understand exactly
> what's going on, it's annoying having to make the mental adjustment
> every time.

Would it help if the traceback showed the ?repr()? of each of the
arguments received? That way it would be much clearer when the instance
was received as the first argument.

-- 
 \          ?Any sufficiently advanced bug is indistinguishable from a |
  `\                                          feature.? ?Rich Kulawiec |
_o__)                                                                  |
Ben Finney


From floris.bruynooghe at gmail.com  Thu May 20 10:02:07 2010
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Thu, 20 May 2010 09:02:07 +0100
Subject: [Python-Dev] Reasons behind misleading TypeError message
	when	passing the wrong number of arguments to a method
In-Reply-To: <87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20100520080207.GA16837@laurie.devork>

On Thu, May 20, 2010 at 11:55:02AM +0900, Stephen J. Turnbull wrote:
> Giampaolo Rodol? writes:
>  > >>> class A:
>  > ...     def echo(self, x):
>  > ...             return x
>  > ...
>  > >>> a = A()
>  > >>> a.echo()
>  > Traceback (most recent call last):
>  >   File "<stdin>", line 1, in <module>
>  > TypeError: echo() takes exactly 2 arguments (1 given)
>  > >>>
>  > 
>  > I bet my last 2 cents this has already been raised in past but I want
>  > to give it a try and revamp the subject anyway.
>  > Is there a reason why the error shouldn't be adjusted to state that
>  > *1* argument is actually required instead of 2?
> 
> As a function, it does take two arguments, and can be called
> explicitly that way, no?  Adjustment is not enough, the message needs
> to be substantially rewritten.  Something like
> 
> TypeError: invoked as a method, echo() takes exactly 1 argument (0 given)
> 
> captures the semantics, but is perhaps too verbose.

How about:

TypeError: bound method echo() takes exactly 1 argument (0 given)

That way you can also have: "unbound method echo() ...".  And it's as
semantically correct as the short "echo() takes ..."

Not having looked at the code I don't know how hard it is for the code
that raises this traceback to notice if it's a bound or unbound method
tough.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From me at gustavonarea.net  Wed May 19 10:28:28 2010
From: me at gustavonarea.net (Gustavo Narea)
Date: Wed, 19 May 2010 09:28:28 +0100
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <AANLkTinPopVWOT7K-AK5qESo4G51IHUdN89tohuTGB2P@mail.gmail.com>
References: <201005182313.42935.me@gustavonarea.net>
	<AANLkTinPopVWOT7K-AK5qESo4G51IHUdN89tohuTGB2P@mail.gmail.com>
Message-ID: <AANLkTildRRXSqyuCq2W-am5jmr7YFa4ySssaUuhGfF8d@mail.gmail.com>

Hi, Guido.

On Wed, May 19, 2010 at 12:11 AM, Guido van Rossum <guido at python.org> wrote:

> This is typically called a "bag". Maybe searching for that will help
> you find a recipe?
>

A bag/multiset is close to what I need, except for one thing: I need to
iterate over the elements in the original order, not in a random order.

The data structure I'm proposing is basically a list/tuple, and the only
thing that changes is comparison with another unordered list/tuple: If they
both have the same elements with the same multiplicity, they are equivalent
(regardless of the order).

Cheers,

 - Gustavo.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100519/fd679c55/attachment.html>

From me at gustavonarea.net  Wed May 19 10:56:03 2010
From: me at gustavonarea.net (Gustavo Narea)
Date: Wed, 19 May 2010 09:56:03 +0100
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <201005182313.42935.me@gustavonarea.net>
References: <201005182313.42935.me@gustavonarea.net>
Message-ID: <AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>

Hello, Oleg.


> class UnorderedList(list):
>    def __eq__(self, other):
>        if not isinstance(other, UnorderedList):
>            return False
>        return sorted(self) == sorted(other)
>
>    def __ne__(self, other):
>        return not self.__eq__(other)
>
>   Do you need more than that?
>
> Oleg.



That's what I had in mind.

I think it'd be useful enough to go in the standard library. Now that
there's a sample implementation, should I still try to demonstrate why I
believe it's worth adding to the stdlib and get support?

Cheers,

 - Gustavo.


On Tue, May 18, 2010 at 11:13 PM, Gustavo Narea <me at gustavonarea.net> wrote:

> Hello, everybody.
>
> I've been searching for a data structure like a tuple/list *but* unordered
> --
> like a set, but duplicated elements shouldn't be removed. I have not even
> found a recipe, so I'd like to write an implementation and contribute it to
> the "collections" module in the standard library.
>
> This is the situation I have: I have a data structure that represents an
> arithmetic/boolean operation. Operations can be commutative, which means
> that
> the order of their operands don't change the result of the operation. This
> is,
> the following operations are equivalent:
>    operation(a, b, c) == operation(c, b, a) == operation(b, a, c)
>    operation(a, b, a) == operation(a, a, b) == operation(b, a, a)
>    operation(a, a) == operation(a, a)
>
> So, I need a type to store the arguments/operands so that if two of these
> collections have the same elements with the same multiplicity, they are
> equivalent, regardless of the order.
>
> A multiset is not exactly what I need: I still need to use the elements in
> the
> order they were given. For example, the logical conjuction (aka "and"
> operator) has a left and right operands; I need to evaluate the first/left
> one
> and, if it returned True, then call the second/right one. They must not be
> evaluated in a random order.
>
> To sum up, it would behave like a tuple or a list, except when it's
> compared
> with another object: They would be equivalent if they're both unordered
> tuples/lists, and have the same elements. There can be mutable and
> immutable
> editions (UnorderedList and UnorderedTuple, respectively).
>
> I will write a PEP to elaborate on this if you think it'd be nice to have.
> Or,
> should I have written the PEP first?
>
> Cheers,
> --
> Gustavo Narea <xri://=Gustavo>.
> | Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100519/217907d1/attachment.html>

From g.rodola at gmail.com  Thu May 20 11:49:02 2010
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Thu, 20 May 2010 11:49:02 +0200
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the wrong number of arguments to a method
In-Reply-To: <4BF479DF.4020102@gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<4BF479DF.4020102@gmail.com>
Message-ID: <AANLkTinuAofWa7O0l8capciOJeJRUzv2-DNgoGhqDNkj@mail.gmail.com>

2010/5/20 John Arbash Meinel <john.arbash.meinel at gmail.com>:
> Giampaolo Rodol? wrote:
>>>>> class A:
>> ... ? ? def echo(self, x):
>> ... ? ? ? ? ? ? return x
>> ...
>>>>> a = A()
>>>>> a.echo()
>> Traceback (most recent call last):
>> ? File "<stdin>", line 1, in <module>
>> TypeError: echo() takes exactly 2 arguments (1 given)
>>
>> I bet my last 2 cents this has already been raised in past but I want
>> to give it a try and revamp the subject anyway.
>> Is there a reason why the error shouldn't be adjusted to state that
>> *1* argument is actually required instead of 2?
>>
>
> Because you wouldn't want to have
>
> A.echo()
>
> Say that it takes 1 argument and (-1 given) ?
>
> John
> =:->
>
>

I see that as a different error type: what you're doing there is
calling a method of a class which you haven't instantiated in the
first place.
Actually the error message returned in this other case is not very
clear as well:

"unbound method echo() must be called with A instance as first
argument (got nothing instead)"

It talks about "arguments" while no arguments are actually involved in
the problem: just a class I forgot to initialize.


--- Giampaolo
http://code.google.com/p/pyftpdlib
http://code.google.com/p/psutil

From phd at phd.pp.ru  Thu May 20 12:40:25 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Thu, 20 May 2010 14:40:25 +0400
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
References: <201005182313.42935.me@gustavonarea.net>
	<AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
Message-ID: <20100520104025.GA9757@phd.pp.ru>

On Wed, May 19, 2010 at 09:56:03AM +0100, Gustavo Narea wrote:
> I think it'd be useful enough to go in the standard library. Now that
> there's a sample implementation, should I still try to demonstrate why I
> believe it's worth adding to the stdlib and get support?

   I think yes. How many developers would find it useful?..

Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From fuzzyman at voidspace.org.uk  Thu May 20 12:40:51 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 20 May 2010 11:40:51 +0100
Subject: [Python-Dev] Reasons behind misleading TypeError message when
 passing the wrong number of arguments to a method
In-Reply-To: <AANLkTinuAofWa7O0l8capciOJeJRUzv2-DNgoGhqDNkj@mail.gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>	<4BF479DF.4020102@gmail.com>
	<AANLkTinuAofWa7O0l8capciOJeJRUzv2-DNgoGhqDNkj@mail.gmail.com>
Message-ID: <4BF511B3.1090203@voidspace.org.uk>

On 20/05/2010 10:49, Giampaolo Rodol? wrote:
> 2010/5/20 John Arbash Meinel<john.arbash.meinel at gmail.com>:
>    
>> Giampaolo Rodol? wrote:
>>      
>>>>>> class A:
>>>>>>              
>>> ...     def echo(self, x):
>>> ...             return x
>>> ...
>>>        
>>>>>> a = A()
>>>>>> a.echo()
>>>>>>              
>>> Traceback (most recent call last):
>>>    File "<stdin>", line 1, in<module>
>>> TypeError: echo() takes exactly 2 arguments (1 given)
>>>
>>> I bet my last 2 cents this has already been raised in past but I want
>>> to give it a try and revamp the subject anyway.
>>> Is there a reason why the error shouldn't be adjusted to state that
>>> *1* argument is actually required instead of 2?
>>>
>>>        
>> Because you wouldn't want to have
>>
>> A.echo()
>>
>> Say that it takes 1 argument and (-1 given) ?
>>
>> John
>> =:->
>>
>>
>>      
> I see that as a different error type: what you're doing there is
> calling a method of a class which you haven't instantiated in the
> first place.
> Actually the error message returned in this other case is not very
> clear as well:
>
> "unbound method echo() must be called with A instance as first
> argument (got nothing instead)"
>
> It talks about "arguments" while no arguments are actually involved in
> the problem: just a class I forgot to initialize.
>    

Although the pattern of calling an unbound method with an instance as 
the first argument (self) is not uncommon - and if you're doing 
Class.method() that not only looks like what you're doing but is also an 
accurate error message.

I would also expect that accidentally calling unbound methods is a much 
less common error than calling bound methods with the wrong number of 
arguments...

All the best,

Michael Foord

>
> --- Giampaolo
> http://code.google.com/p/pyftpdlib
> http://code.google.com/p/psutil
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From steve at pearwood.info  Thu May 20 13:37:08 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 20 May 2010 21:37:08 +1000
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <20100520104025.GA9757@phd.pp.ru>
References: <201005182313.42935.me@gustavonarea.net>
	<AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
	<20100520104025.GA9757@phd.pp.ru>
Message-ID: <201005202137.08852.steve@pearwood.info>

On Thu, 20 May 2010 08:40:25 pm Oleg Broytman wrote:
> On Wed, May 19, 2010 at 09:56:03AM +0100, Gustavo Narea wrote:
> > I think it'd be useful enough to go in the standard library. Now
> > that there's a sample implementation, should I still try to
> > demonstrate why I believe it's worth adding to the stdlib and get
> > support?
>
>    I think yes. How many developers would find it useful?..

Adding classes to the standard library doesn't come without cost. 
There's the maintenance and testing burden, and the cognitive burden of 
having to choose between similar classes with slight variations of 
behaviour. I don't think the benefit of this proposed subclass is great 
enough to outweigh the costs.

It is a trivial subclass -- Oleg gave a seven line subclass of list -- 
with a fairly specialist use-case (it is a regular ordered list except 
for the purposes of equality comparisons). I don't believe it is a 
burden on developers to add it to their own application should they 
need it.

-1 for this.

If anyone wishes to support this proposal, please come up with a less 
misleading name than "UnorderedList".



-- 
Steven D'Aprano

From debatem1 at gmail.com  Thu May 20 18:02:47 2010
From: debatem1 at gmail.com (geremy condra)
Date: Thu, 20 May 2010 09:02:47 -0700
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
References: <201005182313.42935.me@gustavonarea.net>
	<AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
Message-ID: <AANLkTikHuGBBxgFBVP7mY5iYJ6p58i2xXoen91bgoDuF@mail.gmail.com>

On Wed, May 19, 2010 at 1:56 AM, Gustavo Narea <me at gustavonarea.net> wrote:
> Hello, Oleg.
>
>>
>> class UnorderedList(list):
>> ? ?def __eq__(self, other):
>> ? ? ? ?if not isinstance(other, UnorderedList):
>> ? ? ? ? ? ?return False
>> ? ? ? ?return sorted(self) == sorted(other)
>>
>> ? ?def __ne__(self, other):
>> ? ? ? ?return not self.__eq__(other)
>>
>> ? Do you need more than that?
>>
>> Oleg.
>
> That's what I had in mind.
>
> I think it'd be useful enough to go in the standard library. Now that
> there's a sample implementation, should I still try to demonstrate why I
> believe it's worth adding to the stdlib and get support?
>
> Cheers,
>
> ?- Gustavo.

I'm generally in favor of adding more data structures to Python,
but I'm at best -0 on this. Besides being trivial to code and
questionably useful, a much better implementation could be
written using heap.

Maybe with a better implementation I would go +0, but I'm
hard pressed to see a case where this would be needed and
could not be trivially written.

Geremy Condra

From rdmurray at bitdance.com  Thu May 20 18:06:32 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 20 May 2010 12:06:32 -0400
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the wrong number of arguments to a method
In-Reply-To: <AANLkTinuAofWa7O0l8capciOJeJRUzv2-DNgoGhqDNkj@mail.gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<4BF479DF.4020102@gmail.com>
	<AANLkTinuAofWa7O0l8capciOJeJRUzv2-DNgoGhqDNkj@mail.gmail.com>
Message-ID: <20100520160633.0301A1F9F81@kimball.webabinitio.net>

On Thu, 20 May 2010 11:49:02 +0200, =?ISO-8859-1?Q?Giampaolo_Rodol=E0?= <g.rodola at gmail.com> wrote:
> 2010/5/20 John Arbash Meinel <john.arbash.meinel at gmail.com>:
> >>>>> a.echo()
> >> Traceback (most recent call last):
> >> =A0 File "<stdin>", line 1, in <module>
> >> TypeError: echo() takes exactly 2 arguments (1 given)
> >>
> >> I bet my last 2 cents this has already been raised in past but I want
> >> to give it a try and revamp the subject anyway.
> >> Is there a reason why the error shouldn't be adjusted to state that
> >> *1* argument is actually required instead of 2?
> >
> > Because you wouldn't want to have
> >
> > A.echo()
> >
> > Say that it takes 1 argument and (-1 given) ?
> 
> I see that as a different error type: what you're doing there is
> calling a method of a class which you haven't instantiated in the
> first place.
> Actually the error message returned in this other case is not very
> clear as well:
> 
> "unbound method echo() must be called with A instance as first
> argument (got nothing instead)"
> 
> It talks about "arguments" while no arguments are actually involved in
> the problem: just a class I forgot to initialize.

That second message is entirely accurate and IMO should not be changed.
As Michael said, calling an unbound method is not that uncommon.

The problem with fixing the first message (as you will see if you read
issue 2516) is that it turns out to be non-trivial to do it right.
Otherwise I think it would have been fixed already.

--
R. David Murray                                      www.bitdance.com

From fuzzyman at voidspace.org.uk  Thu May 20 18:14:24 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 20 May 2010 17:14:24 +0100
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <AANLkTikHuGBBxgFBVP7mY5iYJ6p58i2xXoen91bgoDuF@mail.gmail.com>
References: <201005182313.42935.me@gustavonarea.net>	<AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
	<AANLkTikHuGBBxgFBVP7mY5iYJ6p58i2xXoen91bgoDuF@mail.gmail.com>
Message-ID: <4BF55FE0.8070707@voidspace.org.uk>

On 20/05/2010 17:02, geremy condra wrote:
> On Wed, May 19, 2010 at 1:56 AM, Gustavo Narea<me at gustavonarea.net>  wrote:
>    
>> Hello, Oleg.
>>
>>      
>>> class UnorderedList(list):
>>>     def __eq__(self, other):
>>>         if not isinstance(other, UnorderedList):
>>>             return False
>>>         return sorted(self) == sorted(other)
>>>
>>>     def __ne__(self, other):
>>>         return not self.__eq__(other)
>>>
>>>    Do you need more than that?
>>>
>>> Oleg.
>>>        
>> That's what I had in mind.
>>
>> I think it'd be useful enough to go in the standard library. Now that
>> there's a sample implementation, should I still try to demonstrate why I
>> believe it's worth adding to the stdlib and get support?
>>
>> Cheers,
>>
>>   - Gustavo.
>>      
> I'm generally in favor of adding more data structures to Python,
> but I'm at best -0 on this. Besides being trivial to code and
> questionably useful, a much better implementation could be
> written using heap.
>
>    

-1 from me, a trivial implementation of an uncommon use case. At this 
point the discussion belongs on python-ideas anyway.

(The trouble with sorted is how it breaks with un-orderable types. There 
are at least two places now in the py3k standard library that have 
techniques for comparing sequences of heterogenous types - both 
prettyprint and unittest I believe. Perhaps a standard recipe for 
comparing or sorting such containers *does* belong somewhere more obvious.)

All the best,

Michael

> Maybe with a better implementation I would go +0, but I'm
> hard pressed to see a case where this would be needed and
> could not be trivially written.
>
> Geremy Condra
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From steven.bethard at gmail.com  Thu May 20 18:18:45 2010
From: steven.bethard at gmail.com (Steven Bethard)
Date: Thu, 20 May 2010 09:18:45 -0700
Subject: [Python-Dev] bug or feature? fixing argparse's default help value
	for version actions
Message-ID: <AANLkTil3Ay1Q8xTDRUNDVuqnnwaEEFP05GfEcKPMaTXD@mail.gmail.com>

Sorry I haven't had time to get around to the argparse issues. I
should have time this weekend. I need a release manager call on one of
the issues though. Two things I assume are fine to fix at this stage:

* In the documentation, the '--version' example should either not use
a shorthand, or should use the conventional '-V'
* In the documentation, the difference between the argparse and
optparse ways of specifying versions needs to be mentioned in the
section on migrating from optparse.

One thing I'm not sure about:

* Right now, the default help value for the version action is None,
which is pretty useless since it means that everyone has to type in
the same help="show program's version number and exit" when they use
the version action. Instead, the string "show program's version number
and exit" should have been the default value for the help argument. To
be explicit, right now everyone who adds a version to their argument
parser has to write:

  parser.add_argument('-V', action='version', version='<the version>',
help="show program's version number and exit")

With the fixed default value you would only have to write:

  parser.add_argument('-V', action='version', version='<the version>')

Can this be considered a bug, i.e. something that can be committed
before the RC? Or does it need to be considered a feature, i.e. at
this point can never be added to Python 2.7?

Thanks, and sorry again for not having time for this until now,

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

From tjreedy at udel.edu  Thu May 20 18:55:13 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 20 May 2010 12:55:13 -0400
Subject: [Python-Dev] Reasons behind misleading TypeError message when
 passing the wrong number of arguments to a method
In-Reply-To: <20100520080207.GA16837@laurie.devork>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>	<87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20100520080207.GA16837@laurie.devork>
Message-ID: <ht3phe$346$1@dough.gmane.org>

On 5/20/2010 4:02 AM, Floris Bruynooghe wrote:

>> TypeError: invoked as a method, echo() takes exactly 1 argument (0 given)
>>
>> captures the semantics, but is perhaps too verbose.
>
> How about:
>
> TypeError: bound method echo() takes exactly 1 argument (0 given)
>
> That way you can also have: "unbound method echo() ...".  And it's as
> semantically correct as the short "echo() takes ..."
>
> Not having looked at the code I don't know how hard it is for the code
> that raises this traceback to notice if it's a bound or unbound method
> tough.

In 3.x, there are no unbound method objects, just functions. But that 
should make the difference *easier* to detect, as bound/unbound method 
objects were actually the same class, differing only in an attribute.

I notice
 >>> list.append()
Traceback (most recent call last):
   File "<pyshell#0>", line 1, in <module>
     list.append()
TypeError: descriptor 'append' of 'list' object needs an argument

So here the message is specific to the type of the callable.


From brett at python.org  Thu May 20 18:56:29 2010
From: brett at python.org (Brett Cannon)
Date: Thu, 20 May 2010 09:56:29 -0700
Subject: [Python-Dev] bug or feature? fixing argparse's default help
	value for version actions
In-Reply-To: <AANLkTil3Ay1Q8xTDRUNDVuqnnwaEEFP05GfEcKPMaTXD@mail.gmail.com>
References: <AANLkTil3Ay1Q8xTDRUNDVuqnnwaEEFP05GfEcKPMaTXD@mail.gmail.com>
Message-ID: <AANLkTinCOPdr8fBwk1QDPQPr13H0OyhMT_9PMwuWZW3i@mail.gmail.com>

On Thu, May 20, 2010 at 09:18, Steven Bethard <steven.bethard at gmail.com> wrote:
> Sorry I haven't had time to get around to the argparse issues. I
> should have time this weekend. I need a release manager call on one of
> the issues though. Two things I assume are fine to fix at this stage:
>
> * In the documentation, the '--version' example should either not use
> a shorthand, or should use the conventional '-V'
> * In the documentation, the difference between the argparse and
> optparse ways of specifying versions needs to be mentioned in the
> section on migrating from optparse.
>
> One thing I'm not sure about:
>
> * Right now, the default help value for the version action is None,
> which is pretty useless since it means that everyone has to type in
> the same help="show program's version number and exit" when they use
> the version action. Instead, the string "show program's version number
> and exit" should have been the default value for the help argument. To
> be explicit, right now everyone who adds a version to their argument
> parser has to write:
>
> ?parser.add_argument('-V', action='version', version='<the version>',
> help="show program's version number and exit")
>
> With the fixed default value you would only have to write:
>
> ?parser.add_argument('-V', action='version', version='<the version>')
>
> Can this be considered a bug, i.e. something that can be committed
> before the RC? Or does it need to be considered a feature, i.e. at
> this point can never be added to Python 2.7?

In the end it's Benjamin's call, but my vote is to make the change.
The chances someone wanted None as their help message is so bloody
small and this is such a good UX change that I'm +1 on making the
change.

-Brett

From eric at trueblade.com  Thu May 20 19:13:47 2010
From: eric at trueblade.com (Eric Smith)
Date: Thu, 20 May 2010 13:13:47 -0400
Subject: [Python-Dev] bug or feature? fixing argparse's default
 help	value for version actions
In-Reply-To: <AANLkTinCOPdr8fBwk1QDPQPr13H0OyhMT_9PMwuWZW3i@mail.gmail.com>
References: <AANLkTil3Ay1Q8xTDRUNDVuqnnwaEEFP05GfEcKPMaTXD@mail.gmail.com>
	<AANLkTinCOPdr8fBwk1QDPQPr13H0OyhMT_9PMwuWZW3i@mail.gmail.com>
Message-ID: <4BF56DCB.40207@trueblade.com>

Brett Cannon wrote:

> In the end it's Benjamin's call, but my vote is to make the change.
> The chances someone wanted None as their help message is so bloody
> small and this is such a good UX change that I'm +1 on making the
> change.

I completely agree.

-- 
Eric.

From martin at v.loewis.de  Thu May 20 19:35:03 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Thu, 20 May 2010 19:35:03 +0200
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
References: <201005182313.42935.me@gustavonarea.net>
	<AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
Message-ID: <4BF572C7.9040703@v.loewis.de>

> I think it'd be useful enough to go in the standard library. Now that
> there's a sample implementation, should I still try to demonstrate why I
> believe it's worth adding to the stdlib and get support?

Most definitely. Just in case it isn't clear: nobody else seems to think
this is useful (let alone useful enough to go into the standard
library). In addition, it's trivial to implement, more reason not to add it.

Regards,
Martin

From me at gustavonarea.net  Thu May 20 21:26:36 2010
From: me at gustavonarea.net (Gustavo Narea)
Date: Thu, 20 May 2010 20:26:36 +0100
Subject: [Python-Dev] Unordered tuples/lists
In-Reply-To: <4BF572C7.9040703@v.loewis.de>
References: <201005182313.42935.me@gustavonarea.net>
	<AANLkTimPQZXdqmzDhnnDzTr3etboq5HSlYU7dB0bOpeY@mail.gmail.com>
	<4BF572C7.9040703@v.loewis.de>
Message-ID: <201005202026.36640.me@gustavonarea.net>

Martin said:
> Most definitely. Just in case it isn't clear: nobody else seems to think
> this is useful (let alone useful enough to go into the standard
> library). In addition, it's trivial to implement, more reason not to add
> it.

Yeah, fair enough. Thanks for your responses! :)
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

From solipsis at pitrou.net  Thu May 20 21:32:53 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 20 May 2010 21:32:53 +0200
Subject: [Python-Dev] Adding a new C API function in 2.6
Message-ID: <20100520213253.48f7dfdd@pitrou.net>


Hello,

I would like to check that it's possible to a new C API function in the
2.6 branch, on the basis that it would help solve what seems to be
reported as a security problem by several vendors (including Linux
distributions) -- see http://bugs.python.org/issue5753 for a thorough
discussion.

The change is rather minimal at the code level; it adds a new function
PySys_SetArgvEx which has an additional flag telling it whether to
update sys.path or not. The existing PySys_SetArgv function
unconditionally updates sys.path, which can allow shadowing of stdlib
or third-party library modules by an attacker.

Thank you

Antoine.



From guido at python.org  Thu May 20 21:53:31 2010
From: guido at python.org (Guido van Rossum)
Date: Thu, 20 May 2010 12:53:31 -0700
Subject: [Python-Dev] Adding a new C API function in 2.6
In-Reply-To: <20100520213253.48f7dfdd@pitrou.net>
References: <20100520213253.48f7dfdd@pitrou.net>
Message-ID: <AANLkTiktT8d6eYo_4np2ZzrQF3Ik6d_kKd2z3hifbE5p@mail.gmail.com>

Sounds good to me, since this is (a) a security fix that will make
some vendors happy, and (b) only a C-level API. I expect that some
apps embedding Python will use this API unconditionally and this break
with earlier Python versions; this could be intentional because of the
vulnerability (else why would they change their code to call the new
API), or they can use an #if to check for a version >= 2.6.6.

--Guido

On Thu, May 20, 2010 at 12:32 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> Hello,
>
> I would like to check that it's possible to a new C API function in the
> 2.6 branch, on the basis that it would help solve what seems to be
> reported as a security problem by several vendors (including Linux
> distributions) -- see http://bugs.python.org/issue5753 for a thorough
> discussion.
>
> The change is rather minimal at the code level; it adds a new function
> PySys_SetArgvEx which has an additional flag telling it whether to
> update sys.path or not. The existing PySys_SetArgv function
> unconditionally updates sys.path, which can allow shadowing of stdlib
> or third-party library modules by an attacker.
>
> Thank you
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From me at gustavonarea.net  Thu May 20 22:59:00 2010
From: me at gustavonarea.net (Gustavo Narea)
Date: Thu, 20 May 2010 21:59:00 +0100
Subject: [Python-Dev] Incorrect length of collections.Counter objects /
	Multiplicity function
In-Reply-To: <201005182300.21079.me@gustavonarea.net>
References: <201005182300.21079.me@gustavonarea.net>
Message-ID: <201005202159.01277.me@gustavonarea.net>

Anyone?


Gustavo said:
> Hello, everyone.
> 
> I've checked the new collections.Counter class and I think I've found a bug:
> > >>> from collections import Counter
> > >>> c1 = Counter([1, 2, 1, 3, 2])
> > >>> c2 = Counter([1, 1, 2, 2, 3])
> > >>> c3 = Counter([1, 1, 2, 3])
> > >>> c1 == c2 and c3 not in (c1, c2)
> > 
> > True
> > 
> > >>> # Perfect, so far. But... There's always a "but":
> > ...
> > 
> > >>> len(c1)
> > 
> > 3
> 
> The length of a Counter is the amount of unique elements. But the length
> must be the cardinality, and the cardinality of a multiset is the total
> number of elements (including duplicates) [1] [2]. The source code
> mentions that the recipe on ActiveState [3] was one of the references, but
> that recipe has this right.
> 
> Also, why is it indexed? The indexes of a multiset call to mind the
> position of its elements, but there's no such thing in sets. I think this
> is inconsistent with the built-in set. I would have implemented the
> multiplicity function as a method instead of the indexes:
>     c1.get_multiplicity(element)
>     # instead of
>     c1[element]
> 
> Is this the intended behavior? If so, I'd like to propose a proper multiset
> implementation for the standard library (preferably called "Multiset";
> should I create a PEP?). If not, I can write a patch to fix it, although
> I'm afraid it'd be a backwards incompatible change.
> 
> Cheers,
> 
> [1] http://en.wikipedia.org/wiki/Multiset#Overview
> [2] http://preview.tinyurl.com/smalltalk-bag
> [3] http://code.activestate.com/recipes/259174/
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

From facundobatista at gmail.com  Thu May 20 23:11:51 2010
From: facundobatista at gmail.com (Facundo Batista)
Date: Thu, 20 May 2010 18:11:51 -0300
Subject: [Python-Dev] Incorrect length of collections.Counter objects /
	Multiplicity function
In-Reply-To: <201005202159.01277.me@gustavonarea.net>
References: <201005182300.21079.me@gustavonarea.net>
	<201005202159.01277.me@gustavonarea.net>
Message-ID: <AANLkTilBhj2bCuU_s89_4AEska6qFt1qLyY4NsgNDSeM@mail.gmail.com>

On Thu, May 20, 2010 at 5:59 PM, Gustavo Narea <me at gustavonarea.net> wrote:

> Anyone?

The best place to post a bug is the bug tracker [0]: you'll surely
receive proper attention there.

Regards,

[0] http://bugs.python.org/

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From dickinsm at gmail.com  Thu May 20 23:18:56 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 20 May 2010 22:18:56 +0100
Subject: [Python-Dev] Incorrect length of collections.Counter objects /
	Multiplicity function
In-Reply-To: <201005182300.21079.me@gustavonarea.net>
References: <201005182300.21079.me@gustavonarea.net>
Message-ID: <AANLkTinKx2YUlTcL9CducMhN4ZAhR7LBTRWhqtZWPi71@mail.gmail.com>

On Tue, May 18, 2010 at 11:00 PM, Gustavo Narea <me at gustavonarea.net> wrote:
> I've checked the new collections.Counter class and I think I've found a bug:
>
>> >>> from collections import Counter
>> >>> c1 = Counter([1, 2, 1, 3, 2])
>> >>> c2 = Counter([1, 1, 2, 2, 3])
>> >>> c3 = Counter([1, 1, 2, 3])
>> >>> c1 == c2 and c3 not in (c1, c2)
>> True
>> >>> # Perfect, so far. But... There's always a "but":
>> ...
>> >>> len(c1)
>> 3

This is the intended behaviour;  it also agrees with what you get when
you iterate
over a Counter object:

>>> list(c1)
[1, 2, 3]

As I understand it, there are other uses for Counter objects besides
treating them
as multisets;  I think the choices for len() and iter() reflected
those other uses.

> Is this the intended behavior? If so, I'd like to propose a proper multiset
> implementation for the standard library (preferably called "Multiset"; should
> I create a PEP?).

Feel free!  The proposal should probably go to python-list or
python-ideas rather
than here, though.

See also this recent thread on python-list, and in particular the messages
from Raymond Hettinger in that thread:

http://mail.python.org/pipermail/python-list/2010-March/thread.html

Mark

From dickinsm at gmail.com  Thu May 20 23:20:40 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 20 May 2010 22:20:40 +0100
Subject: [Python-Dev] Incorrect length of collections.Counter objects /
	Multiplicity function
In-Reply-To: <AANLkTinKx2YUlTcL9CducMhN4ZAhR7LBTRWhqtZWPi71@mail.gmail.com>
References: <201005182300.21079.me@gustavonarea.net>
	<AANLkTinKx2YUlTcL9CducMhN4ZAhR7LBTRWhqtZWPi71@mail.gmail.com>
Message-ID: <AANLkTinGrRJgo3MZaAsTIkzCswNHZ-kXffBXHcXvzpD8@mail.gmail.com>

On Thu, May 20, 2010 at 10:18 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> See also this recent thread on python-list, and in particular the messages
> from Raymond Hettinger in that thread:
>
> http://mail.python.org/pipermail/python-list/2010-March/thread.html

Sorry, bad thread link.  Try:

http://mail.python.org/pipermail/python-list/2010-March/1238730.html

instead.

From barry at python.org  Thu May 20 23:52:25 2010
From: barry at python.org (Barry Warsaw)
Date: Thu, 20 May 2010 17:52:25 -0400
Subject: [Python-Dev] Adding a new C API function in 2.6
In-Reply-To: <AANLkTiktT8d6eYo_4np2ZzrQF3Ik6d_kKd2z3hifbE5p@mail.gmail.com>
References: <20100520213253.48f7dfdd@pitrou.net>
	<AANLkTiktT8d6eYo_4np2ZzrQF3Ik6d_kKd2z3hifbE5p@mail.gmail.com>
Message-ID: <20100520175225.54611105@heresy>

On May 20, 2010, at 12:53 PM, Guido van Rossum wrote:

>Sounds good to me, since this is (a) a security fix that will make
>some vendors happy, and (b) only a C-level API. I expect that some
>apps embedding Python will use this API unconditionally and this break
>with earlier Python versions; this could be intentional because of the
>vulnerability (else why would they change their code to call the new
>API), or they can use an #if to check for a version >= 2.6.6.

I concur, as long as the new API availability is clearly spelled out in the
release notes, NEWS, and C API documentation.  Maybe even provide an example
#ifdef in the latter.

Should we start thinking about releasing 2.6.6 soonish?

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100520/267b26d7/attachment.pgp>

From tjreedy at udel.edu  Fri May 21 00:01:52 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 20 May 2010 18:01:52 -0400
Subject: [Python-Dev] Adding a new C API function in 2.6
In-Reply-To: <20100520175225.54611105@heresy>
References: <20100520213253.48f7dfdd@pitrou.net>	<AANLkTiktT8d6eYo_4np2ZzrQF3Ik6d_kKd2z3hifbE5p@mail.gmail.com>
	<20100520175225.54611105@heresy>
Message-ID: <ht4bgd$btv$1@dough.gmane.org>

On 5/20/2010 5:52 PM, Barry Warsaw wrote:
> On May 20, 2010, at 12:53 PM, Guido van Rossum wrote:
>
>> Sounds good to me, since this is (a) a security fix that will make
>> some vendors happy, and (b) only a C-level API. I expect that some
>> apps embedding Python will use this API unconditionally and this break
>> with earlier Python versions; this could be intentional because of the
>> vulnerability (else why would they change their code to call the new
>> API), or they can use an #if to check for a version>= 2.6.6.
>
> I concur, as long as the new API availability is clearly spelled out in the
> release notes, NEWS, and C API documentation.  Maybe even provide an example
> #ifdef in the latter.
>
> Should we start thinking about releasing 2.6.6 soonish?

By tradition, it should come out soon after 2.7 and be the last bugfix 
(except for security patches).



From barry at python.org  Fri May 21 00:11:42 2010
From: barry at python.org (Barry Warsaw)
Date: Thu, 20 May 2010 18:11:42 -0400
Subject: [Python-Dev] Adding a new C API function in 2.6
In-Reply-To: <ht4bgd$btv$1@dough.gmane.org>
References: <20100520213253.48f7dfdd@pitrou.net>
	<AANLkTiktT8d6eYo_4np2ZzrQF3Ik6d_kKd2z3hifbE5p@mail.gmail.com>
	<20100520175225.54611105@heresy> <ht4bgd$btv$1@dough.gmane.org>
Message-ID: <20100520181142.75e3c51b@heresy>

On May 20, 2010, at 06:01 PM, Terry Reedy wrote:

>On 5/20/2010 5:52 PM, Barry Warsaw wrote:
>> On May 20, 2010, at 12:53 PM, Guido van Rossum wrote:
>>
>>> Sounds good to me, since this is (a) a security fix that will make
>>> some vendors happy, and (b) only a C-level API. I expect that some
>>> apps embedding Python will use this API unconditionally and this break
>>> with earlier Python versions; this could be intentional because of the
>>> vulnerability (else why would they change their code to call the new
>>> API), or they can use an #if to check for a version>= 2.6.6.
>>
>> I concur, as long as the new API availability is clearly spelled out in the
>> release notes, NEWS, and C API documentation.  Maybe even provide an example
>> #ifdef in the latter.
>>
>> Should we start thinking about releasing 2.6.6 soonish?
>
>By tradition, it should come out soon after 2.7 and be the last bugfix 
>(except for security patches).

I guess what I mean is, should we have (at least) one more point release
before the post-2.7 last-bug-fix-release?

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100520/59f59b55/attachment.pgp>

From martin at v.loewis.de  Fri May 21 01:05:02 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 21 May 2010 01:05:02 +0200
Subject: [Python-Dev] Adding a new C API function in 2.6
In-Reply-To: <20100520181142.75e3c51b@heresy>
References: <20100520213253.48f7dfdd@pitrou.net>	<AANLkTiktT8d6eYo_4np2ZzrQF3Ik6d_kKd2z3hifbE5p@mail.gmail.com>	<20100520175225.54611105@heresy>
	<ht4bgd$btv$1@dough.gmane.org> <20100520181142.75e3c51b@heresy>
Message-ID: <4BF5C01E.5000406@v.loewis.de>

>>> Should we start thinking about releasing 2.6.6 soonish?
>>
>> By tradition, it should come out soon after 2.7 and be the last bugfix
>> (except for security patches).
>
> I guess what I mean is, should we have (at least) one more point release
> before the post-2.7 last-bug-fix-release?

Because it's a security fix? No. This issue has been sitting in the 
tracker for more than a year now, so it's not really relevant (IMO) 
whether the bug fix arrives two weeks earlier.

Regards,
Martin

From benjamin at python.org  Fri May 21 01:14:24 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 20 May 2010 18:14:24 -0500
Subject: [Python-Dev] bug or feature? fixing argparse's default help
	value for version actions
In-Reply-To: <AANLkTil3Ay1Q8xTDRUNDVuqnnwaEEFP05GfEcKPMaTXD@mail.gmail.com>
References: <AANLkTil3Ay1Q8xTDRUNDVuqnnwaEEFP05GfEcKPMaTXD@mail.gmail.com>
Message-ID: <AANLkTikLiZNDcFBRUZK82645_xcVJmouR0WQLysix3E0@mail.gmail.com>

2010/5/20 Steven Bethard <steven.bethard at gmail.com>:
> Sorry I haven't had time to get around to the argparse issues. I
> should have time this weekend. I need a release manager call on one of
> the issues though. Two things I assume are fine to fix at this stage:
>
> * In the documentation, the '--version' example should either not use
> a shorthand, or should use the conventional '-V'
> * In the documentation, the difference between the argparse and
> optparse ways of specifying versions needs to be mentioned in the
> section on migrating from optparse.

Documentation changes are always okay.

> Can this be considered a bug, i.e. something that can be committed
> before the RC? Or does it need to be considered a feature, i.e. at
> this point can never be added to Python 2.7?

That sounds fine to me.

>
> Thanks, and sorry again for not having time for this until now,



-- 
Regards,
Benjamin

From greg.ewing at canterbury.ac.nz  Fri May 21 02:53:16 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 21 May 2010 12:53:16 +1200
Subject: [Python-Dev] Reasons behind misleading TypeError message when
 passing the wrong number of arguments to a method
In-Reply-To: <87iq6jgglt.fsf@benfinney.id.au>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<4BF479DF.4020102@gmail.com> <4BF4E41E.3050800@canterbury.ac.nz>
	<87iq6jgglt.fsf@benfinney.id.au>
Message-ID: <4BF5D97C.6080607@canterbury.ac.nz>

Ben Finney wrote:

>>Something like "1 argument in addition to 'self'" would be reasonably
>>clear and would cover both situations.
> 
> Except that there's nothing special to the syntax or parser about the
> name ?self?.

That's true, but the use of the word 'self' here isn't meant
to refer to the name of a parameter. The message is aimed at
the caller of the function, who doesn't necessarily know or
care what the parameter is actually called.

The important thing is that it represents the object the method
is being called for, and 'self' is the traditional term used
when talking about that. I can't think of anything that would
be more accurate without being excessively verbose or pedantic.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri May 21 03:23:15 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 21 May 2010 13:23:15 +1200
Subject: [Python-Dev] Reasons behind misleading TypeError message	when
 passing the wrong number of arguments to a method
In-Reply-To: <20100520080207.GA16837@laurie.devork>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<87k4qzgu49.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20100520080207.GA16837@laurie.devork>
Message-ID: <4BF5E083.6000201@canterbury.ac.nz>

Floris Bruynooghe wrote:

> Not having looked at the code I don't know how hard it is for the code
> that raises this traceback to notice if it's a bound or unbound method
> tough.

The way things currently work, it would be quite difficult.
The exception is raised when attempting to call the function
underlying the bound method, at which point it can't be
distinguished from any other way of calling it.

However, it could be made to raise a special subclass of
TypeError that the bound method wrapper could catch and
replace with its own exception.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri May 21 03:31:33 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 21 May 2010 13:31:33 +1200
Subject: [Python-Dev] Reasons behind misleading TypeError message when
 passing the wrong number of arguments to a method
In-Reply-To: <AANLkTinuAofWa7O0l8capciOJeJRUzv2-DNgoGhqDNkj@mail.gmail.com>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<4BF479DF.4020102@gmail.com>
	<AANLkTinuAofWa7O0l8capciOJeJRUzv2-DNgoGhqDNkj@mail.gmail.com>
Message-ID: <4BF5E275.6000902@canterbury.ac.nz>

Giampaolo Rodol? wrote:

> "unbound method echo() must be called with A instance as first
> argument (got nothing instead)"
> 
> It talks about "arguments" while no arguments are actually involved in
> the problem: just a class I forgot to initialize.

It's hard to see how this could be improved. If you had been
intending to call an unbound method, the comment about arguments
would have been relevant -- you were supposed to supply one
but didn't. Python is unable to read your mind and figure out
that you really meant something else altogether.

BTW, unbound methods no longer exist in Py3, so you would get
the standard message about incorrect number of arguments
instead. Not sure whether that's better or worse in this
situation.

-- 
Greg

From cs at zip.com.au  Fri May 21 03:46:38 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 21 May 2010 11:46:38 +1000
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the	wrong number of arguments to a method
In-Reply-To: <87iq6jgglt.fsf@benfinney.id.au>
References: <87iq6jgglt.fsf@benfinney.id.au>
Message-ID: <20100521014638.GA28209@cskk.homeip.net>

On 20May2010 17:46, Ben Finney <ben+python at benfinney.id.au> wrote:
| Would it help if the traceback showed the ?repr()? of each of the
| arguments received? That way it would be much clearer when the instance
| was received as the first argument.

I've occasionally passed large or deep dicts etc to functions and foolishly
printed their repr in debug statements. They can be... wordy.  Maybe a
cropped repr, or the .__class__ attribute?
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Every technical corrigendum is met by an equally troublesome new defect report.
        - Norman Diamond <diamond at jit.dec.com>

From ben+python at benfinney.id.au  Fri May 21 04:24:37 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 21 May 2010 12:24:37 +1000
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the	wrong number of arguments to a method
References: <87iq6jgglt.fsf@benfinney.id.au>
	<20100521014638.GA28209@cskk.homeip.net>
Message-ID: <87aarugffe.fsf@benfinney.id.au>

Cameron Simpson <cs at zip.com.au> writes:

> On 20May2010 17:46, Ben Finney <ben+python at benfinney.id.au> wrote:
> | Would it help if the traceback showed the ?repr()? of each of the
> | arguments received? That way it would be much clearer when the instance
> | was received as the first argument.
>
> I've occasionally passed large or deep dicts etc to functions and
> foolishly printed their repr in debug statements. They can be...
> wordy. Maybe a cropped repr, or the .__class__ attribute?

Okay. My main point is that I'm offering this as a way of avoiding a
special-case message for method calls versus non-method calls.

In Python, all function calls are essentially the same, and I think it's
too complicated for the message to differ depending on the syntax used.
Emitting the arguments in some form, or even the types of each argument,
in the message would make it clearer without a special case.

-- 
 \         ?We now have access to so much information that we can find |
  `\  support for any prejudice or opinion.? ?David Suzuki, 2008-06-27 |
_o__)                                                                  |
Ben Finney


From brian at sweetapp.com  Fri May 21 05:00:35 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 21 May 2010 13:00:35 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
Message-ID: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>

The PEP is here:
http://www.python.org/dev/peps/pep-3148/

I think the PEP is ready for pronouncement, and the code is pretty  
much ready for submission into py3k (I will have to make some minor  
changes in the patch like changing the copyright assignment):
http://code.google.com/p/pythonfutures/source/browse/#svn/branches/ 
feedback/python3/futures%3Fstate%3Dclosed

The tests are here and pass on W2K, Mac OS X and Linux:
http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/test_futures.py

The docs (which also need some minor changes) are here:
http://code.google.com/p/pythonfutures/source/browse/branches/feedback/docs/index.rst

Cheers,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100521/7035fede/attachment.html>

From steve at pearwood.info  Fri May 21 05:54:56 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 21 May 2010 13:54:56 +1000
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the wrong number of arguments to a method
In-Reply-To: <4BF5D97C.6080607@canterbury.ac.nz>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<87iq6jgglt.fsf@benfinney.id.au>
	<4BF5D97C.6080607@canterbury.ac.nz>
Message-ID: <201005211354.57237.steve@pearwood.info>

On Fri, 21 May 2010 10:53:16 am Greg Ewing wrote:
> Ben Finney wrote:
> >>Something like "1 argument in addition to 'self'" would be
> >> reasonably clear and would cover both situations.
> >
> > Except that there's nothing special to the syntax or parser about
> > the name ?self?.
>
> That's true, but the use of the word 'self' here isn't meant
> to refer to the name of a parameter. The message is aimed at
> the caller of the function, who doesn't necessarily know or
> care what the parameter is actually called.

And who doesn't necessarily know that:

> The important thing is that it represents the object the method
> is being called for, and 'self' is the traditional term used
> when talking about that. 

Referring to self in any such error message is going to confuse newbies, 
and occasional Python programmers who use it without learning about 
object oriented code. E.g. sys admins who use Python as a scripting 
language without ever writing their own classes.

Error messages should be aimed at the least capable users (within 
reason!) rather than the most capable, since the most capable are:

(1) less likely to make the error in the first place;
(2) more able to interpret the error message correctly;
(3) more likely to understand if the error relates to something "under 
the hood"; and
(4) better able to debug the error even in the absence of a useful error 
message.

Given this, it is better for the error message to relate to what 
the "average" user most likely has in his source code (i.e. a bound 
method) rather than what the better-than-average user occasionally uses 
(an unbound method).


> I can't think of anything that would 
> be more accurate without being excessively verbose or pedantic.


In Python 3.1, strings seem to behave more sensibly IMO:

>>> 'abc'.find()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: find() takes at least 1 argument (0 given)

That looks right to me. I'm calling a *method* with zero arguments 
inside the parentheses when I need to call it with one.

If I call it as an unbound method (yes, I know it's actually a function) 
I get something more confusing:

>>> str.find('abc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: find() takes at least 1 argument (0 given)

While calling unbound methods isn't exactly rare, it is a more advanced 
thing to do than calling bound methods. If we're stuck with confusing 
messages, better to have it occur for the users who are more likely to 
be able to interpret it, that is, those advanced enough to use unbound 
methods rather than "ordinary" users.

Note that str avoids the "-1 argument" trap:

>>> str.find()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: descriptor 'find' of 'str' object needs an argument



Classes behave in the opposite manner: the error message is technically 
correct for the less common, more advanced case, but misleading for the 
more common, simple case:


>>> class C:
...     def method(self, x):
...             return x+1
...
>>> C.method()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: method() takes exactly 2 positional arguments (0 given)

This is exactly right, since C.method is a function object which takes 
two arguments, self and x. Because this is a function, we can't change 
this (not that we want to!) without impacting the errors you get when 
calling functions not associated with a class.


>>> C().method()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: method() takes exactly 2 positional arguments (1 given)

This is misleading, since C().method is a bound method which takes one 
argument, x, not two. I find myself wishing that Python distinguished 
between ArgumentError and other TypeErrors, so that the method wrapper 
of bound methods could simply catch ArgumentError and subtract 1 from 
each argument count.



-- 
Steven D'Aprano

From yaniv at aknin.name  Fri May 21 07:18:43 2010
From: yaniv at aknin.name (Yaniv Aknin)
Date: Fri, 21 May 2010 15:18:43 +1000
Subject: [Python-Dev] Documenting [C]Python's Internals
In-Reply-To: <ht2c6b$1o4$1@dough.gmane.org>
References: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
	<ht2c6b$1o4$1@dough.gmane.org>
Message-ID: <AANLkTikq5VmCtfBDcLLOzbV1rlzo4BykwV4LNXRDDTdU@mail.gmail.com>

>
> This link has all post concatenated together in reverse order of how they
>> should be read. The tags link returns the same page. Does your blog software
>> allow you to make a master post and update with new links as available?
>
>
Ugh, either it doesn't or I couldn't find the feature (I'm using
wordpress.com, if someone has advice, let me know). I can clumsily suggest
scrolling from the end. Also see below about reworking this into a single
multi-chapter document with coherent form.

I would if I were qualified, but I an mot. One way to get people to help
> with details is to publish mistakes. This happens all the time on
> python-list ;-). Pre-review would be nice though.
>

I don't mind so much the 'humiliation' of published mistakes, but since I
want this to be perceived as reference grade material, I prefer pre-review.
Yesterday my first mistake was found (ugh), I published an 'Errata Policy'
and will stick to it from now on (see the blog itself for details of the
mistake). Thankfully, I've been approached already about pre-review, we'll
see how this develops (this doesn't mean other people can't also offer
themselves, six eyeballs are better than four).

People have asked for an internals-doc since I started over a decade ago. A
> coherent CPython3.2 Internals would be nice to have with the 3.2 release
> next December or so, whether or not it was made 'official'.
>

I'm targeting py3k anyway, and while I expect a bug lull in my writing
between early June and early September, I think December is a realistic date
for me to have good coverage CPython 3.2's core and rework the content into
a more reference-material-ish form. That said, working things into
reference-material form could be significant work, so if python-dev doesn't
show interest in this I think the blog posts are good enough. Other people,
this is your queue to chime in and state your opinion about this appearing
on python.org somewhere.

Cheers!
 - Yaniv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100521/8a5a049a/attachment.html>

From list at qtrac.plus.com  Fri May 21 13:29:58 2010
From: list at qtrac.plus.com (Mark Summerfield)
Date: Fri, 21 May 2010 12:29:58 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
Message-ID: <201005211229.58970.list@qtrac.plus.com>

On 2010-05-21, Brian Quinlan wrote:
> The PEP is here:
> http://www.python.org/dev/peps/pep-3148/
[snip]

Hi Brian,

Could I suggest a small subtle changing in naming: replace "executor"
with "executer"? I guess this suggestion is doomed though since Java
uses executor:-(

I'd also be tempted to rename "submit()" to "apply()" in view of
Python's history.

Also, maybe change "done()" to "finished()" since the function returns
True if the call was cancelled (so the job can't have been "done"), as
well as if the call was finished. Actually, having read further, maybe
the best name would be "completed()" since that's a term used
throughout.

Perhaps call the "not_finished" set "pending" since presumably these are
still in progress? (My understanding is that if they were cancelled or
finished they'd be in the "finished" set. I'd also rename "finished" to
"completed" if you have a "completed()" method.)

I think FIRST_COMPLETED is misleading since it implies (to me anyway)
the first one passed. How about ONE_COMPLETED; and similarly
ONE_EXCEPTION?

I think it would be helpful to clarify whether the timout value (which
you specify as being in seconds) can meaningfully accept a float, e.g.,
0.5?

Anyway, it looks like it will be a really nice addition to the standard
library:-)

-- 
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
    C++, Python, Qt, PyQt - training and consultancy
        "C++ GUI Programming with Qt 4" - ISBN 0132354160

From john.arbash.meinel at gmail.com  Fri May 21 13:44:35 2010
From: john.arbash.meinel at gmail.com (John Arbash Meinel)
Date: Fri, 21 May 2010 06:44:35 -0500
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
Message-ID: <4BF67223.5080304@gmail.com>

Brian Quinlan wrote:
> The PEP is here:
> http://www.python.org/dev/peps/pep-3148/
> 
> I think the PEP is ready for pronouncement, and the code is pretty much
> ready for submission into py3k (I will have to make some minor changes
> in the patch like changing the copyright assignment):
> http://code.google.com/p/pythonfutures/source/browse/#svn/branches/feedback/python3/futures%3Fstate%3Dclosed
> 

Your example here:
    for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
        print('%d is prime: %s' % (number, is_prime))

Overwrites the 'is_prime' function with the return value of the
function. Probably better to use a different variable name.

John
=:->


From brian at sweetapp.com  Fri May 21 14:13:11 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 21 May 2010 22:13:11 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <201005211229.58970.list@qtrac.plus.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<201005211229.58970.list@qtrac.plus.com>
Message-ID: <785B69C5-9DBE-4049-A1F5-D36BA5EACCFC@sweetapp.com>

Hey Mark,

This really isn't the time to propose changes.

The PEP has been discussed extensively on stdlib-sig and python-dev.

On May 21, 2010, at 9:29 PM, Mark Summerfield wrote:

> On 2010-05-21, Brian Quinlan wrote:
>> The PEP is here:
>> http://www.python.org/dev/peps/pep-3148/
> [snip]
>
> Hi Brian,
>
> Could I suggest a small subtle changing in naming: replace "executor"
> with "executer"? I guess this suggestion is doomed though since Java
> uses executor:-(
>
> I'd also be tempted to rename "submit()" to "apply()" in view of
> Python's history.
>
> Also, maybe change "done()" to "finished()" since the function returns
> True if the call was cancelled (so the job can't have been "done"), as
> well as if the call was finished. Actually, having read further, maybe
> the best name would be "completed()" since that's a term used
> throughout.
>
> Perhaps call the "not_finished" set "pending" since presumably these  
> are
> still in progress? (My understanding is that if they were cancelled or
> finished they'd be in the "finished" set. I'd also rename "finished"  
> to
> "completed" if you have a "completed()" method.)
>
> I think FIRST_COMPLETED is misleading since it implies (to me anyway)
> the first one passed. How about ONE_COMPLETED; and similarly
> ONE_EXCEPTION?
>
> I think it would be helpful to clarify whether the timout value (which
> you specify as being in seconds) can meaningfully accept a float,  
> e.g.,
> 0.5?

I've updated the docs to clarify that float args are acceptable.

Cheers,
Brian

> Anyway, it looks like it will be a really nice addition to the  
> standard
> library:-)
>
> -- 
> Mark Summerfield, Qtrac Ltd, www.qtrac.eu
>    C++, Python, Qt, PyQt - training and consultancy
>        "C++ GUI Programming with Qt 4" - ISBN 0132354160


From brian at sweetapp.com  Fri May 21 14:14:11 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 21 May 2010 22:14:11 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BF67223.5080304@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF67223.5080304@gmail.com>
Message-ID: <F6A20E41-ECD1-4FF7-99D5-DA1891CC145B@sweetapp.com>


On May 21, 2010, at 9:44 PM, John Arbash Meinel wrote:

> Brian Quinlan wrote:
>> The PEP is here:
>> http://www.python.org/dev/peps/pep-3148/
>>
>> I think the PEP is ready for pronouncement, and the code is pretty  
>> much
>> ready for submission into py3k (I will have to make some minor  
>> changes
>> in the patch like changing the copyright assignment):
>> http://code.google.com/p/pythonfutures/source/browse/#svn/branches/ 
>> feedback/python3/futures%3Fstate%3Dclosed
>>
>
> Your example here:
>    for number, is_prime in zip(PRIMES, executor.map(is_prime,  
> PRIMES)):
>        print('%d is prime: %s' % (number, is_prime))
>
> Overwrites the 'is_prime' function with the return value of the
> function. Probably better to use a different variable name.

Good catch. I've updated the example.

Cheers,
Brian

From john.meinel at canonical.com  Fri May 21 13:47:52 2010
From: john.meinel at canonical.com (John Arbash Meinel)
Date: Fri, 21 May 2010 06:47:52 -0500
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
Message-ID: <4BF672E8.1080001@canonical.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian Quinlan wrote:
> The PEP is here:
> http://www.python.org/dev/peps/pep-3148/
> 
> I think the PEP is ready for pronouncement, and the code is pretty much
> ready for submission into py3k (I will have to make some minor changes
> in the patch like changing the copyright assignment):
> http://code.google.com/p/pythonfutures/source/browse/#svn/branches/feedback/python3/futures%3Fstate%3Dclosed
> 
> The tests are here and pass on W2K, Mac OS X and Linux:
> http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/test_futures.py
> 
> The docs (which also need some minor changes) are here:
> http://code.google.com/p/pythonfutures/source/browse/branches/feedback/docs/index.rst
> 
> Cheers,
> Brian
> 

I also just noticed that your example uses:
 zip(PRIMES, executor.map(is_prime, PRIMES))

But your doc explicitly says:
map(func, *iterables, timeout=None)

    Equivalent to map(func, *iterables) but executed asynchronously and
possibly out-of-order.

So it isn't safe to zip() against something that can return out of order.

Which opens up a discussion about how these things should be used.

Given that your other example uses a dict to get back to the original
arguments, and this example uses zip() [incorrectly], it seems that the
Futures object should have the arguments easily accessible. It certainly
seems like a common use case that if things are going to be returned in
arbitrary order, you'll want an easy way to distinguish which one you
have.  Having to write a dict map before each call can be done, but
seems unoptimal.

John
=:->


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkv2cugACgkQJdeBCYSNAAPWzACdE6KepgEmjwhCD1M4bSSVrI97
NIYAn1z5U3CJqZnBSn5XgQ/DyLvcKtbf
=TKO7
-----END PGP SIGNATURE-----

From murman at gmail.com  Fri May 21 14:25:33 2010
From: murman at gmail.com (Michael Urman)
Date: Fri, 21 May 2010 07:25:33 -0500
Subject: [Python-Dev] Reasons behind misleading TypeError message when
	passing the wrong number of arguments to a method
In-Reply-To: <201005211354.57237.steve@pearwood.info>
References: <AANLkTimdWvrqHNC13oMfb8ruFcbUEzm7ahIFAiZqMTDz@mail.gmail.com>
	<87iq6jgglt.fsf@benfinney.id.au> <4BF5D97C.6080607@canterbury.ac.nz>
	<201005211354.57237.steve@pearwood.info>
Message-ID: <AANLkTik22WJAv1PIdaKMWaV_83vtNUar_sqJMSx3C92I@mail.gmail.com>

On Thu, May 20, 2010 at 22:54, Steven D'Aprano <steve at pearwood.info> wrote:
> This is misleading, since C().method is a bound method which takes one
> argument, x, not two. I find myself wishing that Python distinguished
> between ArgumentError and other TypeErrors, so that the method wrapper
> of bound methods could simply catch ArgumentError and subtract 1 from
> each argument count.

But how exactly could it distinguish between various levels of call nesting?

    class C:
        def a(self, i): return self.b(i)
        def b(self, j): return self.c(j)
        def c(self, k): return self.a()

What should C.a(), C().a(), and C().a(1) each yield? Does it change if
c(self, k) calls C.a(self)?

-- 
Michael Urman

From brian at sweetapp.com  Fri May 21 14:26:13 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 21 May 2010 22:26:13 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BF672E8.1080001@canonical.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
Message-ID: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>


On May 21, 2010, at 9:47 PM, John Arbash Meinel wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Brian Quinlan wrote:
>> The PEP is here:
>> http://www.python.org/dev/peps/pep-3148/
>>
>> I think the PEP is ready for pronouncement, and the code is pretty  
>> much
>> ready for submission into py3k (I will have to make some minor  
>> changes
>> in the patch like changing the copyright assignment):
>> http://code.google.com/p/pythonfutures/source/browse/#svn/branches/ 
>> feedback/python3/futures%3Fstate%3Dclosed
>>
>> The tests are here and pass on W2K, Mac OS X and Linux:
>> http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/test_futures.py
>>
>> The docs (which also need some minor changes) are here:
>> http://code.google.com/p/pythonfutures/source/browse/branches/feedback/docs/index.rst
>>
>> Cheers,
>> Brian
>>
>
> I also just noticed that your example uses:
> zip(PRIMES, executor.map(is_prime, PRIMES))
>
> But your doc explicitly says:
> map(func, *iterables, timeout=None)
>
>    Equivalent to map(func, *iterables) but executed asynchronously and
> possibly out-of-order.
>
> So it isn't safe to zip() against something that can return out of  
> order.

The docs don't say that the return value can be out-of-order, just  
that execution can be out-of-order. But I agree that the phrasing is  
confusing so I've changed it to:

     Equivalent to ``map(func, *iterables)`` but *func* is executed
     asynchronously and several calls to *func *may be made  
concurrently.


> Which opens up a discussion about how these things should be used.

Except that now isn't the time for that discussion. This PEP has  
discussed on-and-off for several months on both stdlib-sig and python- 
dev.

If you think that storing the args (e.g. with the future) is a good  
idea then you can propose a patch after the PEP is integrated (if it  
is rejected then it probably isn't worth discussing ;-)).

Cheers,
Brian

> Given that your other example uses a dict to get back to the original
> arguments, and this example uses zip() [incorrectly], it seems that  
> the
> Futures object should have the arguments easily accessible. It  
> certainly
> seems like a common use case that if things are going to be returned  
> in
> arbitrary order, you'll want an easy way to distinguish which one you
> have.  Having to write a dict map before each call can be done, but
> seems unoptimal.
>
> John
> =:->
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (Cygwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkv2cugACgkQJdeBCYSNAAPWzACdE6KepgEmjwhCD1M4bSSVrI97
> NIYAn1z5U3CJqZnBSn5XgQ/DyLvcKtbf
> =TKO7
> -----END PGP SIGNATURE-----


From lie.1296 at gmail.com  Fri May 21 14:42:09 2010
From: lie.1296 at gmail.com (Lie Ryan)
Date: Fri, 21 May 2010 22:42:09 +1000
Subject: [Python-Dev] Documenting [C]Python's Internals
In-Reply-To: <AANLkTikq5VmCtfBDcLLOzbV1rlzo4BykwV4LNXRDDTdU@mail.gmail.com>
References: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>	<ht2c6b$1o4$1@dough.gmane.org>
	<AANLkTikq5VmCtfBDcLLOzbV1rlzo4BykwV4LNXRDDTdU@mail.gmail.com>
Message-ID: <ht5v66$901$1@dough.gmane.org>

On 05/21/10 15:18, Yaniv Aknin wrote:
> I would if I were qualified, but I an mot. One way to get people to help
>> with details is to publish mistakes. This happens all the time on
>> python-list ;-). Pre-review would be nice though.
>>
> 
> I don't mind so much the 'humiliation' of published mistakes, but since I
> want this to be perceived as reference grade material, I prefer pre-review.
> Yesterday my first mistake was found (ugh), I published an 'Errata Policy'
> and will stick to it from now on (see the blog itself for details of the
> mistake). Thankfully, I've been approached already about pre-review, we'll
> see how this develops (this doesn't mean other people can't also offer
> themselves, six eyeballs are better than four).

How about a separate blog (or wiki) for alpha-quality articles? After an
article is written, it is first posted to the alpha blog, and after some
time and eyeballs, moved to the original blog. Of course with an open
comment system, so people can easily suggest corrections.


From fuzzyman at voidspace.org.uk  Fri May 21 14:52:09 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 21 May 2010 13:52:09 +0100
Subject: [Python-Dev] Documenting [C]Python's Internals
In-Reply-To: <ht5v66$901$1@dough.gmane.org>
References: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>	<ht2c6b$1o4$1@dough.gmane.org>	<AANLkTikq5VmCtfBDcLLOzbV1rlzo4BykwV4LNXRDDTdU@mail.gmail.com>
	<ht5v66$901$1@dough.gmane.org>
Message-ID: <4BF681F9.1070305@voidspace.org.uk>

On 21/05/2010 13:42, Lie Ryan wrote:
> On 05/21/10 15:18, Yaniv Aknin wrote:
>    
>> I would if I were qualified, but I an mot. One way to get people to help
>>      
>>> with details is to publish mistakes. This happens all the time on
>>> python-list ;-). Pre-review would be nice though.
>>>
>>>        
>> I don't mind so much the 'humiliation' of published mistakes, but since I
>> want this to be perceived as reference grade material, I prefer pre-review.
>> Yesterday my first mistake was found (ugh), I published an 'Errata Policy'
>> and will stick to it from now on (see the blog itself for details of the
>> mistake). Thankfully, I've been approached already about pre-review, we'll
>> see how this develops (this doesn't mean other people can't also offer
>> themselves, six eyeballs are better than four).
>>      
> How about a separate blog (or wiki) for alpha-quality articles? After an
> article is written, it is first posted to the alpha blog, and after some
> time and eyeballs, moved to the original blog. Of course with an open
> comment system, so people can easily suggest corrections.
>    

Separate blog is confusing I think - you then duplicate your content and 
people are just as likely to be referred to the "alpha quality" version 
as the final version.

Just publish and improve the articles based on feedback - I think your 
current approach with an established errata policy is well beyond what 
most people do or expect. When you have established the sort of coverage 
of the topic you are aiming for you can then take your blog articles, 
along with all feedback, and turn them into documentation.

All the best,

Michael

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From loitiere at gmail.com  Fri May 21 17:25:45 2010
From: loitiere at gmail.com (=?utf-8?Q?Yannick_Loiti=C3=A8re?=)
Date: Fri, 21 May 2010 17:25:45 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
Message-ID: <66D23619-7D57-4B87-AE43-DA6E0C5A0EFD@gmail.com>

The description of Future.add_done_callback() does not specify whether  
the callbacks are done in any particular order (such as the order they  
were added) or not. I'm guessing they are not, but believe it should  
be documented either way.

-- Yannick Loiti?re

From status at bugs.python.org  Fri May 21 18:07:59 2010
From: status at bugs.python.org (Python tracker)
Date: Fri, 21 May 2010 18:07:59 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20100521160759.4DB6C78139@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2010-05-14 - 2010-05-21)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2693 open (+46) / 17908 closed (+24) / 20601 total (+70)

Open issues with patches:  1098

Average duration of open issues: 719 days.
Median duration of open issues: 497 days.

Open Issues Breakdown
       open  2672 (+46)
languishing    12 ( +0)
    pending     8 ( +0)

Issues Created Or Reopened (73)
_______________________________

heapq change breaking compatibility                            2010-05-17
CLOSED http://bugs.python.org/issue3051    reopened exarkun                              
       patch                                                                   

subprocess.Popen.__del__ causes AttributeError (os module == N 2010-05-14
CLOSED http://bugs.python.org/issue5099    reopened haypo                                
       patch, needs review                                                     

Buggy _strerror in asyncore                                    2010-05-18
CLOSED http://bugs.python.org/issue8573    reopened merwok                               
       easy                                                                    

multiprocessing needs option to eschew fork() under Linux      2010-05-14
       http://bugs.python.org/issue8713    created  brandon-rhodes                       
                                                                               

Delayed signals in the REPL on OpenBSD (possibly libpthread re 2010-05-14
       http://bugs.python.org/issue8714    created  skrah                                
                                                                               

Create PyUnicode_EncodeFSDefault() function                    2010-05-14
CLOSED http://bugs.python.org/issue8715    created  haypo                                
       patch                                                                   

test_tk fails on OS X if run from buildbot slave daemon -- cra 2010-05-14
       http://bugs.python.org/issue8716    created  janssen                              
       patch                                                                   

Subprocess.py broken in trunk                                  2010-05-14
CLOSED http://bugs.python.org/issue8717    created  pakal                                
       patch                                                                   

subprocess: NameError: name 'WaitForSingleObject' is not def   2010-05-14
CLOSED http://bugs.python.org/issue8718    created  srid                                 
                                                                               

buildbot: segfault on FreeBSD (signal 11)                      2010-05-14
       http://bugs.python.org/issue8719    created  haypo                                
                                                                               

undo findsource regression/change                              2010-05-14
       http://bugs.python.org/issue8720    created  hpk                                  
       patch                                                                   

urlparse.urlsplit regression in 2.7                            2010-05-15
CLOSED http://bugs.python.org/issue8721    created  srid                                 
                                                                               

Documentation for __getattr__                                  2010-05-15
       http://bugs.python.org/issue8722    created  Paul.Davis                           
                                                                               

IDLE won't start import os error                               2010-05-15
CLOSED http://bugs.python.org/issue8723    created  gromij                               
                                                                               

bind_and_activate parameter is missed from directive           2010-05-15
       http://bugs.python.org/issue8724    created  naoki                                
                                                                               

Python3: use ASCII for the file system encoding on initfsencod 2010-05-15
       http://bugs.python.org/issue8725    created  haypo                                
       patch                                                                   

test_capi failure                                              2010-05-15
CLOSED http://bugs.python.org/issue8726    created  pitrou                               
                                                                               

test_import failure                                            2010-05-15
CLOSED http://bugs.python.org/issue8727    created  pitrou                               
                                                                               

2.7 regression in httplib.py: AttributeError: 'NoneType' objec 2010-05-15
       http://bugs.python.org/issue8728    created  srid                                 
                                                                               

The Mapping ABC's __eq__ method should return NotImplemented i 2010-05-15
       http://bugs.python.org/issue8729    created  stutzbach                            
       patch                                                                   

Spurious test failure in distutils                             2010-05-16
       http://bugs.python.org/issue8730    created  ronaldoussoren                       
                                                                               

BeOS for 2.7 - add to unsupported                              2010-05-16
       http://bugs.python.org/issue8731    created  rpetrov                              
       patch                                                                   

Should urrllib2.urlopen send an Accept-Encoding header?        2010-05-16
       http://bugs.python.org/issue8732    created  dabrahams                            
                                                                               

list type and UserList do not call super in __init__ and there 2010-05-16
       http://bugs.python.org/issue8733    created  alonho                               
                                                                               

msvcrt get_osfhandle crash on bad FD                           2010-05-16
       http://bugs.python.org/issue8734    created  pakal                                
       patch                                                                   

optparse: parse_args(values=...) does not set up default value 2010-05-16
       http://bugs.python.org/issue8735    created  john at nmt.edu                         
                                                                               

*= is undocumented for lists                                   2010-05-16
CLOSED http://bugs.python.org/issue8736    created  stutzbach                            
       easy                                                                    

ssl.RAND_add() should only accept bytes (not str)              2010-05-16
CLOSED http://bugs.python.org/issue8737    created  haypo                                
       patch                                                                   

cPickle dumps(tuple) != dumps(loads(dumps(tuple)))             2010-05-17
       http://bugs.python.org/issue8738    created  Alberto.Planas.Dom??nguez            
                                                                               

Update to smtpd.py to RFC 5321                                 2010-05-17
       http://bugs.python.org/issue8739    created  alfmel                               
       patch                                                                   

infinite recursion with setfilesystemencoding and pdb          2010-05-17
CLOSED http://bugs.python.org/issue8740    created  ccomb                                
                                                                               

2.7 regression in tarfile: IOError: link could not be created  2010-05-17
       http://bugs.python.org/issue8741    created  srid                                 
                                                                               

broken asdl link in Parser/asdl.py                             2010-05-17
CLOSED http://bugs.python.org/issue8742    created  pyfex                                
                                                                               

set() operators don't work with collections.Set instances      2010-05-17
       http://bugs.python.org/issue8743    created  stutzbach                            
                                                                               

Maybe typo in doc                                              2010-05-18
CLOSED http://bugs.python.org/issue8744    created  naoki                                
                                                                               

zipimport is a bit slow                                        2010-05-18
       http://bugs.python.org/issue8745    created  Goplat                               
       patch                                                                   

*chflags detection broken on FreeBSD 9-CURRENT                 2010-05-18
       http://bugs.python.org/issue8746    created  yaneurabeya                          
                                                                               

Autoconf tests in python not portably correct                  2010-05-18
       http://bugs.python.org/issue8747    created  yaneurabeya                          
                                                                               

integer-to-complex comparisons give incorrect results          2010-05-18
       http://bugs.python.org/issue8748    created  mark.dickinson                       
       patch                                                                   

Cruft in object.c: PyObject_GenericGetAttr                     2010-05-18
       http://bugs.python.org/issue8749    created  Yaniv.Aknin                          
                                                                               

Many of MutableSet's methods assume that the other parameter i 2010-05-18
       http://bugs.python.org/issue8750    created  stutzbach                            
       patch, needs review                                                     

Threading and KeyError: 51                                     2010-05-18
CLOSED http://bugs.python.org/issue8751    created  Vignesh.K                            
                                                                               

set_contains, etc. should check for collections.Set in additio 2010-05-18
CLOSED http://bugs.python.org/issue8752    created  stutzbach                            
                                                                               

Py_ReprEnter and Py_ReprLeave are undocumented                 2010-05-18
       http://bugs.python.org/issue8753    created  stutzbach                            
                                                                               

ImportError: quote bad module name in message                  2010-05-18
       http://bugs.python.org/issue8754    created  tjreedy                              
       easy                                                                    

idle crash  UnicodeDecodeError utf-8 codec                     2010-05-18
       http://bugs.python.org/issue8755    created  andyharrington                       
                                                                               

Multiprocessing and UUID bug on Mac OSX                        2010-05-18
CLOSED http://bugs.python.org/issue8756    created  Gavin.Roy                            
                                                                               

Race condition when checking for set in set                    2010-05-18
       http://bugs.python.org/issue8757    created  stutzbach                            
                                                                               

BUILD_LIST followed by BINARY_SUBSCR can be optimized to a BUI 2010-05-18
       http://bugs.python.org/issue8758    created  alex                                 
                                                                               

2.7: wrong user site directory on Linux                        2010-05-18
CLOSED http://bugs.python.org/issue8759    created  srid                                 
                                                                               

Python 2.6.5 fails to build on AIX 5.3                         2010-05-18
CLOSED http://bugs.python.org/issue8760    created  oirraza                              
                                                                               

PyUnicode_CompareWithASCIIString name is not mangled (UCS2, UC 2010-05-18
       http://bugs.python.org/issue8761    created  haypo                                
       patch                                                                   

default value in constructor not unique across objects         2010-05-19
CLOSED http://bugs.python.org/issue8762    created  bolt                                 
                                                                               

py3K bdist_msi wrongly installs itself in ALL python versions  2010-05-19
       http://bugs.python.org/issue8763    created  pakal                                
                                                                               

logging RotatingFileHandler cause too long to fininsh          2010-05-19
CLOSED http://bugs.python.org/issue8764    created  davy.zhang                           
                                                                               

Tests unwillingly writing unicocde to raw streams              2010-05-19
       http://bugs.python.org/issue8765    created  pakal                                
       patch                                                                   

Segmentation fault with empty "encodings" subdirectory of dire 2010-05-19
CLOSED http://bugs.python.org/issue8766    created  Arfrever                             
       patch                                                                   

Configure: Cannot disable unicode                              2010-05-19
CLOSED http://bugs.python.org/issue8767    created  redcomet                             
                                                                               

The checkempty_symmetric_difference test is never called       2010-05-19
       http://bugs.python.org/issue8768    created  stutzbach                            
       patch                                                                   

Straightforward usage of email package fails to round-trip     2010-05-19
       http://bugs.python.org/issue8769    created  akuchling                            
       patch                                                                   

Make 'python -m sysconfig' print something useful              2010-05-19
       http://bugs.python.org/issue8770    created  srid                                 
                                                                               

Socket freezing under load issue on Mac.                       2010-05-19
       http://bugs.python.org/issue8771    created  amcgregor                            
                                                                               

sysconfig: _get_default_scheme can be made public?             2010-05-20
       http://bugs.python.org/issue8772    created  srid                                 
                                                                               

mailbox module is needlessly executable                        2010-05-20
       http://bugs.python.org/issue8773    created  meatballhat                          
                                                                               

tabnanny improperly handles non-ascii source files             2010-05-20
CLOSED http://bugs.python.org/issue8774    created  meatballhat                          
       patch                                                                   

Use locale encoding to decode sys.argv, not the file system en 2010-05-20
       http://bugs.python.org/issue8775    created  haypo                                
                                                                               

Bytes version of sys.argv                                      2010-05-20
       http://bugs.python.org/issue8776    created  haypo                                
                                                                               

Add threading.Barrier                                          2010-05-20
       http://bugs.python.org/issue8777    created  krisvale                             
       patch, patch                                                            

typo in docs for symtable.SymbolTable.has_import_star          2010-05-20
CLOSED http://bugs.python.org/issue8778    created  terrence                             
       patch                                                                   

utf8 codec fails to parse a character                          2010-05-20
CLOSED http://bugs.python.org/issue8779    created  Roman.Gershman                       
                                                                               

py3k: child process don't inherit stdout / stdout on Windows   2010-05-21
       http://bugs.python.org/issue8780    created  haypo                                
       patch                                                                   

32-bit wchar_t doesn't need to be unsigned to be usable (I thi 2010-05-21
       http://bugs.python.org/issue8781    created  stutzbach                            
                                                                               

inspect.getsource returns invalid source for non-newline-endin 2010-05-21
       http://bugs.python.org/issue8782    created  hpk                                  
       patch                                                                   



Issues Now Closed (66)
______________________

Removal of stale code in pyconfig.h                             904 days
       http://bugs.python.org/issue1512    loewis                               
                                                                               

Codepage unset in msilib.init_database()                        855 days
       http://bugs.python.org/issue1855    loewis                               
                                                                               

msilib.SetProperty(msilib.PID_CODEPAGE, '1252') raises 0x65d =  850 days
       http://bugs.python.org/issue1884    loewis                               
                                                                               

add ftp-tls support to ftplib - RFC 4217                        832 days
       http://bugs.python.org/issue2054    giampaolo.rodola                     
       patch                                                                   

Make sysmodule.c compatible with Bazaar                         786 days
       http://bugs.python.org/issue2456    barry                                
                                                                               

heapq change breaking compatibility                               2 days
       http://bugs.python.org/issue3051    r.david.murray                       
       patch                                                                   

Py_FatalError causes infinite loop                              633 days
       http://bugs.python.org/issue3605    haypo                                
       needs review                                                            

PKG-INFO file should differentiate between authors and maintai  628 days
       http://bugs.python.org/issue3686    tarek                                
                                                                               

urllib2 sends Basic auth across redirects                       617 days
       http://bugs.python.org/issue3819    kiilerix                             
       patch, easy                                                             

ssl module is missing SSL_OP_NO_SSLv2                           499 days
       http://bugs.python.org/issue4870    pitrou                               
       patch                                                                   

subprocess.Popen.__del__ causes AttributeError (os module == N    0 days
       http://bugs.python.org/issue5099    haypo                                
       patch, needs review                                                     

csv sniffer                                                     454 days
       http://bugs.python.org/issue5332    r.david.murray                       
       patch                                                                   

tarfile normalizes arcname                                      364 days
       http://bugs.python.org/issue6054    srid                                 
                                                                               

2to3 fails to fix test.test_support                             295 days
       http://bugs.python.org/issue6583    pakal                                
                                                                               

file_close() ignores return value of close_the_file             222 days
       http://bugs.python.org/issue7079    pitrou                               
       patch, needs review                                                     

distutils makefile from python.org installer doesn't work	on O  144 days
       http://bugs.python.org/issue7580    loewis                               
                                                                               

buffered io seek() buggy                                        130 days
       http://bugs.python.org/issue7640    pitrou                               
       patch                                                                   

newgil backport                                                 117 days
       http://bugs.python.org/issue7753    jcea                                 
       patch, needs review                                                     

relative import broken                                           99 days
       http://bugs.python.org/issue7902    brett.cannon                         
       patch, needs review                                                     

_ssl: support surrogates in filenames, and bytes/bytearray fil   26 days
       http://bugs.python.org/issue8477    haypo                                
       patch                                                                   

subprocess: support bytes program name (POSIX)                   25 days
       http://bugs.python.org/issue8513    haypo                                
       patch, needs review                                                     

Expose SSL contexts                                              19 days
       http://bugs.python.org/issue8550    pitrou                               
       patch                                                                   

test_gdb: test_strings() fails with ASCII locale                 22 days
       http://bugs.python.org/issue8559    haypo                                
       patch                                                                   

Buggy _strerror in asyncore                                       0 days
       http://bugs.python.org/issue8573    giampaolo.rodola                     
       easy                                                                    

test_warnings.CEnvironmentVariableTests.test_nonascii fails un   18 days
       http://bugs.python.org/issue8589    haypo                                
       patch                                                                   

Python3/POSIX:  errors if file system encoding is None           11 days
       http://bugs.python.org/issue8610    haypo                                
       patch                                                                   

Doc bug for urllib.request._urlopener in Python 3.1+             13 days
       http://bugs.python.org/issue8619    orsenthil                            
                                                                               

tarfile doesn't support undecodable filename in PAX format       12 days
       http://bugs.python.org/issue8633    lars.gustaebel                       
                                                                               

subprocess: canonicalize env to bytes on Unix (Python3)          12 days
       http://bugs.python.org/issue8640    haypo                                
       patch                                                                   

Failed compile in a non-ASCII path                               11 days
       http://bugs.python.org/issue8663    haypo                                
       patch                                                                   

"make pycremoval" fails                                           8 days
       http://bugs.python.org/issue8665    pitrou                               
                                                                               

distutils sdist is too laze w.r.t. recalculating MANIFEST         6 days
       http://bugs.python.org/issue8688    tarek                                
       patch, needs review                                                     

sqlite3 parameter substitution breaks with multiple parameters    3 days
       http://bugs.python.org/issue8689    tjreedy                              
                                                                               

Use divide-and-conquer for faster factorials                      5 days
       http://bugs.python.org/issue8692    mark.dickinson                       
       patch, patch, needs review                                              

OpenID blunder                                                    2 days
       http://bugs.python.org/issue8708    brett.cannon                         
                                                                               

Document	PyUnicode_DecodeFSDefault()	and	PyUnicode_DecodeFSDef    0 days
       http://bugs.python.org/issue8711    haypo                                
       patch                                                                   

Create PyUnicode_EncodeFSDefault() function                       1 days
       http://bugs.python.org/issue8715    haypo                                
       patch                                                                   

Subprocess.py broken in trunk                                     0 days
       http://bugs.python.org/issue8717    haypo                                
       patch                                                                   

subprocess: NameError: name 'WaitForSingleObject' is not def      0 days
       http://bugs.python.org/issue8718    haypo                                
                                                                               

urlparse.urlsplit regression in 2.7                               5 days
       http://bugs.python.org/issue8721    r.david.murray                       
                                                                               

IDLE won't start import os error                                  0 days
       http://bugs.python.org/issue8723    gromij                               
                                                                               

test_capi failure                                                 2 days
       http://bugs.python.org/issue8726    jyasskin                             
                                                                               

test_import failure                                               3 days
       http://bugs.python.org/issue8727    barry                                
                                                                               

*= is undocumented for lists                                      1 days
       http://bugs.python.org/issue8736    stutzbach                            
       easy                                                                    

ssl.RAND_add() should only accept bytes (not str)                 0 days
       http://bugs.python.org/issue8737    haypo                                
       patch                                                                   

infinite recursion with setfilesystemencoding and pdb             2 days
       http://bugs.python.org/issue8740    ccomb                                
                                                                               

broken asdl link in Parser/asdl.py                                1 days
       http://bugs.python.org/issue8742    orsenthil                            
                                                                               

Maybe typo in doc                                                 0 days
       http://bugs.python.org/issue8744    orsenthil                            
                                                                               

Threading and KeyError: 51                                        0 days
       http://bugs.python.org/issue8751    draghuram                            
                                                                               

set_contains, etc. should check for collections.Set in additio    0 days
       http://bugs.python.org/issue8752    rhettinger                           
                                                                               

Multiprocessing and UUID bug on Mac OSX                           0 days
       http://bugs.python.org/issue8756    r.david.murray                       
                                                                               

2.7: wrong user site directory on Linux                           2 days
       http://bugs.python.org/issue8759    r.david.murray                       
                                                                               

Python 2.6.5 fails to build on AIX 5.3                            3 days
       http://bugs.python.org/issue8760    amaury.forgeotdarc                   
                                                                               

default value in constructor not unique across objects            0 days
       http://bugs.python.org/issue8762    ysj.ray                              
                                                                               

logging RotatingFileHandler cause too long to fininsh             0 days
       http://bugs.python.org/issue8764    vinay.sajip                          
                                                                               

Segmentation fault with empty "encodings" subdirectory of dire    1 days
       http://bugs.python.org/issue8766    haypo                                
       patch                                                                   

Configure: Cannot disable unicode                                 0 days
       http://bugs.python.org/issue8767    redcomet                             
                                                                               

tabnanny improperly handles non-ascii source files                1 days
       http://bugs.python.org/issue8774    haypo                                
       patch                                                                   

typo in docs for symtable.SymbolTable.has_import_star             0 days
       http://bugs.python.org/issue8778    benjamin.peterson                    
       patch                                                                   

utf8 codec fails to parse a character                             0 days
       http://bugs.python.org/issue8779    benjamin.peterson                    
                                                                               

csv module cannot handle embedded \r                           2174 days
       http://bugs.python.org/issue967934  r.david.murray                       
                                                                               

ftplib: add support for MDTM command                           2033 days
       http://bugs.python.org/issue1053369 giampaolo.rodola                     
       patch                                                                   

Bugs in _csv module - lineterminator                           2004 days
       http://bugs.python.org/issue1072404 r.david.murray                       
                                                                               

Missing trailing newline with comment raises SyntaxError       1860 days
       http://bugs.python.org/issue1184112 benjamin.peterson                    
       easy                                                                    

urllib.quote is too slow                                       1712 days
       http://bugs.python.org/issue1285086 flox                                 
       patch                                                                   

Py_FileSystemDefaultEncoding can be non-canonical              1262 days
       http://bugs.python.org/issue1608805 haypo                                
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 26 Use divide-and-conquer for faster factorials                       5 days
closed      http://bugs.python.org/issue8692   

 19 implement PEP 3118 struct changes                                703 days
open        http://bugs.python.org/issue3132   

 16 integer-to-complex comparisons give incorrect results              3 days
open        http://bugs.python.org/issue8748   

 10 tabnanny improperly handles non-ascii source files                 1 days
closed      http://bugs.python.org/issue8774   

 10 urlparse.urlsplit regression in 2.7                                5 days
closed      http://bugs.python.org/issue8721   

 10 Check that _PyUnicode_AsString() result is not NULL              281 days
open        http://bugs.python.org/issue6697   

  9 Python 2.6.5 fails to build on AIX 5.3                             3 days
closed      http://bugs.python.org/issue8760   

  9 Failed compile in a non-ASCII path                                11 days
closed      http://bugs.python.org/issue8663   

  8 2.7: wrong user site directory on Linux                            2 days
closed      http://bugs.python.org/issue8759   

  8 test_import failure                                                3 days
closed      http://bugs.python.org/issue8727   




From andrew.svetlov at gmail.com  Fri May 21 20:18:00 2010
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Fri, 21 May 2010 22:18:00 +0400
Subject: [Python-Dev] __package__ attribute
Message-ID: <AANLkTilvwrKGM53NHMw4W52_PSNV8Ap2DBGLH4amCHww@mail.gmail.com>

What are expected values for module.__package__?
I see two different behaviors: importlib and standard python import.
importlib processes __package__ pretty simple and clean:
- for toplevel modules __package__ is empty string
- for packages it's package name
- for modules inside packages it's again package name

Standard import follows another strategy:
- it tries to setup __package__ only in case of relative import (see
first 'if' in import.c:get_parent)
- otherwise __package__ is untouched and None by default.

For me importlib's way is better.
I don't see any PEP specifying __package__ behavior.

From barry at python.org  Fri May 21 21:19:04 2010
From: barry at python.org (Barry Warsaw)
Date: Fri, 21 May 2010 15:19:04 -0400
Subject: [Python-Dev] Adding a new C API function in 2.6
In-Reply-To: <4BF5C01E.5000406@v.loewis.de>
References: <20100520213253.48f7dfdd@pitrou.net>
	<AANLkTiktT8d6eYo_4np2ZzrQF3Ik6d_kKd2z3hifbE5p@mail.gmail.com>
	<20100520175225.54611105@heresy> <ht4bgd$btv$1@dough.gmane.org>
	<20100520181142.75e3c51b@heresy> <4BF5C01E.5000406@v.loewis.de>
Message-ID: <20100521151904.6c1ef180@heresy>

On May 21, 2010, at 01:05 AM, Martin v. L?wis wrote:

>>>> Should we start thinking about releasing 2.6.6 soonish?
>>>
>>> By tradition, it should come out soon after 2.7 and be the last bugfix
>>> (except for security patches).
>>
>> I guess what I mean is, should we have (at least) one more point release
>> before the post-2.7 last-bug-fix-release?
>
>Because it's a security fix? No. This issue has been sitting in the 
>tracker for more than a year now, so it's not really relevant (IMO) 
>whether the bug fix arrives two weeks earlier.

Partly that, yes.  But also, 2.7 final is not scheduled until July, so we
could fit one more release in I think.  If there's no clamor for it, I'm also
happy to just wait for 2.6.6 until after Python 2.7 is released.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100521/e3293102/attachment.pgp>

From digitalxero at gmail.com  Fri May 21 21:30:13 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Fri, 21 May 2010 15:30:13 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
Message-ID: <AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>

On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan <brian at sweetapp.com> wrote:
> Except that now isn't the time for that discussion. This PEP has discussed
> on-and-off for several months on both stdlib-sig and python-dev.

I think any time till the PEP is accepted is a good time to discuss
changes to the API


Issues with the PEP:
1) Examples as written fail on windows. Patch to fix @
http://code.google.com/p/pythonfutures/issues/detail?id=5

Issues with Implementation:
1) Globals are used for tracking running threads (but not processes)
and shutdown state, but are not accessed as a globals every where they
are modified so it could be inconsistent.

2) The atexit handle randomly throws an exception on exit on windows
when running the tests or examples.

Error in atexit._run_exitfuncs:
TypeError: print_exception(): Exception expected for value, str found

Issues 1 & 2 would be solved by moving thread tracking back into the
executor responsible for the threads, or making a singleton that
tracked threads / processes for all executors.
http://code.google.com/p/pythonfutures/issues/detail?id=6 is one such
implementation

From andrew.svetlov at gmail.com  Fri May 21 23:35:20 2010
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sat, 22 May 2010 01:35:20 +0400
Subject: [Python-Dev] __package__ attribute
In-Reply-To: <AANLkTilvwrKGM53NHMw4W52_PSNV8Ap2DBGLH4amCHww@mail.gmail.com>
References: <AANLkTilvwrKGM53NHMw4W52_PSNV8Ap2DBGLH4amCHww@mail.gmail.com>
Message-ID: <AANLkTimIZAa_z4_BOuyt-rURhMc3oUD9XgU8igeKnIxy@mail.gmail.com>

For me it's the real bug in standard python import machinery.
I don't see any backward incompatibilities.
There are very hard to write any import-depended code based on
decision: was module imported in absolute or relative way.

If it's a bug I can create issue in python bugtracker and provide a
patch (for Python 3.2 and for 2.7 if later is required).

On Fri, May 21, 2010 at 10:18 PM, Andrew Svetlov
<andrew.svetlov at gmail.com> wrote:
> What are expected values for module.__package__?
> I see two different behaviors: importlib and standard python import.
> importlib processes __package__ pretty simple and clean:
> - for toplevel modules __package__ is empty string
> - for packages it's package name
> - for modules inside packages it's again package name
>
> Standard import follows another strategy:
> - it tries to setup __package__ only in case of relative import (see
> first 'if' in import.c:get_parent)
> - otherwise __package__ is untouched and None by default.
>
> For me importlib's way is better.
> I don't see any PEP specifying __package__ behavior.
>

From luhon at healthcare.uiowa.edu  Sat May 22 01:32:23 2010
From: luhon at healthcare.uiowa.edu (Lu, Hongchao (UI Health Care))
Date: Fri, 21 May 2010 18:32:23 -0500
Subject: [Python-Dev] Ask a question for a script about re.findall Modlue
Message-ID: <368F25BB4F425B4087B4B4C5285559FB06476857@HC-MAIL11.healthcare.uiowa.edu>

Hi All,

     I am working on a script to use re.findall,
But the result seems wield.

--------------------------------------------------------------------
import re
p=re.compile("\S+",re.M)

pirfile="""
>DL;mm9|
>DL;petMar1|
>DL;cavPor3|
>DL;mm9|:
"""

for m in p.findall(pirfile,re.M):
        print m

Results:

|
>DL;petMar1|
>DL;cavPor3|
>DL;mm9|

The first result is missing some characters, Could you explain why? Any
wrong with the script?

I tried Python2.4/2.6/2.7, the result is same.


Thank you.


Hongchao Lu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100521/13941900/attachment.html>

From guido at python.org  Sat May 22 02:09:37 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 21 May 2010 17:09:37 -0700
Subject: [Python-Dev] Ask a question for a script about re.findall Modlue
In-Reply-To: <368F25BB4F425B4087B4B4C5285559FB06476857@HC-MAIL11.healthcare.uiowa.edu>
References: <368F25BB4F425B4087B4B4C5285559FB06476857@HC-MAIL11.healthcare.uiowa.edu>
Message-ID: <AANLkTinOY4OBAIWsIFAvrp35vit4Wo8Xt6PQRsRg5fv4@mail.gmail.com>

You shouldn't be asking questions about using Python on python-dev -- this
list is for development of Python.

Your problem is easily explained however: the second argument to p.findall()
should be an offset, not a flag set. (You are confusing re.findall() and
p.findall().)

--Guido

On Fri, May 21, 2010 at 4:32 PM, Lu, Hongchao (UI Health Care) <
luhon at healthcare.uiowa.edu> wrote:

>  Hi All,
>
>      I am working on a script to use re.findall,
>
> But the result seems wield.
>
> --------------------------------------------------------------------
>
> import re
>
> p=re.compile("\S+",re.M)
>
> pirfile="""
>
> >DL;mm9|
>
> >DL;petMar1|
>
> >DL;cavPor3|
>
> >DL;mm9|:
>
> """
>
> for m in p.findall(pirfile,re.M):
>
>         print m
>
> Results:
>
> |
>
> >DL;petMar1|
>
> >DL;cavPor3|
>
> >DL;mm9|
>
> The first result is missing some characters, Could you explain why? Any
> wrong with the script?
>
> I tried Python2.4/2.6/2.7, the result is same.
>
> Thank you.
>
> Hongchao Lu
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100521/87d05100/attachment-0001.html>

From hagen at zhuliguan.net  Sat May 22 10:07:41 2010
From: hagen at zhuliguan.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=)
Date: Sat, 22 May 2010 10:07:41 +0200
Subject: [Python-Dev] Ask a question for a script about re.findall Modlue
In-Reply-To: <AANLkTinOY4OBAIWsIFAvrp35vit4Wo8Xt6PQRsRg5fv4@mail.gmail.com>
References: <368F25BB4F425B4087B4B4C5285559FB06476857@HC-MAIL11.healthcare.uiowa.edu>
	<AANLkTinOY4OBAIWsIFAvrp35vit4Wo8Xt6PQRsRg5fv4@mail.gmail.com>
Message-ID: <4BF790CD.1060909@zhuliguan.net>

> Your problem is easily explained however: the second argument to
> p.findall() should be an offset, not a flag set. (You are confusing
> re.findall() and p.findall().)

I filed a doc bug for this:

http://bugs.python.org/issue8785

Cheers,
Hagen

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 489 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100522/94664fb3/attachment.pgp>

From brian at sweetapp.com  Sat May 22 11:12:05 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 22 May 2010 19:12:05 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
Message-ID: <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>


On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote:

> On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> Except that now isn't the time for that discussion. This PEP has  
>> discussed
>> on-and-off for several months on both stdlib-sig and python-dev.
>
> I think any time till the PEP is accepted is a good time to discuss
> changes to the API

I disagree. If a PEP is being updated continuously then there is  
nothing stable to pronounce on.

>
> Issues with the PEP:
> 1) Examples as written fail on windows. Patch to fix @
> http://code.google.com/p/pythonfutures/issues/detail?id=5

Updated, thanks!

> Issues with Implementation:
> 1) Globals are used for tracking running threads (but not processes)
> and shutdown state, but are not accessed as a globals every where they
> are modified so it could be inconsistent.



> 2) The atexit handle randomly throws an exception on exit on windows
> when running the tests or examples.
>
> Error in atexit._run_exitfuncs:
> TypeError: print_exception(): Exception expected for value, str found

Lets take this off-list.

Cheers,
Brian

> Issues 1 & 2 would be solved by moving thread tracking back into the
> executor responsible for the threads, or making a singleton that
> tracked threads / processes for all executors.
> http://code.google.com/p/pythonfutures/issues/detail?id=6 is one such
> implementation
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com


From rdmurray at bitdance.com  Sat May 22 15:59:46 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 22 May 2010 09:59:46 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
Message-ID: <20100522135946.8F5451F960D@kimball.webabinitio.net>

On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan <brian at sweetapp.com> wrote:
> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote:
> > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan <brian at sweetapp.com>  
> > wrote:
> >> Except that now isn't the time for that discussion. This PEP has  
> >> discussed
> >> on-and-off for several months on both stdlib-sig and python-dev.
> >
> > I think any time till the PEP is accepted is a good time to discuss
> > changes to the API
> 
> I disagree. If a PEP is being updated continuously then there is  
> nothing stable to pronounce on.

Well, you've been making updates as a result of this round of
discussion.

If there is still discussion then perhaps the PEP isn't ready for
pronouncement yet.  At some point someone can decide it is all
bikeshedding and ask for pronouncement on that basis, but I don't
think it is appropriate to cut off discussion by saying "it's ready for
pronouncement" unless you want increase the chances of its getting
rejected.

The usual way of doing this (at least so far as I have observed, which
granted hasn't been too many cases) is to say something like "I think
this PEP is ready for pronouncement" and then wait for feedback on that
assertion or for the pronouncement.  It's especially good if you can answer
any concerns that are raised with "that was discussed already and we
concluded X".  Bonus points for finding a thread reference and adding
it to the PEP :)

--
R. David Murray                                      www.bitdance.com

From jnoller at gmail.com  Sat May 22 16:09:37 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 22 May 2010 10:09:37 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100522135946.8F5451F960D@kimball.webabinitio.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
Message-ID: <AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>

On Sat, May 22, 2010 at 9:59 AM, R. David Murray <rdmurray at bitdance.com> wrote:
> On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan <brian at sweetapp.com> wrote:
>> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote:
>> > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan <brian at sweetapp.com>
>> > wrote:
>> >> Except that now isn't the time for that discussion. This PEP has
>> >> discussed
>> >> on-and-off for several months on both stdlib-sig and python-dev.
>> >
>> > I think any time till the PEP is accepted is a good time to discuss
>> > changes to the API
>>
>> I disagree. If a PEP is being updated continuously then there is
>> nothing stable to pronounce on.
>
> Well, you've been making updates as a result of this round of
> discussion.
>
> If there is still discussion then perhaps the PEP isn't ready for
> pronouncement yet. ?At some point someone can decide it is all
> bikeshedding and ask for pronouncement on that basis, but I don't
> think it is appropriate to cut off discussion by saying "it's ready for
> pronouncement" unless you want increase the chances of its getting
> rejected.

I commiserate with Brian here - he's been very patient, and has been
working on things, taking in input, etc for awhile now on this. In his
mind, it is done (or at least incredibly close to done) and opening
the door in the conversation for more API nit picking and debate about
the exact verbiage on method names means we're never going to be done
splashing paint.

> The usual way of doing this (at least so far as I have observed, which
> granted hasn't been too many cases) is to say something like "I think
> this PEP is ready for pronouncement" and then wait for feedback on that
> assertion or for the pronouncement. ?It's especially good if you can answer
> any concerns that are raised with "that was discussed already and we
> concluded X". ?Bonus points for finding a thread reference and adding
> it to the PEP :)

While sure, this is true - I'd actually back Brian up on trying to
avoid more "why didn't you call it a banana" style discussions. At
some point the constant back and forth has to stop, and to his credit,
Brian has made a lot of changes, listened to a lot of feedback, etc.
It's fair for him to just ask that a decision be made.

jesse

From guido at python.org  Sat May 22 17:38:27 2010
From: guido at python.org (Guido van Rossum)
Date: Sat, 22 May 2010 08:38:27 -0700
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> 
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com> 
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net> 
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
Message-ID: <AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>

On Sat, May 22, 2010 at 7:09 AM, Jesse Noller <jnoller at gmail.com> wrote:
> On Sat, May 22, 2010 at 9:59 AM, R. David Murray <rdmurray at bitdance.com> wrote:
>> On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan <brian at sweetapp.com> wrote:
>>> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote:
>>> > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan <brian at sweetapp.com>
>>> > wrote:
>>> >> Except that now isn't the time for that discussion. This PEP has
>>> >> discussed
>>> >> on-and-off for several months on both stdlib-sig and python-dev.
>>> >
>>> > I think any time till the PEP is accepted is a good time to discuss
>>> > changes to the API
>>>
>>> I disagree. If a PEP is being updated continuously then there is
>>> nothing stable to pronounce on.
>>
>> Well, you've been making updates as a result of this round of
>> discussion.
>>
>> If there is still discussion then perhaps the PEP isn't ready for
>> pronouncement yet. ?At some point someone can decide it is all
>> bikeshedding and ask for pronouncement on that basis, but I don't
>> think it is appropriate to cut off discussion by saying "it's ready for
>> pronouncement" unless you want increase the chances of its getting
>> rejected.
>
> I commiserate with Brian here - he's been very patient, and has been
> working on things, taking in input, etc for awhile now on this. In his
> mind, it is done (or at least incredibly close to done) and opening
> the door in the conversation for more API nit picking and debate about
> the exact verbiage on method names means we're never going to be done
> splashing paint.
>
>> The usual way of doing this (at least so far as I have observed, which
>> granted hasn't been too many cases) is to say something like "I think
>> this PEP is ready for pronouncement" and then wait for feedback on that
>> assertion or for the pronouncement. ?It's especially good if you can answer
>> any concerns that are raised with "that was discussed already and we
>> concluded X". ?Bonus points for finding a thread reference and adding
>> it to the PEP :)
>
> While sure, this is true - I'd actually back Brian up on trying to
> avoid more "why didn't you call it a banana" style discussions. At
> some point the constant back and forth has to stop, and to his credit,
> Brian has made a lot of changes, listened to a lot of feedback, etc.
> It's fair for him to just ask that a decision be made.

Great points Jesse! Since I really don't have the time or expertise to
make a judgment on this PEP, I hereby appoint you chair of the
approval process for this PEP. That basically means that when you
think it's ready to be approved, you say so, and it's a done deal. The
remaining feedback cycle is up to you now -- it sounds like you're
ready for closure, which sounds good to me (again, without having read
the PEP or tried to write something using the proposed code). You can
do it however you like: you can declare it approved now, or read it
over once more yourself and suggest some final changes, or set a
period (e.g. 48 hours) during which final comments have to be
received, which you then will judge by merit or by your whim, or you
can flip a coin or say a prayer... (I've tried most of those myself in
the past and haven't done too badly if I say so myself. :-) You're the
boss now. I know you will do the right thing for this PEP.

-- 
--Guido van Rossum (python.org/~guido)

From rdmurray at bitdance.com  Sat May 22 22:40:16 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 22 May 2010 16:40:16 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
Message-ID: <20100522204016.DA8541F9FA2@kimball.webabinitio.net>

On Sat, 22 May 2010 10:09:37 -0400, Jesse Noller <jnoller at gmail.com> wrote:
> On Sat, May 22, 2010 at 9:59 AM, R. David Murray <rdmurray at bitdance.com> wr=
> ote:
> > On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan <brian at sweetapp.com> wr=
> ote:
> >> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote:
> >> > On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan <brian at sweetapp.com>
> >> > wrote:
> >> >> Except that now isn't the time for that discussion. This PEP has
> >> >> discussed
> >> >> on-and-off for several months on both stdlib-sig and python-dev.
> >> >
> >> > I think any time till the PEP is accepted is a good time to discuss
> >> > changes to the API
> >>
> >> I disagree. If a PEP is being updated continuously then there is
> >> nothing stable to pronounce on.
> >
> > Well, you've been making updates as a result of this round of
> > discussion.
> >
> > If there is still discussion then perhaps the PEP isn't ready for
> > pronouncement yet. =A0At some point someone can decide it is all
> > bikeshedding and ask for pronouncement on that basis, but I don't
> > think it is appropriate to cut off discussion by saying "it's ready for
> > pronouncement" unless you want increase the chances of its getting
> > rejected.
> 
> I commiserate with Brian here - he's been very patient, and has been
> working on things, taking in input, etc for awhile now on this. In his
> mind, it is done (or at least incredibly close to done) and opening
> the door in the conversation for more API nit picking and debate about
> the exact verbiage on method names means we're never going to be done
> splashing paint.

OK, so you are saying that the comments in question are bikeshedding.
I can accept that easily.  What I was trying to point out was that saying
"discussion is closed" is not the best way to nurture community consensus.
Saying "we've reached the bikesheedding point, let's pronounce" is a very
different thing to my mind, even if it is only a matter of tone.

Have fun pronouncing ;)

--
R. David Murray                                      www.bitdance.com

From brian at sweetapp.com  Sun May 23 01:12:46 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 09:12:46 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100522135946.8F5451F960D@kimball.webabinitio.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
Message-ID: <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com>


On 22 May 2010, at 23:59, R. David Murray wrote:

> On Sat, 22 May 2010 19:12:05 +1000, Brian Quinlan  
> <brian at sweetapp.com> wrote:
>> On May 22, 2010, at 5:30 AM, Dj Gilcrease wrote:
>>> On Fri, May 21, 2010 at 8:26 AM, Brian Quinlan <brian at sweetapp.com>
>>> wrote:
>>>> Except that now isn't the time for that discussion. This PEP has
>>>> discussed
>>>> on-and-off for several months on both stdlib-sig and python-dev.
>>>
>>> I think any time till the PEP is accepted is a good time to discuss
>>> changes to the API
>>
>> I disagree. If a PEP is being updated continuously then there is
>> nothing stable to pronounce on.
>
> Well, you've been making updates as a result of this round of
> discussion.

Yes, I've been making documentation and PEP updates to clarify points  
that people found confusing and will continue to do so.

> If there is still discussion then perhaps the PEP isn't ready for
> pronouncement yet.  At some point someone can decide it is all
> bikeshedding and ask for pronouncement on that basis, but I don't
> think it is appropriate to cut off discussion by saying "it's ready  
> for
> pronouncement" unless you want increase the chances of its getting
> rejected.

Here are the new proposed non-documentation changes that I've  
collected (let me know if I've missed any):

Rename "executor" => "executer"
Rename "submit" to "apply"
Rename "done" to "finished"
Rename "not_finished" to "pending"
Rename "FIRST_COMPLETED" to "ONE_COMPLETED"

We can discuss naming for all eternity and never reach a point where  
even half of the participants are satisfied.

Since naming has been discussed extensively here and in stdlib-sig, I  
think that we have to decide that it is good enough and move on. Or  
decide that it isn't good enough and reject the PEP.

Cheers,
Brian

> The usual way of doing this (at least so far as I have observed, which
> granted hasn't been too many cases) is to say something like "I think
> this PEP is ready for pronouncement" and then wait for feedback on  
> that
> assertion or for the pronouncement.  It's especially good if you can  
> answer
> any concerns that are raised with "that was discussed already and we
> concluded X".  Bonus points for finding a thread reference and adding
> it to the PEP :)
>
> --
> R. David Murray                                      www.bitdance.com


From jyasskin at gmail.com  Sun May 23 01:43:41 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sat, 22 May 2010 16:43:41 -0700
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> 
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com> 
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net> 
	<2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com>
Message-ID: <AANLkTimYZ3KyN-jSg0byBIg_ujg2hp7Tob6MJNh9iwIL@mail.gmail.com>

I think the PEP's overall API is good to go.

On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan <brian at sweetapp.com> wrote:
>
> On 22 May 2010, at 23:59, R. David Murray wrote:
>> If there is still discussion then perhaps the PEP isn't ready for
>> pronouncement yet. ?At some point someone can decide it is all
>> bikeshedding and ask for pronouncement on that basis, but I don't
>> think it is appropriate to cut off discussion by saying "it's ready for
>> pronouncement" unless you want increase the chances of its getting
>> rejected.
>
> Here are the new proposed non-documentation changes that I've collected (let
> me know if I've missed any):
>
> ...

I propose to rename the Future.result method to Future.get. "get" is
what Java (http://java.sun.com/javase/7/docs/api/java/util/concurrent/Future.html)
and C++ (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3092.pdf
section 30.6.6 para 12) use, and the word "result" doesn't seem
particularly better or worse than "get" for our purposes, which
inclines me to stay consistent.

> We can discuss naming for all eternity and never reach a point where even
> half of the participants are satisfied.

Agreed. To reduce the length of the discussion, I'm not going to reply
to counter-arguments to my proposal, but I think it'll be useful to
Jesse if people who agree or disagree speak up briefly. I'll reply the
other naming proposals in another message.

Jeffrey

From jyasskin at gmail.com  Sun May 23 02:06:30 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sat, 22 May 2010 17:06:30 -0700
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> 
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com> 
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net> 
	<2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com>
Message-ID: <AANLkTilcTJj92lmNQJYtzd7yiaolHw1xSevE8g64aTz_@mail.gmail.com>

On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Rename "executor" => "executer"

-1 for consistency with Java.

> Rename "submit" to "apply"

"apply" focuses attention on the function object, while "submit"
focuses attention, properly I think, on the fact that you're handing
something to the executor to run. So -1.

> Rename "done" to "finished"

"done" is nice and short, and I don't think "finished" or "completed"
will be any less prone to people thinking the task actually ran. So
-1.

> Rename "not_finished" to "pending"

+0.5. Doesn't matter that much, but pending is used elsewhere in the
proposal for this concept. On the other hand, "pending" could be
thought to refer to the state before "running". Possibly "finished"
should be renamed to "done" here, since it's described as '"finished",
contains the futures that completed (finished or were cancelled)',
which uses "finished" for two different concepts.

> Rename "FIRST_COMPLETED" to "ONE_COMPLETED"

"ONE_COMPLETED" could imply that the first result set must contain
exactly one element, but in fact, if multiple tasks finish before the
waiting thread has a chance to wake up, multiple futures could be
returned as done. So -1.


And like my other post, I won't argue about these, leaving the actual
decision up to Brian and Jesse.

From brian at sweetapp.com  Sun May 23 02:47:08 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 10:47:08 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
Message-ID: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>

Hey all,

Jesse, the designated pronouncer for this PEP, has decided to keep  
discussion open for a few more days.

So fire away!

Cheers,
Brian


From brian at sweetapp.com  Sun May 23 03:04:49 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 11:04:49 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTimYZ3KyN-jSg0byBIg_ujg2hp7Tob6MJNh9iwIL@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com>
	<AANLkTimYZ3KyN-jSg0byBIg_ujg2hp7Tob6MJNh9iwIL@mail.gmail.com>
Message-ID: <54EFD5A6-295A-4622-947F-FC57EBA554A5@sweetapp.com>

On May 23, 2010, at 9:43 AM, Jeffrey Yasskin wrote:

> I think the PEP's overall API is good to go.
>
> On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>>
>> On 22 May 2010, at 23:59, R. David Murray wrote:
>>> If there is still discussion then perhaps the PEP isn't ready for
>>> pronouncement yet.  At some point someone can decide it is all
>>> bikeshedding and ask for pronouncement on that basis, but I don't
>>> think it is appropriate to cut off discussion by saying "it's  
>>> ready for
>>> pronouncement" unless you want increase the chances of its getting
>>> rejected.
>>
>> Here are the new proposed non-documentation changes that I've  
>> collected (let
>> me know if I've missed any):
>>
>> ...
>
> I propose to rename the Future.result method to Future.get. "get" is
> what Java (http://java.sun.com/javase/7/docs/api/java/util/concurrent/Future.html 
> )
> and C++ (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3092.pdf
> section 30.6.6 para 12) use, and the word "result" doesn't seem
> particularly better or worse than "get" for our purposes, which
> inclines me to stay consistent.

In C++ and Java, there is only one result-retrieving method so "get"  
seems like a reasonable name.

My implementation has a second method .exception(), which returns the  
exception raised by the submitted function (or None if no exception  
was raised). I thought that having multiple getter methods, where one  
is called .get() would be a bit confusing.

But I don't really care so I'm -0.

Cheers,
Brian

From jnoller at gmail.com  Sun May 23 03:07:19 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 22 May 2010 21:07:19 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
Message-ID: <AANLkTinuYTnsIR_1-AIYEXsY9TczR5FK-J9E6ZiGjHHD@mail.gmail.com>

On Sat, May 22, 2010 at 8:47 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
>
> Jesse, the designated pronouncer for this PEP, has decided to keep
> discussion open for a few more days.
>
> So fire away!

Man, everyone's faster on the email thing lately than me :)

Yes, I spoke to Brian, and since we're not in a rush - please do bring
up serious issues you might have, assume for the moment that the
names, unless they're "def lol_python" are going to stay pretty
consistent unless everyone breaks out the pitchforks, and also assume
the API won't see much changing.

So please, do bring up issues. I'm obviously biased towards accepting
it - however, nothing is ever set in stone.

jesse

From brian at sweetapp.com  Sun May 23 03:21:51 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 11:21:51 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTilcTJj92lmNQJYtzd7yiaolHw1xSevE8g64aTz_@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com>
	<AANLkTilcTJj92lmNQJYtzd7yiaolHw1xSevE8g64aTz_@mail.gmail.com>
Message-ID: <CE3E9442-ABF9-49F1-BA67-1F89CCBD85B4@sweetapp.com>


On May 23, 2010, at 10:06 AM, Jeffrey Yasskin wrote:

> On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> Rename "executor" => "executer"
>
> -1 for consistency with Java.
>

-1 pending an explanation of why "executer" is better

>> Rename "submit" to "apply"
>
> "apply" focuses attention on the function object, while "submit"
> focuses attention, properly I think, on the fact that you're handing
> something to the executor to run. So -1.

-1

>> Rename "done" to "finished"
>
> "done" is nice and short, and I don't think "finished" or "completed"
> will be any less prone to people thinking the task actually ran. So
> -1.

-0

>> Rename "not_finished" to "pending"
>
> +0.5. Doesn't matter that much, but pending is used elsewhere in the
> proposal for this concept. On the other hand, "pending" could be
> thought to refer to the state before "running". Possibly "finished"
> should be renamed to "done" here, since it's described as '"finished",
> contains the futures that completed (finished or were cancelled)',
> which uses "finished" for two different concepts.

I think that using "finished" is bad terminology here.

So +1 to "finished" => "done".
I don't have a preference for "not_done" vs. "pending".


>> Rename "FIRST_COMPLETED" to "ONE_COMPLETED"
>
> "ONE_COMPLETED" could imply that the first result set must contain
> exactly one element, but in fact, if multiple tasks finish before the
> waiting thread has a chance to wake up, multiple futures could be
> returned as done. So -1.

A logician would probably call it "SOME_COMPLETED".
What about "ANY_COMPLETED"? Though I think that "FIRST_COMPLETED"  
still reads better.

Cheers,
Brian

From glyph at twistedmatrix.com  Sun May 23 06:44:50 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Sun, 23 May 2010 00:44:50 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
Message-ID: <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>


On May 22, 2010, at 8:47 PM, Brian Quinlan wrote:

> Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days.
> 
> So fire away!

As you wish!

The PEP should be consistent in its usage of terminology about callables.  It alternately calls them "callables", "functions", and "functions or methods".  It would be nice to clean this up and be consistent about what can be called where.  I personally like "callables".

The execution context of callable code is not made clear.  Implicitly, submit() or map() would run the code in threads or processes as defined by the executor, but that's not spelled out clearly.

More relevant to my own interests, the execution context of the callables passed to add_done_callback and remove_done_callback is left almost completely to the imagination.  If I'm reading the sample implementation correctly, <http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/process.py#241>, it looks like in the multiprocessing implementation, the done callbacks are invoked in a random local thread.  The fact that they are passed the future itself *sort* of implies that this is the case, but the multiprocessing module plays fast and loose with object identity all over the place, so it would be good to be explicit and say that it's *not* a pickled copy of the future sitting in some arbitrary process (or even on some arbitrary machine).

This is really minor, I know, but why does it say "NOTE: This method can be used to create adapters from Futures to Twisted Deferreds"?  First of all, what's the deal with "NOTE"; it's the only "NOTE" in the whole PEP, and it doesn't seem to add anything.  This sentence would read exactly the same if that word were deleted.  Without more clarity on the required execution context of the callbacks, this claim might not actually be true anyway; Deferred callbacks can only be invoked in the main reactor thread in Twisted.  But even if it is perfectly possible, why leave so much of the adapter implementation up to the imagination?  If it's important enough to mention, why not have a reference to such an adapter in the reference Futures implementation, since it *should* be fairly trivial to write?

The fact that add_done_callback is implemented using a set is weird, since it means you can't add the same callback more than once.  The set implementation also means that the callbacks get called in a semi-random order, potentially creating even _more_ hard-to-debug order of execution issues than you'd normally have with futures.  And I think that this documentation will be unclear to a lot of novice developers: many people have trouble with the idea that "a = Foo(); b = Foo(); a.bar_method != b.bar_method", but "import foo_module; foo_module.bar_function == foo_module.bar_function".

It's also weird that you can remove callbacks - what's the use case?  Deferreds have no callback-removal mechanism and nobody has ever complained of the need for one, as far as I know.  (But lots of people do add the same callback multiple times.)

I suggest having have add_done_callback, implementing it with a list so that callbacks are always invoked in the order that they're added, and getting rid of remove_done_callback.

futures._base.Executor isn't exposed publicly, but it needs to be.  The PEP kinda makes it sound like it is ("Executor is an abstract class...").  Plus, A third party library wanting to implement an executor of its own shouldn't have to copy and paste the implementation of Executor.map.

One minor suggestion on the "internal future methods" bit - something I wish we'd done with Deferreds was to put 'callback()' and 'addCallbacks()' on separate objects, so that it was very explicit whether you were on the emitting side of a Deferred or the consuming side.  That seems to be the case with these internal methods - they are not so much "internal" as they are for the producer of the Future (whether a unit test or executor) so you might want to put them on a different object that it's easy for the thing creating a Future() to get at but hard for any subsequent application code to fiddle with by accident.  Off the top of my head, I suggest naming it "Invoker()".  A good way to do this would be to have an Invoker class which can't be instantiated (raises an exception from __init__ or somesuch), then a Future.create() method which returns an Invoker, which itself has a '.future' attribute.

Finally, why isn't this just a module on PyPI?  It doesn't seem like there's any particular benefit to making this a stdlib module and going through the whole PEP process - except maybe to prompt feedback like this :).  Issues like the ones I'm bringing up could be fixed pretty straightforwardly if it were just a matter of filing a bug on a small package, but fixing a stdlib module is a major undertaking.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100523/c119c5ae/attachment.html>

From brian at sweetapp.com  Sun May 23 08:37:31 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 16:37:31 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
Message-ID: <E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>


On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:

>
> On May 22, 2010, at 8:47 PM, Brian Quinlan wrote:
>
>> Jesse, the designated pronouncer for this PEP, has decided to keep  
>> discussion open for a few more days.
>>
>> So fire away!
>
> As you wish!

I retract my request ;-)

> The PEP should be consistent in its usage of terminology about  
> callables.  It alternately calls them "callables", "functions", and  
> "functions or methods".  It would be nice to clean this up and be  
> consistent about what can be called where.  I personally like  
> "callables".

Did you find the terminology confusing? If not then I propose not  
changing it.

But changing it in the user docs is probably a good idea. I like  
"callables" too.

>
> The execution context of callable code is not made clear.   
> Implicitly, submit() or map() would run the code in threads or  
> processes as defined by the executor, but that's not spelled out  
> clearly.
>
> More relevant to my own interests, the execution context of the  
> callables passed to add_done_callback and remove_done_callback is  
> left almost completely to the imagination.  If I'm reading the  
> sample implementation correctly, <http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/process.py#241 
> >, it looks like in the multiprocessing implementation, the done  
> callbacks are invoked in a random local thread.  The fact that they  
> are passed the future itself *sort* of implies that this is the  
> case, but the multiprocessing module plays fast and loose with  
> object identity all over the place, so it would be good to be  
> explicit and say that it's *not* a pickled copy of the future  
> sitting in some arbitrary process (or even on some arbitrary machine).

The callbacks will always be called in a thread other than the main  
thread in the process that created the executor. Is that a strong  
enough contract?

> This is really minor, I know, but why does it say "NOTE: This method  
> can be used to create adapters from Futures to Twisted Deferreds"?   
> First of all, what's the deal with "NOTE"; it's the only "NOTE" in  
> the whole PEP, and it doesn't seem to add anything.  This sentence  
> would read exactly the same if that word were deleted.  Without more  
> clarity on the required execution context of the callbacks, this  
> claim might not actually be true anyway; Deferred callbacks can only  
> be invoked in the main reactor thread in Twisted.  But even if it is  
> perfectly possible, why leave so much of the adapter implementation  
> up to the imagination?  If it's important enough to mention, why not  
> have a reference to such an adapter in the reference Futures  
> implementation, since it *should* be fairly trivial to write?

I'm a bit surprised that this doesn't allow for better  
interoperability with Deferreds given this discussion:

At 02:36 PM 3/16/2010 -0700, Brian Quinlan wrote:

"""
 From P.J Eby:
> On Mar 7, 2010, at 11:56 AM, P.J. Eby wrote:
>
>> At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>>> Given a way to register "on-done" callbacks with the future, it  
>>> would be straightforward to wait for a future without blocking, too.
>>
>> Yes, and with a few more additions besides that one, you might be  
>> on the way to an actual competitor for Deferreds.  For example:  
>> retry support, chaining, logging, API for transparent result  
>> processing, coroutine support, co-ordination tools like locks,  
>> sempaphores and queues, etc.
>
> OK, but lets just think about making the APIs compatible e.g. you  
> have some code that uses Futures and now you want to integrate it  
> with some code that uses Deferreds.
>
> I think Jeff's suggestion of having a completion callback on Futures  
> would make it possible to write a Future-to-Deferred adapter. Is  
> that correct?

As long as the callback signature included a way to pass in an error,  
then yes, that'd probably be sufficient.
"""

If add_done_callback doesn't help with twisted interoperability then  
I'd suggest removing it to allow for something that may be more useful  
to be added later.

>
> The fact that add_done_callback is implemented using a set is weird,  
> since it means you can't add the same callback more than once.  The  
> set implementation also means that the callbacks get called in a  
> semi-random order, potentially creating even _more_ hard-to-debug  
> order of execution issues than you'd normally have with futures.   
> And I think that this documentation will be unclear to a lot of  
> novice developers: many people have trouble with the idea that "a =  
> Foo(); b = Foo(); a.bar_method != b.bar_method", but "import  
> foo_module; foo_module.bar_function == foo_module.bar_function".
>
> It's also weird that you can remove callbacks - what's the use  
> case?  Deferreds have no callback-removal mechanism and nobody has  
> ever complained of the need for one, as far as I know.  (But lots of  
> people do add the same callback multiple times.)
>
> I suggest having have add_done_callback, implementing it with a list  
> so that callbacks are always invoked in the order that they're  
> added, and getting rid of remove_done_callback.

Sounds good to me!

> futures._base.Executor isn't exposed publicly, but it needs to be.   
> The PEP kinda makes it sound like it is ("Executor is an abstract  
> class...").  Plus, A third party library wanting to implement an  
> executor of its own shouldn't have to copy and paste the  
> implementation of Executor.map.

That was a bug that I've fixed. Thanks!

>
> One minor suggestion on the "internal future methods" bit -  
> something I wish we'd done with Deferreds was to put 'callback()'  
> and 'addCallbacks()' on separate objects, so that it was very  
> explicit whether you were on the emitting side of a Deferred or the  
> consuming side.  That seems to be the case with these internal  
> methods - they are not so much "internal" as they are for the  
> producer of the Future (whether a unit test or executor) so you  
> might want to put them on a different object that it's easy for the  
> thing creating a Future() to get at but hard for any subsequent  
> application code to fiddle with by accident.  Off the top of my  
> head, I suggest naming it "Invoker()".  A good way to do this would  
> be to have an Invoker class which can't be instantiated (raises an  
> exception from __init__ or somesuch), then a Future.create() method  
> which returns an Invoker, which itself has a '.future' attribute.
>
> Finally, why isn't this just a module on PyPI?  It doesn't seem like  
> there's any particular benefit to making this a stdlib module and  
> going through the whole PEP process - except maybe to prompt  
> feedback like this :).

We've already had this discussion before. Could you explain why this  
module should *not* be in the stdlib e.g. does it have significantly  
less utility than other modules in stdlib? Is it significantly higher  
risk? etc?


> Issues like the ones I'm bringing up could be fixed pretty  
> straightforwardly if it were just a matter of filing a bug on a  
> small package, but fixing a stdlib module is a major undertaking.

True but I don't think that is a convincing argument. A subset of the  
functionality provided by this module is already available in Java and  
C++ and (at least in Java) it is used extensively and without too much  
trouble. If there are implementation bugs then we can fix them just  
like we would with any other module.

Cheers,
Brian


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100523/0751b7f5/attachment-0001.html>

From debatem1 at gmail.com  Sun May 23 11:15:12 2010
From: debatem1 at gmail.com (geremy condra)
Date: Sun, 23 May 2010 05:15:12 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
Message-ID: <AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>

On Sun, May 23, 2010 at 2:37 AM, Brian Quinlan <brian at sweetapp.com> wrote:

<snip>

> Finally, why isn't this just a module on PyPI? ?It doesn't seem like there's
> any particular benefit to making this a stdlib module and going through the
> whole PEP process - except maybe to prompt feedback like this :).
>
> We've already had this discussion before. Could you explain why this module
> should *not* be in the stdlib e.g. does it have significantly less utility
> than other modules in stdlib? Is it significantly higher risk? etc?

Inclusion in the stdlib is the exception, not the rule, and every
exception should be issued for a good reason. I'd like to know
what that reason is in this case, if only to get a clearer
understanding of why the PEP was accepted.

> Issues like the ones I'm bringing up could be fixed pretty straightforwardly
> if it were just a matter of filing a bug on a small package, but fixing a
> stdlib module is a major undertaking.
>
> True but I don't think that is a convincing argument. A subset of the
> functionality provided by this module is already available in Java and C++
> and (at least in Java) it is used extensively and without too much trouble.
> If there are implementation bugs then we can fix them just like we would
> with any other module.

Guido made exactly the opposite argument during his keynote at PyCon.
It seemed fairly reasonable at the time- why do you think it doesn't apply
here?

Geremy Condra

From brian at sweetapp.com  Sun May 23 11:39:11 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 19:39:11 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
Message-ID: <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>

On May 23, 2010, at 7:15 PM, geremy condra wrote:

> On Sun, May 23, 2010 at 2:37 AM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>
> <snip>
>
>> Finally, why isn't this just a module on PyPI?  It doesn't seem  
>> like there's
>> any particular benefit to making this a stdlib module and going  
>> through the
>> whole PEP process - except maybe to prompt feedback like this :).
>>
>> We've already had this discussion before. Could you explain why  
>> this module
>> should *not* be in the stdlib e.g. does it have significantly less  
>> utility
>> than other modules in stdlib? Is it significantly higher risk? etc?
>
> Inclusion in the stdlib is the exception, not the rule, and every
> exception should be issued for a good reason. I'd like to know
> what that reason is in this case,

This package eliminates the need to construct the boilerplate present  
in many Python applications i.e. a thread or process pool, a work  
queue and result queue.  It also makes it easy to take an existing  
Python application that executes (e.g. IO operations) in sequence and  
execute them in parallel. It package provides common idioms for two  
existing modules i.e. multiprocessing offers map functionality while  
threading doesn't. Those idioms are well understood and already  
present in Java and C++.

> if only to get a clearer
> understanding of why the PEP was accepted.

It hasn't been accepted.

>> Issues like the ones I'm bringing up could be fixed pretty  
>> straightforwardly
>> if it were just a matter of filing a bug on a small package, but  
>> fixing a
>> stdlib module is a major undertaking.
>>
>> True but I don't think that is a convincing argument. A subset of the
>> functionality provided by this module is already available in Java  
>> and C++
>> and (at least in Java) it is used extensively and without too much  
>> trouble.
>> If there are implementation bugs then we can fix them just like we  
>> would
>> with any other module.
>
> Guido made exactly the opposite argument during his keynote at PyCon.
> It seemed fairly reasonable at the time- why do you think it doesn't  
> apply
> here?


Could you be a little more specific about Guido's argument at PyCon?

Cheers,
Brian

From regebro at gmail.com  Sun May 23 11:54:27 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Sun, 23 May 2010 11:54:27 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net> 
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com> 
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com> 
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> 
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com> 
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
Message-ID: <AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>

On Sun, May 23, 2010 at 11:39, Brian Quinlan <brian at sweetapp.com> wrote:
> This package eliminates the need to construct the boilerplate present in
> many Python applications i.e. a thread or process pool, a work queue and
> result queue. ?It also makes it easy to take an existing Python application
> that executes (e.g. IO operations) in sequence and execute them in parallel.
> It package provides common idioms for two existing modules i.e.
> multiprocessing offers map functionality while threading doesn't. Those
> idioms are well understood and already present in Java and C++.

It can do that as a separate package as well. And not only that, it
could then be available on PyPI for earlier versions of Python as
well, making it much more likely to gain widespread acceptance.

> Could you be a little more specific about Guido's argument at PyCon?

A module in stdlib has to be "dead". After it's included in the stdlib
it can not go through any major changes since that would mean loss of
backwards incompatibility. Also, you can't fix bugs except by
releasing new versions of Python. Therefore the API must be completely
stable, and the product virtually bugfree before it should be in
stdlib. The best way of ensuring that is to release it as a separate
module on PyPI, and let it stabilize for a couple of years.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From brian at sweetapp.com  Sun May 23 12:15:39 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 20:15:39 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
Message-ID: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>


On May 23, 2010, at 7:54 PM, Lennart Regebro wrote:

> On Sun, May 23, 2010 at 11:39, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> This package eliminates the need to construct the boilerplate  
>> present in
>> many Python applications i.e. a thread or process pool, a work  
>> queue and
>> result queue.  It also makes it easy to take an existing Python  
>> application
>> that executes (e.g. IO operations) in sequence and execute them in  
>> parallel.
>> It package provides common idioms for two existing modules i.e.
>> multiprocessing offers map functionality while threading doesn't.  
>> Those
>> idioms are well understood and already present in Java and C++.
>
> It can do that as a separate package as well.

You could make the same argument about any module in the stdlib.

> And not only that, it
> could then be available on PyPI for earlier versions of Python as
> well, making it much more likely to gain widespread acceptance.

I doubt it. Simple modules are unlikely to develop a following because  
it is too easy to partially replicate their functionality. urlparse  
and os.path are very useful modules but I doubt that they would have  
been successful on PyPI.

>> Could you be a little more specific about Guido's argument at PyCon?
>
> A module in stdlib has to be "dead". After it's included in the stdlib
> it can not go through any major changes since that would mean loss of
> backwards incompatibility.

The good news in this case is that the same API has been used  
successfully in Java and C++ for years so it is unlikely that any  
major changes will need to be made.

> Also, you can't fix bugs except by
> releasing new versions of Python. Therefore the API must be completely
> stable, and the product virtually bugfree before it should be in
> stdlib. The best way of ensuring that is to release it as a separate
> module on PyPI, and let it stabilize for a couple of years.


Yeah but that model isn't likely to work with this package.

Cheers,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100523/34e75e9d/attachment.html>

From robertc at robertcollins.net  Sun May 23 12:43:22 2010
From: robertc at robertcollins.net (Robert Collins)
Date: Sun, 23 May 2010 22:43:22 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
Message-ID: <AANLkTimng5BbElsJmTCNvL3nAozAAjvMJkQXBmhxe9Td@mail.gmail.com>

On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan <brian at sweetapp.com> wrote:

>> Also, you can't fix bugs except by
>> releasing new versions of Python. Therefore the API must be completely
>> stable, and the product virtually bugfree before it should be in
>> stdlib. The best way of ensuring that is to release it as a separate
>> module on PyPI, and let it stabilize for a couple of years.
>
> Yeah but that model isn't likely to work with this package.
> Cheers,
> Brian

Forgive my ignorance, but why do you say that that model won't work
with this package?

From dirkjan at ochtman.nl  Sun May 23 12:43:57 2010
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sun, 23 May 2010 12:43:57 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com> 
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com> 
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> 
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com> 
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com> 
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
Message-ID: <AANLkTinBPbj-8I6ZgK6s9gFSEBz3hQLR7plO4ApNAWy6@mail.gmail.com>

On Sun, May 23, 2010 at 12:15, Brian Quinlan <brian at sweetapp.com> wrote:
> I doubt it. Simple modules are unlikely to develop a following because it is
> too easy to partially replicate their functionality. urlparse and os.path
> are very useful modules but I doubt that they would have been successful on
> PyPI.

simplejson was also fairly simple, but still developed a following.

> The good news in this case is that the same API has been used successfully
> in Java and C++ for years so it is unlikely that any major changes will need
> to be made.

I would agree that having prior versions in other languages should
make the API more stable, but I wouldn't agree that it doesn't need
changes (and even minor changes can be a PITA in the stdlib).

>> Also, you can't fix bugs except by
>> releasing new versions of Python. Therefore the API must be completely
>> stable, and the product virtually bugfree before it should be in
>> stdlib. The best way of ensuring that is to release it as a separate
>> module on PyPI, and let it stabilize for a couple of years.
>
> Yeah but that model isn't likely to work with this package.

Okay, I'll bite: why is your package different?

In general, this reminds me of the ipaddr discussions. I read through
the thread from March real quick to see if there was reasoning there
why this package should be an exception from the "normal" standards
track (that is, ripen on PyPI, then moving it in the stdlib when it's
mature -- where "mature" is another word for dead, really). But then
this is just another instance of the fat-stdlib vs lean-stdlib
discussion, I guess, so we can go on at length.

Cheers,

Dirkjan

From brian at sweetapp.com  Sun May 23 12:47:27 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 20:47:27 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTimng5BbElsJmTCNvL3nAozAAjvMJkQXBmhxe9Td@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTimng5BbElsJmTCNvL3nAozAAjvMJkQXBmhxe9Td@mail.gmail.com>
Message-ID: <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com>


On May 23, 2010, at 8:43 PM, Robert Collins wrote:

> On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>
>>> Also, you can't fix bugs except by
>>> releasing new versions of Python. Therefore the API must be  
>>> completely
>>> stable, and the product virtually bugfree before it should be in
>>> stdlib. The best way of ensuring that is to release it as a separate
>>> module on PyPI, and let it stabilize for a couple of years.
>>
>> Yeah but that model isn't likely to work with this package.
>> Cheers,
>> Brian
>
> Forgive my ignorance, but why do you say that that model won't work
> with this package?

As I said in my last message:

"""Simple modules are unlikely to develop a following because it is  
too easy to partially replicate their functionality. urlparse and  
os.path are very useful modules but I doubt that they would have been  
successful on PyPI."""

Cheers,
Brian

From solipsis at pitrou.net  Sun May 23 12:51:55 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 23 May 2010 12:51:55 +0200
Subject: [Python-Dev] Dead modules
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTinBPbj-8I6ZgK6s9gFSEBz3hQLR7plO4ApNAWy6@mail.gmail.com>
Message-ID: <20100523125155.53959f2f@pitrou.net>

On Sun, 23 May 2010 12:43:57 +0200
Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:
> 
> In general, this reminds me of the ipaddr discussions. I read through
> the thread from March real quick to see if there was reasoning there
> why this package should be an exception from the "normal" standards
> track (that is, ripen on PyPI, then moving it in the stdlib when it's
> mature -- where "mature" is another word for dead, really).

I disagree that a stdlib module is a dead module. It is perfectly
possible to augment the API with new functionality without breaking
compatibility. You can also deprecate old APIs if you want.

Regards

Antoine.



From brian at sweetapp.com  Sun May 23 13:02:48 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 21:02:48 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTinBPbj-8I6ZgK6s9gFSEBz3hQLR7plO4ApNAWy6@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTinBPbj-8I6ZgK6s9gFSEBz3hQLR7plO4ApNAWy6@mail.gmail.com>
Message-ID: <FE24980F-8458-4427-9082-9A0EB58663D3@sweetapp.com>

On May 23, 2010, at 8:43 PM, Dirkjan Ochtman wrote:
> On Sun, May 23, 2010 at 12:15, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> I doubt it. Simple modules are unlikely to develop a following  
>> because it is
>> too easy to partially replicate their functionality. urlparse and  
>> os.path
>> are very useful modules but I doubt that they would have been  
>> successful on
>> PyPI.
>
> simplejson was also fairly simple, but still developed a following.

The API is simple but writing a JSON parser is hard enough that people  
will check to see if someone has already done the work for them  
(especially since JSON is fairly topical).

If you are familiar with threads then writing a "good enough" solution  
without futures probably won't take you very long. Also, unless you  
are familiar with another futures implementation, you aren't likely to  
know where to look.

>> The good news in this case is that the same API has been used  
>> successfully
>> in Java and C++ for years so it is unlikely that any major changes  
>> will need
>> to be made.
>
> I would agree that having prior versions in other languages should
> make the API more stable, but I wouldn't agree that it doesn't need
> changes (and even minor changes can be a PITA in the stdlib).

Some changes are hard (i.e. changing the semantics of existing method)  
but some are pretty easy (i.e. adding new methods).

Cheers,
Brian
>
>>> Also, you can't fix bugs except by
>>> releasing new versions of Python. Therefore the API must be  
>>> completely
>>> stable, and the product virtually bugfree before it should be in
>>> stdlib. The best way of ensuring that is to release it as a separate
>>> module on PyPI, and let it stabilize for a couple of years.
>>
>> Yeah but that model isn't likely to work with this package.
>
> Okay, I'll bite: why is your package different?
>
> In general, this reminds me of the ipaddr discussions. I read through
> the thread from March real quick to see if there was reasoning there
> why this package should be an exception from the "normal" standards
> track (that is, ripen on PyPI, then moving it in the stdlib when it's
> mature -- where "mature" is another word for dead, really). But then
> this is just another instance of the fat-stdlib vs lean-stdlib
> discussion, I guess, so we can go on at length.



From dirkjan at ochtman.nl  Sun May 23 13:00:05 2010
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Sun, 23 May 2010 13:00:05 +0200
Subject: [Python-Dev] Dead modules
In-Reply-To: <20100523125155.53959f2f@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> 
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com> 
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com> 
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTinBPbj-8I6ZgK6s9gFSEBz3hQLR7plO4ApNAWy6@mail.gmail.com> 
	<20100523125155.53959f2f@pitrou.net>
Message-ID: <AANLkTinoQwp0Yg7jT_meIJTXYcS5M6oMwv1E8L3OetoZ@mail.gmail.com>

On Sun, May 23, 2010 at 12:51, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I disagree that a stdlib module is a dead module. It is perfectly
> possible to augment the API with new functionality without breaking
> compatibility. You can also deprecate old APIs if you want.

Right, it wasn't intended as that harsh... but it does come with a
rather impressive set of constraints in terms of what you can do with
the API.

Cheers,

Dirkjan

From andrew.svetlov at gmail.com  Sun May 23 13:16:27 2010
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sun, 23 May 2010 15:16:27 +0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTimng5BbElsJmTCNvL3nAozAAjvMJkQXBmhxe9Td@mail.gmail.com>
	<7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com>
Message-ID: <AANLkTik-ewslhUa6bd0pwXyAq_gUiOuXpwgjJ3GkNAV3@mail.gmail.com>

Is there any reason to have Future .cancelled, .done, .running as methods?
>From my perspective they are really readonly properties.

BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l'
form: 'canceled'.

On Sun, May 23, 2010 at 2:47 PM, Brian Quinlan <brian at sweetapp.com> wrote:
>
> On May 23, 2010, at 8:43 PM, Robert Collins wrote:
>
>> On Sun, May 23, 2010 at 10:15 PM, Brian Quinlan <brian at sweetapp.com>
>> wrote:
>>
>>>> Also, you can't fix bugs except by
>>>> releasing new versions of Python. Therefore the API must be completely
>>>> stable, and the product virtually bugfree before it should be in
>>>> stdlib. The best way of ensuring that is to release it as a separate
>>>> module on PyPI, and let it stabilize for a couple of years.
>>>
>>> Yeah but that model isn't likely to work with this package.
>>> Cheers,
>>> Brian
>>
>> Forgive my ignorance, but why do you say that that model won't work
>> with this package?
>
> As I said in my last message:
>
> """Simple modules are unlikely to develop a following because it is too easy
> to partially replicate their functionality. urlparse and os.path are very
> useful modules but I doubt that they would have been successful on PyPI."""
>
> Cheers,
> Brian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>

From regebro at gmail.com  Sun May 23 13:17:51 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Sun, 23 May 2010 13:17:51 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com> 
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com> 
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> 
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com> 
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com> 
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
Message-ID: <AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>

On Sun, May 23, 2010 at 12:15, Brian Quinlan <brian at sweetapp.com> wrote:
> You could make the same argument about any module in the stdlib.

Yeah, and that's exactly what I did.

> I doubt it. Simple modules are unlikely to develop a following because it is
> too easy to partially replicate their functionality. urlparse and os.path
> are very useful modules but I doubt that they would have been successful on
> PyPI.

Are you saying your proposed module is so simple that anyone can
easily replicate it with just a couple of lines of code?

> The good news in this case is that the same API has been used successfully
> in Java and C++ for years so it is unlikely that any major changes will need
> to be made.

Good. Then the time it takes to "mature" on PyPI would be very short.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From jon+python-dev at unequivocal.co.uk  Sun May 23 13:20:26 2010
From: jon+python-dev at unequivocal.co.uk (Jon Ribbens)
Date: Sun, 23 May 2010 12:20:26 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTik-ewslhUa6bd0pwXyAq_gUiOuXpwgjJ3GkNAV3@mail.gmail.com>
References: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTimng5BbElsJmTCNvL3nAozAAjvMJkQXBmhxe9Td@mail.gmail.com>
	<7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com>
	<AANLkTik-ewslhUa6bd0pwXyAq_gUiOuXpwgjJ3GkNAV3@mail.gmail.com>
Message-ID: <20100523112026.GM6976@snowy.squish.net>

On Sun, May 23, 2010 at 03:16:27PM +0400, Andrew Svetlov wrote:
> Is there any reason to have Future .cancelled, .done, .running as methods?
> >From my perspective they are really readonly properties.
> 
> BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l'
> form: 'canceled'.

In English, only "cancelled" is correct. In American, either is correct.

From brian at sweetapp.com  Sun May 23 13:29:45 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 23 May 2010 21:29:45 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
Message-ID: <E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>


On 23 May 2010, at 21:17, Lennart Regebro wrote:

> On Sun, May 23, 2010 at 12:15, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> You could make the same argument about any module in the stdlib.
>
> Yeah, and that's exactly what I did.
>
>> I doubt it. Simple modules are unlikely to develop a following  
>> because it is
>> too easy to partially replicate their functionality. urlparse and  
>> os.path
>> are very useful modules but I doubt that they would have been  
>> successful on
>> PyPI.
>
> Are you saying your proposed module is so simple that anyone can
> easily replicate it with just a couple of lines of code?

Parts of it, yes. Just like I can replace most operations in os.path  
and urlparse with a few lines of code.

>> The good news in this case is that the same API has been used  
>> successfully
>> in Java and C++ for years so it is unlikely that any major changes  
>> will need
>> to be made.
>
> Good. Then the time it takes to "mature" on PyPI would be very short.

How would you define "very short"? I've had the project on PyPI for  
about a year now:
http://pypi.python.org/pypi/futures3

Cheers,
Brian

From steve at holdenweb.com  Sun May 23 13:52:10 2010
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 23 May 2010 07:52:10 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<4BF672E8.1080001@canonical.com>	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
Message-ID: <4BF916EA.2090507@holdenweb.com>

Brian Quinlan wrote:
> 
> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:
> 
[...]
>> One minor suggestion on the "internal future methods" bit - something
>> I wish we'd done with Deferreds was to put 'callback()' and
>> 'addCallbacks()' on separate objects, so that it was very explicit
>> whether you were on the emitting side of a Deferred or the consuming
>> side.  That seems to be the case with these internal methods - they
>> are not so much "internal" as they are for the producer of the Future
>> (whether a unit test or executor) so you might want to put them on a
>> different object that it's easy for the thing creating a Future() to
>> get at but hard for any subsequent application code to fiddle with by
>> accident.  Off the top of my head, I suggest naming it "Invoker()".  A
>> good way to do this would be to have an Invoker class which can't be
>> instantiated (raises an exception from __init__ or somesuch), then a
>> Future.create() method which returns an Invoker, which itself has a
>> '.future' attribute.
>>
>> Finally, why isn't this just a module on PyPI?  It doesn't seem like
>> there's any particular benefit to making this a stdlib module and
>> going through the whole PEP process - except maybe to prompt feedback
>> like this :).
> 
> We've already had this discussion before. Could you explain why this
> module should *not* be in the stdlib e.g. does it have significantly
> less utility than other modules in stdlib? Is it significantly higher
> risk? etc?
> 
Given that its author was ready to go for pronouncement and is still
responding to pretty serious philosophical questions about the API I'd
say that it was at least worth talking about. The thing that's needed
(isn't it?) of stdlib modules is API stability.

> 
>> Issues like the ones I'm bringing up could be fixed pretty
>> straightforwardly if it were just a matter of filing a bug on a small
>> package, but fixing a stdlib module is a major undertaking.
> 
> True but I don't think that is a convincing argument. A subset of the
> functionality provided by this module is already available in Java and
> C++ and (at least in Java) it is used extensively and without too much
> trouble. If there are implementation bugs then we can fix them just like
> we would with any other module.
> 
I don't see the availability of this functionality in those languages as
any kind of reason why this needs to go into the stdlib now. Is there
some desperate rush to get it in? If it were used extensively from PyPi
*that* would be a recommendation ...

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See Python Video!       http://python.mirocommunity.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/
"All I want for my birthday is another birthday" -
                                     Ian Dury, 1942-2000


From regebro at gmail.com  Sun May 23 13:56:05 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Sun, 23 May 2010 13:56:05 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com> 
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com> 
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com> 
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com> 
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
Message-ID: <AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>

On Sun, May 23, 2010 at 13:29, Brian Quinlan <brian at sweetapp.com> wrote:
> Parts of it, yes. Just like I can replace most operations in os.path and
> urlparse with a few lines of code.

Yeah, but "parts of" is not the question. I've read the PEP, and I do
*not* know how to implement it. That means it's not a trivial module,
so that argument doesn't hold up here, even if we accept it as valid
(which I actually don't). I don't think any module in the stdlib is
entirely trivial. Yes, even parsing an URL is non-trivial, as shown by
the fact that the urlparse module apparently has a bug in it for urls
like svn+ssh://foo.bar/frotz. ;-)

Also, even trivial modules can be useful if you use them a lot.

> How would you define "very short"?

That's not up to me to decide.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From jnoller at gmail.com  Sun May 23 14:34:22 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Sun, 23 May 2010 08:34:22 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BF916EA.2090507@holdenweb.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4BF916EA.2090507@holdenweb.com>
Message-ID: <AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>

On Sun, May 23, 2010 at 7:52 AM, Steve Holden <steve at holdenweb.com> wrote:
...snip...

>>
>>> Issues like the ones I'm bringing up could be fixed pretty
>>> straightforwardly if it were just a matter of filing a bug on a small
>>> package, but fixing a stdlib module is a major undertaking.
>>
>> True but I don't think that is a convincing argument. A subset of the
>> functionality provided by this module is already available in Java and
>> C++ and (at least in Java) it is used extensively and without too much
>> trouble. If there are implementation bugs then we can fix them just like
>> we would with any other module.
>>
> I don't see the availability of this functionality in those languages as
> any kind of reason why this needs to go into the stdlib now. Is there
> some desperate rush to get it in? If it were used extensively from PyPi
> *that* would be a recommendation ...
>

Not picking Steve's particular comments out - but Brian cites the
previous discussions in the PEP itself:

http://www.python.org/dev/peps/pep-3148/

All of you questioning "Why should this be in the standard library"
should go read those old threads, where that question was answered
numerous times. Now I suddenly regret leaving the floodgates open, as
we're rapidly rehashing discussions from months ago.

For this same mailing list only a few months ago (brian, I think this
link should be added to the PEP, I didn't see it):
http://mail.python.org/pipermail/python-dev/2010-March/098169.html

Specifically:
http://mail.python.org/pipermail/python-dev/2010-March/098173.html

Quote:

"Baloney. A young library providing some syntactic sugar which uses
primitives in the standard library to implement a common pattern is
fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig
list prior to bringing it here. By the same argument, we should shunt
all of the recent unittest changes and improvements into space, since
golly, other people did it, why should we.

This is something relatively simple, which I would gladly add in an
instant to the multiprocessing package - but Brian's one-upped me in
that regard and is providing something which works with both threads
and processes handily. Take a look at multiprocessing.Pool for example
- all that is some sugar on top of the primitives, but it's good
sugar, and is used by a fair number of people.

Let me also state - "my" vision of where futures would live would be
in a concurrent package - for example:

from concurrent import futures

The reason *why* is that I would like to also move the abstractions I
have in multiprocessing *out* of that module, make them work with both
threads and processes (if it makes sense) and reduce the
multiprocessing module to the base primitive Process object. A
concurrent package which implements common patterns built on top of
the primitives we support is an objectively Good Thing.

For example, how many of us have sat down and implemented a thread
pool on top of threading, I would hazard to say that most of us who
use threading have done this, and probably more than once. It stands
to reason that this is a common enough pattern to include in the
standard library.
"

Brian has already agreed to name spacing it to "concurrent.futures" -
this means it will be a small part to a much larger concurrent.*
implementation ala Java.

So, in short - given we've already hashed the reasoning out.

jesse

From solipsis at pitrou.net  Sun May 23 14:47:54 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 23 May 2010 14:47:54 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4BF916EA.2090507@holdenweb.com>
	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
Message-ID: <20100523144754.1c6f1afc@pitrou.net>

On Sun, 23 May 2010 08:34:22 -0400
Jesse Noller <jnoller at gmail.com> wrote:
> 
> Brian has already agreed to name spacing it to "concurrent.futures" -
> this means it will be a small part to a much larger concurrent.*
> implementation ala Java.

What I would question here is what other things will be part
of the "concurrent" package, and who will implement them. Are there
plans for that? (or even tracker issues open?)

Apart from that, it seems to me that the only serious issues
blocking PEP approval are Glyph's interesting remarks.

Regards

Antoine.



From steve at holdenweb.com  Sun May 23 15:11:00 2010
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 23 May 2010 09:11:00 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	
	<20100522135946.8F5451F960D@kimball.webabinitio.net>	
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>	
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	
	<4BF916EA.2090507@holdenweb.com>
	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
Message-ID: <4BF92964.1040101@holdenweb.com>

Jesse Noller wrote:
> On Sun, May 23, 2010 at 7:52 AM, Steve Holden <steve at holdenweb.com> wrote:
> ...snip...
> 
>>>> Issues like the ones I'm bringing up could be fixed pretty
>>>> straightforwardly if it were just a matter of filing a bug on a small
>>>> package, but fixing a stdlib module is a major undertaking.
>>> True but I don't think that is a convincing argument. A subset of the
>>> functionality provided by this module is already available in Java and
>>> C++ and (at least in Java) it is used extensively and without too much
>>> trouble. If there are implementation bugs then we can fix them just like
>>> we would with any other module.
>>>
>> I don't see the availability of this functionality in those languages as
>> any kind of reason why this needs to go into the stdlib now. Is there
>> some desperate rush to get it in? If it were used extensively from PyPi
>> *that* would be a recommendation ...
>>
> 
> Not picking Steve's particular comments out - but Brian cites the
> previous discussions in the PEP itself:
> 
> http://www.python.org/dev/peps/pep-3148/
> 
> All of you questioning "Why should this be in the standard library"
> should go read those old threads, where that question was answered
> numerous times. Now I suddenly regret leaving the floodgates open, as
> we're rapidly rehashing discussions from months ago.
> 
Yes, it might have been better to call for participation from those who
had contributed to the original discussion, and therefore knew what they
were talking about. No flood from me, though, all my questions have been
answered.

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See Python Video!       http://python.mirocommunity.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/
"All I want for my birthday is another birthday" -
                                     Ian Dury, 1942-2000

From tjreedy at udel.edu  Sun May 23 21:01:59 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 23 May 2010 15:01:59 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTilcTJj92lmNQJYtzd7yiaolHw1xSevE8g64aTz_@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<2EF114F8-585B-4204-A7ED-1FE7F8226D2D@sweetapp.com>
	<AANLkTilcTJj92lmNQJYtzd7yiaolHw1xSevE8g64aTz_@mail.gmail.com>
Message-ID: <htbu38$q3q$1@dough.gmane.org>

On 5/22/2010 8:06 PM, Jeffrey Yasskin wrote:
> On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan<brian at sweetapp.com>  wrote:
>> Rename "executor" =>  "executer"
>
> -1 for consistency with Java.

-10 for 'executer'. As far as I can find out, it is a misspelling of 
'executor'. If the designers of some other language made a stupid 
mistake, let them correct it instead of us following them over a cliff.

Unlike this one, the other name suggestions look at least plausible and 
worth a couple of minutes of consideration.

As for the module itself, part of the justification to me for accepting 
it would be if it is part of a larger plan, even if currently vague, to 
refactor and improve Python's support for concurrent execution, as 
implied by the name 'concurrent.futures'. If Jesse accepts it, I would 
take it as some kind of commitment to help with at least one other 
concurrent.x module so this one is not an orphan. While concurrent 
execution does not *currently* affect me, I am convinced that better 
support will be important for Python's future.

Terry Jan Reedy


From glyph at twistedmatrix.com  Sun May 23 21:16:35 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Sun, 23 May 2010 15:16:35 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
Message-ID: <4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com>


On May 23, 2010, at 2:37 AM, Brian Quinlan wrote:

> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:
> 
>> 
>> On May 22, 2010, at 8:47 PM, Brian Quinlan wrote:
>> 
>>> Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days.
>>> 
>>> So fire away!
>> 
>> As you wish!
> 
> I retract my request ;-)

May you get what you wish for, may you find what you are seeking :).

>> The PEP should be consistent in its usage of terminology about callables.  It alternately calls them "callables", "functions", and "functions or methods".  It would be nice to clean this up and be consistent about what can be called where.  I personally like "callables".
> 
> Did you find the terminology confusing? If not then I propose not changing it.

Yes, actually.  Whenever I see references to the multiprocessing module, I picture a giant "HERE BE (serialization) DRAGONS" sign.  When I saw that some things were documented as being "functions", I thought that maybe there was intended to be a restriction like the "these can only be top-level functions so they're easy for different executors to locate and serialize".  I didn't realize that the intent was "arbitrary callables" until I carefully re-read the document and noticed that the terminology was inconsistent.

> But changing it in the user docs is probably a good idea. I like "callables" too.

Great.  Still, users will inevitably find the PEP and use it as documentation too.

>> The execution context of callable code is not made clear.  Implicitly, submit() or map() would run the code in threads or processes as defined by the executor, but that's not spelled out clearly.

Any response to this bit?  Did I miss something in the PEP?

>> More relevant to my own interests, the execution context of the callables passed to add_done_callback and remove_done_callback is left almost completely to the imagination.  If I'm reading the sample implementation correctly, <http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/process.py#241>, it looks like in the multiprocessing implementation, the done callbacks are invoked in a random local thread.  The fact that they are passed the future itself *sort* of implies that this is the case, but the multiprocessing module plays fast and loose with object identity all over the place, so it would be good to be explicit and say that it's *not* a pickled copy of the future sitting in some arbitrary process (or even on some arbitrary machine).
> 
> The callbacks will always be called in a thread other than the main thread in the process that created the executor. Is that a strong enough contract?

Sure.  Really, almost any contract would work, it just needs to be spelled out.  It might be nice to know whether the thread invoking the callbacks is a daemon thread or not, but I suppose it's not strictly necessary.

>> This is really minor, I know, but why does it say "NOTE: This method can be used to create adapters from Futures to Twisted Deferreds"?  First of all, what's the deal with "NOTE"; it's the only "NOTE" in the whole PEP, and it doesn't seem to add anything.  This sentence would read exactly the same if that word were deleted.  Without more clarity on the required execution context of the callbacks, this claim might not actually be true anyway; Deferred callbacks can only be invoked in the main reactor thread in Twisted.  But even if it is perfectly possible, why leave so much of the adapter implementation up to the imagination?  If it's important enough to mention, why not have a reference to such an adapter in the reference Futures implementation, since it *should* be fairly trivial to write?
> 
> I'm a bit surprised that this doesn't allow for better interoperability with Deferreds given this discussion:

> <discussion snipped>

I did not communicate that well.  As implemented, it's quite possible to implement a translation layer which turns a Future into a Deferred.  What I meant by that comment was, the specification in the PEP was to loose to be sure that such a layer would work with arbitrary executors.

For what it's worth, the Deferred translator would look like this, if you want to include it in the PEP (untested though, you may want to run it first):

    from twisted.internet.defer import Deferred
    from twisted.internet.reactor import callFromThread

    def future2deferred(future):
        d = Deferred()
        def invoke_deferred():
            try:
                result = future.result()
            except:
                d.errback()
            else:
                d.callback(result)
        def done_callback(same_future):
            callFromThread(invoke_deferred)
        future.add_done_callback(done_callback)
        return d

This does beg the question of what the traceback will look like in that except: block though.  I guess the multi-threaded executor will use python3 exception chaining so Deferred should be able to show a sane traceback in case of an error, but what about exceptions in other processes?

>> I suggest having have add_done_callback, implementing it with a list so that callbacks are always invoked in the order that they're added, and getting rid of remove_done_callback.
> 
> Sounds good to me!

Great! :-)

>> futures._base.Executor isn't exposed publicly, but it needs to be.  The PEP kinda makes it sound like it is ("Executor is an abstract class...").  Plus, A third party library wanting to implement an executor of its own shouldn't have to copy and paste the implementation of Executor.map.
> 
> That was a bug that I've fixed. Thanks!

Double-great!

>> One minor suggestion on the "internal future methods" bit - something I wish we'd done with Deferreds was to put 'callback()' and 'addCallbacks()' on separate objects, so that it was very explicit whether you were on the emitting side of a Deferred or the consuming side.  That seems to be the case with these internal methods - they are not so much "internal" as they are for the producer of the Future (whether a unit test or executor) so you might want to put them on a different object that it's easy for the thing creating a Future() to get at but hard for any subsequent application code to fiddle with by accident.  Off the top of my head, I suggest naming it "Invoker()".  A good way to do this would be to have an Invoker class which can't be instantiated (raises an exception from __init__ or somesuch), then a Future.create() method which returns an Invoker, which itself has a '.future' attribute.

No reaction on this part?  I think you'll wish you did this in a couple of years when you start bumping into application code that calls "set_result" :).

>> Finally, why isn't this just a module on PyPI?  It doesn't seem like there's any particular benefit to making this a stdlib module and going through the whole PEP process - except maybe to prompt feedback like this :).
> 
> We've already had this discussion before. Could you explain why this module should *not* be in the stdlib e.g. does it have significantly less utility than other modules in stdlib? Is it significantly higher risk? etc?

You've convinced me, mainly because I noticed later on in the discussion that it *has* been released to pypi for several months, and does have a bunch of downloads.  It doesn't have quite the popularity I'd personally like to see for stdlib modules, but it's not like you didn't try, and you do (sort of) have a point about small modules being hard to get adoption.  I'm sorry that this, my least interesting point in my opinion, is what has seen the most discussion so far.

I'd appreciate it if you could do a release to pypi with the bugfixes you mentioned here, to make sure that the released version is consistent with what eventually gets into Python.

Oh, and one final nitpick: <http://www.rfc-editor.org/rfc/rfc2606.txt> says you really should not put real domain names into your "web crawl example", especially not "some-made-up-domain.com".

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100523/cfd9d0ba/attachment.html>

From greg.ewing at canterbury.ac.nz  Mon May 24 01:33:47 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 24 May 2010 11:33:47 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
Message-ID: <4BF9BB5B.50809@canterbury.ac.nz>

Glyph Lefkowitz wrote:

> Finally, why isn't this just a module on PyPI?  It doesn't seem like 
> there's any particular benefit to making this a stdlib module and going 
> through the whole PEP process

I'm inclined to agree. This needs to be field-tested before
being considered for stdlib inclusion.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon May 24 01:51:26 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 24 May 2010 11:51:26 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
Message-ID: <4BF9BF7E.5030300@canterbury.ac.nz>

Brian Quinlan wrote:

> The good news in this case is that the same API has been used 
> successfully in Java and C++ for years so it is unlikely that any major 
> changes will need to be made.

That doesn't follow. An API that's appropriate for Java or
C++ is not necessarily appropriate for Python. Slavishly
copying an API from another language is often not the best
approach when designing an API for a Python module.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon May 24 02:00:05 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 24 May 2010 12:00:05 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTimng5BbElsJmTCNvL3nAozAAjvMJkQXBmhxe9Td@mail.gmail.com>
	<7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com>
Message-ID: <4BF9C185.5010200@canterbury.ac.nz>

Brian Quinlan wrote:

> Simple modules are unlikely to develop a following because it is  too 
> easy to partially replicate their functionality.

I don't think it needs a particularly large following.
What it does need is at least a few people using it in
some real projects.

No matter how much discussion there is and how much
apparent agreement is reached, it's no substitute for
practical experience. Often API design mistakes are only
found when trying to use the library for real.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon May 24 02:06:33 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 24 May 2010 12:06:33 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTik-ewslhUa6bd0pwXyAq_gUiOuXpwgjJ3GkNAV3@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTimng5BbElsJmTCNvL3nAozAAjvMJkQXBmhxe9Td@mail.gmail.com>
	<7251439E-F687-490C-A7A5-29018ECC3861@sweetapp.com>
	<AANLkTik-ewslhUa6bd0pwXyAq_gUiOuXpwgjJ3GkNAV3@mail.gmail.com>
Message-ID: <4BF9C309.1020300@canterbury.ac.nz>

Andrew Svetlov wrote:

> BTW, is 'cancelled' correct name? Spell-checkers likes only single 'l'
> form: 'canceled'.

I think this is an English vs. American thing. Double 'l'
looks right to me, but then I was brought up as a loyal
subject of the antipodean branch of the British Empire. :-)

-- 
Greg

From stephen at xemacs.org  Mon May 24 03:47:34 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 24 May 2010 10:47:34 +0900
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <FE24980F-8458-4427-9082-9A0EB58663D3@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTinBPbj-8I6ZgK6s9gFSEBz3hQLR7plO4ApNAWy6@mail.gmail.com>
	<FE24980F-8458-4427-9082-9A0EB58663D3@sweetapp.com>
Message-ID: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp>

Brian Quinlan writes:

 > If you are familiar with threads then writing a "good enough" solution  
 > without futures probably won't take you very long. Also, unless you  
 > are familiar with another futures implementation, you aren't likely to  
 > know where to look.

That looks like an argument *against* your module, to me.  Why would
people look for it in the stdlib if they're not looking for it at all,
and specifically because anybody who would know enough to look for
"something like" it is also able to devise a good-enough solution?
You're describing a solution in search of a user, not a user in search
of a solution, and it would appear to violate "not every three-line
function" as well as TOOWTDI.

I personally plan to defer to the people who know and use such
constructs (specifically Glyph and Jesse), and who seem to be in favor
(at least +0) of stabilizing an API for this in the stdlib.  But you
may want to rethink your sales pitch if you want to avoid giving ammo
to the opposition.  It sounds like you agree with them, except on the
vote you cast.<wink>


From cs at zip.com.au  Mon May 24 04:58:05 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Mon, 24 May 2010 12:58:05 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20100524025805.GA29197@cskk.homeip.net>

On 24May2010 10:47, Stephen J. Turnbull <stephen at xemacs.org> wrote:
| Brian Quinlan writes:
|  > If you are familiar with threads then writing a "good enough" solution  
|  > without futures probably won't take you very long. Also, unless you  
|  > are familiar with another futures implementation, you aren't likely to  
|  > know where to look.
| 
| That looks like an argument *against* your module, to me.  Why would
| people look for it in the stdlib if they're not looking for it at all,
| and specifically because anybody who would know enough to look for
| "something like" it is also able to devise a good-enough solution?
| You're describing a solution in search of a user, not a user in search
| of a solution, and it would appear to violate "not every three-line
| function" as well as TOOWTDI.

This might be a terminology problem. I think, above, Brian means "good
enough" to mean "looks ok at first cut but doesn't handle the corner
cases". Which usually means obscure breakage later.

I almost am Brian's hypothetical user. I've got a "FuncMultiQueue" that
accepts callables in synchronous and asynchronous modes for future
possibly-concurrent execution, just as the futures module does. I've
spent a _lot_ of time debugging it.

There's a lot to be said for a robust implementation of a well defined
problem. Brian's module, had it been present and presuming it robust and
debugged, would have been quite welcome.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

I am a Bear of Very Little Brain and long words Bother Me.
        - Winnie-the-Pooh

From steven.bethard at gmail.com  Mon May 24 05:52:19 2010
From: steven.bethard at gmail.com (Steven Bethard)
Date: Sun, 23 May 2010 20:52:19 -0700
Subject: [Python-Dev] moving issues from argparse tracker to python tracker?
Message-ID: <AANLkTimg9c2ZrY3XAjkt6qJC9lAJWmdfk58VWeO986eV@mail.gmail.com>

Before I go and add about 30 open issues to the Python tracker, I
figured I should ask. What's the normal process for the bug trackers
of modules that move to the standard library? I have a few feature
requests, etc. for argparse, and I was planning to just copy them over
to the Python bug tracker (and close them on the Google code tracker).
Is this what people normally do? (It should be easy enough to do - I
just don't want to mess up the tracker if this is usually done some
other way.)

Thanks,

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

From orsenthil at gmail.com  Mon May 24 06:08:33 2010
From: orsenthil at gmail.com (Senthil Kumaran)
Date: Mon, 24 May 2010 09:38:33 +0530
Subject: [Python-Dev] moving issues from argparse tracker to python
 tracker?
In-Reply-To: <AANLkTimg9c2ZrY3XAjkt6qJC9lAJWmdfk58VWeO986eV@mail.gmail.com>
References: <AANLkTimg9c2ZrY3XAjkt6qJC9lAJWmdfk58VWeO986eV@mail.gmail.com>
Message-ID: <20100524040832.GA5722@remy>

On Sun, May 23, 2010 at 08:52:19PM -0700, Steven Bethard wrote:
> requests, etc. for argparse, and I was planning to just copy them over
> to the Python bug tracker (and close them on the Google code tracker).

I think, this is a good idea. +1 from me. There is only one module,
bsddb, which is maintained outside due to release constraints and
other issues, but other than anything part of stdlib, better be
tracked through python tracker.

An additional advantage of moving to python tracker is, people who are
subscribed to python-bugs will be notified and can pitch in their
ideas/efforts too.

-- 
Senthil

OK, enough hype.
		-- Larry Wall in the perl man page

From p.f.moore at gmail.com  Mon May 24 09:17:21 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 24 May 2010 08:17:21 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100524025805.GA29197@cskk.homeip.net>
References: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20100524025805.GA29197@cskk.homeip.net>
Message-ID: <AANLkTilHGVbyl1iUvIfI799vnvw2ZrLgGV9RuujkNdJ3@mail.gmail.com>

On 24 May 2010 03:58, Cameron Simpson <cs at zip.com.au> wrote:
> I almost am Brian's hypothetical user. I've got a "FuncMultiQueue" that
> accepts callables in synchronous and asynchronous modes for future
> possibly-concurrent execution, just as the futures module does. I've
> spent a _lot_ of time debugging it.

I pretty much am that user as well (whether or not I am hypothetical,
I'll leave to others to determine...)

I have a set of scripts that needed to do precisely the sort of thing
that the futures module offers. I searched for a fair while for a
suitable offering (this was before futures had been published) and
found nothing suitable. So in the end I implemented my own - and I hit
corner cases, and they needed a lot of work to fix. I now have a
working solution, but it's too tangled in the application logic to be
reusable :-(

If futures had been in the stdlib, I'd have used it like a shot, and
saved myself a lot of wasted time.

> There's a lot to be said for a robust implementation of a well defined
> problem. Brian's module, had it been present and presuming it robust and
> debugged, would have been quite welcome.

Precisely my view.

Paul.

From g.brandl at gmx.net  Mon May 24 09:24:32 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 24 May 2010 09:24:32 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BF9BF7E.5030300@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<4BF9BF7E.5030300@canterbury.ac.nz>
Message-ID: <htd9l5$5oo$1@dough.gmane.org>

Am 24.05.2010 01:51, schrieb Greg Ewing:
> Brian Quinlan wrote:
> 
>> The good news in this case is that the same API has been used 
>> successfully in Java and C++ for years so it is unlikely that any major 
>> changes will need to be made.
> 
> That doesn't follow. An API that's appropriate for Java or
> C++ is not necessarily appropriate for Python. Slavishly
> copying an API from another language is often not the best
> approach when designing an API for a Python module.

*cough* unittest *cough*

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From vinay_sajip at yahoo.co.uk  Mon May 24 10:49:56 2010
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 24 May 2010 08:49:56 +0000 (UTC)
Subject: [Python-Dev] urllib2/urllib incompatibility?
Message-ID: <loom.20100524T104324-206@post.gmane.org>

> Not top-posting, but gmane seems too finicky in this area.

I encountered what seems like an incompatibility between urllib and urllib2 in
the way they handle file:// URLs, is this a bug? I had a look on the bug tracker
and via Google to see if there were prior reports, but perhaps my search-fu is
deficient. Here's a console session to illustrate:

vinay at eta-karmic:/tmp$ echo Hello, world! >hello.txt
vinay at eta-karmic:/tmp$ cat hello.txt 
Hello, world!
vinay at eta-karmic:/tmp$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib,urllib2
>>> s = 'file:////tmp/hello.txt'
>>> f1 = urllib.urlopen(s)
>>> f1.read()
'Hello, world!\n'
>>> f2 = urllib2.urlopen(s)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 389, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 407, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1240, in file_open
    return self.parent.open(req)
  File "/usr/lib/python2.6/urllib2.py", line 389, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 407, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1287, in ftp_open
    raise URLError('ftp error: no host given')
urllib2.URLError: <urlopen error ftp error: no host given>
>>> 

Anyone seen this before? Should I file an issue on the tracker? If I've missed
something obvious, sorry for the noise, but please do tell!

Regards,

Vinay Sajip


From list at qtrac.plus.com  Mon May 24 11:30:59 2010
From: list at qtrac.plus.com (Mark Summerfield)
Date: Mon, 24 May 2010 10:30:59 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <htbu38$q3q$1@dough.gmane.org>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTilcTJj92lmNQJYtzd7yiaolHw1xSevE8g64aTz_@mail.gmail.com>
	<htbu38$q3q$1@dough.gmane.org>
Message-ID: <201005241030.59713.list@qtrac.plus.com>

On 2010-05-23, Terry Reedy wrote:
> On 5/22/2010 8:06 PM, Jeffrey Yasskin wrote:
> > On Sat, May 22, 2010 at 4:12 PM, Brian Quinlan<brian at sweetapp.com>  wrote:
> >> Rename "executor" =>  "executer"
> >
> > -1 for consistency with Java.
> 
> -10 for 'executer'. As far as I can find out, it is a misspelling of
> 'executor'. If the designers of some other language made a stupid
> mistake, let them correct it instead of us following them over a cliff.

I'd suggested this because it seemed obvious to me, but clearly not.
Compare:
http://www.thefreedictionary.com/executor
http://www.thefreedictionary.com/executer

However, as I mentioned in the first place I didn't expect any change of
this since Java uses the first spelling.

[snip]

-- 
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
    C++, Python, Qt, PyQt - training and consultancy
        "Advanced Qt Programming" - ISBN 0321635906

From brian at sweetapp.com  Mon May 24 11:36:50 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Mon, 24 May 2010 19:36:50 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com>
Message-ID: <64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com>


On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote:

>
> On May 23, 2010, at 2:37 AM, Brian Quinlan wrote:
>
>> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:
>>
>>>
>>> On May 22, 2010, at 8:47 PM, Brian Quinlan wrote:
>>>
>>>> Jesse, the designated pronouncer for this PEP, has decided to  
>>>> keep discussion open for a few more days.
>>>>
>>>> So fire away!
>>>
>>> As you wish!
>>
>> I retract my request ;-)
>
> May you get what you wish for, may you find what you are seeking :).
>
>>> The PEP should be consistent in its usage of terminology about  
>>> callables.  It alternately calls them "callables", "functions",  
>>> and "functions or methods".  It would be nice to clean this up and  
>>> be consistent about what can be called where.  I personally like  
>>> "callables".
>>
>> Did you find the terminology confusing? If not then I propose not  
>> changing it.
>
> Yes, actually.  Whenever I see references to the multiprocessing  
> module, I picture a giant "HERE BE (serialization) DRAGONS" sign.   
> When I saw that some things were documented as being "functions", I  
> thought that maybe there was intended to be a restriction like the  
> "these can only be top-level functions so they're easy for different  
> executors to locate and serialize".  I didn't realize that the  
> intent was "arbitrary callables" until I carefully re-read the  
> document and noticed that the terminology was inconsistent.

ProcessPoolExecutor has the same serialization perils that  
multiprocessing does. My original plan was to link to the  
multiprocessing docs to explain them but I couldn't find them listed.

>> But changing it in the user docs is probably a good idea. I like  
>> "callables" too.
>
> Great.  Still, users will inevitably find the PEP and use it as  
> documentation too.
>
>>> The execution context of callable code is not made clear.   
>>> Implicitly, submit() or map() would run the code in threads or  
>>> processes as defined by the executor, but that's not spelled out  
>>> clearly.
>
> Any response to this bit?  Did I miss something in the PEP?

Yes, the execution context is Executor-dependent. The section under  
ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think.

>>> More relevant to my own interests, the execution context of the  
>>> callables passed to add_done_callback and remove_done_callback is  
>>> left almost completely to the imagination.  If I'm reading the  
>>> sample implementation correctly, <http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/process.py#241 
>>> >, it looks like in the multiprocessing implementation, the done  
>>> callbacks are invoked in a random local thread.  The fact that  
>>> they are passed the future itself *sort* of implies that this is  
>>> the case, but the multiprocessing module plays fast and loose with  
>>> object identity all over the place, so it would be good to be  
>>> explicit and say that it's *not* a pickled copy of the future  
>>> sitting in some arbitrary process (or even on some arbitrary  
>>> machine).
>>
>> The callbacks will always be called in a thread other than the main  
>> thread in the process that created the executor. Is that a strong  
>> enough contract?
>
> Sure.  Really, almost any contract would work, it just needs to be  
> spelled out.  It might be nice to know whether the thread invoking  
> the callbacks is a daemon thread or not, but I suppose it's not  
> strictly necessary.

Your concerns is that the thread will be killed when the interpreter  
exits? It won't be.

>>> This is really minor, I know, but why does it say "NOTE: This  
>>> method can be used to create adapters from Futures to Twisted  
>>> Deferreds"?  First of all, what's the deal with "NOTE"; it's the  
>>> only "NOTE" in the whole PEP, and it doesn't seem to add  
>>> anything.  This sentence would read exactly the same if that word  
>>> were deleted.  Without more clarity on the required execution  
>>> context of the callbacks, this claim might not actually be true  
>>> anyway; Deferred callbacks can only be invoked in the main reactor  
>>> thread in Twisted.  But even if it is perfectly possible, why  
>>> leave so much of the adapter implementation up to the  
>>> imagination?  If it's important enough to mention, why not have a  
>>> reference to such an adapter in the reference Futures  
>>> implementation, since it *should* be fairly trivial to write?
>>
>> I'm a bit surprised that this doesn't allow for better  
>> interoperability with Deferreds given this discussion:
>
>> <discussion snipped>
>
> I did not communicate that well.  As implemented, it's quite  
> possible to implement a translation layer which turns a Future into  
> a Deferred.  What I meant by that comment was, the specification in  
> the PEP was to loose to be sure that such a layer would work with  
> arbitrary executors.
>
> For what it's worth, the Deferred translator would look like this,  
> if you want to include it in the PEP (untested though, you may want  
> to run it first):
>
>     from twisted.internet.defer import Deferred
>     from twisted.internet.reactor import callFromThread
>
>     def future2deferred(future):
>         d = Deferred()
>         def invoke_deferred():
>             try:
>                 result = future.result()
>             except:
>                 d.errback()
>             else:
>                 d.callback(result)
>         def done_callback(same_future):
>             callFromThread(invoke_deferred)
>         future.add_done_callback(done_callback)
>         return d
>
> This does beg the question of what the traceback will look like in  
> that except: block though.  I guess the multi-threaded executor will  
> use python3 exception chaining so Deferred should be able to show a  
> sane traceback in case of an error, but what about exceptions in  
> other processes?
>
>>> I suggest having have add_done_callback, implementing it with a  
>>> list so that callbacks are always invoked in the order that  
>>> they're added, and getting rid of remove_done_callback.
>>
>> Sounds good to me!
>
> Great! :-)
>
>>> futures._base.Executor isn't exposed publicly, but it needs to  
>>> be.  The PEP kinda makes it sound like it is ("Executor is an  
>>> abstract class...").  Plus, A third party library wanting to  
>>> implement an executor of its own shouldn't have to copy and paste  
>>> the implementation of Executor.map.
>>
>> That was a bug that I've fixed. Thanks!
>
> Double-great!
>
>>> One minor suggestion on the "internal future methods" bit -  
>>> something I wish we'd done with Deferreds was to put 'callback()'  
>>> and 'addCallbacks()' on separate objects, so that it was very  
>>> explicit whether you were on the emitting side of a Deferred or  
>>> the consuming side.  That seems to be the case with these internal  
>>> methods - they are not so much "internal" as they are for the  
>>> producer of the Future (whether a unit test or executor) so you  
>>> might want to put them on a different object that it's easy for  
>>> the thing creating a Future() to get at but hard for any  
>>> subsequent application code to fiddle with by accident.  Off the  
>>> top of my head, I suggest naming it "Invoker()".  A good way to do  
>>> this would be to have an Invoker class which can't be instantiated  
>>> (raises an exception from __init__ or somesuch), then a  
>>> Future.create() method which returns an Invoker, which itself has  
>>> a '.future' attribute.
>
> No reaction on this part?  I think you'll wish you did this in a  
> couple of years when you start bumping into application code that  
> calls "set_result" :).

My reactions are mixed ;-)

Your proposal is to add a level of indirection to make it harder for  
people to call implementation methods. The downside is that it makes  
it a bit harder to write tests and Executors. I also can't see a big  
problem in letting people call set_result in client code though it is  
documented as being only for Executor implementations and tests.

On the implementation side, I don't see why an Invoker needs a  
reference to the future. Each Invoker could own one Future. A  
reference to the Invoker is kept by the Executor and its future is  
returned to the client i.e.

class Invoker(object):
   def __init__(self):
     """Should only be called by Executor implementations."""
     self.future = Future()

   def set_running_or_notify_cancel(self):
     # Messes with self.future's internals

   def set_result(self):
     # Messes with self.future's internals

   def set_exception(self):
     # Messes with self.future's internals


Cheers,
Brian


>>> Finally, why isn't this just a module on PyPI?  It doesn't seem  
>>> like there's any particular benefit to making this a stdlib module  
>>> and going through the whole PEP process - except maybe to prompt  
>>> feedback like this :).
>>
>> We've already had this discussion before. Could you explain why  
>> this module should *not* be in the stdlib e.g. does it have  
>> significantly less utility than other modules in stdlib? Is it  
>> significantly higher risk? etc?
>
> You've convinced me, mainly because I noticed later on in the  
> discussion that it *has* been released to pypi for several months,  
> and does have a bunch of downloads.  It doesn't have quite the  
> popularity I'd personally like to see for stdlib modules, but it's  
> not like you didn't try, and you do (sort of) have a point about  
> small modules being hard to get adoption.  I'm sorry that this, my  
> least interesting point in my opinion, is what has seen the most  
> discussion so far.
>
> I'd appreciate it if you could do a release to pypi with the  
> bugfixes you mentioned here, to make sure that the released version  
> is consistent with what eventually gets into Python.
>
> Oh, and one final nitpick: <http://www.rfc-editor.org/rfc/ 
> rfc2606.txt> says you really should not put real domain names into  
> your "web crawl example", especially not "some-made-up-domain.com".
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100524/ec371d18/attachment-0001.html>

From orsenthil at gmail.com  Mon May 24 11:37:16 2010
From: orsenthil at gmail.com (Senthil Kumaran)
Date: Mon, 24 May 2010 15:07:16 +0530
Subject: [Python-Dev] urllib2/urllib incompatibility?
In-Reply-To: <loom.20100524T104324-206@post.gmane.org>
References: <loom.20100524T104324-206@post.gmane.org>
Message-ID: <20100524093716.GA5284@remy>

On Mon, May 24, 2010 at 08:49:56AM +0000, Vinay Sajip wrote:
> I encountered what seems like an incompatibility between urllib and urllib2 in
> the way they handle file:// URLs, is this a bug? I had a look on the bug tracker

> >>> s = 'file:////tmp/hello.txt'
> >>> f1 = urllib.urlopen(s)

The actual (and Valid) file:// url in your case is 'file:///tmp/hello.txt'
And this was fine and consistent.

>>> s = 'file:///tmp/hello.txt'
>>> import urllib2
>>> import urllib
>>> o1 = urllib.urlopen(s)
>>> o2 = urllib2.urlopen(s)


The extra '/' is making it in invalid url in urllib2, I think that be
tracked as bug at least to show a consistent behaviour.  The local
file open methods of urllib2 and urllib are different.

-- 
Senthil

You may my glories and my state dispose,
But not my griefs; still am I king of those.
		-- William Shakespeare, "Richard II"

From vinay_sajip at yahoo.co.uk  Mon May 24 12:00:07 2010
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 24 May 2010 10:00:07 +0000 (UTC)
Subject: [Python-Dev] urllib2/urllib incompatibility?
References: <loom.20100524T104324-206@post.gmane.org>
	<20100524093716.GA5284@remy>
Message-ID: <loom.20100524T115720-267@post.gmane.org>

Senthil Kumaran <orsenthil <at> gmail.com> writes:

> 
> The actual (and Valid) file:// url in your case is 'file:///tmp/hello.txt'
> And this was fine and consistent.
> 
> The extra '/' is making it in invalid url in urllib2, I think that be
> tracked as bug at least to show a consistent behaviour.  The local
> file open methods of urllib2 and urllib are different.
> 

Yes, it seems to be a bug in urllib which permits an invalid URL to pass. I came
across it by accident, I normally use urls like file://localhost/tmp/hello.txt
so I don't trip over the /// hurdles :-)

I'll raise an issue on the tracker.

Regards,

Vinay Sajip



From stephen at xemacs.org  Mon May 24 12:46:02 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 24 May 2010 19:46:02 +0900
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100524025805.GA29197@cskk.homeip.net>
References: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20100524025805.GA29197@cskk.homeip.net>
Message-ID: <871vd1mvbp.fsf@uwakimon.sk.tsukuba.ac.jp>

Cameron Simpson writes:

 > There's a lot to be said for a robust implementation of a well defined
 > problem. Brian's module, had it been present and presuming it robust and
 > debugged, would have been quite welcome.

That, of course, is the consensus view, both in general and with
respect to this particular module.

The difference is over what constitutes sufficient evidence for your
presumption of "robust and debugged" from the point of view of the
users of the stdlib.

From solipsis at pitrou.net  Mon May 24 13:01:33 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 24 May 2010 13:01:33 +0200
Subject: [Python-Dev] moving issues from argparse tracker to python
	tracker?
References: <AANLkTimg9c2ZrY3XAjkt6qJC9lAJWmdfk58VWeO986eV@mail.gmail.com>
Message-ID: <20100524130133.43438a0e@pitrou.net>

On Sun, 23 May 2010 20:52:19 -0700
Steven Bethard <steven.bethard at gmail.com> wrote:
> I have a few feature
> requests, etc. for argparse, and I was planning to just copy them over
> to the Python bug tracker (and close them on the Google code tracker).

Yes, I think this is desireable. You should also maintain argparse
directly in the Python SVN (even if you plan to do separate releases).

Regards

Antoine.



From brett at python.org  Tue May 25 04:57:56 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 24 May 2010 19:57:56 -0700
Subject: [Python-Dev] __package__ attribute
In-Reply-To: <AANLkTimIZAa_z4_BOuyt-rURhMc3oUD9XgU8igeKnIxy@mail.gmail.com>
References: <AANLkTilvwrKGM53NHMw4W52_PSNV8Ap2DBGLH4amCHww@mail.gmail.com> 
	<AANLkTimIZAa_z4_BOuyt-rURhMc3oUD9XgU8igeKnIxy@mail.gmail.com>
Message-ID: <AANLkTilon_1ETNiKKgds1ZW3g3qJP3YVkRy3UCKNPg4R@mail.gmail.com>

On Fri, May 21, 2010 at 14:35, Andrew Svetlov <andrew.svetlov at gmail.com> wrote:
> For me it's the real bug in standard python import machinery.
> I don't see any backward incompatibilities.
> There are very hard to write any import-depended code based on
> decision: was module imported in absolute or relative way.
>
> If it's a bug I can create issue in python bugtracker and provide a
> patch (for Python 3.2 and for 2.7 if later is required).

It's definitely not a bug. The definition of __package__ is such that
not bothering to set it is still valid (see PEP 366 for the exact
details of the attribute). It just happens that importlib puts in the
extra effort to add/set the attribute as much as possible. Now if you
want to write a patch for import.c to add the attribute properly then
that could get reviewed and added as a feature request, but as it
stands the current semantics are correct.

-Brett

>
> On Fri, May 21, 2010 at 10:18 PM, Andrew Svetlov
> <andrew.svetlov at gmail.com> wrote:
>> What are expected values for module.__package__?
>> I see two different behaviors: importlib and standard python import.
>> importlib processes __package__ pretty simple and clean:
>> - for toplevel modules __package__ is empty string
>> - for packages it's package name
>> - for modules inside packages it's again package name
>>
>> Standard import follows another strategy:
>> - it tries to setup __package__ only in case of relative import (see
>> first 'if' in import.c:get_parent)
>> - otherwise __package__ is untouched and None by default.
>>
>> For me importlib's way is better.
>> I don't see any PEP specifying __package__ behavior.
>>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>

From ncoghlan at gmail.com  Wed May 26 01:54:13 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 09:54:13 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100523144754.1c6f1afc@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<4BF916EA.2090507@holdenweb.com>	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
	<20100523144754.1c6f1afc@pitrou.net>
Message-ID: <4BFC6325.5050700@gmail.com>

On 23/05/10 22:47, Antoine Pitrou wrote:
> On Sun, 23 May 2010 08:34:22 -0400
> Jesse Noller<jnoller at gmail.com>  wrote:
>>
>> Brian has already agreed to name spacing it to "concurrent.futures" -
>> this means it will be a small part to a much larger concurrent.*
>> implementation ala Java.
>
> What I would question here is what other things will be part
> of the "concurrent" package, and who will implement them. Are there
> plans for that? (or even tracker issues open?)

I'm not sure it is called out explicitly in the PEP, but the specific 
example that came up in the previous discussions was something like 
"concurrent.pool" to hold a thread vs process agnostic worker pool 
interface based on the existing Pool interface in multiprocessing (with 
concrete implementations for both threading and multiprocessing).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May 26 02:10:25 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 10:10:25 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
Message-ID: <4BFC66F1.3060107@gmail.com>

On 23/05/10 21:56, Lennart Regebro wrote:
> On Sun, May 23, 2010 at 13:29, Brian Quinlan<brian at sweetapp.com>  wrote:
>> Parts of it, yes. Just like I can replace most operations in os.path and
>> urlparse with a few lines of code.
>
> Yeah, but "parts of" is not the question. I've read the PEP, and I do
> *not* know how to implement it. That means it's not a trivial module,
> so that argument doesn't hold up here, even if we accept it as valid
> (which I actually don't). I don't think any module in the stdlib is
> entirely trivial. Yes, even parsing an URL is non-trivial, as shown by
> the fact that the urlparse module apparently has a bug in it for urls
> like svn+ssh://foo.bar/frotz. ;-)

In this case, the "trivial" refers to being able to write something that 
will get the job done for a specific task or application, but that isn't 
as solid from a reliability/maintainability/portability/scalability 
point of view.

By providing solid infrastructure in the standard library, we can remove 
that choice between "do it fast" vs "do it right", by providing 
ready-made robust infrastructure.

Those that say "just put it on PyPI" may not recognise the additional 
overhead that can be involved in identifying, obtaining approval to use 
and ongoing management of additional dependencies in a corporate 
environment that is actually performing appropriate due diligence in 
regards to IP licensing. This overhead can be especially significant 
(and, depending on licence and contract details, even a dealbreaker) for 
companies with specific IP licensing provisions in their contracts with 
their clients. It doesn't matter *how* easy we make it to download PyPI 
packages, we can't do anything about such company IP management policies 
(except for making it easier for programmers to violate them 
thoughtlessly, of course).

To use myself as an example, I have several utilities that I probably 
would have written differently if the futures module had been available 
in the standard library at the time I wrote them. As it is, they work 
well enough, but their usage of the threading module is fairly ad hoc 
(and migrating them to use multiprocessing would be a fairly complex 
task, and making that decision run-time selectable even more complicated).

In the near-term, backports of future standard library modules are much 
easier to get through a corporate review process as the licensing is 
typically similar to the PSF license (or is even the PSF license itself) 
and the modules come with a clear roadmap for eliminating the dependency 
(i.e. once the baseline Python version employed by the company includes 
the module in the standard library, the external dependency is no longer 
needed).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May 26 02:12:43 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 10:12:43 +1000
Subject: [Python-Dev] Dead modules
In-Reply-To: <AANLkTinoQwp0Yg7jT_meIJTXYcS5M6oMwv1E8L3OetoZ@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>	
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	
	<AANLkTinBPbj-8I6ZgK6s9gFSEBz3hQLR7plO4ApNAWy6@mail.gmail.com>	
	<20100523125155.53959f2f@pitrou.net>	
	<AANLkTinoQwp0Yg7jT_meIJTXYcS5M6oMwv1E8L3OetoZ@mail.gmail.com>
Message-ID: <4BFC677B.9060107@gmail.com>

(Sending again - I didn't mean to drop python-dev from the cc list when 
I originally sent this via the gmail web interface)

On Sun, May 23, 2010 at 9:00 PM, Dirkjan Ochtman <dirkjan at ochtman.nl 
<mailto:dirkjan at ochtman.nl>> wrote:

    Right, it wasn't intended as that harsh... but it does come with a
    rather impressive set of constraints in terms of what you can do with
    the API.

True, but in some cases (especially low level infrastructure), it is 
worth accepting those constraints in order to achieve other aims (such 
as standardisation of techniques). Things like itertools, collections, 
functools, unittest owe their existence largely to the choice of gains 
in standardisation over flexibility of API updates.
Besides, popular PyPI modules don't have that much more freedom than the 
stdlib when it comes to API changes. The only real difference is that 
the 18-24 month release cycle for the stdlib is a lot *slower* than that 
of many PyPI packages, so feedback on any changes we make is 
correspondingly delayed. Hence the existence of projects like distutils2 
and unittest2 to enable that faster feedback cycle to inform the updates 
passed back into the more slowly evolving stdlib modules, as well as the 
desire to copy prior art wherever it makes sense to do so (whether that 
is other languages, existing PyPI modules or the internal code bases of 
large corporate contributors).

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100526/49e5e3a6/attachment.html>

From ncoghlan at gmail.com  Wed May 26 02:25:22 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 10:25:22 +1000
Subject: [Python-Dev] Documenting [C]Python's Internals
In-Reply-To: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
References: <AANLkTinBQEJFhzJ1kH07jSPMXKh7lKYfKioAMYQFKBZ1@mail.gmail.com>
Message-ID: <4BFC6A72.2060608@gmail.com>

On 20/05/10 08:13, Yaniv Aknin wrote:
> Hi,
>
> I wanted to let python-dev know about a series of articles about
> CPython's internals I'm publishing under the collective title "Guido's
> Python"* (http://tech.blog.aknin.name/tag/guidos-python/).

A resource that may be useful to you is a 2.5 focused manuscript I put 
together a few years ago trying to bridge the gap between the library 
reference and the language reference:
http://svn.python.org/projects/sandbox/trunk/userref/

It's obviously a little dated in some areas and doesn't delve as deeply 
into the source code as you apparently plan to, but hopefully it may 
prove useful as a resource (I still have vague intentions of exporting 
that document to ReST markup and updating it to current Python, but that 
doesn't look like actually happening any time soon)

Cheers,
Nick.

P.S. For the record, the relevant URL is now 
http://tech.blog.aknin.name/tag/pythons-innards/


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From jnoller at gmail.com  Wed May 26 02:42:05 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Tue, 25 May 2010 20:42:05 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFC6325.5050700@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4BF916EA.2090507@holdenweb.com>
	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
	<20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com>
Message-ID: <AANLkTimySGJTYTkuEkLszhfWo4jHm2O8bZg6SSfCTx_r@mail.gmail.com>

On Tue, May 25, 2010 at 7:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 23/05/10 22:47, Antoine Pitrou wrote:
>>
>> On Sun, 23 May 2010 08:34:22 -0400
>> Jesse Noller<jnoller at gmail.com> ?wrote:
>>>
>>> Brian has already agreed to name spacing it to "concurrent.futures" -
>>> this means it will be a small part to a much larger concurrent.*
>>> implementation ala Java.
>>
>> What I would question here is what other things will be part
>> of the "concurrent" package, and who will implement them. Are there
>> plans for that? (or even tracker issues open?)
>
> I'm not sure it is called out explicitly in the PEP, but the specific
> example that came up in the previous discussions was something like
> "concurrent.pool" to hold a thread vs process agnostic worker pool interface
> based on the existing Pool interface in multiprocessing (with concrete
> implementations for both threading and multiprocessing).
>

Nick is correct - there's plenty of things in multiprocessing which
belong in a more abstract package as they're useful for more things
than just multiprocessing. I don't think they need to be called out as
part of the PEP though.

jesse

From regebro at gmail.com  Wed May 26 04:29:05 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 26 May 2010 04:29:05 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFC66F1.3060107@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com> 
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com> 
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com> 
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com> 
	<4BFC66F1.3060107@gmail.com>
Message-ID: <AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>

On Wed, May 26, 2010 at 02:10, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Those that say "just put it on PyPI" may not recognise the additional ...

Just a note, so we don't get sidelined by misunderstandings: I don't
think anybody said that. ;-)

There are two issues here, one generic and one specific:

Generic: Modules should go on PyPI first, for a time, to stabilize
(and so they can be used in earlier versions of Python) before they
end up in stdlib. I suspect everyone actually agrees on that (but I
could be wrong).

Specific:Has futures been long enough on PyPI, and is it stable? I'm
staying out of that discussion. :-)

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From ncoghlan at gmail.com  Wed May 26 05:34:35 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 13:34:35 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
Message-ID: <4BFC96CB.9090600@gmail.com>

On 26/05/10 12:29, Lennart Regebro wrote:
> On Wed, May 26, 2010 at 02:10, Nick Coghlan<ncoghlan at gmail.com>  wrote:
>> Those that say "just put it on PyPI" may not recognise the additional ...
>
> Just a note, so we don't get sidelined by misunderstandings: I don't
> think anybody said that. ;-)

Nah, that pseudo-quote wasn't from this discussion in particular. It's a 
reference to the ongoing tension between the "batteries included" 
advocates and the "make the standard library as streamlined as possible" 
crowd. Both sides have valid points, so the "included battery" vs 
"optional download" question needs to be decided on a case-by-case basis.

> There are two issues here, one generic and one specific:
>
> Generic: Modules should go on PyPI first, for a time, to stabilize
> (and so they can be used in earlier versions of Python) before they
> end up in stdlib. I suspect everyone actually agrees on that (but I
> could be wrong).

That's the point I'm disagreeing with. For most modules it makes sense 
to do things that way, but for some low-level infrastructure elements, 
it is going to be less effective (because people will quickly throw 
together their own solutions instead of adding a new dependency for 
something "simple").

Other times we'll invent a new module because *we* need it for something 
(e.g. runpy).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May 26 05:38:33 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 13:38:33 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <871vd1mvbp.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <874ohym5op.fsf@uwakimon.sk.tsukuba.ac.jp>	<20100524025805.GA29197@cskk.homeip.net>
	<871vd1mvbp.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4BFC97B9.3080901@gmail.com>

On 24/05/10 20:46, Stephen J. Turnbull wrote:
> Cameron Simpson writes:
>
>   >  There's a lot to be said for a robust implementation of a well defined
>   >  problem. Brian's module, had it been present and presuming it robust and
>   >  debugged, would have been quite welcome.
>
> That, of course, is the consensus view, both in general and with
> respect to this particular module.
>
> The difference is over what constitutes sufficient evidence for your
> presumption of "robust and debugged" from the point of view of the
> users of the stdlib.

At the very least, we'll be offering a promise to be "more robust and 
more debugged than what you came up with in that coding marathon last 
night" ;)

Having a decent test suite that is regularly executed on multiple 
platforms (which will be the case for any accepted module by the time it 
is included in a Python release) also places anything we release a cut 
above a *lot* of in-house code.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From stephen at xemacs.org  Wed May 26 05:51:07 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 26 May 2010 12:51:07 +0900
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFC66F1.3060107@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
Message-ID: <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > Those that say "just put it on PyPI"

Nobody is saying that, AFAICS.  Nobody is saying that *some* futures
module shouldn't *eventually* go into the stdlib.

The question is whether *this* futures module should go into the
stdlib *now*.  And it is clear that more time on PyPI would provide
valuable information.  This is a general principle that has served us
well: put best current practice backed up by actual widespread usage
in the stdlib, not theoretical wins based on the developer's
experience.  PyPI is a way to broaden usage to determine BCP, not an
end in itself.

People have been asking "what's special about this module, to violate
the BCP principle?"  There's nothing special about the fact that
several people would use a "robust and debugged" futures module if it
were in the stdlib.  That's true of *every* module that is worth a
PEP.  But remember, in the case of ipaddr it was the people who wanted
some such module badly who were also the most vocal opponents, because
they could see offhand that it was going to serve their use cases
badly.  (It turned out that this was equally trivial to fix despite a
week of hot debate, and everyone lived happily ever after.  But that
was smiling Luck, not inexorable Fate.)  For this module, three people
have said "I 'would have' used it if it were available," but none of
you has announced that you've started refactoring and the PEP 3148 API
meets all expectations.  I call that "damning with faint praise".

OTOH, Glyph has changed from "why not more time on PyPI?" to "let's
see if we can improve this a bit, then let's do it".  He has published
code (showing how to turn futures into Twisted Deferreds), and argues
that based on download stats to date and the nature of the use cases
it would take a lot of time on PyPI to demonstrate a BCP.  Those are
good arguments for an exception, IMHO.



From ncoghlan at gmail.com  Wed May 26 06:22:01 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 14:22:01 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>
	<87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4BFCA1E9.1070808@gmail.com>

On 26/05/10 13:51, Stephen J. Turnbull wrote:
> People have been asking "what's special about this module, to violate
> the BCP principle?"  There's nothing special about the fact that
> several people would use a "robust and debugged" futures module if it
> were in the stdlib.  That's true of *every* module that is worth a
> PEP.

I actually wrote a reply to that question earlier in the week, but 
failed at using gmail's web interface correctly and only sent it to 
Steve Holden.

===================
The trick with futures and executor pools is that they're a *better* way 
of programming with threads in many cases.

However, given the choices of:
- hack together something simple with some worker Threads and a Queue 
(or two)
- write my own futures and executor infrastructure
- download a futures module from PyPI and live with the additional 
dependency

I'll choose the first option every time, and my programs will be the 
worse for it.

Put the capability to use futures and an executor into the stdlib, and 
it becomes something I can reach for without having to worry about 
additional dependencies beyond specifying a minimal Python version. It 
provides a higher level API that can be more readily switched between 
threading and multiprocessing back ends. It becomes something that can 
be taught as a standard Python technique for enabling concurrency in 
imperative code.

This is something that is irrelevant to me as a module on PyPI, but has 
the potential to significantly affect my programming in the future as a 
standard library module. Even in the near term, backports of future 
standard library modules are often perceived differently when being 
discussed as potential additional dependencies for an application (i.e. 
I believe it would be worthwhile for a backport of the module to earlier 
Python versions than 3.2 to be made available on PyPI).
===================

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From regebro at gmail.com  Wed May 26 09:11:58 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 26 May 2010 09:11:58 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFCA1E9.1070808@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com> 
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com> 
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com> 
	<4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BFCA1E9.1070808@gmail.com>
Message-ID: <AANLkTin_h58313_-WQfkn36BUFTXTmtsLUgLBnOP3MYW@mail.gmail.com>

On Wed, May 26, 2010 at 06:22, Nick Coghlan <ncoghlan at gmail.com> wrote:
> - download a futures module from PyPI and live with the additional
> dependency

Why would that be a problem?

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From p.f.moore at gmail.com  Wed May 26 09:37:04 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 26 May 2010 08:37:04 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTin_h58313_-WQfkn36BUFTXTmtsLUgLBnOP3MYW@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BFCA1E9.1070808@gmail.com>
	<AANLkTin_h58313_-WQfkn36BUFTXTmtsLUgLBnOP3MYW@mail.gmail.com>
Message-ID: <AANLkTinZKkUIQ9yfJ9utlC_zdrA26ecFhi8cXpgmgnsy@mail.gmail.com>

On 26 May 2010 08:11, Lennart Regebro <regebro at gmail.com> wrote:
> On Wed, May 26, 2010 at 06:22, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> - download a futures module from PyPI and live with the additional
>> dependency
>
> Why would that be a problem?

That has been hashed out repeatedly on this and other lists. Can it
please be stipulated that for *some* people, in *some* cases, it is a
problem?

It seems to me that if you've experienced the sort of culture that
makes it a problem, you understand the point immediately, but if you
haven't, you never will (that's not disparaging anyone, the
idiosyncracies of corporate culture are widespread and bizarre - if it
helps, just remember that Dilbert is a documentary :-))

Paul.

From solipsis at pitrou.net  Wed May 26 09:38:20 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 26 May 2010 09:38:20 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4BF916EA.2090507@holdenweb.com>
	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
	<20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com>
Message-ID: <20100526093820.4fbf2f87@pitrou.net>

On Wed, 26 May 2010 09:54:13 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > What I would question here is what other things will be part
> > of the "concurrent" package, and who will implement them. Are there
> > plans for that? (or even tracker issues open?)
> 
> I'm not sure it is called out explicitly in the PEP, but the specific 
> example that came up in the previous discussions was something like 
> "concurrent.pool" to hold a thread vs process agnostic worker pool
> interface based on the existing Pool interface in multiprocessing
> (with concrete implementations for both threading and
> multiprocessing).

Ha, I'm a bit surprised. Isn't it what "futures" already provides?
(except that for some reason it insists on the "SomeExecutor" naming
scheme)
http://www.python.org/dev/peps/pep-3148/#processpoolexecutor

Regards

Antoine.



From glyph at twistedmatrix.com  Wed May 26 10:09:17 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Wed, 26 May 2010 04:09:17 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com>
	<64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com>
Message-ID: <1AE1A9E7-F821-45FF-8396-62EE8297ADE3@twistedmatrix.com>


On May 24, 2010, at 5:36 AM, Brian Quinlan wrote:
> On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote:
>> On May 23, 2010, at 2:37 AM, Brian Quinlan wrote:
>>> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:

> ProcessPoolExecutor has the same serialization perils that multiprocessing does. My original plan was to link to the multiprocessing docs to explain them but I couldn't find them listed.

Linking to the pickle documentation might be a good start.

> Yes, the execution context is Executor-dependent. The section under ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think.

I suppose so.  I guess I'm just looking for more precise usage of terminology. (This is a PEP, after all.  It's a specification that multiple VMs may have to follow, not just some user documentation for a package, even if they'll *probably* be using your code in all cases.)  I'd be happier if there were a clearer term than "calls" for the things being scheduled ("submissions"?), since the done callbacks aren't called in the subprocess for ProcessPoolExecutor, as we just discussed.

>> Sure.  Really, almost any contract would work, it just needs to be spelled out.  It might be nice to know whether the thread invoking the callbacks is a daemon thread or not, but I suppose it's not strictly necessary.
> 
> Your concerns is that the thread will be killed when the interpreter exits? It won't be.

Good to know.  Tell it to the PEP though, not me ;).

>> No reaction on [invoker vs. future]?  I think you'll wish you did this in a couple of years when you start bumping into application code that calls "set_result" :).
> 
> My reactions are mixed ;-)

Well, you are not obliged to take my advice, as long as I am not obliged to refrain from mocking you mercilessly if it happens that I was right in a couple of years ;-).

> Your proposal is to add a level of indirection to make it harder for people to call implementation methods. The downside is that it makes it a bit harder to write tests and Executors.

Both tests and executors will still create and invoke methods directly on one object; the only additional difficulty seems to be the need to type '.future' every so often on the executor/testing side of things, and that seems a cost well worth paying to avoid confusion over who is allowed to call those methods and when.

> I also can't see a big problem in letting people call set_result in client code though it is documented as being only for Executor implementations and tests. 
> 
> On the implementation side, I don't see why an Invoker needs a reference to the future.

Well, uh...

> class Invoker(object):
>   def __init__(self):
>     """Should only be called by Executor implementations."""
>     self.future = Future()
         ^ this is what I'd call a "reference to the future"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100526/44d901e1/attachment.html>

From regebro at gmail.com  Wed May 26 10:15:25 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 26 May 2010 10:15:25 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTinZKkUIQ9yfJ9utlC_zdrA26ecFhi8cXpgmgnsy@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com> 
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com> 
	<4BFC66F1.3060107@gmail.com> <87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BFCA1E9.1070808@gmail.com>
	<AANLkTin_h58313_-WQfkn36BUFTXTmtsLUgLBnOP3MYW@mail.gmail.com> 
	<AANLkTinZKkUIQ9yfJ9utlC_zdrA26ecFhi8cXpgmgnsy@mail.gmail.com>
Message-ID: <AANLkTinXDRw77Pae3yscmomNnz9B2OwRAseX8IXduJwv@mail.gmail.com>

On Wed, May 26, 2010 at 09:37, Paul Moore <p.f.moore at gmail.com> wrote:
> It seems to me that if you've experienced the sort of culture that
> makes it a problem,

Ah, it's a culture problem.

In a heterogenous world, every action will benefit some and hurt some.
Another arbitrary corporate ruleset could also mean you might be stuck
on ancient python versions, and might not see a new module added to
stdlib in 3.2 until 2015 or so. Some corporations go through a lot of
trouble to prevent their employees from doing their job. Pythons core
developers can not and should not let that hinder *them* from doing
what is best for Python.

Decisions on inclusion in stdlib must be made on what benefits Python
and it's users in general. Since even small mistakes in a stdlib
module will hurt far more people than having to having the module
mature on PyPI until the worst API issues and bugs are ironed out,
it's clear to me that letting a module mature on PyPI before inclusion
is the better policy here, although how long obviously must be decided
on a case by case basis.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From glyph at twistedmatrix.com  Wed May 26 10:25:18 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Wed, 26 May 2010 04:25:18 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTinZKkUIQ9yfJ9utlC_zdrA26ecFhi8cXpgmgnsy@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BFCA1E9.1070808@gmail.com>
	<AANLkTin_h58313_-WQfkn36BUFTXTmtsLUgLBnOP3MYW@mail.gmail.com>
	<AANLkTinZKkUIQ9yfJ9utlC_zdrA26ecFhi8cXpgmgnsy@mail.gmail.com>
Message-ID: <F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>

On May 26, 2010, at 3:37 AM, Paul Moore wrote:

> On 26 May 2010 08:11, Lennart Regebro <regebro at gmail.com> wrote:
>> On Wed, May 26, 2010 at 06:22, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> - download a futures module from PyPI and live with the additional
>>> dependency
>> 
>> Why would that be a problem?
> 
> That has been hashed out repeatedly on this and other lists. Can it
> please be stipulated that for *some* people, in *some* cases, it is a
> problem?

Sure, but I for one fully support Lennart asking the question, because while in the short term this *is* a problem with packaging tools in the Python ecosystem, in the long term (as you do note) it's an organizational dysfunction that can be addressed with better tools.

I think it would be bad to ever concede the point that sane factoring of dependencies and code re-use aren't worth it because some jerk in Accounting or System Operations wants you to fill out a requisition form for a software component that's free and liberally licensed anyway.

To support the unfortunate reality that such jerks in such departments really do in fact exist, there should be simple tools to glom a set of small, nicely factored dependencies into a giant monolithic ball of crud that installs all at once, and slap a sticker on the side of it that says "I am only filling out your stupid form once, okay".  This should be as distant as possible from the actual decision to package things in sensibly-sized chunks.

In other words, while I kinda-sorta buy Brian's argument that having this module in easy reach will motivate more people to use a standard, tested idiom for parallelization, I *don't* think that the stdlib should be expanded simply to accommodate those who just don't want to install additional packages for anything.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100526/1f600358/attachment.html>

From stephen at xemacs.org  Wed May 26 10:44:02 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 26 May 2010 17:44:02 +0900
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFCA1E9.1070808@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BFCA1E9.1070808@gmail.com>
Message-ID: <87hblvkq7h.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:
 > On 26/05/10 13:51, Stephen J. Turnbull wrote:

 > > People have been asking "what's special about this module, to violate
 > > the BCP principle?"  There's nothing special about the fact that
 > > several people would use a "robust and debugged" futures module if it
 > > were in the stdlib.  That's true of *every* module that is worth a
 > > PEP.
 > 
 > The trick with futures and executor pools is that they're a *better* way 
 > of programming with threads in many cases.

and

 > However, given the choices of [...].  I'll choose the first option
 > every time, and my programs will be the worse for it.

Again, nothing all that special about those; lots of proposed changes
satisfy similar conditions.  I don't think anyone denies the truth or
applicability of those arguments.  But are they enough?

Really, what you're arguing is "now is better than never."  Indeed,
that is so.  But you shouldn't forget that is immediately followed by
"although never is often better than *right* now."


From brian at sweetapp.com  Wed May 26 10:52:44 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 26 May 2010 18:52:44 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <87hblvkq7h.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BFCA1E9.1070808@gmail.com>
	<87hblvkq7h.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <BFD00783-D5C9-443E-8D59-25BAF33693FA@sweetapp.com>


On 26 May 2010, at 18:44, Stephen J. Turnbull wrote:

> Nick Coghlan writes:
>> On 26/05/10 13:51, Stephen J. Turnbull wrote:
>
>>> People have been asking "what's special about this module, to  
>>> violate
>>> the BCP principle?"  There's nothing special about the fact that
>>> several people would use a "robust and debugged" futures module if  
>>> it
>>> were in the stdlib.  That's true of *every* module that is worth a
>>> PEP.
>>
>> The trick with futures and executor pools is that they're a  
>> *better* way
>> of programming with threads in many cases.
>
> and
>
>> However, given the choices of [...].  I'll choose the first option
>> every time, and my programs will be the worse for it.
>
> Again, nothing all that special about those; lots of proposed changes
> satisfy similar conditions.  I don't think anyone denies the truth or
> applicability of those arguments.  But are they enough?
>
> Really, what you're arguing is "now is better than never."  Indeed,
> that is so.  But you shouldn't forget that is immediately followed by
> "although never is often better than *right* now."

I've been trying to stay out of the meta-discussions but "*right* now"  
would be >6 months if it applies in this context.

If that is what "*right* now" means to you then I hope that I never  
have a heart attack in your presence and need an ambulance *right*  
now :-)

Cheers,
Brian

From brian at sweetapp.com  Wed May 26 10:55:42 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 26 May 2010 18:55:42 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <1AE1A9E7-F821-45FF-8396-62EE8297ADE3@twistedmatrix.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com>
	<64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com>
	<1AE1A9E7-F821-45FF-8396-62EE8297ADE3@twistedmatrix.com>
Message-ID: <27B98FA1-A2FC-41D6-88AD-B44B9BF90DDC@sweetapp.com>


On 26 May 2010, at 18:09, Glyph Lefkowitz wrote:

>
> On May 24, 2010, at 5:36 AM, Brian Quinlan wrote:
>> On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote:
>>> On May 23, 2010, at 2:37 AM, Brian Quinlan wrote:
>>>> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:
>
>> ProcessPoolExecutor has the same serialization perils that  
>> multiprocessing does. My original plan was to link to the  
>> multiprocessing docs to explain them but I couldn't find them listed.
>
> Linking to the pickle documentation might be a good start.

Will do.

>> Yes, the execution context is Executor-dependent. The section under  
>> ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think.
>
> I suppose so.  I guess I'm just looking for more precise usage of  
> terminology. (This is a PEP, after all.  It's a specification that  
> multiple VMs may have to follow, not just some user documentation  
> for a package, even if they'll *probably* be using your code in all  
> cases.)  I'd be happier if there were a clearer term than "calls"  
> for the things being scheduled ("submissions"?), since the done  
> callbacks aren't called in the subprocess for ProcessPoolExecutor,  
> as we just discussed.
>
>>> Sure.  Really, almost any contract would work, it just needs to be  
>>> spelled out.  It might be nice to know whether the thread invoking  
>>> the callbacks is a daemon thread or not, but I suppose it's not  
>>> strictly necessary.
>>
>> Your concerns is that the thread will be killed when the  
>> interpreter exits? It won't be.
>
> Good to know.  Tell it to the PEP though, not me ;).

Will do.

>>> No reaction on [invoker vs. future]?  I think you'll wish you did  
>>> this in a couple of years when you start bumping into application  
>>> code that calls "set_result" :).
>>
>> My reactions are mixed ;-)
>
> Well, you are not obliged to take my advice, as long as I am not  
> obliged to refrain from mocking you mercilessly if it happens that I  
> was right in a couple of years ;-).

I was looking for your reasoning rather than trying to negotiate the  
circumstances under which you would mock me.
>
>> Your proposal is to add a level of indirection to make it harder  
>> for people to call implementation methods. The downside is that it  
>> makes it a bit harder to write tests and Executors.
>
> Both tests and executors will still create and invoke methods  
> directly on one object; the only additional difficulty seems to be  
> the need to type '.future' every so often on the executor/testing  
> side of things, and that seems a cost well worth paying to avoid  
> confusion over who is allowed to call those methods and when.
>
>> I also can't see a big problem in letting people call set_result in  
>> client code though it is documented as being only for Executor  
>> implementations and tests.
>>
>> On the implementation side, I don't see why an Invoker needs a  
>> reference to the future.
>
> Well, uh...
>
>> class Invoker(object):
>>   def __init__(self):
>>     """Should only be called by Executor implementations."""
>>     self.future = Future()
>          ^ this is what I'd call a "reference to the future"

I said exactly the opposite of what I meant: futures don't need a  
reference to the invoker.

Cheers,
Brian


From glyph at twistedmatrix.com  Wed May 26 11:02:33 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Wed, 26 May 2010 05:02:33 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <27B98FA1-A2FC-41D6-88AD-B44B9BF90DDC@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4FD0AFC2-F0FF-4190-867D-3A2C35FD850E@twistedmatrix.com>
	<64B5ACEE-0B5C-40A8-AEC8-4D40BC7FE65B@sweetapp.com>
	<1AE1A9E7-F821-45FF-8396-62EE8297ADE3@twistedmatrix.com>
	<27B98FA1-A2FC-41D6-88AD-B44B9BF90DDC@sweetapp.com>
Message-ID: <D6BDC3EC-76B4-4CD2-ABA6-2A40E0E7637A@twistedmatrix.com>


On May 26, 2010, at 4:55 AM, Brian Quinlan wrote:

> I said exactly the opposite of what I meant: futures don't need a reference to the invoker.

Indeed they don't, and they really shouldn't have one.  If I wrote that they did, then it was an error.

... and that appears to be it!  Thank you for your very gracious handling of a pretty huge pile of criticism :).

Good luck with the PEP,

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100526/d826f023/attachment-0001.html>

From hawkett at gmail.com  Wed May 26 11:15:02 2010
From: hawkett at gmail.com (Colin H)
Date: Wed, 26 May 2010 10:15:02 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
Message-ID: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>

Hi,

   issue991196 was closed being described as intentional.  I've added
a comment in that issue which argues that this is a serious bug (also
aserted by a previous commenter - Armin Rigo), because it creates a
unique, undocumented, oddly behaving scope that doesn't apply closures
correctly. At the very least I think this should be acknowledged as a
plain old bug (rather than a feature), and then a discussion about
whether it will be fixed or not.  Appreciate your thoughts - cheers,

Colin

From solipsis at pitrou.net  Wed May 26 11:39:15 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 26 May 2010 11:39:15 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<87k4qrl3ro.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BFCA1E9.1070808@gmail.com>
	<AANLkTin_h58313_-WQfkn36BUFTXTmtsLUgLBnOP3MYW@mail.gmail.com>
	<AANLkTinZKkUIQ9yfJ9utlC_zdrA26ecFhi8cXpgmgnsy@mail.gmail.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
Message-ID: <20100526113915.5bf46665@pitrou.net>

On Wed, 26 May 2010 04:25:18 -0400
Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> 
> In other words, while I kinda-sorta buy Brian's argument that having this module in easy reach
> will motivate more people to use a standard, tested idiom for parallelization, I *don't* think
> that the stdlib should be expanded simply to accommodate those who just don't want to install
> additional packages for anything.

+1.  Why don't the castrated-by-the-corporation people offer to maintain
a "Sumo" distribution of Python on python.org instead?  The rest of the
world shouldn't have to be impacted by their corporate culture woes.

cheers

Antoine.



From dickinsm at gmail.com  Wed May 26 11:48:51 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 26 May 2010 10:48:51 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
Message-ID: <AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>

On Wed, May 26, 2010 at 10:15 AM, Colin H <hawkett at gmail.com> wrote:
> ? issue991196 was closed being described as intentional. ?I've added
> a comment in that issue which argues that this is a serious bug (also
> aserted by a previous commenter - Armin Rigo), because it creates a
> unique, undocumented, oddly behaving scope that doesn't apply closures
> correctly. At the very least I think this should be acknowledged as a
> plain old bug (rather than a feature), and then a discussion about
> whether it will be fixed or not.

Here's a quick recap of the issue so that people don't have to go
searching through the bug archive.  In Python 2.x, we get the
following behaviour:

>>> code = """\
... y = 3
... def f():
...     return y
... f()
... """
>>> exec code in {}   # works fine
>>> exec code in {}, {}   # dies with a NameError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 4, in <module>
  File "<string>", line 3, in f
NameError: global name 'y' is not defined

The issue is whether the second example should work, given that two
different dictionaries have been passed.

The cause of the NameError can be seen by looking at the bytecode: y
is bound using STORE_NAME, which stores y into the locals dictionary
(which here is *not* the same as the globals dictionary) but the
attempt to retrieve the value of y uses LOAD_GLOBAL, which only looks
in the globals.

>>> co = compile(code, 'mycode', 'exec')
>>> dis.dis(co)
  1           0 LOAD_CONST               0 (3)
              3 STORE_NAME               0 (y)

  2           6 LOAD_CONST               1 (<code object f at
0xa22b40, file "mycode", line 2>)
              9 MAKE_FUNCTION            0
             12 STORE_NAME               1 (f)

  4          15 LOAD_NAME                1 (f)
             18 CALL_FUNCTION            0
             21 POP_TOP
             22 LOAD_CONST               2 (None)
             25 RETURN_VALUE
>>> dis.dis(co.co_consts[1])  # disassembly of 'f'
  3           0 LOAD_GLOBAL              0 (y)
              3 RETURN_VALUE

This is a long way from my area of expertise (I'm commenting here
because it was me who sent Colin here in the first place), and it's
not clear to me whether this is a bug, and if it is a bug, how it
could be resolved.  What would the impact be of having the compiler
produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here?

Mark

From steve at pearwood.info  Wed May 26 12:42:12 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 26 May 2010 20:42:12 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100526113915.5bf46665@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
Message-ID: <201005262042.12825.steve@pearwood.info>

On Wed, 26 May 2010 07:39:15 pm Antoine Pitrou wrote:
> On Wed, 26 May 2010 04:25:18 -0400
>
> Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> > In other words, while I kinda-sorta buy Brian's argument that
> > having this module in easy reach will motivate more people to use a
> > standard, tested idiom for parallelization, I *don't* think that
> > the stdlib should be expanded simply to accommodate those who just
> > don't want to install additional packages for anything.
>
> +1.  Why don't the castrated-by-the-corporation people offer to
> maintain a "Sumo" distribution of Python on python.org instead?  The
> rest of the world shouldn't have to be impacted by their corporate
> culture woes.


It's not just the corporate culture. For many people, the standard 
library is the first introduction to even the existence of a particular 
technique or technology. You can't go looking for something on PyPI if 
you don't know that there's a something to look for. And for many 
beginners and not-so-beginners, the idea and practice of installing 
additional packages is simply problematic.

I'm not saying that Python-Dev should bend over backwards to accommodate 
such people to the exclusion of all else, but these folks are 
stakeholders too, and their wants and needs are just as worthy as the 
wants and needs of those who prefer a more conservative approach to the 
standard library.

This is a Python implementation of a stable Java API, Brian has said the 
futures package has been on PyPI for about a year, and it's been 
flagged as a production/stable release since October last year.

http://pypi.python.org/pypi/futures3

Given that there does seem to be a general agreement that futures should 
go into the std lib at some point, is this not sufficient exposure?


-- 
Steven D'Aprano

From greg.ewing at canterbury.ac.nz  Wed May 26 12:57:15 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 26 May 2010 22:57:15 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
Message-ID: <4BFCFE8B.1080205@canterbury.ac.nz>

Having read through the PEP again, here are my thoughts.

* I'm bothered by the term "future". To my mind, it's
too long on cleverness and too short on explanativeness.

I think that the standard library is no place for
cuteness of naming. The name of a stdlib module should
reflect its functionality in some straightforward and
obvious way. If I were looking for a thread pool or
process pool implementation, the word "future" is not
something that would spring readily to mind.

The stated purpose of the module is to "execute
computations asynchronously", so perhaps a name such
as "asyntask" would be appropriate, following the
pattern of existing modules dealing with ansynchronous
matters, ansyncore and asynchat. For the Future object
itself, I'd suggest something like "Task" or "Job".

* It seems unnecessarily verbose to tack "Executor"
onto the end of every Executor subclass. They could
simply be called ThreadPool and ProcessPool without
losing anything.

* I don't see a strong reason to put this module
inside a newly-created namespace. If there were a
namespace called "concurrent", I would expect to find
other existing concurrency-related modules there as
well, such as threading and multiprocessing. But we
can't move them there without breaking existing code.

(More generally, I'm inclined to think that introducing
a namespace package for a category of modules having
existing members in the stdlib is an anti-pattern,
unless it's done during the kind of namespace refactoring
that we won't get another chance to perform until Py4k.)

Concerning the structure of the PEP:

* A section titled 'Specification' should *not* start
with a bunch of examples. It may be appropriate to include
short examples *following* items in the specification in
order to illustrate the features concerned. Extended
examples such as these belong in a section of their own.

* I found the examples included to be rather difficult
to follow, and they served more to confuse than elucidate.
I think this is partly because they are written in a
fairly compressed style, burying important things being
illustrated inside complicated expressions. Rewriting
them in a more straightforward style might help.

Concerning details of the specification:

* Is it possible to have more than one Executor active
at a time? The fact that as_completed() is a module-level
function rather than an Executor method suggests that it
is, but it would be good to have this spelled out one
way or the other in the PEP.

-- 
Greg

From solipsis at pitrou.net  Wed May 26 12:56:14 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 26 May 2010 12:56:14 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
Message-ID: <20100526125614.0a7ae470@pitrou.net>

On Wed, 26 May 2010 20:42:12 +1000
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> I'm not saying that Python-Dev should bend over backwards to accommodate 
> such people to the exclusion of all else, but these folks are 
> stakeholders too, and their wants and needs are just as worthy as the 
> wants and needs of those who prefer a more conservative approach to the 
> standard library.

Well, my "Sumo" proposal was a serious one.
(not serious in that I would offer to give a hand, but in that I think
it could help those people; also, wouldn't it be sensible for users in
a corporate environment to share their efforts and produce something
that can benefit all of them? it's the free software spirit after all)

> This is a Python implementation of a stable Java API, Brian has said the 
> futures package has been on PyPI for about a year, and it's been 
> flagged as a production/stable release since October last year.

I'm not against futures being in the stdlib, I was just pointing out
that I don't agree with the "corporate culture issues should be
accomodated by including more modules in the stdlib" argument.

Regards

Antoine.



From brian at sweetapp.com  Wed May 26 13:05:45 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 26 May 2010 21:05:45 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFCFE8B.1080205@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
Message-ID: <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>

On May 26, 2010, at 8:57 PM, Greg Ewing wrote:

> Having read through the PEP again, here are my thoughts.
>
> * I'm bothered by the term "future". To my mind, it's
> too long on cleverness and too short on explanativeness.
>
> I think that the standard library is no place for
> cuteness of naming. The name of a stdlib module should
> reflect its functionality in some straightforward and
> obvious way. If I were looking for a thread pool or
> process pool implementation, the word "future" is not
> something that would spring readily to mind.
>
> The stated purpose of the module is to "execute
> computations asynchronously", so perhaps a name such
> as "asyntask" would be appropriate, following the
> pattern of existing modules dealing with ansynchronous
> matters, ansyncore and asynchat. For the Future object
> itself, I'd suggest something like "Task" or "Job".

"future" is a computing science term of art, like "thread". Anyway,  
this has been discussed in the past and Guido was happy with the name.

> * It seems unnecessarily verbose to tack "Executor"
> onto the end of every Executor subclass. They could
> simply be called ThreadPool and ProcessPool without
> losing anything.

You could have general thread pools that aren't related to executors  
(actually, it would be great if Python had a good built-in thread pool  
implementation) and I'd like to avoid using an overly generic name.

> * I don't see a strong reason to put this module
> inside a newly-created namespace. If there were a
> namespace called "concurrent", I would expect to find
> other existing concurrency-related modules there as
> well, such as threading and multiprocessing. But we
> can't move them there without breaking existing code.

I think that Jesse was planning to add some functionality to this  
namespace. I don't really have an opinion on this.

> (More generally, I'm inclined to think that introducing
> a namespace package for a category of modules having
> existing members in the stdlib is an anti-pattern,
> unless it's done during the kind of namespace refactoring
> that we won't get another chance to perform until Py4k.)
>
> Concerning the structure of the PEP:
>
> * A section titled 'Specification' should *not* start
> with a bunch of examples. It may be appropriate to include
> short examples *following* items in the specification in
> order to illustrate the features concerned. Extended
> examples such as these belong in a section of their own.

I thought that the specification would be difficult to follow without  
examples to pave the way. Anyone else have an opinion on this?

> * I found the examples included to be rather difficult
> to follow, and they served more to confuse than elucidate.
> I think this is partly because they are written in a
> fairly compressed style, burying important things being
> illustrated inside complicated expressions. Rewriting
> them in a more straightforward style might help.

Do you think starting with a simpler example would help? I think that  
idiomatic future use will end up looking similar to my examples. If  
that is too complex for most users then we have a problem.

> Concerning details of the specification:
>
> * Is it possible to have more than one Executor active
> at a time?

Of course.

> The fact that as_completed() is a module-level
> function rather than an Executor method suggests that it
> is, but it would be good to have this spelled out one
> way or the other in the PEP.

I'll add a note to the global functions that they can accept futures  
from different in the same call.

Cheers,
Brian

>
> -- 
> Greg
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com


From steve at pearwood.info  Wed May 26 13:18:22 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 26 May 2010 21:18:22 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFCFE8B.1080205@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
Message-ID: <201005262118.23006.steve@pearwood.info>

On Wed, 26 May 2010 08:57:15 pm Greg Ewing wrote:

> * I'm bothered by the term "future". To my mind, it's
> too long on cleverness and too short on explanativeness.

"Futures" is a standard term in computer science which has been around 
for 33 years.

http://en.wikipedia.org/wiki/Futures_and_promises


> I think that the standard library is no place for
> cuteness of naming.

You mean like pickle, marshal, shelve, turtle, and even dict?


> * I don't see a strong reason to put this module
> inside a newly-created namespace. If there were a
> namespace called "concurrent", I would expect to find
> other existing concurrency-related modules there as
> well, such as threading and multiprocessing. But we
> can't move them there without breaking existing code.

I'm sure that it can be done easily, although not quickly. For instance, 
we move threading into the concurrent namespace, and leave behind in 
its place a stub:

from concurrent.threading import *

Then for (say) 3.3 the stub could gain a PendingDeprecation warning, 
then in 3.4 a Deprecation warning, and finally in 3.5 or 3.6 it could 
be removed.



-- 
Steven D'Aprano

From floris.bruynooghe at gmail.com  Wed May 26 14:06:51 2010
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Wed, 26 May 2010 13:06:51 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
Message-ID: <20100526120651.GA12751@laurie.devork.be>

Hi

On Sun, May 23, 2010 at 10:47:08AM +1000, Brian Quinlan wrote:
> Jesse, the designated pronouncer for this PEP, has decided to keep
> discussion open for a few more days.
> 
> So fire away!

In thread.py the module automatically registers a handler with atexit.
I don't think I'm alone in thinking libraries should not be doing this
sort of thing unconditionally behind a user's back.  I'm also not so
sure how comfortable I am with the module-level globals.

Would it not be possible to have an exit handler on each thread pool
which the documentation reccomends you register with atexit if it
suits your application?  I think that would get rid of the global
singletons and hidden atexit in a fairly elegant way.

Lastly _base.py creates a LOGGER (connected to sys.stderr if I
understand correctly) and only logs a critical message to it at the
same time as a RuntimeError is raised.  While I don't necessarily
dislike that it uses a logger, I don't like that it's wired up to
sys.stderr I rather think it's the application's duty to create a
handler if it wants one.  But given that it's only used at the same
time as a RuntimeError it does seem redundant.

Regards
Floris

PS: I've only looked at the threading part of the implementation.

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From p.f.moore at gmail.com  Wed May 26 14:19:01 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 26 May 2010 13:19:01 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100526125614.0a7ae470@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
Message-ID: <AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>

On 26 May 2010 11:56, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 26 May 2010 20:42:12 +1000
> Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> I'm not saying that Python-Dev should bend over backwards to accommodate
>> such people to the exclusion of all else, but these folks are
>> stakeholders too, and their wants and needs are just as worthy as the
>> wants and needs of those who prefer a more conservative approach to the
>> standard library.
>
> Well, my "Sumo" proposal was a serious one.
> (not serious in that I would offer to give a hand, but in that I think
> it could help those people; also, wouldn't it be sensible for users in
> a corporate environment to share their efforts and produce something
> that can benefit all of them? it's the free software spirit after all)

I'm not sure how a "Sumo" approach would work in practical terms, and
this thread isn't really the place to discuss, but there's a couple of
points I think are worth making:

* For a "Sumo" distribution to make sense, some relatively substantial
chunk of the standard library would need to be moved *out* to reside
in the sumo distribution. Otherwise it's not really a "sumo", just a
couple of modules that "nearly made it into the stdlib", at least for
the near-to-medium term. I've yet to see any sort of consensus that
python-dev is willing to undertake that decoupling work. (Which would
include extracting the various tests, migrating bugs out of the
pythion tracker, etc etc).

* If the decoupled modules aren't simply being abandoned, python-dev
needs to continue to commit to supporting them "in the wild" (i.e., on
PyPI and in the sumo distribution). Otherwise we're just abandoning
existing users and saying "support it yourself". I've seen no
indication that python-dev members would expect to follow bug trackers
for various decoupled modules - so in practice, this sounds more like
abandonment than decoupling.

Until a stdlib-decoupling proposal which takes these aspects into
account is on the table, I'm afraid that suggesting there's a "Sumo
distribution" style middle ground between stdlib and PyPI isn't really
true...

Paul.

From jnoller at gmail.com  Wed May 26 14:25:08 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 26 May 2010 08:25:08 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
Message-ID: <AANLkTinD66Nj81kyzXRahM7DYfRyzauUNzVAEi-u3_gl@mail.gmail.com>

On Wed, May 26, 2010 at 8:19 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 26 May 2010 11:56, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Wed, 26 May 2010 20:42:12 +1000
>> Steven D'Aprano <steve at pearwood.info> wrote:
>>>
>>> I'm not saying that Python-Dev should bend over backwards to accommodate
>>> such people to the exclusion of all else, but these folks are
>>> stakeholders too, and their wants and needs are just as worthy as the
>>> wants and needs of those who prefer a more conservative approach to the
>>> standard library.
>>
>> Well, my "Sumo" proposal was a serious one.
>> (not serious in that I would offer to give a hand, but in that I think
>> it could help those people; also, wouldn't it be sensible for users in
>> a corporate environment to share their efforts and produce something
>> that can benefit all of them? it's the free software spirit after all)
>
> I'm not sure how a "Sumo" approach would work in practical terms, and
> this thread isn't really the place to discuss, but there's a couple of
> points I think are worth making:
>
> * For a "Sumo" distribution to make sense, some relatively substantial
> chunk of the standard library would need to be moved *out* to reside
> in the sumo distribution. Otherwise it's not really a "sumo", just a
> couple of modules that "nearly made it into the stdlib", at least for
> the near-to-medium term. I've yet to see any sort of consensus that
> python-dev is willing to undertake that decoupling work. (Which would
> include extracting the various tests, migrating bugs out of the
> pythion tracker, etc etc).
>
> * If the decoupled modules aren't simply being abandoned, python-dev
> needs to continue to commit to supporting them "in the wild" (i.e., on
> PyPI and in the sumo distribution). Otherwise we're just abandoning
> existing users and saying "support it yourself". I've seen no
> indication that python-dev members would expect to follow bug trackers
> for various decoupled modules - so in practice, this sounds more like
> abandonment than decoupling.
>
> Until a stdlib-decoupling proposal which takes these aspects into
> account is on the table, I'm afraid that suggesting there's a "Sumo
> distribution" style middle ground between stdlib and PyPI isn't really
> true...
>
> Paul.

The fat vs. thin stdlib was discussed on stdlib-sig some time ago (I
am generally +1 to having a thin dist and a secondary "fatter" dist),
however right now, it doesn't make sense packaging and dependency
management is still a mess (but getting better), and there's a ton of
other things to take into consideration, some of which has been
iterated in this thread.

That being said, we've now evolved into meta-meta-meta-discussion - if
people seriously want to discuss the fat vs. thin subject, it should
probably go to stdlib-sig.

jesse

From ncoghlan at gmail.com  Wed May 26 14:26:17 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 22:26:17 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100526125614.0a7ae470@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>	<20100526113915.5bf46665@pitrou.net>	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
Message-ID: <4BFD1369.8090803@gmail.com>

On 26/05/10 20:56, Antoine Pitrou wrote:
> On Wed, 26 May 2010 20:42:12 +1000
> Steven D'Aprano<steve at pearwood.info>  wrote:
>>
>> I'm not saying that Python-Dev should bend over backwards to accommodate
>> such people to the exclusion of all else, but these folks are
>> stakeholders too, and their wants and needs are just as worthy as the
>> wants and needs of those who prefer a more conservative approach to the
>> standard library.
>
> Well, my "Sumo" proposal was a serious one.
> (not serious in that I would offer to give a hand, but in that I think
> it could help those people; also, wouldn't it be sensible for users in
> a corporate environment to share their efforts and produce something
> that can benefit all of them? it's the free software spirit after all)

That's actually what happens with groups like Enthought and ActiveState 
- bundles with extra batteries.

However, note that this isn't just a dysfunctional corporate culture 
issue, and I object to it being characterised as such (although 
dysfunctional cultures can certainly make it much, much worse). Vetting 
licenses for due diligence reasons, tracking releases of an external 
module, familiarising yourself with an additional API and code base, the 
risk of encountering bugs in that code base... these are all real costs 
that don't go away no matter how good the Python packaging ecosystem 
becomes. There is a trade off between "do the simplest thing that could 
possibly work (but may cause you problems later)" and spending the time 
to investigate third party solutions (with the risk that you end up 
rolling your own later anyway if you don't find anything suitable or, 
worse, find something that initially seems suitable but proves 
unworkable in practice).

A module that makes it into the standard library, however, carries 
python-dev's stamp of approval. Except for some older legacy libraries, 
that means a module will have at least half decent documentation and an 
automated test suite that is regularly run on multiple platforms. Its 
design will also have run the gauntlet of python-dev approval.

If we identify a good solution to a standard problem, and we have reason 
to believe that posting it in on PyPI as a separate module won't lead to 
a significant amount of additional real world testing, then it makes 
sense for it to go straight into the standard library.

Such modules are going to be rare (since most non-trivial modules *will* 
benefit from some time on PyPI, and most trivial modules won't be added 
to the standard library at all), but they do exist (runpy, contextlib, 
collections, itertools and abc spring to mind).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From fuzzyman at voidspace.org.uk  Wed May 26 14:28:38 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Wed, 26 May 2010 13:28:38 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>	<20100526113915.5bf46665@pitrou.net>	<201005262042.12825.steve@pearwood.info>	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
Message-ID: <4BFD13F6.1080300@voidspace.org.uk>

On 26/05/2010 13:19, Paul Moore wrote:
> On 26 May 2010 11:56, Antoine Pitrou<solipsis at pitrou.net>  wrote:
>    
>> On Wed, 26 May 2010 20:42:12 +1000
>> Steven D'Aprano<steve at pearwood.info>  wrote:
>>      
>>> I'm not saying that Python-Dev should bend over backwards to accommodate
>>> such people to the exclusion of all else, but these folks are
>>> stakeholders too, and their wants and needs are just as worthy as the
>>> wants and needs of those who prefer a more conservative approach to the
>>> standard library.
>>>        
>> Well, my "Sumo" proposal was a serious one.
>> (not serious in that I would offer to give a hand, but in that I think
>> it could help those people; also, wouldn't it be sensible for users in
>> a corporate environment to share their efforts and produce something
>> that can benefit all of them? it's the free software spirit after all)
>>      
> I'm not sure how a "Sumo" approach would work in practical terms, and
> this thread isn't really the place to discuss, but there's a couple of
> points I think are worth making:
>
> * For a "Sumo" distribution to make sense, some relatively substantial
> chunk of the standard library would need to be moved *out* to reside
> in the sumo distribution. Otherwise it's not really a "sumo", just a
> couple of modules that "nearly made it into the stdlib", at least for
> the near-to-medium term. I've yet to see any sort of consensus that
> python-dev is willing to undertake that decoupling work. (Which would
> include extracting the various tests, migrating bugs out of the
> pythion tracker, etc etc).
>
> * If the decoupled modules aren't simply being abandoned, python-dev
> needs to continue to commit to supporting them "in the wild" (i.e., on
> PyPI and in the sumo distribution). Otherwise we're just abandoning
> existing users and saying "support it yourself". I've seen no
> indication that python-dev members would expect to follow bug trackers
> for various decoupled modules - so in practice, this sounds more like
> abandonment than decoupling.
>
> Until a stdlib-decoupling proposal which takes these aspects into
> account is on the table, I'm afraid that suggesting there's a "Sumo
> distribution" style middle ground between stdlib and PyPI isn't really
> true...
>    

Well... a middle ground certainly could exist; perhaps in the form of an 
"Extended Standard Library" (community distribution), with simple 
installation and management tools.

It could be "blessed" by python-dev and maintain a high standard (only 
well established best-of-breed modules with a commitment of ongoing 
maintenance and more than one maintainer - something that the stdlib 
itself doesn't stick to). A common license could even be chosen, 
potentially allowing corporations to approve the extended package in a 
single pass.

Lot of details to flesh out obviously - but it would be great to see 
something like this come into being. Obviously this would need to be a 
community initiative and would take some time to establish. A "fat" 
distribution like this, based on tools like pip and distribute would be 
great for both newbies and for experienced programmers in making it 
easier to find "best" solutions for standard problems. It could also act 
as an incubator for the standard library (perhaps with stable and 
experimental streams where stable has a more conservative update policy).

All the best,

Michael Foord
> Paul.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From brian at sweetapp.com  Wed May 26 14:31:06 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 26 May 2010 22:31:06 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100526120651.GA12751@laurie.devork.be>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<20100526120651.GA12751@laurie.devork.be>
Message-ID: <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>

On 26 May 2010, at 22:06, Floris Bruynooghe wrote:

> Hi
>
> On Sun, May 23, 2010 at 10:47:08AM +1000, Brian Quinlan wrote:
>> Jesse, the designated pronouncer for this PEP, has decided to keep
>> discussion open for a few more days.
>>
>> So fire away!
>
> In thread.py the module automatically registers a handler with atexit.
> I don't think I'm alone in thinking libraries should not be doing this
> sort of thing unconditionally behind a user's back.  I'm also not so
> sure how comfortable I am with the module-level globals.
>
> Would it not be possible to have an exit handler on each thread pool
> which the documentation reccomends you register with atexit if it
> suits your application?  I think that would get rid of the global
> singletons and hidden atexit in a fairly elegant way.

First let me explain why I install at atexit handler.

Imagine that the you write a script like this:
t = ThreadPoolExecutor(1)
t.submit(lambda url: print(urllib.open(url).read()), 'http://www.apple.com/')

You have two semantic choices here:
1. let the interpreter exit with the future still running
2. wait until the future finishes and then exit

I chose (2) but can be convinced otherwise. The obvious way to  
accomplish this is to make the worker thread non-daemon so the  
interpreter won't exit while it is running. But since the worker  
thread is part of a pool, it won't stop while it's executor is alive.

So my approach was to make worker threads daemon and install an atexit  
handler that sets a global indicating that the interpreter is exiting  
so any workers should exit when when their work queues are empty. It  
then calls join on each worker thread so the interpreter will not exit  
until they are finished.

I think that this approach is reasonable assuming that you want (2). I  
also don't have the aversion to globals that you do :-)

> Lastly _base.py creates a LOGGER (connected to sys.stderr if I
> understand correctly) and only logs a critical message to it at the
> same time as a RuntimeError is raised.  While I don't necessarily
> dislike that it uses a logger, I don't like that it's wired up to
> sys.stderr I rather think it's the application's duty to create a
> handler if it wants one.  But given that it's only used at the same
> time as a RuntimeError it does seem redundant.

The LOGGER is only use for "impossible" exceptions (exceptions in the  
machinery of the module itself) that won't be propagated because they  
occur in worker threads.

Cheers,
Brian


> Regards
> Floris
>
> PS: I've only looked at the threading part of the implementation.
>
> -- 
> Debian GNU/Linux -- The Power of Freedom
> www.debian.org | www.gnu.org | www.kernel.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com


From ncoghlan at gmail.com  Wed May 26 14:32:33 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 22:32:33 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100526093820.4fbf2f87@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<4BF916EA.2090507@holdenweb.com>	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>	<20100523144754.1c6f1afc@pitrou.net>
	<4BFC6325.5050700@gmail.com> <20100526093820.4fbf2f87@pitrou.net>
Message-ID: <4BFD14E1.6000305@gmail.com>

On 26/05/10 17:38, Antoine Pitrou wrote:
> On Wed, 26 May 2010 09:54:13 +1000
> Nick Coghlan<ncoghlan at gmail.com>  wrote:
>>>
>>> What I would question here is what other things will be part
>>> of the "concurrent" package, and who will implement them. Are there
>>> plans for that? (or even tracker issues open?)
>>
>> I'm not sure it is called out explicitly in the PEP, but the specific
>> example that came up in the previous discussions was something like
>> "concurrent.pool" to hold a thread vs process agnostic worker pool
>> interface based on the existing Pool interface in multiprocessing
>> (with concrete implementations for both threading and
>> multiprocessing).
>
> Ha, I'm a bit surprised. Isn't it what "futures" already provides?
> (except that for some reason it insists on the "SomeExecutor" naming
> scheme)
> http://www.python.org/dev/peps/pep-3148/#processpoolexecutor

Not really - a general purpose pool would be a lot more agnostic about 
how you give the pooled threads/processes work to do and get the results 
back.

Executors are the kind of thing you would build on top of one though. If 
concurrent.pool was added, then the existing processing pools in 
multiprocessing and the executors in concurrent.future would be the 
first use cases for it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May 26 14:42:35 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 22:42:35 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFCFE8B.1080205@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
Message-ID: <4BFD173B.9010703@gmail.com>

On 26/05/10 20:57, Greg Ewing wrote:
> Having read through the PEP again, here are my thoughts.
> * It seems unnecessarily verbose to tack "Executor"
> onto the end of every Executor subclass. They could
> simply be called ThreadPool and ProcessPool without
> losing anything.

We would lose the ability to add general purpose thread and process 
pools under the obvious names later.

> * I don't see a strong reason to put this module
> inside a newly-created namespace. If there were a
> namespace called "concurrent", I would expect to find
> other existing concurrency-related modules there as
> well, such as threading and multiprocessing. But we
> can't move them there without breaking existing code.
>
> (More generally, I'm inclined to think that introducing
> a namespace package for a category of modules having
> existing members in the stdlib is an anti-pattern,
> unless it's done during the kind of namespace refactoring
> that we won't get another chance to perform until Py4k.)

_thread, threading, Queue and multiprocessing do likely belong here, but 
moving them isn't likely to be worth the pain. Does it help to know that 
at least Jesse and I (and probably others) would like to see 
concurrent.pool added eventually with robust general purpose ThreadPool 
and ProcessPool implementations?

The specific reason the new package namespace was added was to help 
avoid confusion with stock market futures without using an unduly 
cumbersome module name, but I don't know how well the PEP explains that.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Wed May 26 14:46:03 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 26 May 2010 14:46:03 +0200
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
Message-ID: <1274877963.3162.20.camel@localhost.localdomain>

Le mercredi 26 mai 2010 ? 13:19 +0100, Paul Moore a ?crit :
> 
> I'm not sure how a "Sumo" approach would work in practical terms, and
> this thread isn't really the place to discuss, but there's a couple of
> points I think are worth making:
> 
> * For a "Sumo" distribution to make sense, some relatively substantial
> chunk of the standard library would need to be moved *out* to reside
> in the sumo distribution. Otherwise it's not really a "sumo", just a
> couple of modules that "nearly made it into the stdlib", at least for
> the near-to-medium term.

This is not what I'm suggesting at all. The stdlib wouldn't shrink
(well, we could dump outdated modules but that's a separate decision).

A hypothetical "Sumo" distribution would be made of those more or less
useful modules that many people (application or framework developers;
for example, take a look at the dozens of direct and indirect
dependencies TurboGears pulls in) need and install.

The whole point is that it would *not* be supported by python-dev itself
but by a separate group of people (such as you :-)).
And it would have its own rules (surely more relaxed) over inclusion or
deprecation delays, the ability to make compatibility-breaking changes,
release timings, multiple obvious ways to do the same thing, etc.

And it means it could really bundle a lot of third-party stuff: small
helpers (things like the decorator module), event loops, template
engines, network address abstractions, argument parsers, ORMs, UI
toolkits, etc.

(a side-effect would be that it could, if it works well, behave as a
good intermediate stress test - a purgatory, if you want - for modules
before they integrate the stdlib)

If you want an existing analogy, you could look at EasyPHP. Or think of
Python as the Gnome or KDE project (consistent and aiming at providing
the most important everyday tools, but quite focused), and "Sumo" as an
entire distribution of disparate Linux GUI apps.

Regards

Antoine.



From solipsis at pitrou.net  Wed May 26 14:50:33 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 26 May 2010 14:50:33 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4BF916EA.2090507@holdenweb.com>
	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
	<20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com>
	<20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com>
Message-ID: <20100526145033.3b61d95b@pitrou.net>

On Wed, 26 May 2010 22:32:33 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > Ha, I'm a bit surprised. Isn't it what "futures" already provides?
> > (except that for some reason it insists on the "SomeExecutor" naming
> > scheme)
> > http://www.python.org/dev/peps/pep-3148/#processpoolexecutor
> 
> Not really - a general purpose pool would be a lot more agnostic about 
> how you give the pooled threads/processes work to do and get the results 
> back.
> 
> Executors are the kind of thing you would build on top of one though. If 
> concurrent.pool was added, then the existing processing pools in 
> multiprocessing and the executors in concurrent.future would be the 
> first use cases for it.

I think I'm a bit ignorant, but how is the Executor abstraction (and
its proposed implementations) not generic enough? You have a pool,
submit one or several tasks, and can either repeatedly poll for
completion or do a blocking wait.

(after all, Glyph pointed out that it should be quite easy to wrap the
resulting Futures into Deferred objects)

cheers

Antoine.



From ncoghlan at gmail.com  Wed May 26 14:51:42 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 22:51:42 +1000
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
Message-ID: <4BFD195E.2060007@gmail.com>

On 26/05/10 19:48, Mark Dickinson wrote:
> This is a long way from my area of expertise (I'm commenting here
> because it was me who sent Colin here in the first place), and it's
> not clear to me whether this is a bug, and if it is a bug, how it
> could be resolved.  What would the impact be of having the compiler
> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here?

exec with a single argument = module namespace
exec with two arguments = class namespace

Class namespaces are deliberately exempted from lexical scoping so that 
methods can't see class attributes, hence the example in the tracker 
issue works exactly as it would if the code was written as a class body.

   class C:
     y = 3
     def execfunc():
       print y
     execfunc()

With this code, y would end up in C.__dict__ rather than the module 
globals (at least, it would if it wasn't for the exception) and the call 
to execfunc fails with a NameError when attempting to find y.

I know I've closed other bug reports that were based on the same 
misunderstanding, and I didn't understand it myself until Guido 
explained it to me a few years back, so suggestions for improving the 
exec documentation in this area would be appreciated.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From jnoller at gmail.com  Wed May 26 15:01:18 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 26 May 2010 09:01:18 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFD173B.9010703@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com>
Message-ID: <AANLkTinWhP8lYKXGQ2mNoY-TpmbIbuY6_CnOR9CBrxsA@mail.gmail.com>

On Wed, May 26, 2010 at 8:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 26/05/10 20:57, Greg Ewing wrote:
>>
>> Having read through the PEP again, here are my thoughts.
>> * It seems unnecessarily verbose to tack "Executor"
>> onto the end of every Executor subclass. They could
>> simply be called ThreadPool and ProcessPool without
>> losing anything.
>
> We would lose the ability to add general purpose thread and process pools
> under the obvious names later.
>
>> * I don't see a strong reason to put this module
>> inside a newly-created namespace. If there were a
>> namespace called "concurrent", I would expect to find
>> other existing concurrency-related modules there as
>> well, such as threading and multiprocessing. But we
>> can't move them there without breaking existing code.
>>
>> (More generally, I'm inclined to think that introducing
>> a namespace package for a category of modules having
>> existing members in the stdlib is an anti-pattern,
>> unless it's done during the kind of namespace refactoring
>> that we won't get another chance to perform until Py4k.)
>
> _thread, threading, Queue and multiprocessing do likely belong here, but
> moving them isn't likely to be worth the pain. Does it help to know that at
> least Jesse and I (and probably others) would like to see concurrent.pool
> added eventually with robust general purpose ThreadPool and ProcessPool
> implementations?
>
> The specific reason the new package namespace was added was to help avoid
> confusion with stock market futures without using an unduly cumbersome
> module name, but I don't know how well the PEP explains that.


I assume(d) it's sufficient to link to the mailing list threads where
we hashed this out already ;)

The namespace serves a few purposes:

1 > We put futures where it makes sense - into a concurrent package.
Futures are a concurrency construct; therefore it simply makes sense
to put them within a sub package rather on the top level.

2 > We carve out a box to move to, and add other concurrent things,
such as generic pools, Actor implementations, etc. See
java.util.concurrent. Things within multiprocessing that don't start
with P and rhyme with "rocess" can go here too.

Admittedly, it's mainly my own long-term vision to see python-core
grow more concurrency tidbits - although I don't know too many people
who would complain about it.

jesse

From brian at sweetapp.com  Wed May 26 15:01:15 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 26 May 2010 23:01:15 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFD173B.9010703@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com>
Message-ID: <1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com>


On 26 May 2010, at 22:42, Nick Coghlan wrote:

> On 26/05/10 20:57, Greg Ewing wrote:
>> Having read through the PEP again, here are my thoughts.
>> * It seems unnecessarily verbose to tack "Executor"
>> onto the end of every Executor subclass. They could
>> simply be called ThreadPool and ProcessPool without
>> losing anything.
>
> We would lose the ability to add general purpose thread and process  
> pools under the obvious names later.
>
>> * I don't see a strong reason to put this module
>> inside a newly-created namespace. If there were a
>> namespace called "concurrent", I would expect to find
>> other existing concurrency-related modules there as
>> well, such as threading and multiprocessing. But we
>> can't move them there without breaking existing code.
>>
>> (More generally, I'm inclined to think that introducing
>> a namespace package for a category of modules having
>> existing members in the stdlib is an anti-pattern,
>> unless it's done during the kind of namespace refactoring
>> that we won't get another chance to perform until Py4k.)
>
> _thread, threading, Queue and multiprocessing do likely belong here,  
> but moving them isn't likely to be worth the pain. Does it help to  
> know that at least Jesse and I (and probably others) would like to  
> see concurrent.pool added eventually with robust general purpose  
> ThreadPool and ProcessPool implementations?
>
> The specific reason the new package namespace was added was to help  
> avoid confusion with stock market futures without using an unduly  
> cumbersome module name, but I don't know how well the PEP explains  
> that.

It doesn't at all. Are these plans formalized anywhere that I can link  
to?

Cheers,
Brian


From thomas at python.org  Wed May 26 15:02:41 2010
From: thomas at python.org (Thomas Wouters)
Date: Wed, 26 May 2010 15:02:41 +0200
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
Message-ID: <AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>

On Wed, May 26, 2010 at 11:48, Mark Dickinson <dickinsm at gmail.com> wrote:

> On Wed, May 26, 2010 at 10:15 AM, Colin H <hawkett at gmail.com> wrote:
> >   issue991196 was closed being described as intentional.  I've added
> > a comment in that issue which argues that this is a serious bug (also
> > aserted by a previous commenter - Armin Rigo), because it creates a
> > unique, undocumented, oddly behaving scope that doesn't apply closures
> > correctly. At the very least I think this should be acknowledged as a
> > plain old bug (rather than a feature), and then a discussion about
> > whether it will be fixed or not.
>
> Here's a quick recap of the issue so that people don't have to go
> searching through the bug archive.  In Python 2.x, we get the
> following behaviour:
>
> >>> code = """\
> ... y = 3
> ... def f():
> ...     return y
> ... f()
> ... """
> >>> exec code in {}   # works fine
> >>> exec code in {}, {}   # dies with a NameError
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "<string>", line 4, in <module>
>  File "<string>", line 3, in f
> NameError: global name 'y' is not defined
>
> The issue is whether the second example should work, given that two
> different dictionaries have been passed.
>
> The cause of the NameError can be seen by looking at the bytecode: y
> is bound using STORE_NAME, which stores y into the locals dictionary
> (which here is *not* the same as the globals dictionary) but the
> attempt to retrieve the value of y uses LOAD_GLOBAL, which only looks
> in the globals.
>
> >>> co = compile(code, 'mycode', 'exec')
> >>> dis.dis(co)
>  1           0 LOAD_CONST               0 (3)
>              3 STORE_NAME               0 (y)
>
>  2           6 LOAD_CONST               1 (<code object f at
> 0xa22b40, file "mycode", line 2>)
>              9 MAKE_FUNCTION            0
>             12 STORE_NAME               1 (f)
>
>  4          15 LOAD_NAME                1 (f)
>             18 CALL_FUNCTION            0
>             21 POP_TOP
>             22 LOAD_CONST               2 (None)
>             25 RETURN_VALUE
> >>> dis.dis(co.co_consts[1])  # disassembly of 'f'
>  3           0 LOAD_GLOBAL              0 (y)
>              3 RETURN_VALUE
>
> This is a long way from my area of expertise (I'm commenting here
> because it was me who sent Colin here in the first place), and it's
> not clear to me whether this is a bug, and if it is a bug, how it
> could be resolved.  What would the impact be of having the compiler
> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here?


It wouldn't matter. The 'f' function only knows about its own namespace
(separate from the surrounding code's local namespace), and the global
namespace. LOAD_NAME is only different from LOAD_GLOBAL in that it looks in
the local namespace first, but in this case the local namespace contains
nothing. Here's what happens:

'exec code in d1, d2' executes code with 'd1' as locals and 'd2' as globals.
Assignment always operates on the local namespace (barring the 'global'
declaration.) The function definition also assigns to the local namespace,
but the created function knows nothing about that local namespace -- it only
cares about its own namespace and the global namespace, 'd1'.

'exec code in d1' does the same thing as 'exec code in d1, d1': it uses the
same dict for the locals and the globals. The execution of the code doesn't
change -- the assignment to 'y' still assigns to the locals, and the 'f'
function still looks it up in globals, but now *they are the same dict*.
Using the same dict for locals and globals is how modules work, as well.

The main confusion here is the fact that 'exec' doesn't generate closures.
(Nobody was ever confused about this behaviour back in Python
2.0-and-earlier! :-) The reason for that is the disconnect between the
compiler and the exec statement: the compiler sees no enclosing function, so
it doesn't generate a closure. The exec statement, because it gets two
different namespaces, then executes it like a function, with a distinct
local namespace.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100526/0c536c93/attachment-0001.html>

From fuzzyman at voidspace.org.uk  Wed May 26 15:08:35 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Wed, 26 May 2010 14:08:35 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFD195E.2060007@gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<4BFD195E.2060007@gmail.com>
Message-ID: <4BFD1D53.2080502@voidspace.org.uk>

On 26/05/2010 13:51, Nick Coghlan wrote:
> On 26/05/10 19:48, Mark Dickinson wrote:
>> This is a long way from my area of expertise (I'm commenting here
>> because it was me who sent Colin here in the first place), and it's
>> not clear to me whether this is a bug, and if it is a bug, how it
>> could be resolved. What would the impact be of having the compiler
>> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here?
>
> exec with a single argument = module namespace
> exec with two arguments = class namespace
>
> Class namespaces are deliberately exempted from lexical scoping so 
> that methods can't see class attributes, hence the example in the 
> tracker issue works exactly as it would if the code was written as a 
> class body.
>
> class C:
> y = 3
> def execfunc():
> print y
> execfunc()
>
> With this code, y would end up in C.__dict__ rather than the module 
> globals (at least, it would if it wasn't for the exception) and the 
> call to execfunc fails with a NameError when attempting to find y.
>
> I know I've closed other bug reports that were based on the same 
> misunderstanding, and I didn't understand it myself until Guido 
> explained it to me a few years back, so suggestions for improving the 
> exec documentation in this area would be appreciated.

Your explanation here is very clear. Is this in the documentation?

Michael

>
> Cheers,
> Nick.
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From jnoller at gmail.com  Wed May 26 15:17:54 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 26 May 2010 09:17:54 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com>
	<1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com>
Message-ID: <AANLkTimozeO2FjcUihkNCeGJWxZ2KtTCakrFKGm5FPeT@mail.gmail.com>

On Wed, May 26, 2010 at 9:01 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>
> On 26 May 2010, at 22:42, Nick Coghlan wrote:
>
>> On 26/05/10 20:57, Greg Ewing wrote:
>>>
>>> Having read through the PEP again, here are my thoughts.
>>> * It seems unnecessarily verbose to tack "Executor"
>>> onto the end of every Executor subclass. They could
>>> simply be called ThreadPool and ProcessPool without
>>> losing anything.
>>
>> We would lose the ability to add general purpose thread and process pools
>> under the obvious names later.
>>
>>> * I don't see a strong reason to put this module
>>> inside a newly-created namespace. If there were a
>>> namespace called "concurrent", I would expect to find
>>> other existing concurrency-related modules there as
>>> well, such as threading and multiprocessing. But we
>>> can't move them there without breaking existing code.
>>>
>>> (More generally, I'm inclined to think that introducing
>>> a namespace package for a category of modules having
>>> existing members in the stdlib is an anti-pattern,
>>> unless it's done during the kind of namespace refactoring
>>> that we won't get another chance to perform until Py4k.)
>>
>> _thread, threading, Queue and multiprocessing do likely belong here, but
>> moving them isn't likely to be worth the pain. Does it help to know that at
>> least Jesse and I (and probably others) would like to see concurrent.pool
>> added eventually with robust general purpose ThreadPool and ProcessPool
>> implementations?
>>
>> The specific reason the new package namespace was added was to help avoid
>> confusion with stock market futures without using an unduly cumbersome
>> module name, but I don't know how well the PEP explains that.
>
> It doesn't at all. Are these plans formalized anywhere that I can link to?
>
> Cheers,
> Brian

Nope; and I don't think we need to worry about it right now.

From brian at sweetapp.com  Wed May 26 15:29:05 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 26 May 2010 23:29:05 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100526145033.3b61d95b@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4BF916EA.2090507@holdenweb.com>
	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
	<20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com>
	<20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com>
	<20100526145033.3b61d95b@pitrou.net>
Message-ID: <A3704B8E-2B63-4B97-90EA-B1137FF6566C@sweetapp.com>


On 26 May 2010, at 22:50, Antoine Pitrou wrote:

> On Wed, 26 May 2010 22:32:33 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>> Ha, I'm a bit surprised. Isn't it what "futures" already provides?
>>> (except that for some reason it insists on the "SomeExecutor" naming
>>> scheme)
>>> http://www.python.org/dev/peps/pep-3148/#processpoolexecutor
>>
>> Not really - a general purpose pool would be a lot more agnostic  
>> about
>> how you give the pooled threads/processes work to do and get the  
>> results
>> back.
>>
>> Executors are the kind of thing you would build on top of one  
>> though. If
>> concurrent.pool was added, then the existing processing pools in
>> multiprocessing and the executors in concurrent.future would be the
>> first use cases for it.
>
> I think I'm a bit ignorant, but how is the Executor abstraction (and
> its proposed implementations) not generic enough? You have a pool,
> submit one or several tasks, and can either repeatedly poll for
> completion or do a blocking wait.
>
> (after all, Glyph pointed out that it should be quite easy to wrap the
> resulting Futures into Deferred objects)

Interesting. Executor.submit() return a Future, which might not be  
useful in some ThreadPool fire-and-forget use cases but having them  
doesn't seem harmful.

Java does take this approach and it gives you a lot more ways to  
customize the Executor thread pool i.e. the minimum number of threads  
running, the maximum number, the amount of time that a thread can be  
idle before it is killed, the queueing strategy to use (e.g. LIFO,  
FIFO, priority).

Cheers,
Brian

From ncoghlan at gmail.com  Wed May 26 15:33:03 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 23:33:03 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com>
	<1F22277E-150D-43D1-A332-3DD1BC2B11A0@sweetapp.com>
Message-ID: <4BFD230F.8010100@gmail.com>

On 26/05/10 23:01, Brian Quinlan wrote:
>> _thread, threading, Queue and multiprocessing do likely belong here,
>> but moving them isn't likely to be worth the pain. Does it help to
>> know that at least Jesse and I (and probably others) would like to see
>> concurrent.pool added eventually with robust general purpose
>> ThreadPool and ProcessPool implementations?
>>
>> The specific reason the new package namespace was added was to help
>> avoid confusion with stock market futures without using an unduly
>> cumbersome module name, but I don't know how well the PEP explains that.
>
> It doesn't at all. Are these plans formalized anywhere that I can link to?

Just the previous lot of discussions. The main point that should be 
mentioned in the PEP is that "futures" on its own was ambiguous as to 
the applicable domain, but "concurrent.futures" was perfectly clear, 
without causing any readability problems the way a longer name could.

Moving the general purpose pools out to their own module was just an 
example that occurred to us as something else that could go in that 
package rather than a concrete plan for implementation.

Yes, we're setting ourselves up for inevitable questions as to why the 
existing modules are top level rather than part of this package, but the 
minimal pain response there would be to link to them from the package 
documentation with a note along the lines of "for historical reasons, 
some modules you might reasonably expect to find in this package are 
instead provided as top level modules".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May 26 15:43:35 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 23:43:35 +1000
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFD1D53.2080502@voidspace.org.uk>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<4BFD195E.2060007@gmail.com> <4BFD1D53.2080502@voidspace.org.uk>
Message-ID: <4BFD2587.7060903@gmail.com>

On 26/05/10 23:08, Michael Foord wrote:
> On 26/05/2010 13:51, Nick Coghlan wrote:
>> On 26/05/10 19:48, Mark Dickinson wrote:
>>> This is a long way from my area of expertise (I'm commenting here
>>> because it was me who sent Colin here in the first place), and it's
>>> not clear to me whether this is a bug, and if it is a bug, how it
>>> could be resolved. What would the impact be of having the compiler
>>> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here?
>>
>> exec with a single argument = module namespace
>> exec with two arguments = class namespace
>>
>> Class namespaces are deliberately exempted from lexical scoping so
>> that methods can't see class attributes, hence the example in the
>> tracker issue works exactly as it would if the code was written as a
>> class body.
>>
>> class C:
>> y = 3
>> def execfunc():
>> print y
>> execfunc()
>>
>> With this code, y would end up in C.__dict__ rather than the module
>> globals (at least, it would if it wasn't for the exception) and the
>> call to execfunc fails with a NameError when attempting to find y.
>>
>> I know I've closed other bug reports that were based on the same
>> misunderstanding, and I didn't understand it myself until Guido
>> explained it to me a few years back, so suggestions for improving the
>> exec documentation in this area would be appreciated.
>
> Your explanation here is very clear. Is this in the documentation?

It isn't actually - I think Thomas may be right that the current exec 
documentation was largely written prior to the introduction of lexical 
scoping. So adding something along these lines would probably be a good 
place to start.

Unfortunately, even my explanation above isn't 100% correct - it misses 
a subtle distinction where lexical scoping will pass through a class 
definition nested inside a function definition and see the function 
scope, but the use of strings for exec code means that the scopes of any 
containing functions are ignored completely.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed May 26 15:48:24 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 26 May 2010 23:48:24 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <A3704B8E-2B63-4B97-90EA-B1137FF6566C@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<4BF916EA.2090507@holdenweb.com>	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>	<20100523144754.1c6f1afc@pitrou.net>
	<4BFC6325.5050700@gmail.com>	<20100526093820.4fbf2f87@pitrou.net>
	<4BFD14E1.6000305@gmail.com>	<20100526145033.3b61d95b@pitrou.net>
	<A3704B8E-2B63-4B97-90EA-B1137FF6566C@sweetapp.com>
Message-ID: <4BFD26A8.9050802@gmail.com>

On 26/05/10 23:29, Brian Quinlan wrote:
> On 26 May 2010, at 22:50, Antoine Pitrou wrote:
>> I think I'm a bit ignorant, but how is the Executor abstraction (and
>> its proposed implementations) not generic enough? You have a pool,
>> submit one or several tasks, and can either repeatedly poll for
>> completion or do a blocking wait.
>>
>> (after all, Glyph pointed out that it should be quite easy to wrap the
>> resulting Futures into Deferred objects)
>
> Interesting. Executor.submit() return a Future, which might not be
> useful in some ThreadPool fire-and-forget use cases but having them
> doesn't seem harmful.
>
> Java does take this approach and it gives you a lot more ways to
> customize the Executor thread pool i.e. the minimum number of threads
> running, the maximum number, the amount of time that a thread can be
> idle before it is killed, the queueing strategy to use (e.g. LIFO, FIFO,
> priority).

I would say it is precisely that extra configurability which separates 
the executor pools in the PEP implementation from more flexible general 
purpose pools.

It's something to investigate somewhere along the line, but, as Jesse 
pointed out, not something we need to worry about specifically for this 
PEP (except as an example of another module that may eventually end up 
in the concurrent package)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From hawkett at gmail.com  Wed May 26 16:03:35 2010
From: hawkett at gmail.com (Colin H)
Date: Wed, 26 May 2010 15:03:35 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
Message-ID: <AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>

Thanks for the details on why the observed behaviour occurs - very
clear. My only query would be why this is considered correct? Why is
it running as a class namespace, when it is not a class? Is there any
reason why this is not considered a mistake? Slightly concerned that
this is being considered not a bug because 'it is how it is'.

A really good reason why you would want to provide a separate locals
dictionary is to get access to the stuff that was defined in the
exec()'d code block.  Unfortunately this use case is broken by the
current behaviour.  The only way to get the definitions from the
exec()'d code block is to supply a single dictionary, and then try to
weed out the definitions from amongst all the other globals, which is
very difficult if you don't know in advance what was in the code block
you exec()'d.

So put simply - the bug is that a class namespace is used, but its not a class.

On 26/05/2010 13:51, Nick Coghlan wrote:
> On 26/05/10 19:48, Mark Dickinson wrote:
>> This is a long way from my area of expertise (I'm commenting here
>> because it was me who sent Colin here in the first place), and it's
>> not clear to me whether this is a bug, and if it is a bug, how it
>> could be resolved. What would the impact be of having the compiler
>> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here?
>
> exec with a single argument = module namespace
> exec with two arguments = class namespace
>
> Class namespaces are deliberately exempted from lexical scoping so
> that methods can't see class attributes, hence the example in the
> tracker issue works exactly as it would if the code was written as a
> class body.
>
> class C:
> y = 3
> def execfunc():
> print y
> execfunc()
>
> With this code, y would end up in C.__dict__ rather than the module
> globals (at least, it would if it wasn't for the exception) and the
> call to execfunc fails with a NameError when attempting to find y.
>
> I know I've closed other bug reports that were based on the same
> misunderstanding, and I didn't understand it myself until Guido
> explained it to me a few years back, so suggestions for improving the
> exec documentation in this area would be appreciated.

From debatem1 at gmail.com  Wed May 26 17:40:56 2010
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 26 May 2010 11:40:56 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100526125614.0a7ae470@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
Message-ID: <AANLkTimrS3gEBQtBFnNzlcx7H30Y6yPZ-E7XrUErbElz@mail.gmail.com>

On Wed, May 26, 2010 at 6:56 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 26 May 2010 20:42:12 +1000
> Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> I'm not saying that Python-Dev should bend over backwards to accommodate
>> such people to the exclusion of all else, but these folks are
>> stakeholders too, and their wants and needs are just as worthy as the
>> wants and needs of those who prefer a more conservative approach to the
>> standard library.
>
> Well, my "Sumo" proposal was a serious one.
> (not serious in that I would offer to give a hand, but in that I think
> it could help those people; also, wouldn't it be sensible for users in
> a corporate environment to share their efforts and produce something
> that can benefit all of them? it's the free software spirit after all)

Not in a corporate environment, but I would gladly help with this.

Geremy Condra

From debatem1 at gmail.com  Wed May 26 18:22:04 2010
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 26 May 2010 09:22:04 -0700
Subject: [Python-Dev] Sumo
In-Reply-To: <1274877963.3162.20.camel@localhost.localdomain>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
Message-ID: <AANLkTik6gGTK7KUJnmeFb5J8dijqdt8Dk0QIZlKg0Izw@mail.gmail.com>

On Wed, May 26, 2010 at 5:46 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le mercredi 26 mai 2010 ? 13:19 +0100, Paul Moore a ?crit :
>>
>> I'm not sure how a "Sumo" approach would work in practical terms, and
>> this thread isn't really the place to discuss, but there's a couple of
>> points I think are worth making:
>>
>> * For a "Sumo" distribution to make sense, some relatively substantial
>> chunk of the standard library would need to be moved *out* to reside
>> in the sumo distribution. Otherwise it's not really a "sumo", just a
>> couple of modules that "nearly made it into the stdlib", at least for
>> the near-to-medium term.
>
> This is not what I'm suggesting at all. The stdlib wouldn't shrink
> (well, we could dump outdated modules but that's a separate decision).
>
> A hypothetical "Sumo" distribution would be made of those more or less
> useful modules that many people (application or framework developers;
> for example, take a look at the dozens of direct and indirect
> dependencies TurboGears pulls in) need and install.
>
> The whole point is that it would *not* be supported by python-dev itself
> but by a separate group of people (such as you :-)).
> And it would have its own rules (surely more relaxed) over inclusion or
> deprecation delays, the ability to make compatibility-breaking changes,
> release timings, multiple obvious ways to do the same thing, etc.
>
> And it means it could really bundle a lot of third-party stuff: small
> helpers (things like the decorator module), event loops, template
> engines, network address abstractions, argument parsers, ORMs, UI
> toolkits, etc.
>
> (a side-effect would be that it could, if it works well, behave as a
> good intermediate stress test - a purgatory, if you want - for modules
> before they integrate the stdlib)
>
> If you want an existing analogy, you could look at EasyPHP. Or think of
> Python as the Gnome or KDE project (consistent and aiming at providing
> the most important everyday tools, but quite focused), and "Sumo" as an
> entire distribution of disparate Linux GUI apps.
>
> Regards
>
> Antoine.

I'd also point out that creating a sumo distribution would focus attention
on the need to port those libraries which were a part of it to python3,
which would help to weaken the argument that there aren't any packages
written for it.

Geremy Condra

From tjreedy at udel.edu  Wed May 26 18:27:17 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 26 May 2010 12:27:17 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFD173B.9010703@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4BFD173B.9010703@gmail.com>
Message-ID: <htji55$qui$1@dough.gmane.org>


> On 26/05/10 20:57, Greg Ewing wrote:

>> (More generally, I'm inclined to think that introducing
>> a namespace package for a category of modules having
>> existing members in the stdlib is an anti-pattern,

As a user, I agree with this.

>> unless it's done during the kind of namespace refactoring
>> that we won't get another chance to perform until Py4k.)

Is Steven D'Aprano's suggestion (in another post) possible?
> I'm sure that it can be done easily, although not quickly. For
> instance, we move threading into the concurrent namespace, and leave
> behind in its place a stub:
>
> from concurrent.threading import *
>
> Then for (say) 3.3 the stub could gain a PendingDeprecation warning,
> then in 3.4 a Deprecation warning, and finally in 3.5 or 3.6 it
> could be removed.

On 5/26/2010 8:42 AM, Nick Coghlan wrote:
> _thread, threading, Queue and multiprocessing do likely belong here, but
> moving them isn't likely to be worth the pain.

As a user, I disagree and think collecting together these and future 
modules (pun intended) would be beneficial. There are, from my 
viewpoint, too many top level modules already.

[and in another thread]
> Yes, we're setting ourselves up for inevitable questions as to why
> the existing modules are top level rather than part of this package,

Yes, forever until they are put in the package.

> but the minimal pain responsethere

For the developers, for the short term

> would be to link to them from the
> package documentation with a note along the lines of "for historical
> reasons, some modules you might reasonably expect to find in this
> package are instead provided as top level modules".

You are, in my opinion, overly discounting the cognitive load and 
confusion on the part of new users. It would be much better to link *to* 
subpackage documentation *from* a top level entries (until deleted) and 
just say that the top level synonyms are present for the obvious 
historical reason that there once no package, just modules.


I am suggesting that if we add a package, we do it right, from the 
beginning.

Terry Jan Reedy






From jyasskin at gmail.com  Wed May 26 18:25:53 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Wed, 26 May 2010 09:25:53 -0700
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFCFE8B.1080205@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com> 
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com> 
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com> 
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com> 
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com> 
	<4BFCFE8B.1080205@canterbury.ac.nz>
Message-ID: <AANLkTila167A43c6gm-0rQP_SdqBvQ52MAlchn_sisek@mail.gmail.com>

On Wed, May 26, 2010 at 3:57 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Having read through the PEP again, here are my thoughts.
>
> * I'm bothered by the term "future". To my mind, it's
> too long on cleverness and too short on explanativeness.
>
> I think that the standard library is no place for
> cuteness of naming. The name of a stdlib module should
> reflect its functionality in some straightforward and
> obvious way. If I were looking for a thread pool or
> process pool implementation, the word "future" is not
> something that would spring readily to mind.

Please go re-read the older threads on this. For example,
http://mail.python.org/pipermail/python-dev/2010-March/098279.html.

> * It seems unnecessarily verbose to tack "Executor"
> onto the end of every Executor subclass. They could
> simply be called ThreadPool and ProcessPool without
> losing anything.

+0

> * I don't see a strong reason to put this module
> inside a newly-created namespace. If there were a
> namespace called "concurrent", I would expect to find
> other existing concurrency-related modules there as
> well, such as threading and multiprocessing. But we
> can't move them there without breaking existing code.

Again, please re-read the older thread (which you participated in).
For example, http://mail.python.org/pipermail/python-dev/2010-March/098200.html.

Jeffrey

From tjreedy at udel.edu  Wed May 26 19:40:43 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 26 May 2010 13:40:43 -0400
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
Message-ID: <htjmer$h7g$1@dough.gmane.org>

Mark Dickinson wrote (with interactice prompts removed)

code = """\
y = 3
def f():
   return y
. f()
"""
exec code in {}   # works fine
exec code in {}, {}   # dies with a NameError

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<string>", line 4, in <module>
   File "<string>", line 3, in f
NameError: global name 'y' is not defined

I verified that 3.1 (with exec adjusted to be a function) operates the same.

On 5/26/2010 8:51 AM, Nick Coghlan wrote:
 >exec with a single argument = module namespace
 >exec with two arguments = class namespace

I verified in 3.1 that indenting 'code' and prepending 'class C():\n' 
gives the same error and that prepending 'def f():\n' now, with nexted 
function namespaces, does not give an error, although it would have been 
an error before Python 2.2, when there were no nested function namespaces.

On 5/26/2010 10:03 AM, Colin H wrote:
> Thanks for the details on why the observed behaviour occurs - very
> clear. My only query would be why this is considered correct? Why is
> it running as a class namespace, when it is not a class?

You are expecting that it run as a function namespace (with post 2.2 
nesting), when it is not a function. Why is that any better?

> Is there any
> reason why this is not considered a mistake? Slightly concerned that
> this is being considered not a bug because 'it is how it is'.

In original Python, the snippet would have given an error whether you 
thought of it as being in a class or function context, which is how 
anyone who knew Python then would have expected. Consistency is not a bug.

When nested function namespaces were introduced, the behavior of exec 
was left unchanged. Backward compatibility is not a bug.

A change could have been proposed for 3.x, but I do not remember such a 
discussion and expect it would have been rejected. One can get the 
nested function behavior by doing what I did in the test mentioned 
above. One could easily write a nested_exec function to do the wrapping 
automatically.

-----

In http://bugs.python.org/issue8824

I suggest that

"In all cases, if the optional parts are omitted, the code is executed 
in the current scope. If only globals is provided, it must be a 
dictionary, which will be used for both the global and the local 
variables. If globals  and locals are given, they are used for the 
global and local variables, respectively. If provided, locals can be any 
mapping object."

be followed by

"If only globals is provided or if onedict is provided as both globals 
and locals, the code is executed in a new top-level scope. If different 
objects are given as globals and locals, the code is executed as if it 
were in a class statement in a new top-level scope."

to make the behavior clearer.

Terry Jan Reedy


From hawkett at gmail.com  Wed May 26 22:07:52 2010
From: hawkett at gmail.com (Colin H)
Date: Wed, 26 May 2010 21:07:52 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
Message-ID: <AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>

The changes to the docs will definitely help in understanding why this
behaves as it does. I would like like to take one last stab though at
justifying why this behaviour isn't correct - will leave it alone if
these arguments don't stack up :)  Appreciate the input and
discussion.

Terry Jan Reedy wrote

> You are expecting that it run as a function namespace (with post 2.2
> nesting), when it is not a function. Why is that any better?

Because a class namespace (as I see it) was implemented to treat a
specific situation - i.e. that functions in classes cannot see class
variables. exec() is a far more generic instrument that has no such
explicit requirement - i.e. it feels like hijacking an edge case to
meet a requirement that doesn't exist. However 'all locals in an
enclosing scope are made available in the function namespace' is
generally understood as python's generic closure implementation, and
would match more effectively the generic nature of the exec()
statement.  A litmus test for this sort of thing - if you polled 100
knowledgeable python devs who hadn't encountered this problem or this
thread and asked if they would expect exec() to run as a class or
function namespace, I think you'd struggle to get 1 of them to expect
a class namespace. Functions are the more generic construct, and thus
more appropriate for the generic nature of exec() (IMHO).

It would appear that the only actual requirement not to make locals in
an enclosing scope available in a nested function scope is for a
class. The situation we are discussing seems have created a similar
requirement for exec(), but with no reason.

> In original Python, the snippet would have given an error whether you
> thought of it as being in a class or function context, which is how
> anyone who knew Python then would have expected. Consistency is not a bug.

> When nested function namespaces were introduced, the behavior of exec
> was left unchanged. Backward compatibility is not a bug.

Generally, most other behaviour did change - locals in enclosing
scopes *did* become available in the nested function namespace, which
was not backward compatible.  Why is a special case made to retain
consistency and backward compatibility for code run using exec()? It's
all python code. Inconsistent backward compatibility might be
considered a bug.

Cheers,

Colin

From p.f.moore at gmail.com  Thu May 27 00:41:51 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 26 May 2010 23:41:51 +0100
Subject: [Python-Dev] Sumo
In-Reply-To: <1274877963.3162.20.camel@localhost.localdomain>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
Message-ID: <AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>

On 26 May 2010 13:46, Antoine Pitrou <solipsis at pitrou.net> wrote:
> This is not what I'm suggesting at all. The stdlib wouldn't shrink
> (well, we could dump outdated modules but that's a separate decision).

Ah, OK. In that case, I see the argument for a "Sumo" distribution as
weak for a different reason - for general use, the standard library is
(nearly!) sufficient, and ignoring specialised use cases, there aren't
enough generally useful modules to warrant a "Sumo" distribution
(you'd essentially be talking about stuff that "nearly made it into
the stdlib", and there's not a huge amount of that).

Specialised distributions are another matter - I can see a "web stack"
distribution comprising your TurboGears example (or should it be
Django, or...?). Enthought essentially do that for a "Scientific
Python" distribution. There could easily be others. But a general
purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical.
(Personally, my "essential extras" are pywin32, cx_Oracle and that's
about it - futures might make it if it doesn't get into the stdlib,
but that's about all).

I'm genuinely struggling to see how a Sumo distribution ever comes
into being under your proposal. There's no evidence that anyone wants
it (otherwise it would have been created by now!!) and until it
exists, it's not a plausible "place" to put modules that don't make it
into the stdlib. So (unless I'm missing something) your argument seems
to be that if enough good stuff is rejected for stdlib inclusion, this
will prompt the people who wanted that stuff included to create a sumo
distribution, which addresses the "too many dependencies is bad"
argument for inclusion in the stdlib. That sounds like a suspiciously
circular argument to me...

Paul.

From solipsis at pitrou.net  Thu May 27 00:51:19 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 27 May 2010 00:51:19 +0200
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
Message-ID: <1274914279.3162.76.camel@localhost.localdomain>

Le mercredi 26 mai 2010 ? 23:41 +0100, Paul Moore a ?crit :
> 
> But a general
> purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical.
> (Personally, my "essential extras" are pywin32, cx_Oracle and that's
> about it - futures might make it if it doesn't get into the stdlib,
> but that's about all).

Well, unless you are short on megabytes or on download bandwidth, would
you really care to get some modules you never use?
(there are probably some in the stdlib too!)

> I'm genuinely struggling to see how a Sumo distribution ever comes
> into being under your proposal. There's no evidence that anyone wants
> it (otherwise it would have been created by now!!) and until it
> exists, it's not a plausible "place" to put modules that don't make it
> into the stdlib.

I don't think all package owners dream of putting their software in the
stdlib. It's likely some don't care, and it's likely some would actively
refuse it (because e.g. they don't want to lose control).
So the suggestion is not to somehow salvage packages which "don't make
it into the stdlib", but to build a broader distribution of packages
without necessarily having people bow to the constraints of stdlib
inclusion.

Of course, I agree with you that someone has to do it if they want it to
happen :-)

Regards

Antoine.



From debatem1 at gmail.com  Thu May 27 01:11:27 2010
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 26 May 2010 16:11:27 -0700
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
Message-ID: <AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>

On Wed, May 26, 2010 at 3:41 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 26 May 2010 13:46, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> This is not what I'm suggesting at all. The stdlib wouldn't shrink
>> (well, we could dump outdated modules but that's a separate decision).
>
> Ah, OK. In that case, I see the argument for a "Sumo" distribution as
> weak for a different reason - for general use, the standard library is
> (nearly!) sufficient, and ignoring specialised use cases, there aren't
> enough generally useful modules to warrant a "Sumo" distribution
> (you'd essentially be talking about stuff that "nearly made it into
> the stdlib", and there's not a huge amount of that).
>
> Specialised distributions are another matter - I can see a "web stack"
> distribution comprising your TurboGears example (or should it be
> Django, or...?). Enthought essentially do that for a "Scientific
> Python" distribution. There could easily be others. But a general
> purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical.
> (Personally, my "essential extras" are pywin32, cx_Oracle and that's
> about it - futures might make it if it doesn't get into the stdlib,
> but that's about all).

I'm not clear, you seem to be arguing that there's a market for many
augmented python distributions but not one. Why not just have one
that includes the best from each domain?

> I'm genuinely struggling to see how a Sumo distribution ever comes
> into being under your proposal. There's no evidence that anyone wants
> it (otherwise it would have been created by now!!)

Everything worth making has already been made?

> and until it exists, it's not a plausible "place" to put modules that don't
> make it into the stdlib.

Of course its implausible to put something somewhere that
doesn't exist... until it does.

> So (unless I'm missing something) your argument seems
> to be that if enough good stuff is rejected for stdlib inclusion, this
> will prompt the people who wanted that stuff included to create a sumo
> distribution, which addresses the "too many dependencies is bad"
> argument for inclusion in the stdlib. That sounds like a suspiciously
> circular argument to me...

I'd say rather that there are a large number of specialized tools which
aren't individually popular enough to be included in Python, but which
when taken together greatly increase its utility, and that sumo offers a
way to provide that additional utility to python's users without forcing
python core devs to shoulder the maintenance burden.

Geremy Condra

From greg.ewing at canterbury.ac.nz  Thu May 27 01:36:07 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 11:36:07 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
Message-ID: <4BFDB067.4060800@canterbury.ac.nz>

Brian Quinlan wrote:

> I think that Jesse was planning to add some functionality to this  
> namespace.

Even if that happens, the existing threading and multiprocessing
modules would remain outside of it.

> You could have general thread pools that aren't related to executors

Yes, but it should be fairly obvious that the ones defined
in the futures module have to do with futures. Namespaces are
only a honking great idea if you actually let them do the job
they're designed for.

> I thought that the specification would be difficult to follow without  
> examples to pave the way.

Well, for me, what happened was that I saw the examples and
thought "WTF is going on here?" Then I read the specification
to figure out how the examples worked.

It might be better to have a tutorial section preceeding the
specification section, containing explanation interspersed
with examples.

 > I think that
> idiomatic future use will end up looking similar to my examples.

Maybe, but code written for pedagogical purposes needs to
meet a particularly high standard of clarity. Remember that
the reader isn't yet familiar with the idioms, so idiomatic
code isn't necessarily going to be easy for him to follow.

>> * Is it possible to have more than one Executor active
>> at a time?
> 
> Of course.

That's good, but I don't think that the "of course" is at
all obvious, considering that things such as GUI event loops
generally can't be mixed easily.

-- 
Greg



From greg.ewing at canterbury.ac.nz  Thu May 27 01:38:50 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 11:38:50 +1200
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
Message-ID: <4BFDB10A.4080002@canterbury.ac.nz>

Mark Dickinson wrote:

>>>>code = """\
> 
> ... y = 3
> ... def f():
> ...     return y
> ... f()
> ... """
> 
>>>>exec code in {}   # works fine
>>>>exec code in {}, {}   # dies with a NameError

Seems to me the whole idea of being able to specify
separate global and local scopes for top-level code is
screwy in the first place. Are there any use cases for
it? Maybe the second scope argument to exec() should
be deprecated?

-- 
Greg

From fuzzyman at voidspace.org.uk  Thu May 27 01:33:46 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 27 May 2010 00:33:46 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFDB10A.4080002@canterbury.ac.nz>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<4BFDB10A.4080002@canterbury.ac.nz>
Message-ID: <4BFDAFDA.7060404@voidspace.org.uk>

On 27/05/2010 00:38, Greg Ewing wrote:
> Mark Dickinson wrote:
>
>>>>> code = """\
>>
>> ... y = 3
>> ... def f():
>> ... return y
>> ... f()
>> ... """
>>
>>>>> exec code in {} # works fine
>>>>> exec code in {}, {} # dies with a NameError
>
> Seems to me the whole idea of being able to specify
> separate global and local scopes for top-level code is
> screwy in the first place. Are there any use cases for
> it? Maybe the second scope argument to exec() should
> be deprecated?
>
Sounds good to me, certainly ends the confusion over this undoubtedly 
unintuitive behaviour. :-)

Michael


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From guido at python.org  Thu May 27 01:44:02 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 May 2010 16:44:02 -0700
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFDAFDA.7060404@voidspace.org.uk>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<4BFDB10A.4080002@canterbury.ac.nz> <4BFDAFDA.7060404@voidspace.org.uk>
Message-ID: <AANLkTinZBeyaGVomnLNHnvd_PLcMj44QboxrerFJwtA3@mail.gmail.com>

Let me quickly jump in before someone actually deletes the feature.
Nick Coghlan and Thomas Wouters had it right; there is still a use
case. Don't kill it -- documenting it better is of course fine.

It *might* be possible to add a closure to the definition of f in the
case where globals != locals, but I doubt that that would be worth it,
and there's probably some code out there that would actually break, so
I'd say this is not a priority.

--Guido

On Wed, May 26, 2010 at 4:33 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> On 27/05/2010 00:38, Greg Ewing wrote:
>>
>> Mark Dickinson wrote:
>>
>>>>>> code = """\
>>>
>>> ... y = 3
>>> ... def f():
>>> ... return y
>>> ... f()
>>> ... """
>>>
>>>>>> exec code in {} # works fine
>>>>>> exec code in {}, {} # dies with a NameError
>>
>> Seems to me the whole idea of being able to specify
>> separate global and local scopes for top-level code is
>> screwy in the first place. Are there any use cases for
>> it? Maybe the second scope argument to exec() should
>> be deprecated?
>>
> Sounds good to me, certainly ends the confusion over this undoubtedly
> unintuitive behaviour. :-)
>
> Michael
>
>
> --
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
> READ CAREFULLY. By accepting and reading this email you agree, on behalf of
> your employer, to release me from all obligations and waivers arising from
> any and all NON-NEGOTIATED agreements, licenses, terms-of-service,
> shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure,
> non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have
> entered into with your employer, its partners, licensors, agents and
> assigns, in perpetuity, without prejudice to my ongoing rights and
> privileges. You further represent that you have the authority to release me
> from any BOGUS AGREEMENTS on behalf of your employer.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Thu May 27 01:57:26 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 May 2010 09:57:26 +1000
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>	<20100526113915.5bf46665@pitrou.net>	<201005262042.12825.steve@pearwood.info>	<20100526125614.0a7ae470@pitrou.net>	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>	<1274877963.3162.20.camel@localhost.localdomain>	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
Message-ID: <4BFDB566.2060502@gmail.com>

On 27/05/10 09:11, geremy condra wrote:
>> Specialised distributions are another matter - I can see a "web stack"
>> distribution comprising your TurboGears example (or should it be
>> Django, or...?). Enthought essentially do that for a "Scientific
>> Python" distribution. There could easily be others. But a general
>> purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical.
>> (Personally, my "essential extras" are pywin32, cx_Oracle and that's
>> about it - futures might make it if it doesn't get into the stdlib,
>> but that's about all).
>
> I'm not clear, you seem to be arguing that there's a market for many
> augmented python distributions but not one. Why not just have one
> that includes the best from each domain?

Because scientists, financial analysts, web designers, etc all have 
different needs.

A targeted distribution like Scientific Python will include nearly all 
the stuff a scientist is likely to need, but a financial analyst or web 
designer would find it lacking.

As Paul points out, the current size of the set of modules that are 
sufficiently general purpose and of high enough quality to qualify for 
python-dev's blessing, but wouldn't be suitable for inclusion in the 
normal standard library is fairly small. Particular when most developers 
are able to get sufficiently valuable modules from PyPI if they 
genuinely need them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu May 27 02:03:23 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 May 2010 10:03:23 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <htji55$qui$1@dough.gmane.org>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>	<4BFD173B.9010703@gmail.com>
	<htji55$qui$1@dough.gmane.org>
Message-ID: <4BFDB6CB.7030104@gmail.com>

On 27/05/10 02:27, Terry Reedy wrote:
> I am suggesting that if we add a package, we do it right, from the
> beginning.

This is a reasonable point of view, but I wouldn't want to hold up PEP 
3148 over it (call it a +0 for the idea in general, but a -1 for linking 
it to the acceptance of PEP 3148).

A separate short PEP proposing a migration plan that could be accepted 
or rejected independently of PEP 3148 would likely be valuable.

E.g.
  - no change in 2.x (obviously)
  - add concurrent.* alternate names in 3.x
  - rearrange documentation in 3.x, with pointers from old names to new 
names
  - put a PendingDeprecationWarning on the old names, but otherwise 
leave them alone indefinitely
  - add 2to3 fixers to translate from the old names to the new names in 
import statements

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From jnoller at gmail.com  Thu May 27 02:04:46 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 26 May 2010 20:04:46 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFDB067.4060800@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<4BFDB067.4060800@canterbury.ac.nz>
Message-ID: <AANLkTinmitR_ateAN0cgcQpUWUnrFAsO6tlXDdBc3Bgb@mail.gmail.com>

On Wed, May 26, 2010 at 7:36 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Brian Quinlan wrote:
>
>> I think that Jesse was planning to add some functionality to this
>> ?namespace.
>
> Even if that happens, the existing threading and multiprocessing
> modules would remain outside of it.

Not entirely; once concurrent.* comes into existence, I will seriously
begin looking at what we can move out of multiprocessing, into
concurrent.* alongside futures.

>> You could have general thread pools that aren't related to executors
>
> Yes, but it should be fairly obvious that the ones defined
> in the futures module have to do with futures. Namespaces are
> only a honking great idea if you actually let them do the job
> they're designed for.

concurrent.* is the namespace, futures is the package within the
namespace - concurrent.futures is highly descriptive of the items
contained therein.

jesse

From jnoller at gmail.com  Thu May 27 02:05:28 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Wed, 26 May 2010 20:05:28 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFDB6CB.7030104@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com>
	<htji55$qui$1@dough.gmane.org> <4BFDB6CB.7030104@gmail.com>
Message-ID: <AANLkTinUdjT2KheM12F4_pJIAfUNjLMU9MaRS-G2IeNS@mail.gmail.com>

On Wed, May 26, 2010 at 8:03 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27/05/10 02:27, Terry Reedy wrote:
>>
>> I am suggesting that if we add a package, we do it right, from the
>> beginning.
>
> This is a reasonable point of view, but I wouldn't want to hold up PEP 3148
> over it (call it a +0 for the idea in general, but a -1 for linking it to
> the acceptance of PEP 3148).
>
> A separate short PEP proposing a migration plan that could be accepted or
> rejected independently of PEP 3148 would likely be valuable.
>
> E.g.
> ?- no change in 2.x (obviously)
> ?- add concurrent.* alternate names in 3.x
> ?- rearrange documentation in 3.x, with pointers from old names to new names
> ?- put a PendingDeprecationWarning on the old names, but otherwise leave
> them alone indefinitely
> ?- add 2to3 fixers to translate from the old names to the new names in
> import statements
>
> Cheers,
> Nick.

Agreed; and intended as a different PEP.

From ncoghlan at gmail.com  Thu May 27 02:12:30 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 May 2010 10:12:30 +1000
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>
Message-ID: <4BFDB8EE.4010307@gmail.com>

On 27/05/10 06:07, Colin H wrote:
>> In original Python, the snippet would have given an error whether you
>> thought of it as being in a class or function context, which is how
>> anyone who knew Python then would have expected. Consistency is not a bug.
>
>> When nested function namespaces were introduced, the behavior of exec
>> was left unchanged. Backward compatibility is not a bug.
>
> Generally, most other behaviour did change - locals in enclosing
> scopes *did* become available in the nested function namespace, which
> was not backward compatible.  Why is a special case made to retain
> consistency and backward compatibility for code run using exec()? It's
> all python code. Inconsistent backward compatibility might be
> considered a bug.

Because strings are opaque to the compiler. The lexical scoping has *no 
idea* what is inside the string, and the exec operation only has 3 
things available to it:
  - the code object compiled from the string
  - the supplied globals namespace
  - the supplied locals namespace

It isn't a special case, it's the only way it can possible work.

Consider a more complex example:

   def get_exec_str():
     y = 3
     return "print(y)"

   exec(get_exec_str())

Should that code work?

Or consider this one:

   def get_exec_str():
     y = 3
     return "print y"

   def run_exec_str(str_to_run):
     y = 5
     exec(str_to_run)

   run_exec_str(get_exec_str())

Should that work? If yes, should it print 3 or 5?

Lexical scoping only works for code that is compiled as part of a single 
operation - the separation between the compilation of the individual 
string and the code defining that string means that the symbol table 
analysis needed for lexical scoping can't cross the boundary.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu May 27 02:19:50 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 27 May 2010 10:19:50 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFDB067.4060800@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<4BFDB067.4060800@canterbury.ac.nz>
Message-ID: <4BFDBAA6.2010104@gmail.com>

On 27/05/10 09:36, Greg Ewing wrote:
> Brian Quinlan wrote:
>
>> I think that Jesse was planning to add some functionality to this
>> namespace.
>
> Even if that happens, the existing threading and multiprocessing
> modules would remain outside of it.
>
>> You could have general thread pools that aren't related to executors
>
> Yes, but it should be fairly obvious that the ones defined
> in the futures module have to do with futures. Namespaces are
> only a honking great idea if you actually let them do the job
> they're designed for.

futures.ThreadPoolExecutor would likely be refactored to inherit from 
the mooted pool.ThreadPool. I'd like to leave that option open, and 
having two classes with the same name from different modules in a single 
inheritance tree is one of the places where module namespacing still 
isn't quite as tidy as we might wish.

I'd also consider a simple thread pool and an actual executor different 
things. I'm fine with the longer names, but if I was going to drop a 
word from the names, it would actually be "Pool" (i.e. ThreadExecutor, 
ProcessExecutor).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From debatem1 at gmail.com  Thu May 27 02:15:41 2010
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 26 May 2010 17:15:41 -0700
Subject: [Python-Dev] Sumo
In-Reply-To: <4BFDB566.2060502@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
	<4BFDB566.2060502@gmail.com>
Message-ID: <AANLkTilrTep2PPYtVPwegXHmzMI5hJl-qE0I670Qzxit@mail.gmail.com>

On Wed, May 26, 2010 at 4:57 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27/05/10 09:11, geremy condra wrote:
>>>
>>> Specialised distributions are another matter - I can see a "web stack"
>>> distribution comprising your TurboGears example (or should it be
>>> Django, or...?). Enthought essentially do that for a "Scientific
>>> Python" distribution. There could easily be others. But a general
>>> purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical.
>>> (Personally, my "essential extras" are pywin32, cx_Oracle and that's
>>> about it - futures might make it if it doesn't get into the stdlib,
>>> but that's about all).
>>
>> I'm not clear, you seem to be arguing that there's a market for many
>> augmented python distributions but not one. Why not just have one
>> that includes the best from each domain?
>
> Because scientists, financial analysts, web designers, etc all have
> different needs.

My point is just that a web designer probably doesn't care if he's
got numpy, nor does a mathematician care if he has cherrypy
onboard. They only care when the tools they need aren't there,
which is where sumo can help.

> A targeted distribution like Scientific Python will include nearly all the
> stuff a scientist is likely to need, but a financial analyst or web designer
> would find it lacking.

Seems like the same point as above, I might be missing something.

> As Paul points out, the current size of the set of modules that are
> sufficiently general purpose and of high enough quality to qualify for
> python-dev's blessing, but wouldn't be suitable for inclusion in the normal
> standard library is fairly small. Particular when most developers are able
> to get sufficiently valuable modules from PyPI if they genuinely need them.

Seems like the point is not to focus on the general purpose ones, but rather
to include domain or task specific libraries, and libraries that are close to
(but not at) the level where they would be considered for inclusion.

Geremy Condra

From solipsis at pitrou.net  Thu May 27 02:29:35 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 27 May 2010 02:29:35 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<4BFDB067.4060800@canterbury.ac.nz> <4BFDBAA6.2010104@gmail.com>
Message-ID: <20100527022935.2cfd04c5@pitrou.net>

On Thu, 27 May 2010 10:19:50 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> futures.ThreadPoolExecutor would likely be refactored to inherit from 
> the mooted pool.ThreadPool.

There still doesn't seem to be reason to have two different thread pool
APIs, though. Shouldn't there be one obvious way to do it?

> I'd also consider a simple thread pool and an actual executor different 
> things. I'm fine with the longer names, but if I was going to drop a 
> word from the names, it would actually be "Pool" (i.e. ThreadExecutor, 
> ProcessExecutor).

To me, ThreadPool looks a lot more obvious than ThreadExecutor
("obvious" in that I can easily find it again, and I don't need to
read some documentation to know what it is).

Regards

Antoine.



From hawkett at gmail.com  Thu May 27 02:37:52 2010
From: hawkett at gmail.com (Colin H)
Date: Thu, 27 May 2010 01:37:52 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTinSKzyWPOqY1oyxqxMo_UZ2oUzsGW3XxcTeJPhN@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTinSKzyWPOqY1oyxqxMo_UZ2oUzsGW3XxcTeJPhN@mail.gmail.com>
Message-ID: <AANLkTilIg9O5DPpft2XkQiU1UFXKXU8jDwC_Ogi92vsF@mail.gmail.com>

Mark Dickinson wrote:

> Seems to me the whole idea of being able to specify
> separate global and local scopes for top-level code is
> screwy in the first place. Are there any use cases for
> it? Maybe the second scope argument to exec() should
> be deprecated?

It is running as class namespace that makes the argument that there's
no use case. I agree - I can't think of any decent use cases for
exec() as class namespace - defining functions and classes only works
for a subset of function and class definitions

However, if exec() ran as function namespace instead, then the locals
dictionary will contain all the definitions from the exec()'d code
block, and only those definitions. Very useful. This is a major use
case for exec() - defining code from strings (e.g. enabling you to
store python code in the database), and using it at runtime. It seems
to me this must have been the point of locals in the first place.

If you just use globals, then your definitions exist amongst a whole
bunch of other python stuff, and unless you know in advance what was
defined in your code block, its very difficult to extract them.

From guido at python.org  Thu May 27 02:38:04 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 May 2010 17:38:04 -0700
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFDB8EE.4010307@gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
Message-ID: <AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>

On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27/05/10 06:07, Colin H wrote:
>>>
>>> In original Python, the snippet would have given an error whether you
>>> thought of it as being in a class or function context, which is how
>>> anyone who knew Python then would have expected. Consistency is not a
>>> bug.
>>
>>> When nested function namespaces were introduced, the behavior of exec
>>> was left unchanged. Backward compatibility is not a bug.
>>
>> Generally, most other behaviour did change - locals in enclosing
>> scopes *did* become available in the nested function namespace, which
>> was not backward compatible. ?Why is a special case made to retain
>> consistency and backward compatibility for code run using exec()? It's
>> all python code. Inconsistent backward compatibility might be
>> considered a bug.
>
> Because strings are opaque to the compiler. The lexical scoping has *no
> idea* what is inside the string, and the exec operation only has 3 things
> available to it:
> ?- the code object compiled from the string
> ?- the supplied globals namespace
> ?- the supplied locals namespace
>
> It isn't a special case, it's the only way it can possible work.
>
> Consider a more complex example:
>
> ?def get_exec_str():
> ? ?y = 3
> ? ?return "print(y)"
>
> ?exec(get_exec_str())
>
> Should that code work?
>
> Or consider this one:
>
> ?def get_exec_str():
> ? ?y = 3
> ? ?return "print y"
>
> ?def run_exec_str(str_to_run):
> ? ?y = 5
> ? ?exec(str_to_run)
>
> ?run_exec_str(get_exec_str())
>
> Should that work? If yes, should it print 3 or 5?
>
> Lexical scoping only works for code that is compiled as part of a single
> operation - the separation between the compilation of the individual string
> and the code defining that string means that the symbol table analysis
> needed for lexical scoping can't cross the boundary.

Hi Nick,

I don't think Colin was asking for such things. His use case is
clarify by this post of his:

On Wed, May 26, 2010 at 7:03 AM, Colin H <hawkett at gmail.com> wrote:
> A really good reason why you would want to provide a separate locals
> dictionary is to get access to the stuff that was defined in the
> exec()'d code block.  Unfortunately this use case is broken by the
> current behaviour.  The only way to get the definitions from the
> exec()'d code block is to supply a single dictionary, and then try to
> weed out the definitions from amongst all the other globals, which is
> very difficult if you don't know in advance what was in the code block
> you exec()'d.

I think here's an example of what he's asking for

def define_stuff(user_code):
  context = {'FOO': 42, 'BAR': 10**100}
  stuff = {}
  exec(user_code, context, stuff)
  return stuff

In some other place, define_stuff() is called like this:

user_code = """
EXTRA = 1.1
def func(): return FOO * BAR * EXTRA
"""
stuff = define_stuff(user_code)
func = stuff['func']
print(func())

This can't find the EXTRA variable found. (Another example would be
defining a recursive function -- the function can't find itself.)

The alternative (which Colin complains about) is for define_stuff() to
use a single namespace, initialized with the context, and to return
that -- but he's right that in that case FOO and BAR would be exported
as part of stuff, which may not be the intention. (Another solution
would be to put FOO and BAR in the builtins dict -- but that has other
problems of course.)

> So put simply - the bug is that a class namespace is used, but its not a class.

Well, really, the bug is that no closure is created even though there
are separate globals and locals. Unfortunately this is because while
at the point where the code is *executed* the two different contexts
are clearly present, at the point where it is *compiled* this is not
the case -- the compiler normally uses syntactic clues to decide
whether to generate code using closures, in particular, the presence
of nested functions. Note that define_stuff() could be equivalently
written like this, where it is more obvious that the compiler doesn't
know about the separate namespaces:

def define_stuff(user_code):
  context = {...}
  stuff = {}
  compiled_code = compile(user_code, "<string>", "exec")
  exec(user_code, context, stuff)
  return stuff

This is not easy to fix. The best short-term work-around is probably a
hack like this:

def define_stuff(user_code):
  context = {...}
  stuff = {}
  stuff.update(context)
  exec(user_code, stuff)
  for key in context:
    if key in stuff and stuff[key] == context[key]:
      del stuff[key]
  return stuff

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Thu May 27 02:44:21 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 May 2010 17:44:21 -0700
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTilIg9O5DPpft2XkQiU1UFXKXU8jDwC_Ogi92vsF@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTinSKzyWPOqY1oyxqxMo_UZ2oUzsGW3XxcTeJPhN@mail.gmail.com> 
	<AANLkTilIg9O5DPpft2XkQiU1UFXKXU8jDwC_Ogi92vsF@mail.gmail.com>
Message-ID: <AANLkTil2BBArN8FTc3NbCNTdvXKHIyefI56We5O-GdQ7@mail.gmail.com>

On Wed, May 26, 2010 at 5:37 PM, Colin H <hawkett at gmail.com> wrote:
> Mark Dickinson wrote:
>
>> Seems to me the whole idea of being able to specify
>> separate global and local scopes for top-level code is
>> screwy in the first place. Are there any use cases for
>> it? Maybe the second scope argument to exec() should
>> be deprecated?
>
> It is running as class namespace that makes the argument that there's
> no use case. I agree - I can't think of any decent use cases for
> exec() as class namespace - defining functions and classes only works
> for a subset of function and class definitions
>
> However, if exec() ran as function namespace instead, then the locals
> dictionary will contain all the definitions from the exec()'d code
> block, and only those definitions. Very useful.

Hmm... I see your point, but fear that the implementation is tricky,
since it would mean every exec'ed block would have to be compiled as
if it were a function body, and that probably violates some other
invariants.

> This is a major use
> case for exec() - defining code from strings (e.g. enabling you to
> store python code in the database), and using it at runtime. It seems
> to me this must have been the point of locals in the first place.
>
> If you just use globals, then your definitions exist amongst a whole
> bunch of other python stuff, and unless you know in advance what was
> defined in your code block, its very difficult to extract them.

Check the reconstruction of your use case I just posted. I think you
re thinking context == globals(), which makes sense.

-- 
--Guido van Rossum (python.org/~guido)

From hawkett at gmail.com  Thu May 27 02:53:15 2010
From: hawkett at gmail.com (Colin H)
Date: Thu, 27 May 2010 01:53:15 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
Message-ID: <AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com>

Hi Guido,

   Thanks for the possible workaround - unfortunately 'stuff' will
contain a whole stack of things that are not in 'context', and were
not defined in 'user_code' - things that python embeds - a (very
small) selection -

{..., 'NameError': <type 'exceptions.NameError'>, 'BytesWarning':
<type 'exceptions.BytesWarning'>, 'dict': <type 'dict'>, 'input':
<function input at 0x10047a9b0>, 'oct': <built-in function oct>,
'bin': <built-in function bin>, ...}

It makes sense why this happens of course, but upon return, the
globals dict is very large, and finding the stuff you defined in your
user_code amongst it is a very difficult task.  Avoiding this problem
is the 'locals' use-case for me.  Cheers,

Colin

On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum <guido at python.org> wrote:
> This is not easy to fix. The best short-term work-around is probably a
> hack like this:
>
> def define_stuff(user_code):
> ?context = {...}
> ?stuff = {}
> ?stuff.update(context)
> ?exec(user_code, stuff)
> ?for key in context:
> ? ?if key in stuff and stuff[key] == context[key]:
> ? ? ?del stuff[key]
> ?return stuff
>
> --
> --Guido van Rossum (python.org/~guido)
>

From pje at telecommunity.com  Thu May 27 03:05:40 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 26 May 2010 21:05:40 -0400
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.c
 om>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
Message-ID: <20100527010538.C7AA83A40A5@sparrow.telecommunity.com>

At 11:41 PM 5/26/2010 +0100, Paul Moore wrote:
>I'm genuinely struggling to see how a Sumo distribution ever comes
>into being under your proposal. There's no evidence that anyone wants
>it (otherwise it would have been created by now!!)

Actually, sumo distributions *have* been created; it's just that 
there are so *many* of them:

* Linux distributions
* BSD & Mac "ports" systems
* ActivePython, the Enthought Distribution, etc.
* certain distutils extensions that I shall not name

When the idea of "sumo" distributions were first being batted around 
(10-12 years ago?), I'm not sure even *distutils* existed yet, let 
alone any of this other stuff.  I mean, yeah, the port systems and 
distros existed, but their coverage wasn't nearly as good as it is today.


From guido at python.org  Thu May 27 03:05:46 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 26 May 2010 18:05:46 -0700
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com>
Message-ID: <AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com>

On Wed, May 26, 2010 at 5:53 PM, Colin H <hawkett at gmail.com> wrote:
> ? Thanks for the possible workaround - unfortunately 'stuff' will
> contain a whole stack of things that are not in 'context', and were
> not defined in 'user_code' - things that python embeds - a (very
> small) selection -
>
> {..., 'NameError': <type 'exceptions.NameError'>, 'BytesWarning':
> <type 'exceptions.BytesWarning'>, 'dict': <type 'dict'>, 'input':
> <function input at 0x10047a9b0>, 'oct': <built-in function oct>,
> 'bin': <built-in function bin>, ...}
>
> It makes sense why this happens of course, but upon return, the
> globals dict is very large, and finding the stuff you defined in your
> user_code amongst it is a very difficult task. ?Avoiding this problem
> is the 'locals' use-case for me. ?Cheers,

No, if taken literally that doesn't make sense. Those are builtins. I
think you are mistaken that each of those (e.g. NameError) is in stuff
-- they are in stuff['__builtins__'] which represents the built-in
namespace. You should remove that key from stuff as well.

--Guido

> Colin
>
> On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum <guido at python.org> wrote:
>> This is not easy to fix. The best short-term work-around is probably a
>> hack like this:
>>
>> def define_stuff(user_code):
>> ?context = {...}
>> ?stuff = {}
>> ?stuff.update(context)
>> ?exec(user_code, stuff)
>> ?for key in context:
>> ? ?if key in stuff and stuff[key] == context[key]:
>> ? ? ?del stuff[key]
>> ?return stuff
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>



-- 
--Guido van Rossum (python.org/~guido)

From hawkett at gmail.com  Thu May 27 03:13:42 2010
From: hawkett at gmail.com (Colin H)
Date: Thu, 27 May 2010 02:13:42 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com> 
	<AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com>
Message-ID: <AANLkTimAT0k1I1ESw8twwBinYeChNB-rQp3SwpSOGH1Z@mail.gmail.com>

Of course :) - I need to pay more attention. Your workaround should do
the trick. It would make sense if locals could be used for this
purpose, but the workaround doesn't add so much overhead in most
situations.  Thanks for the help, much appreciated,

Colin

On Thu, May 27, 2010 at 2:05 AM, Guido van Rossum <guido at python.org> wrote:
> On Wed, May 26, 2010 at 5:53 PM, Colin H <hawkett at gmail.com> wrote:
>> ? Thanks for the possible workaround - unfortunately 'stuff' will
>> contain a whole stack of things that are not in 'context', and were
>> not defined in 'user_code' - things that python embeds - a (very
>> small) selection -
>>
>> {..., 'NameError': <type 'exceptions.NameError'>, 'BytesWarning':
>> <type 'exceptions.BytesWarning'>, 'dict': <type 'dict'>, 'input':
>> <function input at 0x10047a9b0>, 'oct': <built-in function oct>,
>> 'bin': <built-in function bin>, ...}
>>
>> It makes sense why this happens of course, but upon return, the
>> globals dict is very large, and finding the stuff you defined in your
>> user_code amongst it is a very difficult task. ?Avoiding this problem
>> is the 'locals' use-case for me. ?Cheers,
>
> No, if taken literally that doesn't make sense. Those are builtins. I
> think you are mistaken that each of those (e.g. NameError) is in stuff
> -- they are in stuff['__builtins__'] which represents the built-in
> namespace. You should remove that key from stuff as well.
>
> --Guido
>
>> Colin
>>
>> On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum <guido at python.org> wrote:
>>> This is not easy to fix. The best short-term work-around is probably a
>>> hack like this:
>>>
>>> def define_stuff(user_code):
>>> ?context = {...}
>>> ?stuff = {}
>>> ?stuff.update(context)
>>> ?exec(user_code, stuff)
>>> ?for key in context:
>>> ? ?if key in stuff and stuff[key] == context[key]:
>>> ? ? ?del stuff[key]
>>> ?return stuff
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>

From greg.ewing at canterbury.ac.nz  Thu May 27 03:46:07 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 13:46:07 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4BF672E8.1080001@canonical.com>
	<5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<20100526120651.GA12751@laurie.devork.be>
	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>
Message-ID: <4BFDCEDF.1060303@canterbury.ac.nz>

On 27/05/10 00:31, Brian Quinlan wrote:

> You have two semantic choices here:
> 1. let the interpreter exit with the future still running
> 2. wait until the future finishes and then exit

I'd go for (1). I don't think it's unreasonable to
expect a program that wants all its tasks to finish
to explicitly wait for that to happen.

Also, automatically doing (2) would seem to make it
difficult for a program to bail out if something
unexpected happens. It would have to explicitly
shut down the thread pool instead of just letting
an exception propagate.

-- 
Greg

From yaniv at aknin.name  Thu May 27 04:09:07 2010
From: yaniv at aknin.name (Yaniv Aknin)
Date: Thu, 27 May 2010 12:09:07 +1000
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTilrTep2PPYtVPwegXHmzMI5hJl-qE0I670Qzxit@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
	<4BFDB566.2060502@gmail.com>
	<AANLkTilrTep2PPYtVPwegXHmzMI5hJl-qE0I670Qzxit@mail.gmail.com>
Message-ID: <AANLkTiml1xTGQnh2EVlphXYQMlcPmA_DoLLhOwZqJWWK@mail.gmail.com>

>
> > Because scientists, financial analysts, web designers, etc all have
> > different needs.
>
> My point is just that a web designer probably doesn't care if he's
> got numpy, nor does a mathematician care if he has cherrypy
> onboard. They only care when the tools they need aren't there,
> which is where sumo can help.


Why do we want "distributions" (whether 'sumo' or domain specific) in the
first place? Obviously because we have some pains to alleviate, but I think
Linux distributions, particularly Ubuntu, have been coping with similar
pains for a while now with good results. I used to be rather informed on the
state of Linux distributions circa the end of the 90's. Should I use this
distribution or that one? Now I don't care, the answer is always 'Ubuntu,
and apt-get it to fit your needs'.

The pain list, as I see it:
(a) Make it easy to find functionality that you'd typically be tempted to
implement alone
(b) Make it trivial, mind boggling easy, to add this functionality to your
interpreter once you know of it
(b.1) Make it easy to package a Python 'distribution' so end users don't
need to muck around with (b)
(c) Make it easy for software to specify its requirements in code and get
them mostly automagically
(d) Make it *legally* least painful to acquire and use endorsed
functionality sets
 ... (pipe in if I missed something important)

I'm wondering if expanding the efforts in Python packaging ("easy_install"
and PEP376 comes to mind, only on steroids) isn't the correct solution for
alleviating these pains in a much better fashion. I'm not saying it's easy,
but I think that superb packaging technologies are a very well proven
technology, that they're well within the community's capabilities/needs and
that maximizing flexibility is a move in a better direction than creating
one or two or seven 'distributions'.

I hope you would intuitively grasp what I mean, even if I fail to articulate
it well, if you ever used dpkg and apt-get (or equivalents), along with the
ability to add sources and the simple distinctions between sources (core vs.
universe vs. multiverse), along with eco systems like launchpad and PPAs,
etc. To my 1999 self these features of Ubuntu seem like miracles, not so
much because Debian and dpkg/apt-get weren't there back then (they were life
saving back then too), but because of the whole system's huge support base
and centrally-designed decentrally-implemented nature. 'apt-cache search' is
usually how I find new software these days, and the same could be true for
Python when I need futures or multiprocessing or whatever. I could write
lots about what I think we should have in such a system, but maybe it's more
befitting for python-ideas or a blog post than here; for this argument,
'mirror Ubuntu's winnage' is a good enough description. I think a good and
lively packaging system beats 'distributions' any day.

This leaves us only with the legal pain, which I felt harshly on my flesh
following some acquisition by a corporation and it's painful and frustrating
and mostly unreasonable. I suspect, though I'm a legal n00b (and proud of
it), that packaging in multiple repositories like Ubuntu does could help a
lot, along with taking some responsibility over the most recommended repo
(candidates are the 'almost made it into stdlib' packages, maybe even the
giants like Django, NumPy, TurboGears, Twisted, etc). This is not about
development responsibility and surely not about taking legal liability, it's
about actively helping facilitate these packages' use in hostile legal
environments. While the hackers in the trenches working on trunk and py3k
may not care much (and probably needn't care much), I humbly think the PSF
may care more and is best equipped to help. Maybe by requiring certain
licenses for 'recommended' repo inclusion, maybe by working with the likes
of Google or IBM on publishing legal reviews of select Python distributions,
I don't know, depends what would help a corporate lawyer make speedy
approval.

 - Yaniv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100527/c2cb244f/attachment-0001.html>

From yaniv at aknin.name  Thu May 27 04:15:55 2010
From: yaniv at aknin.name (Yaniv Aknin)
Date: Thu, 27 May 2010 12:15:55 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFD13F6.1080300@voidspace.org.uk>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<4BFD13F6.1080300@voidspace.org.uk>
Message-ID: <AANLkTinWslMQp8QV7lr7dTNGNGwTsy7mUc7r4aVMH564@mail.gmail.com>

>
> Well... a middle ground certainly could exist; perhaps in the form of an
> "Extended Standard Library" (community distribution), with simple
> installation and management tools.
>
> It could be "blessed" by python-dev and maintain a high standard (only well
> established best-of-breed modules with a commitment of ongoing maintenance
> and more than one maintainer - something that the stdlib itself doesn't
> stick to). A common license could even be chosen, potentially allowing
> corporations to approve the extended package in a single pass.
>

I read the 'sumo' thread before I read this (and replied in depth there),
but I think Michael and I mean similar things.

 - Yaniv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100527/fe384e69/attachment.html>

From debatem1 at gmail.com  Thu May 27 04:29:18 2010
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 26 May 2010 19:29:18 -0700
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTinWslMQp8QV7lr7dTNGNGwTsy7mUc7r4aVMH564@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<4BFD13F6.1080300@voidspace.org.uk>
	<AANLkTinWslMQp8QV7lr7dTNGNGwTsy7mUc7r4aVMH564@mail.gmail.com>
Message-ID: <AANLkTim4KQx8eK2KNzj5q-OB_yjsgsyk00V0jTQB2Gvl@mail.gmail.com>

On Wed, May 26, 2010 at 7:15 PM, Yaniv Aknin <yaniv at aknin.name> wrote:
>> Well... a middle ground certainly could exist; perhaps in the form of an
>> "Extended Standard Library" (community distribution), with simple
>> installation and management tools.

I'm not sure about the 'installation and management tools' part, but this
is basically the idea I was trying to articulate: a middle ground between
a 'fat' stdlib and a 'lean' one.

>> It could be "blessed" by python-dev and maintain a high standard (only
>> well established best-of-breed modules with a commitment of ongoing
>> maintenance and more than one maintainer - something that the stdlib itself
>> doesn't stick to). A common license could even be chosen, potentially
>> allowing corporations to approve the extended package in a single pass.

If we could do it that would be great, IMHO.

> I read the 'sumo' thread before I read this?(and replied in depth there),
> but I think Michael and I mean similar things.
> ?- Yaniv

I don't think I'm understanding you correctly in that thread then, ISTM
that you're advocating better packaging systems as an alternative to
this. Would you mind clarifying?

Geremy Condra

From greg.ewing at canterbury.ac.nz  Thu May 27 04:29:28 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 14:29:28 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFD26A8.9050802@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4BF916EA.2090507@holdenweb.com>
	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
	<20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com>
	<20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com>
	<20100526145033.3b61d95b@pitrou.net>
	<A3704B8E-2B63-4B97-90EA-B1137FF6566C@sweetapp.com>
	<4BFD26A8.9050802@gmail.com>
Message-ID: <4BFDD908.90609@canterbury.ac.nz>

On 27/05/10 01:48, Nick Coghlan wrote:

> I would say it is precisely that extra configurability which separates
> the executor pools in the PEP implementation from more flexible general
> purpose pools.

Wouldn't this be better addressed by adding the relevant
options to the futures pools, rather than adding another
module that provides almost exactly the same thing with
different options?

-- 
Greg

From scott+python-dev at scottdial.com  Thu May 27 04:48:04 2010
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Wed, 26 May 2010 22:48:04 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFDB6CB.7030104@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>	<4BFD173B.9010703@gmail.com>	<htji55$qui$1@dough.gmane.org>
	<4BFDB6CB.7030104@gmail.com>
Message-ID: <4BFDDD64.9060701@scottdial.com>

On 5/26/2010 8:03 PM, Nick Coghlan wrote:
> On 27/05/10 02:27, Terry Reedy wrote:
>> I am suggesting that if we add a package, we do it right, from the
>> beginning.
> 
> This is a reasonable point of view, but I wouldn't want to hold up PEP
> 3148 over it (call it a +0 for the idea in general, but a -1 for linking
> it to the acceptance of PEP 3148).

That sounds backward. How can you justify accepting PEP 3148 into a
"concurrent" namespace without also accepting the demand for such a
namespace? What is the contingency if this TBD migration PEP is not
accepted, what happens to PEP 3148? After all, there was some complaints
about just calling it "futures", without putting it in a "concurrent"
namespace.

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu

From greg.ewing at canterbury.ac.nz  Thu May 27 05:13:16 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 15:13:16 +1200
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFDAFDA.7060404@voidspace.org.uk>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<4BFDB10A.4080002@canterbury.ac.nz> <4BFDAFDA.7060404@voidspace.org.uk>
Message-ID: <4BFDE34C.1080405@canterbury.ac.nz>

On 27/05/10 11:33, Michael Foord wrote:
> On 27/05/2010 00:38, Greg Ewing wrote:

>> Maybe the second scope argument to exec() should
>> be deprecated?
>>
> Sounds good to me, certainly ends the confusion over this undoubtedly
> unintuitive behaviour. :-)

Although it's a fair point that it can be useful to have
a way of capturing definitions made by the execed code,
while still making an environment of other stuff available
to it.

So, I'd be in favour of changing the behaviour of exec
so that the local scope is made visible inside functions.
However, it would be non-trivial to implement this the
way things are currently structured, which is probably one
of the reasons it hasn't already been done.

I don't think that simply using LOAD_NAME inside the function
would work, because that would only look in the function's
own local namespace first, then in the global one. There is
just no mechanism available at the moment for the function
to know about the local namespace passed in to exec.

The way that functions get access to names in enclosing
local scopes is by having them passed in as cells, but that
mechanism is only available for optimised local namespaces,
not ones implemented as dicts.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu May 27 05:21:14 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 15:21:14 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTinmitR_ateAN0cgcQpUWUnrFAsO6tlXDdBc3Bgb@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<4BFDB067.4060800@canterbury.ac.nz>
	<AANLkTinmitR_ateAN0cgcQpUWUnrFAsO6tlXDdBc3Bgb@mail.gmail.com>
Message-ID: <4BFDE52A.7040209@canterbury.ac.nz>

On 27/05/10 12:04, Jesse Noller wrote:

>> Namespaces are
>> only a honking great idea if you actually let them do the job
>> they're designed for.
>
> concurrent.* is the namespace, futures is the package within the
> namespace - concurrent.futures is highly descriptive of the items
> contained therein.

I was referring to the issue of ThreadPool vs. ThreadPoolExecutor
etc. By your own argument above, concurrent.futures.ThreadPool is
quite descriptive enough of what it provides. It's not a problem
if some other module also provides something called a ThreadPool.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu May 27 05:35:35 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 15:35:35 +1200
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTilIg9O5DPpft2XkQiU1UFXKXU8jDwC_Ogi92vsF@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTinSKzyWPOqY1oyxqxMo_UZ2oUzsGW3XxcTeJPhN@mail.gmail.com>
	<AANLkTilIg9O5DPpft2XkQiU1UFXKXU8jDwC_Ogi92vsF@mail.gmail.com>
Message-ID: <4BFDE887.80904@canterbury.ac.nz>

On 27/05/10 12:37, Colin H wrote:

> This is a major use
> case for exec() - defining code from strings (e.g. enabling you to
> store python code in the database), and using it at runtime. It seems
> to me this must have been the point of locals in the first place.

I suspect that originally it just fell out of the
implementation. The function in the interpreter that
executes code objects requires two namespaces as
arguments, and they were both exposed via exec just
in case anyone found a use for them.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu May 27 05:40:16 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 15:40:16 +1200
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
Message-ID: <4BFDE9A0.8050602@canterbury.ac.nz>

On 27/05/10 12:38, Guido van Rossum wrote:
> the compiler normally uses syntactic clues to decide
> whether to generate code using closures, in particular, the presence
> of nested functions.

Well, the compiler could be passed a flag indicating that
the code is being compiled for an exec statement.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu May 27 05:47:26 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 27 May 2010 15:47:26 +1200
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
	<AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com>
	<AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com>
Message-ID: <4BFDEB4E.50100@canterbury.ac.nz>

Another approach to all this might be to generalise the
mechanism by which a lookup of the globals falls back
to a lookup of __builtins__.

If this were done recursively, then the "stuff" could
be attached to the globals dict, e.g.

   stuff['__builtins__'] = __builtins__
   g = dict(__builtins__ = stuff)
   exec(code, g)
   del g['__builtins__']

-- 
Greg


From yaniv at aknin.name  Thu May 27 06:31:03 2010
From: yaniv at aknin.name (Yaniv Aknin)
Date: Thu, 27 May 2010 14:31:03 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTim4KQx8eK2KNzj5q-OB_yjsgsyk00V0jTQB2Gvl@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<4BFD13F6.1080300@voidspace.org.uk>
	<AANLkTinWslMQp8QV7lr7dTNGNGwTsy7mUc7r4aVMH564@mail.gmail.com>
	<AANLkTim4KQx8eK2KNzj5q-OB_yjsgsyk00V0jTQB2Gvl@mail.gmail.com>
Message-ID: <AANLkTikgbGJl7Wp4lzuvcX1r4IZgbQq4Le_4UqxcYPp3@mail.gmail.com>

>
> I don't think I'm understanding you correctly in that thread then, ISTM
> that you're advocating better packaging systems as an alternative to
> this. Would you mind clarifying?


Gladly.

In my mind, 'better packaging' is not "just" about something that will let
you do 'pypkg install foo' and 'pypkg remove foo' along with dependancies,
it's also about meta-packages (packages that have nothing but dependencies)
but also about separate repositories with different natures, some endorsed
by python-dev, some not. It's not so much the packaging system that does the
trick - it's the eco system around it.

When you install something from the 'recommended' repository (should find a
better name), you know its a 'blessed' package. You know it has unittests,
documentation, active development, good license, a bug tracker and the
general-soundedness which I hope users have come to expect of python-dev. It
is not stdlib, it's not made by python-dev, it doesn't come with python.org's
default build by default, but you have a lot of important assurances with
ease. You know 'it could have been in stdlib, but happens not to be'.

To make this really work, the future tool I called "pypkg" will come with
all future Python versions (so people can rely on it), and will arrive
configured by default to use only the 'recommended' repository. From a
developer's standpoint, relying on package 'foo' of the 'recommended'
repository is safe: even if their end users don't happen to have 'foo', the
developer is certain that 'recommended' is configured on their user's Python
and is a quick (automagical?) download away.

 - Yaniv

(p.s.: again, think Ubuntu core vs universe vs multiverse vs. non-Ubuntu
repositories vs. PPAs; forgive me for waving the Ubuntu flag, from the
packaging systems /and accompanying eco-systems/ I know, they did it best)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100527/52e99cf5/attachment.html>

From morph at debian.org  Thu May 27 07:40:48 2010
From: morph at debian.org (Sandro Tosi)
Date: Thu, 27 May 2010 07:40:48 +0200
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTiml1xTGQnh2EVlphXYQMlcPmA_DoLLhOwZqJWWK@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net> 
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net> 
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com> 
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com> 
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com> 
	<4BFDB566.2060502@gmail.com>
	<AANLkTilrTep2PPYtVPwegXHmzMI5hJl-qE0I670Qzxit@mail.gmail.com> 
	<AANLkTiml1xTGQnh2EVlphXYQMlcPmA_DoLLhOwZqJWWK@mail.gmail.com>
Message-ID: <AANLkTin_C83h82cAR3zDPJ5udQZdZ__5DZgcnmLNcOoN@mail.gmail.com>

Hello,
sorry to interrupt your discussion but..

On Thu, May 27, 2010 at 04:09, Yaniv Aknin <yaniv at aknin.name> wrote:
>> > Because scientists, financial analysts, web designers, etc all have
>> > different needs.
>>
>> My point is just that a web designer probably doesn't care if he's
>> got numpy, nor does a mathematician care if he has cherrypy
>> onboard. They only care when the tools they need aren't there,
>> which is where sumo can help.
>
> Why do we want "distributions" (whether 'sumo' or domain specific) in the
> first place? Obviously because we have some pains to alleviate, but I think
> Linux distributions, particularly Ubuntu, have been coping with similar
> pains for a while now with good results. I used to be rather informed on the
> state of Linux distributions circa the end of the 90's. Should I use this
> distribution or that one? Now I don't care, the answer is always 'Ubuntu,
> and apt-get it to fit your needs'.

please note that the huge bunch of work for Python third-party modules
is done in *Debian* and Ubuntu just takes those packages without
advertise enough where they come from and who did the work (and not
the merchandizing only).

> I hope you would intuitively grasp what I mean, even if I fail to articulate
> it well,?if you ever used dpkg and apt-get (or equivalents), along with the
> ability to add sources and the simple distinctions between sources (core vs.
> universe vs. multiverse), along with eco systems like launchpad and PPAs,
> etc. To my 1999 self these features of Ubuntu seem like miracles, not so
> much because Debian and dpkg/apt-get weren't there back then (they were life
> saving back then too), but because of the whole system's huge support base
> and centrally-designed decentrally-implemented nature.

mh? Debian was not in present in 1999? Debian started in 1993 (dpkg in
1994 and apt-get in 1998) while ubuntu only in mid-2000 as a layer
over Debian packages (and hiring several Debian Developers for its
core positions).

Also, let me remind you that transition to Python 2.6 was a huge mess
for Ubuntu, where several packages were just left broken (f.e. failing
unit tests were disabled to make the package build...) and only now
that Debian started to migrate to 2.6 too, you can see a "flow" of
packages that works for 2.6 too coming to Ubuntu. Debian can be slow,
but we care about quality.

End of "give credit where it's due" post :)

Regards,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

From brian at sweetapp.com  Thu May 27 08:20:04 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Thu, 27 May 2010 16:20:04 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFDE52A.7040209@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<4BFDB067.4060800@canterbury.ac.nz>
	<AANLkTinmitR_ateAN0cgcQpUWUnrFAsO6tlXDdBc3Bgb@mail.gmail.com>
	<4BFDE52A.7040209@canterbury.ac.nz>
Message-ID: <B3CACAFC-1B77-4D03-8EBE-000081D193B5@sweetapp.com>


On May 27, 2010, at 1:21 PM, Greg Ewing wrote:

> On 27/05/10 12:04, Jesse Noller wrote:
>
>>> Namespaces are
>>> only a honking great idea if you actually let them do the job
>>> they're designed for.
>>
>> concurrent.* is the namespace, futures is the package within the
>> namespace - concurrent.futures is highly descriptive of the items
>> contained therein.
>
> I was referring to the issue of ThreadPool vs. ThreadPoolExecutor
> etc. By your own argument above, concurrent.futures.ThreadPool is
> quite descriptive enough of what it provides. It's not a problem
> if some other module also provides something called a ThreadPool.


I think that the "Executor" suffix is a good indicator of the  
interface being provided. "Pool" is not because you can could have  
Executor implementations that don't involve pools.

Cheers,
Brian

From regebro at gmail.com  Thu May 27 09:37:00 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 27 May 2010 09:37:00 +0200
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTin_C83h82cAR3zDPJ5udQZdZ__5DZgcnmLNcOoN@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net> 
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net> 
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com> 
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com> 
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com> 
	<4BFDB566.2060502@gmail.com>
	<AANLkTilrTep2PPYtVPwegXHmzMI5hJl-qE0I670Qzxit@mail.gmail.com> 
	<AANLkTiml1xTGQnh2EVlphXYQMlcPmA_DoLLhOwZqJWWK@mail.gmail.com> 
	<AANLkTin_C83h82cAR3zDPJ5udQZdZ__5DZgcnmLNcOoN@mail.gmail.com>
Message-ID: <AANLkTim7dKO2Xjyut08KRBmGmitrjSpL_XGR8qE7X4Wi@mail.gmail.com>

One  worry with an official sumo distribution is that it could become
an excuse for *not* putting something in the stdlib.
Otherwise it's an interesting idea.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From p.f.moore at gmail.com  Thu May 27 09:46:34 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 27 May 2010 08:46:34 +0100
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
Message-ID: <AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>

On 27 May 2010 00:11, geremy condra <debatem1 at gmail.com> wrote:
> I'm not clear, you seem to be arguing that there's a market for many
> augmented python distributions but not one. Why not just have one
> that includes the best from each domain?

Because that's "bloat". You later argue that a web designer wouldn't
care if his "distribution" included numpy. OK, maybe, but if my needs
are simply futures, cx_Oracle and pywin32, I *would* object to
downloading many megabytes of other stuff just to get those three.
It's a matter of degree.

>> I'm genuinely struggling to see how a Sumo distribution ever comes
>> into being under your proposal. There's no evidence that anyone wants
>> it (otherwise it would have been created by now!!)
>
> Everything worth making has already been made?

Not what I'm saying. But if/when it happens, something will trigger
it. I see no sign of such a trigger. That's all I'm saying.

>> and until it exists, it's not a plausible "place" to put modules that don't
>> make it into the stdlib.
>
> Of course its implausible to put something somewhere that
> doesn't exist... until it does.

Hence my point - people are saying futures don't belong in the stdlib
but they could go in a sumo distribution. The second half of that
statement is (currently) content free if not self-contradictory.

> I'd say rather that there are a large number of specialized tools which
> aren't individually popular enough to be included in Python, but which
> when taken together greatly increase its utility, and that sumo offers a
> way to provide that additional utility to python's users without forcing
> python core devs to shoulder the maintenance burden.

I don't believe that there's evidence that aggregation (except in the
context of specialist areas) does provide additional utility. (In the
context of the discussion that sparked this debate, that contrasts
with inclusion in the stdlib, which *does* offer additional utility -
"batteries included", guaranteed and tested cross-platform
functioning, a statement of best practice, etc etc).

Paul.

PS One thing I haven't seen made clear - in my view, they hypothetical
"sumo" is a single aggregated distribution of Python
modules/packages/extensions. It would NOT include core Python and the
stdlib (in contrast to Enthought or ActivePython). I get the
impression that other people may be thinking in terms of a full Python
distribution, like those 2 cases. We probably ought to be clear which
we're talking about.

From floris.bruynooghe at gmail.com  Thu May 27 09:53:49 2010
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Thu, 27 May 2010 08:53:49 +0100
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFDCEDF.1060303@canterbury.ac.nz>
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<20100526120651.GA12751@laurie.devork.be>
	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>
	<4BFDCEDF.1060303@canterbury.ac.nz>
Message-ID: <20100527075349.GA32291@laurie.devork.be>

On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote:
> On 27/05/10 00:31, Brian Quinlan wrote:
> 
> >You have two semantic choices here:
> >1. let the interpreter exit with the future still running
> >2. wait until the future finishes and then exit
> 
> I'd go for (1). I don't think it's unreasonable to
> expect a program that wants all its tasks to finish
> to explicitly wait for that to happen.

I'd got for (1) as well, it's no more then reasonable that if you want
a result you wait for it.  And I dislike libraries doing magic you
can't see, I'd prefer if I explicitly had to shut a pool down.  And
yes, if you shut the interpreter down while threads are running they
sometimes wake up at the wrong time to find the world around them
destroyed.  But that's part of programming with threads so it's not
like the futures lib suddenly makes things behave differently.

I'm glad I'm not alone in preferring (1) tough.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From brian at sweetapp.com  Thu May 27 10:13:01 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Thu, 27 May 2010 18:13:01 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100527075349.GA32291@laurie.devork.be>
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<20100526120651.GA12751@laurie.devork.be>
	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>
	<4BFDCEDF.1060303@canterbury.ac.nz>
	<20100527075349.GA32291@laurie.devork.be>
Message-ID: <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>


On 27 May 2010, at 17:53, Floris Bruynooghe wrote:

> On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote:
>> On 27/05/10 00:31, Brian Quinlan wrote:
>>
>>> You have two semantic choices here:
>>> 1. let the interpreter exit with the future still running
>>> 2. wait until the future finishes and then exit
>>
>> I'd go for (1). I don't think it's unreasonable to
>> expect a program that wants all its tasks to finish
>> to explicitly wait for that to happen.
>
> I'd got for (1) as well, it's no more then reasonable that if you want
> a result you wait for it.  And I dislike libraries doing magic you
> can't see, I'd prefer if I explicitly had to shut a pool down.  And
> yes, if you shut the interpreter down while threads are running they
> sometimes wake up at the wrong time to find the world around them
> destroyed.  But that's part of programming with threads so it's not
> like the futures lib suddenly makes things behave differently.
>
> I'm glad I'm not alone in preferring (1) tough.

Keep in mind that this library magic is consistent with the library  
magic that the threading module does - unless the user sets  
Thread.daemon to True, the interpreter does *not* exit until the  
thread does.

Cheers,
Brian

From regebro at gmail.com  Thu May 27 11:01:17 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 27 May 2010 11:01:17 +0200
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net> 
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net> 
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com> 
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com> 
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com> 
	<AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>
Message-ID: <AANLkTik8PhfRKC_nygAtVTiHxxxRDMLsAQJRJ9FZb-I-@mail.gmail.com>

OK, I had an idea here:

How about that the people affected by difficulties in getting software
approved got together to put together not a sumo-python, but a
python-extras package? That package could include all the popular
stuff, like SciPy, Numpy, twisted, distribute, buildout, virtualenv,
pip, pytz, PIL, openid, docutils, simplejson, nose, genshi, and tons
of others.

That would be a big download. But here's the trick: You don't *have*
to install them! Just bundle all of it.

If licensing is a problem I guess you'd need to have permission to
relicense them all to the Python license, which would be problematic.
But otherwise having a team of people overseeing and bundling all this
might not be that much work, and you'd avoid the bloat by not
installing all of it. :-)

Or would this not fool the company trolls?

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From solipsis at pitrou.net  Thu May 27 12:25:37 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 27 May 2010 12:25:37 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<4BF916EA.2090507@holdenweb.com>
	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>
	<20100523144754.1c6f1afc@pitrou.net> <4BFC6325.5050700@gmail.com>
	<20100526093820.4fbf2f87@pitrou.net> <4BFD14E1.6000305@gmail.com>
	<20100526145033.3b61d95b@pitrou.net>
	<A3704B8E-2B63-4B97-90EA-B1137FF6566C@sweetapp.com>
	<4BFD26A8.9050802@gmail.com> <4BFDD908.90609@canterbury.ac.nz>
Message-ID: <20100527122537.133899de@pitrou.net>

On Thu, 27 May 2010 14:29:28 +1200
Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> On 27/05/10 01:48, Nick Coghlan wrote:
> 
> > I would say it is precisely that extra configurability which separates
> > the executor pools in the PEP implementation from more flexible general
> > purpose pools.
> 
> Wouldn't this be better addressed by adding the relevant
> options to the futures pools, rather than adding another
> module that provides almost exactly the same thing with
> different options?

+1.



From hawkett at gmail.com  Thu May 27 13:14:05 2010
From: hawkett at gmail.com (Colin H)
Date: Thu, 27 May 2010 12:14:05 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTimAT0k1I1ESw8twwBinYeChNB-rQp3SwpSOGH1Z@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com> 
	<AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com> 
	<AANLkTimAT0k1I1ESw8twwBinYeChNB-rQp3SwpSOGH1Z@mail.gmail.com>
Message-ID: <AANLkTimzzBg2SXcXPGXtPXB8iHvqvR4gvB4h4Wnmyi0Y@mail.gmail.com>

I needed to make a small modification to the workaround - I wasn't
able to delete from 'stuff', as the definitions in exec()'d code won't
run - they're relying on that being present at runtime. In practice
the overhead of doing this is quite noticeable if you run your code
like this a lot, and build up a decent sized context (which I do). It
will obviously depend on the usage scenario though.

def define_stuff(user_code):
 ?context = {...}
 ?stuff = {}
 ?stuff.update(context)

 ?exec(user_code, stuff)

  return_stuff = {}
  return_stuff.update(stuff)

  del return_stuff['__builtins__']
 ?for key in context:
 ? ?if key in return_stuff and return_stuff[key] == context[key]:
 ? ? ?del return_stuff[key]

 ?return return_stuff

On Thu, May 27, 2010 at 2:13 AM, Colin H <hawkett at gmail.com> wrote:
> Of course :) - I need to pay more attention. Your workaround should do
> the trick. It would make sense if locals could be used for this
> purpose, but the workaround doesn't add so much overhead in most
> situations. ?Thanks for the help, much appreciated,
>
> Colin
>
> On Thu, May 27, 2010 at 2:05 AM, Guido van Rossum <guido at python.org> wrote:
>> On Wed, May 26, 2010 at 5:53 PM, Colin H <hawkett at gmail.com> wrote:
>>> ? Thanks for the possible workaround - unfortunately 'stuff' will
>>> contain a whole stack of things that are not in 'context', and were
>>> not defined in 'user_code' - things that python embeds - a (very
>>> small) selection -
>>>
>>> {..., 'NameError': <type 'exceptions.NameError'>, 'BytesWarning':
>>> <type 'exceptions.BytesWarning'>, 'dict': <type 'dict'>, 'input':
>>> <function input at 0x10047a9b0>, 'oct': <built-in function oct>,
>>> 'bin': <built-in function bin>, ...}
>>>
>>> It makes sense why this happens of course, but upon return, the
>>> globals dict is very large, and finding the stuff you defined in your
>>> user_code amongst it is a very difficult task. ?Avoiding this problem
>>> is the 'locals' use-case for me. ?Cheers,
>>
>> No, if taken literally that doesn't make sense. Those are builtins. I
>> think you are mistaken that each of those (e.g. NameError) is in stuff
>> -- they are in stuff['__builtins__'] which represents the built-in
>> namespace. You should remove that key from stuff as well.
>>
>> --Guido
>>
>>> Colin
>>>
>>> On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum <guido at python.org> wrote:
>>>> This is not easy to fix. The best short-term work-around is probably a
>>>> hack like this:
>>>>
>>>> def define_stuff(user_code):
>>>> ?context = {...}
>>>> ?stuff = {}
>>>> ?stuff.update(context)
>>>> ?exec(user_code, stuff)
>>>> ?for key in context:
>>>> ? ?if key in stuff and stuff[key] == context[key]:
>>>> ? ? ?del stuff[key]
>>>> ?return stuff
>>>>
>>>> --
>>>> --Guido van Rossum (python.org/~guido)
>>>>
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>

From scott+python-dev at scottdial.com  Thu May 27 15:09:55 2010
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Thu, 27 May 2010 09:09:55 -0400
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTimzzBg2SXcXPGXtPXB8iHvqvR4gvB4h4Wnmyi0Y@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>
	<4BFDB8EE.4010307@gmail.com>	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
	<AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com>
	<AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com>
	<AANLkTimAT0k1I1ESw8twwBinYeChNB-rQp3SwpSOGH1Z@mail.gmail.com>
	<AANLkTimzzBg2SXcXPGXtPXB8iHvqvR4gvB4h4Wnmyi0Y@mail.gmail.com>
Message-ID: <4BFE6F23.8030504@scottdial.com>

On 5/27/2010 7:14 AM, Colin H wrote:
> def define_stuff(user_code):
>   context = {...}
>   stuff = {}
>   stuff.update(context)
> 
>   exec(user_code, stuff)
> 
>   return_stuff = {}
>   return_stuff.update(stuff)
> 
>   del return_stuff['__builtins__']
>   for key in context:
>     if key in return_stuff and return_stuff[key] == context[key]:
>       del return_stuff[key]
> 
>   return return_stuff

I'm not sure your application, but I suspect you would benefit from
using an identity check instead of an __eq__ check. The equality check
may be expensive (e.g., a large dictionary), and I don't think it
actually is checking what you want -- if the user_code generates an
__eq__-similar dictionary, wouldn't you still want that? The only reason
I can see to use __eq__ is if you are trying to detect user_code
modifying an object passed in, which is something that wouldn't be
addressed by your original complaint about exec (as in, modifying a
global data structure).

Instead of:
>     if key in return_stuff and return_stuff[key] == context[key]:

Use:
>     if key in return_stuff and return_stuff[key] is context[key]:

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu

From hawkett at gmail.com  Thu May 27 15:39:28 2010
From: hawkett at gmail.com (Colin H)
Date: Thu, 27 May 2010 14:39:28 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFE6F23.8030504@scottdial.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com> 
	<AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com> 
	<AANLkTimAT0k1I1ESw8twwBinYeChNB-rQp3SwpSOGH1Z@mail.gmail.com> 
	<AANLkTimzzBg2SXcXPGXtPXB8iHvqvR4gvB4h4Wnmyi0Y@mail.gmail.com> 
	<4BFE6F23.8030504@scottdial.com>
Message-ID: <AANLkTilphYGqcFJ6_k6MT-OQO6IfD6iY8U-zBkrWCFtp@mail.gmail.com>

Yep fair call - was primarily modifying Guido's example to make the
point about not being able to delete from the globals returned from
exec - cheers,

Colin

On Thu, May 27, 2010 at 2:09 PM, Scott Dial
<scott+python-dev at scottdial.com> wrote:
> On 5/27/2010 7:14 AM, Colin H wrote:
>> def define_stuff(user_code):
>> ? context = {...}
>> ? stuff = {}
>> ? stuff.update(context)
>>
>> ? exec(user_code, stuff)
>>
>> ? return_stuff = {}
>> ? return_stuff.update(stuff)
>>
>> ? del return_stuff['__builtins__']
>> ? for key in context:
>> ? ? if key in return_stuff and return_stuff[key] == context[key]:
>> ? ? ? del return_stuff[key]
>>
>> ? return return_stuff
>
> I'm not sure your application, but I suspect you would benefit from
> using an identity check instead of an __eq__ check. The equality check
> may be expensive (e.g., a large dictionary), and I don't think it
> actually is checking what you want -- if the user_code generates an
> __eq__-similar dictionary, wouldn't you still want that? The only reason
> I can see to use __eq__ is if you are trying to detect user_code
> modifying an object passed in, which is something that wouldn't be
> addressed by your original complaint about exec (as in, modifying a
> global data structure).
>
> Instead of:
>> ? ? if key in return_stuff and return_stuff[key] == context[key]:
>
> Use:
>> ? ? if key in return_stuff and return_stuff[key] is context[key]:
>
> --
> Scott Dial
> scott at scottdial.com
> scodial at cs.indiana.edu
>

From hawkett at gmail.com  Thu May 27 16:14:44 2010
From: hawkett at gmail.com (Colin H)
Date: Thu, 27 May 2010 15:14:44 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFE6F23.8030504@scottdial.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<AANLkTikdxmGGXtq0maLNgbSSwbR5rsRGCn0Xbpw4Cqou@mail.gmail.com> 
	<AANLkTikwq94T5Xe7pFX30hhYqDjt95T-ZHEcUkQvnTBQ@mail.gmail.com> 
	<AANLkTimAT0k1I1ESw8twwBinYeChNB-rQp3SwpSOGH1Z@mail.gmail.com> 
	<AANLkTimzzBg2SXcXPGXtPXB8iHvqvR4gvB4h4Wnmyi0Y@mail.gmail.com> 
	<4BFE6F23.8030504@scottdial.com>
Message-ID: <AANLkTimOkt6xQAkiLy5ideKycmrC_oE5Ni4pFENWrmAT@mail.gmail.com>

Just to put a couple of alternatives on the table that don't break
existing code - not necessarily promoting them, or suggesting they
would be easy to do -

1. modify exec() to take an optional third argument - 'scope_type' -
if it is not supplied (but locals is), then it runs as class namespace
- i.e. identical to existing behaviour. If it is supplied then it will
run as whichever is specified, with function namespace being an
option.  The API already operates along these lines, with the second
argument being optional and implying module namespace if it is not
present.

2. a new API exec2() which uses function namespace, and deprecating
the old exec() - assuming there is agreement that function namespace
makes more sense than the class namespace, because there are real use
cases, and developers would generally expect this behaviour when
approaching the API for the first time.

From ncoghlan at gmail.com  Thu May 27 17:42:17 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 May 2010 01:42:17 +1000
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
Message-ID: <4BFE92D9.1060306@gmail.com>

On 27/05/10 10:38, Guido van Rossum wrote:
> On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan<ncoghlan at gmail.com>  wrote:
>> Lexical scoping only works for code that is compiled as part of a single
>> operation - the separation between the compilation of the individual string
>> and the code defining that string means that the symbol table analysis
>> needed for lexical scoping can't cross the boundary.
>
> Hi Nick,
>
> I don't think Colin was asking for such things.

Yes, I realised some time after sending that message that I'd gone off 
on a tangent unrelated to the original question (as a result of earlier 
parts of the discussion I'd been pondering the scoping differences 
between exec with two namespaces and a class definition and ended up 
writing about that instead of the topic Colin originally brought up).

I suspect Thomas is right that the current two namespace exec behaviour 
is mostly a legacy of the standard scoping before nested scopes were added.

To state the problem as succinctly as I can, the basic issue is that a 
code object which includes a function definition that refers to top 
level variables will execute correctly when the same namespace is used 
for both locals and globals (i.e. like module level code) but will fail 
when these namespaces are different (i.e. like code in class definition).

So long as the code being executed doesn't define any functions that 
refer to top level variables in the executed code the two argument form 
is currently perfectly usable, so deprecating it would be an overreaction.

However, attaining the (sensible) behaviour Colin is requesting when 
such top level variable references exist would actually be somewhat 
tricky. Considering Guido's suggestion to treat two argument exec like a 
function rather than a class and generate a closure with full lexical 
scoping a little further, I don't believe this could be done in exec 
itself without breaking code that expects the current behaviour. 
However, something along these lines could probably be managed as a new 
compilation mode for compile() (e.g. compile(code_str, name, 
"closure")), which would then allow these code objects to be passed to 
exec to get the desired behaviour.

Compare and contrast:

 >>> def f():
...   x = 1
...   def g():
...     print x
...   g()
...
 >>> exec f.func_code in globals(), {}
1

 >>> source = """\
... x = 1
... def g():
...   print x
... g()
... """
 >>> exec source in globals(), {}
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<string>", line 4, in <module>
   File "<string>", line 3, in g
NameError: global name 'x' is not defined

Breaking out dis.dis on these examples is fairly enlightening, as they 
generate *very* different bytecode for the definition of g().

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu May 27 17:46:13 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 May 2010 01:46:13 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFDDD64.9060701@scottdial.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>	<4BFD173B.9010703@gmail.com>	<htji55$qui$1@dough.gmane.org>
	<4BFDB6CB.7030104@gmail.com> <4BFDDD64.9060701@scottdial.com>
Message-ID: <4BFE93C5.6050905@gmail.com>

On 27/05/10 12:48, Scott Dial wrote:
> On 5/26/2010 8:03 PM, Nick Coghlan wrote:
>> On 27/05/10 02:27, Terry Reedy wrote:
>>> I am suggesting that if we add a package, we do it right, from the
>>> beginning.
>>
>> This is a reasonable point of view, but I wouldn't want to hold up PEP
>> 3148 over it (call it a +0 for the idea in general, but a -1 for linking
>> it to the acceptance of PEP 3148).
>
> That sounds backward. How can you justify accepting PEP 3148 into a
> "concurrent" namespace without also accepting the demand for such a
> namespace? What is the contingency if this TBD migration PEP is not
> accepted, what happens to PEP 3148? After all, there was some complaints
> about just calling it "futures", without putting it in a "concurrent"
> namespace.

We can accept PEP 3148 by saying that we're happy to add the extra 
namespace level purely for disambiguation purposes, even if we never 
follow through on adding anything else to the package (although I 
consider such an outcome to be highly unlikely).

Any future additions or renames to move things into the concurrent 
package would then be handled as their own PEPs.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu May 27 17:55:21 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 May 2010 01:55:21 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFDD908.90609@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<058F5600-0D81-4431-94EF-554482F80498@twistedmatrix.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<4BF916EA.2090507@holdenweb.com>	<AANLkTimZ6yTLTf6GuU3n7HuQJVNsvn4qYcRqwqLTtG8U@mail.gmail.com>	<20100523144754.1c6f1afc@pitrou.net>
	<4BFC6325.5050700@gmail.com>	<20100526093820.4fbf2f87@pitrou.net>
	<4BFD14E1.6000305@gmail.com>	<20100526145033.3b61d95b@pitrou.net>	<A3704B8E-2B63-4B97-90EA-B1137FF6566C@sweetapp.com>	<4BFD26A8.9050802@gmail.com>
	<4BFDD908.90609@canterbury.ac.nz>
Message-ID: <4BFE95E9.9030504@gmail.com>

On 27/05/10 12:29, Greg Ewing wrote:
> On 27/05/10 01:48, Nick Coghlan wrote:
>
>> I would say it is precisely that extra configurability which separates
>> the executor pools in the PEP implementation from more flexible general
>> purpose pools.
>
> Wouldn't this be better addressed by adding the relevant
> options to the futures pools, rather than adding another
> module that provides almost exactly the same thing with
> different options?

It would depend on the details, but my instinct says no (instead, the 
futures pools would be refactored to be task specific tailorings of the 
general purpose pools).

However, this is all very hypothetical at this point and not really 
relevant to the PEP review. We may never even bother creating these more 
general purpose threading pools - it was just an example of the kind of 
thing that may go alongside the futures module.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From stephen at xemacs.org  Thu May 27 17:56:33 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 28 May 2010 00:56:33 +0900
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
	<AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>
Message-ID: <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp>

Paul Moore writes:
 > On 27 May 2010 00:11, geremy condra <debatem1 at gmail.com> wrote:
 > > I'm not clear, you seem to be arguing that there's a market for many
 > > augmented python distributions but not one. Why not just have one
 > > that includes the best from each domain?
 > 
 > Because that's "bloat". You later argue that a web designer wouldn't
 > care if his "distribution" included numpy. OK, maybe, but if my needs
 > are simply futures, cx_Oracle and pywin32, I *would* object to
 > downloading many megabytes of other stuff just to get those three.
 > It's a matter of degree.

So don't do that.  Go to PyPI and get just what you need.

The point of the sumo is that there are people and organizations with
more bandwidth/diskspace than brains (or to be more accurate, they
have enough bandwidth that optimizing bandwidth is a poor use of their
brains).

XEmacs has used a separate distribution for packages for over a
decade, and it's been a very popular feature.  Since originally all
packages were part of Emacs (and still are in GNU Emacs), the package
distribution is a single source hierarchy (like the Python stdlib).
So there are three ways of acquiring packages: check out the sources
and build and install them, download individual pre-built packages,
and download the sumo of all pre-built packages.  The sumos are very
popular.

The reason is simple.  A distribution of all Emacs packages ever made
would still probably be under 1GB.  This just isn't a lot of bandwidth
or disk space when people are schlepping around DVD images, even BD
images.  A Python sumo would probably be much bigger (multiple GB)
than XEmacs's (about 120MB, IIRC), but it would still be a negligible
amount of resources *for some people/organizations*.

And I have to support the "organizational constraints" argument here.
Several people have told me that (strangely enough, given its rather
random nature, both in what is provided and the quality) getting the
sumo certified by their organization was less trouble than getting
XEmacs itself certified, and neither was all that much more effort
than getting a single package certified.

Maintaining a sumo would be a significant effort.  The XEmacs rule is
that we allow someone to add a package to the distro if they promise
to maintain it for a couple years, or if we think it matters enough
that we'll accept the burden.  We're permissive enough that there are
at least 4 different MUAs in the distribution, several IRC clients,
two TeX modes, etc, etc.  Still, just maintaining contact with
"external maintainers" (who do go AWOL regularly), and dealing with
issues where somebody wants to upgrade (eg) "vcard" which is provided
by "gnus" but doesn't want to deal with "gnus", etc takes time,
thought, and sometimes improvement in the distribution
infrastructure.

It's not clear to me that Python users would benefit that much over
and above the current stdlib, which provides a huge amount of
functionality, of somewhat uneven but generally high quality.  But I
certainly think significant additional benefit would be gained, the
question is is it worth the effort?  It's worth discussing.

 > I don't believe that there's evidence that aggregation (except in the
 > context of specialist areas) does provide additional utility.

We'll just have to agree to disagree, then.  Plenty of evidence has
been provided; it just doesn't happen to apply to you.  Fine, but I
wish you'd make the "to me" part explicit, because I know that it does
apply to others, many of them, from their personal testimony, both
related to XEmacs and to Python.

 > PS One thing I haven't seen made clear - in my view, they hypothetical
 > "sumo" is a single aggregated distribution of Python
 > modules/packages/extensions. It would NOT include core Python and the
 > stdlib (in contrast to Enthought or ActivePython). I get the
 > impression that other people may be thinking in terms of a full Python
 > distribution, like those 2 cases. We probably ought to be clear which
 > we're talking about.

On the XEmacs model, it would not include core Python, but it would
include much of the stdlib.  The reason is that the stdlib makes
commitments to compatibility that the sumo would not need to.  So the
sumo might include (a) recent, relatively experimental versions of
stdlib packages (yes, this kind of duplication is a pain, but (some)
users do want it) and (b) packages which are formally separate but
duplicate functionality in the stdlib (eg, ipaddr and netaddr) -- in
some cases the sumo distro would want to make adjustments so they can
co-exist.

I wouldn't recommend building a production system on top of a sumo in
any case.  But (given resources to maintain multiple Python development
installations) it is a good environment for experimentation, because
not only batteries but screwdrivers and duct tape are supplied.


From ncoghlan at gmail.com  Thu May 27 17:58:55 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 May 2010 01:58:55 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<20100526120651.GA12751@laurie.devork.be>	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>	<4BFDCEDF.1060303@canterbury.ac.nz>	<20100527075349.GA32291@laurie.devork.be>
	<2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
Message-ID: <4BFE96BF.5010400@gmail.com>

On 27/05/10 18:13, Brian Quinlan wrote:
>
> On 27 May 2010, at 17:53, Floris Bruynooghe wrote:
>> I'm glad I'm not alone in preferring (1) tough.
>
> Keep in mind that this library magic is consistent with the library
> magic that the threading module does - unless the user sets
> Thread.daemon to True, the interpreter does *not* exit until the thread
> does.

Along those lines, an Executor.daemon option may be a good idea. That 
way the default behaviour is to wait until things are done (just like 
threading itself), but it is easy for someone to turn that behaviour off 
for a specific executor.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu May 27 18:05:14 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 May 2010 02:05:14 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100527022935.2cfd04c5@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>	<4BFDB067.4060800@canterbury.ac.nz>
	<4BFDBAA6.2010104@gmail.com> <20100527022935.2cfd04c5@pitrou.net>
Message-ID: <4BFE983A.8080909@gmail.com>

On 27/05/10 10:29, Antoine Pitrou wrote:
> On Thu, 27 May 2010 10:19:50 +1000
> Nick Coghlan<ncoghlan at gmail.com>  wrote:
>>
>> futures.ThreadPoolExecutor would likely be refactored to inherit from
>> the mooted pool.ThreadPool.
>
> There still doesn't seem to be reason to have two different thread pool
> APIs, though. Shouldn't there be one obvious way to do it?

Executors and thread pools are not the same thing.

I might create a thread pool for *anything*. An executor will always 
have a specific execution model associated with it (whether it be called 
futures, as in this case, or runnables or something else).

This confusion is making me think that dropping the "Pool" from the 
names might even be beneficial (since, to my mind, it currently 
emphasises a largely irrelevant implementation detail).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu May 27 18:08:02 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 May 2010 02:08:02 +1000
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFDE34C.1080405@canterbury.ac.nz>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>	<4BFDB10A.4080002@canterbury.ac.nz>
	<4BFDAFDA.7060404@voidspace.org.uk>
	<4BFDE34C.1080405@canterbury.ac.nz>
Message-ID: <4BFE98E2.8020302@gmail.com>

On 27/05/10 13:13, Greg Ewing wrote:
> The way that functions get access to names in enclosing
> local scopes is by having them passed in as cells, but that
> mechanism is only available for optimised local namespaces,
> not ones implemented as dicts.

I believe exec already includes the tapdancing needed to make that work.

As Guido pointed out, it's the ability to generate closures directly 
from a source string that is currently missing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From stephen at xemacs.org  Thu May 27 18:12:35 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 28 May 2010 01:12:35 +0900
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTim7dKO2Xjyut08KRBmGmitrjSpL_XGR8qE7X4Wi@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
	<4BFDB566.2060502@gmail.com>
	<AANLkTilrTep2PPYtVPwegXHmzMI5hJl-qE0I670Qzxit@mail.gmail.com>
	<AANLkTiml1xTGQnh2EVlphXYQMlcPmA_DoLLhOwZqJWWK@mail.gmail.com>
	<AANLkTin_C83h82cAR3zDPJ5udQZdZ__5DZgcnmLNcOoN@mail.gmail.com>
	<AANLkTim7dKO2Xjyut08KRBmGmitrjSpL_XGR8qE7X4Wi@mail.gmail.com>
Message-ID: <87vda9jpcc.fsf@uwakimon.sk.tsukuba.ac.jp>

Lennart Regebro writes:

 > One  worry with an official sumo distribution is that it could become
 > an excuse for *not* putting something in the stdlib.
 > Otherwise it's an interesting idea.

On the contrary, that is the meat of why it's an interesting idea.

I really don't think the proponents of ipaddr and futures (to take two
recent PEPs) would have been willing to stop with a hypothetical sumo.
Both of those packages were designed with general use in mind.
Substantial effort was put into making them TOOWTDI-able.  Partly
that's pride ("my stuff is good enough for the stdlib"), and partly
there's a genuine need for it to be there (for your customers or just
to pay back the community).  Of course there was a lot of criticism of
both that they don't really come up to that standard, but even
opponents would credit the proponents for good intentions and making
the necessary effort, I think.  And it's the stdlib that (in a certain
sense) puts the "OO" in "TOOWTDI".

On the other hand, some ideas deserve widespread exposure, but they
need real experience because the appropriate requirements and specs
are unclear.  It would be premature to put in the effort to make them
TOOWTDI.  However, to get the momentum to become BCP, and thus an
obvious candidate for stdlib inclusion, it's helpful to be *already*
available on *typical* installations.  PyPI is great, but it's not
quite there; it's not as discoverable and accessible as simply putting
"import stuff" based on some snippet you found on the web.  And the
stdlib itself can't be the means, it's the end.

At present, such ideas face the alternative "stdlib or die".  The sumo
would give them a place to be.


From solipsis at pitrou.net  Thu May 27 18:16:38 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 27 May 2010 18:16:38 +0200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<4BFDB067.4060800@canterbury.ac.nz> <4BFDBAA6.2010104@gmail.com>
	<20100527022935.2cfd04c5@pitrou.net> <4BFE983A.8080909@gmail.com>
Message-ID: <20100527181638.2164a9e1@pitrou.net>

On Fri, 28 May 2010 02:05:14 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Executors and thread pools are not the same thing.
> 
> I might create a thread pool for *anything*. An executor will always 
> have a specific execution model associated with it (whether it be called 
> futures, as in this case, or runnables or something else).

I'm sorry, but what is the specific execution model you are talking
about, and how is it different from what you usually do with a thread
pool?  Why would you do something other with a thread pool than running
tasks and (optionally) collecting their results?

Thanks

Antoine.



From fuzzyman at voidspace.org.uk  Thu May 27 18:22:10 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 27 May 2010 17:22:10 +0100
Subject: [Python-Dev] Sumo
In-Reply-To: <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>	<20100526113915.5bf46665@pitrou.net>	<201005262042.12825.steve@pearwood.info>	<20100526125614.0a7ae470@pitrou.net>	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>	<1274877963.3162.20.camel@localhost.localdomain>	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>	<AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>
	<87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4BFE9C32.8050209@voidspace.org.uk>

On 27/05/2010 16:56, Stephen J. Turnbull wrote:
> Paul Moore writes:
>   >  On 27 May 2010 00:11, geremy condra<debatem1 at gmail.com>  wrote:
>   >  >  I'm not clear, you seem to be arguing that there's a market for many
>   >  >  augmented python distributions but not one. Why not just have one
>   >  >  that includes the best from each domain?
>   >
>   >  Because that's "bloat". You later argue that a web designer wouldn't
>   >  care if his "distribution" included numpy. OK, maybe, but if my needs
>   >  are simply futures, cx_Oracle and pywin32, I *would* object to
>   >  downloading many megabytes of other stuff just to get those three.
>   >  It's a matter of degree.
>
> So don't do that.  Go to PyPI and get just what you need.
>
> The point of the sumo is that there are people and organizations with
> more bandwidth/diskspace than brains (or to be more accurate, they
> have enough bandwidth that optimizing bandwidth is a poor use of their
> brains).
>    
To my mind one of the most important benefits of a "sumo" style 
distribution is not just that it easily provides a whole bunch of useful 
modules - but that it *highlights* which modules are the community 
blessed "best of breed". At the moment if a new user wants to work out 
how to achieve a particular task (work with images for example) they 
have to google around and try and work out what the right module to use is.

For some problem domains there are a host of modules on PyPI many of 
which are unmaintained, immature or simply rubbish. A standardised 
solution makes choosing solutions for common problems *dramatically* 
easier, and may save people much heartache and frustration. For that to 
work though it needs to be well curated and genuinely have the 
substantial backing of the Python development community.

All the best,

Michael

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From stephen at xemacs.org  Thu May 27 18:31:56 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 28 May 2010 01:31:56 +0900
Subject: [Python-Dev] Sumo
In-Reply-To: <AANLkTik8PhfRKC_nygAtVTiHxxxRDMLsAQJRJ9FZb-I-@mail.gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
	<AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>
	<AANLkTik8PhfRKC_nygAtVTiHxxxRDMLsAQJRJ9FZb-I-@mail.gmail.com>
Message-ID: <87typtjog3.fsf@uwakimon.sk.tsukuba.ac.jp>

Lennart Regebro writes:

 > If licensing is a problem I guess you'd need to have permission to
 > relicense them all to the Python license,

Licensing compatibility is only a problem for copyleft, but most
copyleft licenses have "mere aggregation is not derivation" clauses.

Corporate concern about *knowing* what the license is, is a problem.
The XEmacs experience I've referred to elsewhere doesn't apply because
all our stuff is GPL, and therefore all our stuff has to be GPL. :-(
It's not obvious to me what the resolution is, although lots of
distributions now have some way to find out what licenses are.  GCC
(and soon GNU Emacs) even have a way to check GPL-compatibility at
runtime (inspired by the Linux kernel feature, maybe?)

Perhaps the sumo infrastructure could provide a license-ok-or-fatal
feature.  Eg, the application would do something like

sumo_ok_licenses = ['gplv2','bsd','microsoft_eula']

and the sumo version of the package's __init.py__ would do

    sumo_check_my_license('artistic')

and raise LicenseError if it's not in sumo_ok_licenses.  In theory it
might be able to do more complex stuff like keep track of declared
licenses and barf if they're incompatible.

This scheme probably doesn't save lawyer time.  The lawyers would
still have to go over the actual packages to make sure the licenses
are what they say etc. before going into distribution.  Its selling
point is that the developers would be warned of problems that need
corporate legal's attention early in 90% of the cases, thus not
wasting developer time on using packages that were non-starters
because of license issues.

 > which would be problematic.  But otherwise having a team of people
 > overseeing and bundling all this might not be that much work, and
 > you'd avoid the bloat by not installing all of it. :-)

As I've argued elsewhere, bloat is good, for some purposes.

 > Or would this not fool the company trolls?

It will satisfy some, and not others, in my experience, described
elsewhere.


From ncoghlan at gmail.com  Thu May 27 19:08:40 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 May 2010 03:08:40 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <20100527181638.2164a9e1@pitrou.net>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>	<4BFDB067.4060800@canterbury.ac.nz>
	<4BFDBAA6.2010104@gmail.com>	<20100527022935.2cfd04c5@pitrou.net>
	<4BFE983A.8080909@gmail.com> <20100527181638.2164a9e1@pitrou.net>
Message-ID: <4BFEA718.6070007@gmail.com>

On 28/05/10 02:16, Antoine Pitrou wrote:
> On Fri, 28 May 2010 02:05:14 +1000
> Nick Coghlan<ncoghlan at gmail.com>  wrote:
>>
>> Executors and thread pools are not the same thing.
>>
>> I might create a thread pool for *anything*. An executor will always
>> have a specific execution model associated with it (whether it be called
>> futures, as in this case, or runnables or something else).
>
> I'm sorry, but what is the specific execution model you are talking
> about, and how is it different from what you usually do with a thread
> pool?  Why would you do something other with a thread pool than running
> tasks and (optionally) collecting their results?

Both the execution and communications models may be different. The 
components may be long-lived state machines, they may be active objects, 
they may communicate by message passing or by manipulating shared state, 
who knows. Executors are designed around a model of "go do this and let 
me know when you're done". A general purpose pool needs to support other 
execution models, and hence will look different (and is harder to design 
and write, since it needs to be more flexible).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From stephen at xemacs.org  Thu May 27 19:52:34 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 28 May 2010 02:52:34 +0900
Subject: [Python-Dev] Sumo
In-Reply-To: <4BFE9C32.8050209@voidspace.org.uk>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
	<AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>
	<87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4BFE9C32.8050209@voidspace.org.uk>
Message-ID: <87r5kxjkpp.fsf@uwakimon.sk.tsukuba.ac.jp>

Michael Foord writes:

 > To my mind one of the most important benefits of a "sumo" style 
 > distribution is not just that it easily provides a whole bunch of useful 
 > modules - but that it *highlights* which modules are the community 
 > blessed "best of breed".

That has several problems.

(1) There is a lot of overlap with the mission of the stdlib, and I
    think confusion over roles would be quite costly.

(2) As the stdlib demonstrates, picking winners is expensive.  I
    greatly doubt that running *two* such processes is worthwhile.

(3) Very often there is no best of breed.

From p.f.moore at gmail.com  Thu May 27 21:44:30 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 27 May 2010 20:44:30 +0100
Subject: [Python-Dev] Sumo
In-Reply-To: <87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<F199CF19-EE5C-49A9-B47C-1D04DE6CF3BE@twistedmatrix.com>
	<20100526113915.5bf46665@pitrou.net>
	<201005262042.12825.steve@pearwood.info>
	<20100526125614.0a7ae470@pitrou.net>
	<AANLkTilTBTLBu-Yo3IOHJR6qLnDnjxH-k8Njy4E2maOV@mail.gmail.com>
	<1274877963.3162.20.camel@localhost.localdomain>
	<AANLkTin_UbRbeO0u8s5rja9YuzC5QKdA3s_-P7UJCJIW@mail.gmail.com>
	<AANLkTimjrG1BGpn3TkhI3g1CZxsywdvy-vv_uBclYhvL@mail.gmail.com>
	<AANLkTinA8jiUi58nXXGwC0TJtAeog-D5gYZ-nPEPGk6f@mail.gmail.com>
	<87wrupjq32.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <AANLkTinFnWPdxLU6nrphAsMx_9-IBAkLOpdvo7X-QbbQ@mail.gmail.com>

On 27 May 2010 16:56, Stephen J. Turnbull <stephen at xemacs.org> wrote:

> We'll just have to agree to disagree, then. ?Plenty of evidence has
> been provided; it just doesn't happen to apply to you. ?Fine, but I
> wish you'd make the "to me" part explicit, because I know that it does
> apply to others, many of them, from their personal testimony, both
> related to XEmacs and to Python.

Sorry, you're right. There's a very strong "to me" in all of this, but
I more or less assumed it was obvious, as I was originally responding
to comments implying that a sumo distribution was a solution to a
problem I stated that I have. In trying to trim things, and keep
things concise, I completely lost the context. My apologies.

> I wouldn't recommend building a production system on top of a sumo in
> any case. ?But (given resources to maintain multiple Python development
> installations) it is a good environment for experimentation, because
> not only batteries but screwdrivers and duct tape are supplied.

That's an interesting perspective that I hadn't seen mentioned before.
For experimentation, I'd *love* a sumo distribution as you describe.
But I thought this whole discussion focussed around building
production systems. For that, the stdlib's quality guarantees are a
major benefit, and the costs of locating and validating appropriately
high-quality external packages are (sometimes prohibitively) high.

But I think I'm getting to the point where I'm adding more confusion
than information, so I'll bow out of this discussion at this point.

Paul.

From rnk at mit.edu  Thu May 27 22:05:26 2010
From: rnk at mit.edu (Reid Kleckner)
Date: Thu, 27 May 2010 16:05:26 -0400
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFE92D9.1060306@gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<4BFE92D9.1060306@gmail.com>
Message-ID: <AANLkTimUiP7GGPRM5hTyf50S4Z8yGubzpWvwX3P_Zlvz@mail.gmail.com>

On Thu, May 27, 2010 at 11:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> However, attaining the (sensible) behaviour Colin is requesting when such
> top level variable references exist would actually be somewhat tricky.
> Considering Guido's suggestion to treat two argument exec like a function
> rather than a class and generate a closure with full lexical scoping a
> little further, I don't believe this could be done in exec itself without
> breaking code that expects the current behaviour.

Just to give a concrete example, here is code that would break if exec
were to execute code in a function scope instead of a class scope:

exec """
def len(xs):
    return -1
def foo():
    return len([])
print foo()
""" in globals(), {}

Currently, the call to 'len' inside 'foo' skips the outer scope
(because it's a class scope) and goes straight to globals and
builtins.  If it were switched to a local scope, a cell would be
created for the broken definition of 'len', and the call would resolve
to it.

Honestly, to me, the fact that the above code ever worked (ie prints
"0", not "-1") seems like a bug, so I wouldn't worry about backwards
compatibility.

Reid

From greg.ewing at canterbury.ac.nz  Fri May 28 01:18:47 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 28 May 2010 11:18:47 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <B3CACAFC-1B77-4D03-8EBE-000081D193B5@sweetapp.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<4BFDB067.4060800@canterbury.ac.nz>
	<AANLkTinmitR_ateAN0cgcQpUWUnrFAsO6tlXDdBc3Bgb@mail.gmail.com>
	<4BFDE52A.7040209@canterbury.ac.nz>
	<B3CACAFC-1B77-4D03-8EBE-000081D193B5@sweetapp.com>
Message-ID: <4BFEFDD7.70001@canterbury.ac.nz>

Brian Quinlan wrote:

> I think that the "Executor" suffix is a good indicator of the  interface 
> being provided.

It's not usually considered necessary for the name of a
type to indicate its interface. We don't have 'listsequence'
and 'dictmapping' for example.

I think what bothers me most about these names is their
longwindedness. Two parts to a name is okay, but three or
more starts to sound pedantic. And for me, "Pool" is a
more important piece of information than "Executor".
The fact that it manages a pool is the main reason I'd
use such a module rather than just spawning a thread myself
for each task.

-- 
Greg

From hawkett at gmail.com  Fri May 28 01:32:18 2010
From: hawkett at gmail.com (Colin H)
Date: Fri, 28 May 2010 00:32:18 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <4BFE92D9.1060306@gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<4BFE92D9.1060306@gmail.com>
Message-ID: <AANLkTikkNDTDqot2vC6fapVp6dlmk8EVbQts_2_WBhB8@mail.gmail.com>

This option sounds very promising - seems right to do it at the
compile stage - i.e. compile(code_str, name, "closure") as you have
suggested.  If there were any argument against, it would be that the
most obvious behaviour (function namespace) is the hardest to induce,
but the value in knowing you're not breaking anything is pretty high.

Cheers,
Colin

On Thu, May 27, 2010 at 4:42 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27/05/10 10:38, Guido van Rossum wrote:
>>
>> On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan<ncoghlan at gmail.com> ?wrote:
>>>
>>> Lexical scoping only works for code that is compiled as part of a single
>>> operation - the separation between the compilation of the individual
>>> string
>>> and the code defining that string means that the symbol table analysis
>>> needed for lexical scoping can't cross the boundary.
>>
>> Hi Nick,
>>
>> I don't think Colin was asking for such things.
>
> Yes, I realised some time after sending that message that I'd gone off on a
> tangent unrelated to the original question (as a result of earlier parts of
> the discussion I'd been pondering the scoping differences between exec with
> two namespaces and a class definition and ended up writing about that
> instead of the topic Colin originally brought up).
>
> I suspect Thomas is right that the current two namespace exec behaviour is
> mostly a legacy of the standard scoping before nested scopes were added.
>
> To state the problem as succinctly as I can, the basic issue is that a code
> object which includes a function definition that refers to top level
> variables will execute correctly when the same namespace is used for both
> locals and globals (i.e. like module level code) but will fail when these
> namespaces are different (i.e. like code in class definition).
>
> So long as the code being executed doesn't define any functions that refer
> to top level variables in the executed code the two argument form is
> currently perfectly usable, so deprecating it would be an overreaction.
>
> However, attaining the (sensible) behaviour Colin is requesting when such
> top level variable references exist would actually be somewhat tricky.
> Considering Guido's suggestion to treat two argument exec like a function
> rather than a class and generate a closure with full lexical scoping a
> little further, I don't believe this could be done in exec itself without
> breaking code that expects the current behaviour. However, something along
> these lines could probably be managed as a new compilation mode for
> compile() (e.g. compile(code_str, name, "closure")), which would then allow
> these code objects to be passed to exec to get the desired behaviour.
>
> Compare and contrast:
>
>>>> def f():
> ... ? x = 1
> ... ? def g():
> ... ? ? print x
> ... ? g()
> ...
>>>> exec f.func_code in globals(), {}
> 1
>
>>>> source = """\
> ... x = 1
> ... def g():
> ... ? print x
> ... g()
> ... """
>>>> exec source in globals(), {}
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> ?File "<string>", line 4, in <module>
> ?File "<string>", line 3, in g
> NameError: global name 'x' is not defined
>
> Breaking out dis.dis on these examples is fairly enlightening, as they
> generate *very* different bytecode for the definition of g().
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan ? | ? ncoghlan at gmail.com ? | ? Brisbane, Australia
> ---------------------------------------------------------------
>

From hawkett at gmail.com  Fri May 28 01:33:25 2010
From: hawkett at gmail.com (Colin H)
Date: Fri, 28 May 2010 00:33:25 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTikkNDTDqot2vC6fapVp6dlmk8EVbQts_2_WBhB8@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<4BFE92D9.1060306@gmail.com>
	<AANLkTikkNDTDqot2vC6fapVp6dlmk8EVbQts_2_WBhB8@mail.gmail.com>
Message-ID: <AANLkTilR5FPDfrcSWIshfZNGlMBaKyUJft3erCdxw7JY@mail.gmail.com>

By hardest to induce I mean the default compile exec(code_str, {}, {})
would still be class namespace, but it's pretty insignificant.

On Fri, May 28, 2010 at 12:32 AM, Colin H <hawkett at gmail.com> wrote:
> This option sounds very promising - seems right to do it at the
> compile stage - i.e. compile(code_str, name, "closure") as you have
> suggested. ?If there were any argument against, it would be that the
> most obvious behaviour (function namespace) is the hardest to induce,
> but the value in knowing you're not breaking anything is pretty high.
>
> Cheers,
> Colin
>
> On Thu, May 27, 2010 at 4:42 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 27/05/10 10:38, Guido van Rossum wrote:
>>>
>>> On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan<ncoghlan at gmail.com> ?wrote:
>>>>
>>>> Lexical scoping only works for code that is compiled as part of a single
>>>> operation - the separation between the compilation of the individual
>>>> string
>>>> and the code defining that string means that the symbol table analysis
>>>> needed for lexical scoping can't cross the boundary.
>>>
>>> Hi Nick,
>>>
>>> I don't think Colin was asking for such things.
>>
>> Yes, I realised some time after sending that message that I'd gone off on a
>> tangent unrelated to the original question (as a result of earlier parts of
>> the discussion I'd been pondering the scoping differences between exec with
>> two namespaces and a class definition and ended up writing about that
>> instead of the topic Colin originally brought up).
>>
>> I suspect Thomas is right that the current two namespace exec behaviour is
>> mostly a legacy of the standard scoping before nested scopes were added.
>>
>> To state the problem as succinctly as I can, the basic issue is that a code
>> object which includes a function definition that refers to top level
>> variables will execute correctly when the same namespace is used for both
>> locals and globals (i.e. like module level code) but will fail when these
>> namespaces are different (i.e. like code in class definition).
>>
>> So long as the code being executed doesn't define any functions that refer
>> to top level variables in the executed code the two argument form is
>> currently perfectly usable, so deprecating it would be an overreaction.
>>
>> However, attaining the (sensible) behaviour Colin is requesting when such
>> top level variable references exist would actually be somewhat tricky.
>> Considering Guido's suggestion to treat two argument exec like a function
>> rather than a class and generate a closure with full lexical scoping a
>> little further, I don't believe this could be done in exec itself without
>> breaking code that expects the current behaviour. However, something along
>> these lines could probably be managed as a new compilation mode for
>> compile() (e.g. compile(code_str, name, "closure")), which would then allow
>> these code objects to be passed to exec to get the desired behaviour.
>>
>> Compare and contrast:
>>
>>>>> def f():
>> ... ? x = 1
>> ... ? def g():
>> ... ? ? print x
>> ... ? g()
>> ...
>>>>> exec f.func_code in globals(), {}
>> 1
>>
>>>>> source = """\
>> ... x = 1
>> ... def g():
>> ... ? print x
>> ... g()
>> ... """
>>>>> exec source in globals(), {}
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> ?File "<string>", line 4, in <module>
>> ?File "<string>", line 3, in g
>> NameError: global name 'x' is not defined
>>
>> Breaking out dis.dis on these examples is fairly enlightening, as they
>> generate *very* different bytecode for the definition of g().
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan ? | ? ncoghlan at gmail.com ? | ? Brisbane, Australia
>> ---------------------------------------------------------------
>>
>

From brian at sweetapp.com  Fri May 28 01:37:19 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 28 May 2010 09:37:19 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFEFDD7.70001@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<4BFDB067.4060800@canterbury.ac.nz>
	<AANLkTinmitR_ateAN0cgcQpUWUnrFAsO6tlXDdBc3Bgb@mail.gmail.com>
	<4BFDE52A.7040209@canterbury.ac.nz>
	<B3CACAFC-1B77-4D03-8EBE-000081D193B5@sweetapp.com>
	<4BFEFDD7.70001@canterbury.ac.nz>
Message-ID: <BAA75A83-5256-4DBD-B295-575D537AD51E@sweetapp.com>


On 28 May 2010, at 09:18, Greg Ewing wrote:
> Brian Quinlan wrote:
>
>> I think that the "Executor" suffix is a good indicator of the   
>> interface being provided.
>
> It's not usually considered necessary for the name of a
> type to indicate its interface. We don't have 'listsequence'
> and 'dictmapping' for example.
>
> I think what bothers me most about these names is their
> longwindedness. Two parts to a name is okay, but three or
> more starts to sound pedantic. And for me, "Pool" is a
> more important piece of information than "Executor".
> The fact that it manages a pool is the main reason I'd
> use such a module rather than just spawning a thread myself
> for each task.


Actually, an executor implementation that created a new thread per  
task would still be useful - it would save you the hassle of  
developing a mechanism to wait for the thread to finish and to collect  
the results. We actually have such an implementation at Google and it  
is quite popular.

Cheers,
Brian

From greg.ewing at canterbury.ac.nz  Fri May 28 01:52:52 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 28 May 2010 11:52:52 +1200
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFE93C5.6050905@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz> <4BFD173B.9010703@gmail.com>
	<htji55$qui$1@dough.gmane.org> <4BFDB6CB.7030104@gmail.com>
	<4BFDDD64.9060701@scottdial.com> <4BFE93C5.6050905@gmail.com>
Message-ID: <4BFF05D4.4000202@canterbury.ac.nz>

Nick Coghlan wrote:

> We can accept PEP 3148 by saying that we're happy to add the extra 
> namespace level purely for disambiguation purposes,

If that's the only rationale for the namespace, it makes it
sound like a kludge to work around a poor choice of name.

-- 
Greg

From rnk at mit.edu  Fri May 28 03:57:48 2010
From: rnk at mit.edu (Reid Kleckner)
Date: Thu, 27 May 2010 21:57:48 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> 
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com> 
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net> 
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com> 
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com> 
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<20100526120651.GA12751@laurie.devork.be> 
	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>
	<4BFDCEDF.1060303@canterbury.ac.nz> 
	<20100527075349.GA32291@laurie.devork.be>
	<2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
Message-ID: <AANLkTikcDkK1CxaJ5SZ2-2CKS13PRsOfc2M-ZI7DkrEJ@mail.gmail.com>

On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan <brian at sweetapp.com> wrote:
> Keep in mind that this library magic is consistent with the library magic
> that the threading module does - unless the user sets Thread.daemon to True,
> the interpreter does *not* exit until the thread does.

Is there a compelling to make the threads daemon threads?  If not,
perhaps they can just be normal threads, and you can rely on the
threading module to wait for them to finish.

Unrelatedly, I feel like this behavior of waiting for the thread to
terminate usually manifests as deadlocks when the main thread throws
an uncaught exception.  The application then no longer responds
properly to interrupts, since it's stuck waiting on a semaphore.  I
guess it's better than the alternative of random crashes when daemon
threads wake up during interpreter shutdown, though.

Reid

From brian at sweetapp.com  Fri May 28 05:06:02 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 28 May 2010 13:06:02 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <AANLkTikcDkK1CxaJ5SZ2-2CKS13PRsOfc2M-ZI7DkrEJ@mail.gmail.com>
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<20100526120651.GA12751@laurie.devork.be>
	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>
	<4BFDCEDF.1060303@canterbury.ac.nz>
	<20100527075349.GA32291@laurie.devork.be>
	<2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
	<AANLkTikcDkK1CxaJ5SZ2-2CKS13PRsOfc2M-ZI7DkrEJ@mail.gmail.com>
Message-ID: <31F8D1F3-13C9-4211-B808-388DB2CC705F@sweetapp.com>


On May 28, 2010, at 11:57 AM, Reid Kleckner wrote:

> On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> Keep in mind that this library magic is consistent with the library  
>> magic
>> that the threading module does - unless the user sets Thread.daemon  
>> to True,
>> the interpreter does *not* exit until the thread does.
>
> Is there a compelling to make the threads daemon threads?  If not,
> perhaps they can just be normal threads, and you can rely on the
> threading module to wait for them to finish.

Did you read my explanation of the reasoning behind my approach?

Cheers,
Brian

> Unrelatedly, I feel like this behavior of waiting for the thread to
> terminate usually manifests as deadlocks when the main thread throws
> an uncaught exception.  The application then no longer responds
> properly to interrupts, since it's stuck waiting on a semaphore.  I
> guess it's better than the alternative of random crashes when daemon
> threads wake up during interpreter shutdown, though.
>
> Reid


From jyasskin at gmail.com  Fri May 28 05:16:04 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Thu, 27 May 2010 20:16:04 -0700
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <31F8D1F3-13C9-4211-B808-388DB2CC705F@sweetapp.com>
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com> 
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com> 
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net> 
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com> 
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com> 
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<20100526120651.GA12751@laurie.devork.be> 
	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>
	<4BFDCEDF.1060303@canterbury.ac.nz> 
	<20100527075349.GA32291@laurie.devork.be>
	<2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com> 
	<AANLkTikcDkK1CxaJ5SZ2-2CKS13PRsOfc2M-ZI7DkrEJ@mail.gmail.com> 
	<31F8D1F3-13C9-4211-B808-388DB2CC705F@sweetapp.com>
Message-ID: <AANLkTimLwZRCdOZ1angClU5FtTEARV6_aXni_yS_4Xn3@mail.gmail.com>

On Thu, May 27, 2010 at 8:06 PM, Brian Quinlan <brian at sweetapp.com> wrote:
>
> On May 28, 2010, at 11:57 AM, Reid Kleckner wrote:
>
>> On Thu, May 27, 2010 at 4:13 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>>>
>>> Keep in mind that this library magic is consistent with the library magic
>>> that the threading module does - unless the user sets Thread.daemon to
>>> True,
>>> the interpreter does *not* exit until the thread does.
>>
>> Is there a compelling to make the threads daemon threads? ?If not,
>> perhaps they can just be normal threads, and you can rely on the
>> threading module to wait for them to finish.
>
> Did you read my explanation of the reasoning behind my approach?

You should try to link to explanations. These have been long threads,
and it's often hard to find the previous message where a subject was
addressed.

From scott+python-dev at scottdial.com  Fri May 28 05:39:36 2010
From: scott+python-dev at scottdial.com (Scott Dial)
Date: Thu, 27 May 2010 23:39:36 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<20100526120651.GA12751@laurie.devork.be>	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>	<4BFDCEDF.1060303@canterbury.ac.nz>	<20100527075349.GA32291@laurie.devork.be>
	<2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
Message-ID: <4BFF3AF8.9040504@scottdial.com>

On 5/27/2010 4:13 AM, Brian Quinlan wrote:
> On 27 May 2010, at 17:53, Floris Bruynooghe wrote:
>> On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote:
>>> On 27/05/10 00:31, Brian Quinlan wrote:
>>>> You have two semantic choices here:
>>>> 1. let the interpreter exit with the future still running
>>>> 2. wait until the future finishes and then exit
>>>
>>> I'd go for (1). I don't think it's unreasonable to
>>> expect a program that wants all its tasks to finish
>>> to explicitly wait for that to happen.
>>
>> I'd got for (1) as well, it's no more then reasonable that if you want
>> a result you wait for it.  And I dislike libraries doing magic you
>> can't see, I'd prefer if I explicitly had to shut a pool down.
>>
>> I'm glad I'm not alone in preferring (1) tough.
> 
> Keep in mind that this library magic is consistent with the library
> magic that the threading module does - unless the user sets
> Thread.daemon to True, the interpreter does *not* exit until the thread
> does.

Given your rationale, I don't understand from the PEP:

> shutdown(wait=True)
> 
> Signal the executor that it should free any resources that it is
> using when the currently pending futures are done executing. Calls to
> Executor.submit and Executor.map and made after shutdown will raise
> RuntimeError.
> 
> If wait is True then the executor will not return until all the
> pending futures are done executing and the resources associated with
> the executor have been freed.

Can you tell me what is the expected execution time of the following:

>>> executor = ThreadPoolExecutor(max_workers=1)
>>> executor.submit(lambda: time.sleep(1000))
>>> executor.shutdown(wait=False)
>>> sys.exit(0)

I believe it's 1000 seconds, which seems to defy my request of
shutdown(wait=False) because "secretly" the Python exit is going to wait
anyways. ISTM, it is much easier to get behavior #2 if you have behavior
#1, and it would also seem rather trivial to make ThreadPoolExecutor
take an optional argument specifying which behavior you want.

Your reference implementation does not actually implement the
specification given in the PEP, so it's quite impossible to check this
myself. There is no wait=True option for shutdown() in the reference
implementation, so I can only guess what that implementation might look
like.

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu

From ncoghlan at gmail.com  Fri May 28 06:35:19 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 May 2010 14:35:19 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFF05D4.4000202@canterbury.ac.nz>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4BFD173B.9010703@gmail.com>	<htji55$qui$1@dough.gmane.org>
	<4BFDB6CB.7030104@gmail.com>	<4BFDDD64.9060701@scottdial.com>
	<4BFE93C5.6050905@gmail.com> <4BFF05D4.4000202@canterbury.ac.nz>
Message-ID: <4BFF4807.1050903@gmail.com>

On 28/05/10 09:52, Greg Ewing wrote:
> Nick Coghlan wrote:
>
>> We can accept PEP 3148 by saying that we're happy to add the extra
>> namespace level purely for disambiguation purposes,
>
> If that's the only rationale for the namespace, it makes it
> sound like a kludge to work around a poor choice of name.

It's the right name though (it really is a futures implementation - I 
don't know what else you would even consider calling it). The problem is 
that the same word is used to mean different things in other programming 
domains (most obviously finance).

Resolving that kind of ambiguity is an *excellent* use of a package 
namespace - you remove the ambiguity without imposing any significant 
long term cognitive overhead.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From brian at sweetapp.com  Fri May 28 08:52:39 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 28 May 2010 16:52:39 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4BFF3AF8.9040504@scottdial.com>
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>	<20100522135946.8F5451F960D@kimball.webabinitio.net>	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>	<20100526120651.GA12751@laurie.devork.be>	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>	<4BFDCEDF.1060303@canterbury.ac.nz>	<20100527075349.GA32291@laurie.devork.be>
	<2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
	<4BFF3AF8.9040504@scottdial.com>
Message-ID: <2861971A-8E80-490C-8C50-7FC9C67F2B26@sweetapp.com>


On May 28, 2010, at 1:39 PM, Scott Dial wrote:

> On 5/27/2010 4:13 AM, Brian Quinlan wrote:
>> On 27 May 2010, at 17:53, Floris Bruynooghe wrote:
>>> On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote:
>>>> On 27/05/10 00:31, Brian Quinlan wrote:
>>>>> You have two semantic choices here:
>>>>> 1. let the interpreter exit with the future still running
>>>>> 2. wait until the future finishes and then exit
>>>>
>>>> I'd go for (1). I don't think it's unreasonable to
>>>> expect a program that wants all its tasks to finish
>>>> to explicitly wait for that to happen.
>>>
>>> I'd got for (1) as well, it's no more then reasonable that if you  
>>> want
>>> a result you wait for it.  And I dislike libraries doing magic you
>>> can't see, I'd prefer if I explicitly had to shut a pool down.
>>>
>>> I'm glad I'm not alone in preferring (1) tough.
>>
>> Keep in mind that this library magic is consistent with the library
>> magic that the threading module does - unless the user sets
>> Thread.daemon to True, the interpreter does *not* exit until the  
>> thread
>> does.
>
> Given your rationale, I don't understand from the PEP:
>
>> shutdown(wait=True)
>>
>> Signal the executor that it should free any resources that it is
>> using when the currently pending futures are done executing. Calls to
>> Executor.submit and Executor.map and made after shutdown will raise
>> RuntimeError.
>>
>> If wait is True then the executor will not return until all the
>> pending futures are done executing and the resources associated with
>> the executor have been freed.
>
> Can you tell me what is the expected execution time of the following:
>
>>>> executor = ThreadPoolExecutor(max_workers=1)
>>>> executor.submit(lambda: time.sleep(1000))
>>>> executor.shutdown(wait=False)
>>>> sys.exit(0)
>
> I believe it's 1000 seconds, which seems to defy my request of
> shutdown(wait=False) because "secretly" the Python exit is going to  
> wait
> anyways.

It would take 1000 seconds. "...then the executor will not return..."  
should read "...then the method will not return...".

> ISTM, it is much easier to get behavior #2 if you have behavior
> #1, and it would also seem rather trivial to make ThreadPoolExecutor
> take an optional argument specifying which behavior you want.

Adding a daemon option would be reasonable. If you don't shutdown your  
executors you are pretty much guaranteed to get random traceback  
output on exit through.

> Your reference implementation does not actually implement the
> specification given in the PEP, so it's quite impossible to check this
> myself. There is no wait=True option for shutdown() in the reference
> implementation, so I can only guess what that implementation might  
> look
> like.

Look at around line 129 in:
http://code.google.com/p/pythonfutures/source/browse/branches/feedback/python3/futures/thread.py

Cheers,
Brian




From solipsis at pitrou.net  Fri May 28 12:58:02 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 28 May 2010 12:58:02 +0200
Subject: [Python-Dev] Locks and signals
References: <5A22700C-87E6-4175-BC75-B08578460DF1@sweetapp.com>
	<AANLkTilyvtRjf7BYVsIrLkOI57klyYJoDoYYd2Rx5gvX@mail.gmail.com>
	<70F3B975-4ADC-41E6-8754-F7EEC0AC08A2@sweetapp.com>
	<20100522135946.8F5451F960D@kimball.webabinitio.net>
	<AANLkTinqxMkAOQWenqId2LOmTq7C7TnykKwraYOHeoCj@mail.gmail.com>
	<AANLkTin3PbAeIm3mjq7nr7Hp9jSgiN5aoQ8rM0JUPtNa@mail.gmail.com>
	<55112B16-2C75-4C0F-890D-82C6FDF483FC@sweetapp.com>
	<20100526120651.GA12751@laurie.devork.be>
	<8F210CFF-93A3-48E7-BF7A-C6E99B000FF4@sweetapp.com>
	<4BFDCEDF.1060303@canterbury.ac.nz>
	<20100527075349.GA32291@laurie.devork.be>
	<2A3BF269-D6A9-4A48-A5FC-4939531CDCD7@sweetapp.com>
	<AANLkTikcDkK1CxaJ5SZ2-2CKS13PRsOfc2M-ZI7DkrEJ@mail.gmail.com>
Message-ID: <20100528125802.75f388b4@pitrou.net>

On Thu, 27 May 2010 21:57:48 -0400
Reid Kleckner <rnk at mit.edu> wrote:
> 
> Unrelatedly, I feel like this behavior of waiting for the thread to
> terminate usually manifests as deadlocks when the main thread throws
> an uncaught exception.  The application then no longer responds
> properly to interrupts, since it's stuck waiting on a semaphore.

I think the internal low-level lock implementation should be
fixed so that it runs PyErr_CheckSignals() and is able to signal an
error on function return (rather than the current binary "lock
succeeded" / "lock timed out" status).

Actually, it would be nice if you could open a bug entry for that :)

Regards

Antoine.



From status at bugs.python.org  Fri May 28 18:07:42 2010
From: status at bugs.python.org (Python tracker)
Date: Fri, 28 May 2010 18:07:42 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20100528160742.3EEED7813C@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2010-05-21 - 2010-05-28)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2705 open (+40) / 17957 closed (+21) / 20662 total (+61)

Open issues with patches:  1095

Average duration of open issues: 720 days.
Median duration of open issues: 500 days.

Open Issues Breakdown
       open  2683 (+40)
languishing    12 ( +0)
    pending     9 ( +0)

Issues Created Or Reopened (62)
_______________________________

httplib fails with HEAD requests to pages with "transfer-encod 2010-05-26
       http://bugs.python.org/issue6312    reopened orsenthil                            
       patch                                                                   

The external link to a "Hash Collision FAQ" points to some com 2010-05-21
CLOSED http://bugs.python.org/issue8783    created  stutzbach                            
                                                                               

tarfile/Windows: Don't use mbcs as the default encoding        2010-05-22
       http://bugs.python.org/issue8784    created  haypo                                
       patch                                                                   

findall() and finditer() docs misleading                       2010-05-22
CLOSED http://bugs.python.org/issue8785    created  hagen                                
                                                                               

Add support for IEEE 754 contexts to decimal module.           2010-05-22
       http://bugs.python.org/issue8786    created  mark.dickinson                       
                                                                               

warnings inside PyRun_SimpleString() display argv[1]           2010-05-22
       http://bugs.python.org/issue8787    created  Sebastian                            
       patch                                                                   

urllib.urlencode documentation unclear on doseq                2010-05-22
       http://bugs.python.org/issue8788    created  VeXocide                             
                                                                               

spam                                                           2010-05-22
CLOSED http://bugs.python.org/issue8789    created  renben                               
                                                                               

spam                                                           2010-05-22
CLOSED http://bugs.python.org/issue8790    created  renben                               
                                                                               

spam                                                           2010-05-22
CLOSED http://bugs.python.org/issue8791    created  renben                               
                                                                               

xmlrpclib compatibility issues with Apache XML-RPC library     2010-05-22
       http://bugs.python.org/issue8792    created  bra                                  
                                                                               

IDLE crashes on opening invalid file                           2010-05-23
       http://bugs.python.org/issue8793    created  royf                                 
                                                                               

spam                                                           2010-05-23
CLOSED http://bugs.python.org/issue8794    created  renben                               
                                                                               

Error sending utf8 strings over syslog udp protocol            2010-05-23
CLOSED http://bugs.python.org/issue8795    created  skliarie                             
                                                                               

Deprecate codecs.open()                                        2010-05-23
       http://bugs.python.org/issue8796    created  haypo                                
                                                                               

urllib2 basicauth broken in 2.6.5: RuntimeError: maximum recur 2010-05-24
       http://bugs.python.org/issue8797    created  jonozzz                              
                                                                               

tkinter build variants on OSX                                  2010-05-24
       http://bugs.python.org/issue8798    created  ronaldoussoren                       
                                                                               

Hang in lib/test/test_threading.py                             2010-05-24
       http://bugs.python.org/issue8799    created  krisvale                             
       patch, patch, easy                                                      

add threading.RWLock                                           2010-05-24
       http://bugs.python.org/issue8800    created  krisvale                             
       patch, patch, needs review                                              

Inconsistency in behaviour of urllib and urllib2 with file://  2010-05-24
       http://bugs.python.org/issue8801    created  vinay.sajip                          
                                                                               

about oa module                                                2010-05-24
CLOSED http://bugs.python.org/issue8802    created  sudanti                              
                                                                               

about oa module                                                2010-05-24
CLOSED http://bugs.python.org/issue8803    created  sudanti                              
                                                                               

http.client should support SSL contexts                        2010-05-24
       http://bugs.python.org/issue8804    created  pitrou                               
                                                                               

urllib should support SSL contexts                             2010-05-24
       http://bugs.python.org/issue8805    created  pitrou                               
                                                                               

ftplib should support SSL contexts                             2010-05-24
CLOSED http://bugs.python.org/issue8806    created  pitrou                               
       patch                                                                   

poplib should support SSL contexts                             2010-05-24
       http://bugs.python.org/issue8807    created  pitrou                               
       patch                                                                   

imaplib should support SSL contexts                            2010-05-24
       http://bugs.python.org/issue8808    created  pitrou                               
                                                                               

smptlib should support SSL contexts                            2010-05-24
       http://bugs.python.org/issue8809    created  pitrou                               
                                                                               

TZ offset description is unclear in docs                       2010-05-24
       http://bugs.python.org/issue8810    created  belopolsky                           
       easy                                                                    

fixing sqlite3 docs for py3k                                   2010-05-24
       http://bugs.python.org/issue8811    created  l0nwlf                               
       patch                                                                   

Show package path in repr string for packages installed to use 2010-05-24
CLOSED http://bugs.python.org/issue8812    created  srid                                 
                                                                               

SSLContext doesn't support loading a CRL                       2010-05-24
       http://bugs.python.org/issue8813    created  haypo                                
                                                                               

functools.WRAPPER_ASSIGNMENTS should include __annotations__   2010-05-24
       http://bugs.python.org/issue8814    created  terrence                             
       patch                                                                   

65001                                                          2010-05-24
CLOSED http://bugs.python.org/issue8815    created  haypo                                
                                                                               

Add test cases for builtins                                    2010-05-25
CLOSED http://bugs.python.org/issue8816    created  gnofi                                
       patch                                                                   

Expose round-to-nearest division algorithm in Objects/longobje 2010-05-25
CLOSED http://bugs.python.org/issue8817    created  mark.dickinson                       
       patch                                                                   

urlsplit and urlparse add extra slash when using scheme        2010-05-25
       http://bugs.python.org/issue8818    reopened adamnelson                           
                                                                               

variable resolution within exec() incorrect                    2010-05-25
CLOSED http://bugs.python.org/issue8819    created  hawkett                              
                                                                               

IDLE not launching correctly                                   2010-05-26
       http://bugs.python.org/issue8820    created  NightFalcon                          
                                                                               

Range check on unicode repr                                    2010-05-26
       http://bugs.python.org/issue8821    created  mgiuca                               
       patch                                                                   

datetime naive and aware types should have a well-defined defi 2010-05-26
       http://bugs.python.org/issue8822    created  techtonik                            
                                                                               

urllib2 does not catch httplib.BadStatusLine                   2010-05-26
       http://bugs.python.org/issue8823    created  adamnelson                           
                                                                               

Improve documentation of exec                                  2010-05-26
       http://bugs.python.org/issue8824    created  tjreedy                              
                                                                               

int("0",0) throws exception                                    2010-05-26
CLOSED http://bugs.python.org/issue8825    created  cmoss60                              
                                                                               

Invalid expires date in cookiejar                              2010-05-26
       http://bugs.python.org/issue8826    created  tzulberti                            
                                                                               

poplib.POP3_SSL doesn't have an actual test suite              2010-05-26
CLOSED http://bugs.python.org/issue8827    created  giampaolo.rodola                     
                                                                               

Atomic function to rename a file                               2010-05-26
       http://bugs.python.org/issue8828    created  haypo                                
                                                                               

IDLE editior not opening                                       2010-05-27
       http://bugs.python.org/issue8829    created  Spatch                               
                                                                               

Add nondestructive selection to sets                           2010-05-27
CLOSED http://bugs.python.org/issue8830    created  ipatrol                              
                                                                               

recv and recvfrom on UDP socket do not return/throw exception  2010-05-27
       http://bugs.python.org/issue8831    created  Alessandro.Roat                      
                                                                               

automate minidom.unlink() with a context manager               2010-05-27
       http://bugs.python.org/issue8832    created  krisvale                             
       patch, patch, easy, needs review                                        

tarfile:  broken hardlink handling and testcase.               2010-05-27
       http://bugs.python.org/issue8833    created  jsbronder                            
       patch                                                                   

Define order of Misc/ACKS entries                              2010-05-27
       http://bugs.python.org/issue8834    created  belopolsky                           
       patch                                                                   

buildbot: support.transient_internet() doesn't catch DNS socke 2010-05-27
       http://bugs.python.org/issue8835    created  haypo                                
       patch                                                                   

Conflicting default values of parameter to __import__          2010-05-27
CLOSED http://bugs.python.org/issue8836    created  afoglia                              
                                                                               

PyArg_ParseTuple(): remove old and unused "O?" format          2010-05-27
       http://bugs.python.org/issue8837    created  haypo                                
       patch                                                                   

Remove codecs.readbuffer_encode() and codecs.charbuffer_encode 2010-05-27
       http://bugs.python.org/issue8838    created  haypo                                
                                                                               

PyArg_ParseTuple(): remove "t# format                          2010-05-27
       http://bugs.python.org/issue8839    created  haypo                                
       patch                                                                   

io.StringIO: truncate+print disabled in 3.1.2                  2010-05-28
CLOSED http://bugs.python.org/issue8840    created  tjreedy                              
                                                                               

GetoptError strings should be localized                        2010-05-28
       http://bugs.python.org/issue8841    created  richlowe                             
                                                                               

sqlite3 library outdated in Windows builds                     2010-05-28
CLOSED http://bugs.python.org/issue8842    created  Marko.Kohtala                        
                                                                               

urllib2 Digest Authorization uri must match request URI        2010-05-28
       http://bugs.python.org/issue8843    created  anelis                               
       patch                                                                   



Issues Now Closed (57)
______________________

_winreg.EnumValue sometimes raises WindowsError ("More data is  746 days
       http://bugs.python.org/issue2810    brian.curtin                         
       patch, needs review                                                     

int() lies about base parameter                                 744 days
       http://bugs.python.org/issue2844    mark.dickinson                       
       patch                                                                   

SystemExit incorrectly displays unicode message                 626 days
       http://bugs.python.org/issue3798    haypo                                
       patch                                                                   

cookielib chokes on non-integer cookie version, should ignore   608 days
       http://bugs.python.org/issue3924    georg.brandl                         
       patch                                                                   

Implement PEP 376                                               501 days
       http://bugs.python.org/issue4908    alexis                               
       patch                                                                   

Wrong print() result when unicode error handler is not 'strict  416 days
       http://bugs.python.org/issue5640    haypo                                
       patch                                                                   

CVE-2008-5983 python: untrusted python modules search path      402 days
       http://bugs.python.org/issue5753    pitrou                               
       patch                                                                   

Seeking to the beginning of a text file a second time will ret  345 days
       http://bugs.python.org/issue6268    haypo                                
                                                                               

2to3 fails to fix test.test_support                             305 days
       http://bugs.python.org/issue6583    benjamin.peterson                    
                                                                               

HTMLParser.HTMLParser doesn't handle malformed charrefs         291 days
       http://bugs.python.org/issue6662    haypo                                
       patch                                                                   

Sub-optimal "Locate" button behaviour in Windows CHM file       252 days
       http://bugs.python.org/issue6900    georg.brandl                         
                                                                               

datetime operations spanning MINYEAR give bad results           224 days
       http://bugs.python.org/issue7150    belopolsky                           
       patch                                                                   

Struct incorrectly compiles format strings                      184 days
       http://bugs.python.org/issue7355    mark.dickinson                       
       patch, easy                                                             

A number tests "crash" if python is compiled --without-threads   13 days
       http://bugs.python.org/issue7449    haypo                                
       patch, easy, needs review                                               

Support for cp858                                                89 days
       http://bugs.python.org/issue8016    georg.brandl                         
       patch                                                                   

libffi update to 3.0.9                                           68 days
       http://bugs.python.org/issue8142    flox                                 
       patch, buildbot                                                         

urlparse has a duplicate of urllib.unquote                       72 days
       http://bugs.python.org/issue8143    r.david.murray                       
       patch                                                                   

Unified hash for numeric types.                                  64 days
       http://bugs.python.org/issue8188    mark.dickinson                       
       patch                                                                   

siginterrupt with flag=False is reset when signal received       44 days
       http://bugs.python.org/issue8354    vila                                 
       patch, needs review                                                     

Make Context._clamp public in decimal module                     26 days
       http://bugs.python.org/issue8540    mark.dickinson                       
       patch                                                                   

test/support: don't use localhost as IPv6 host name              25 days
       http://bugs.python.org/issue8598    haypo                                
       patch                                                                   

enumerate() docstring doesn't cover optional start argument      16 days
       http://bugs.python.org/issue8635    georg.brandl                         
       patch, easy                                                             

enumerate() test cases do not cover optional start argument      19 days
       http://bugs.python.org/issue8636    mark.dickinson                       
       patch                                                                   

python3 FAQ mentions unicode()                                   10 days
       http://bugs.python.org/issue8694    georg.brandl                         
                                                                               

Duplicated document in telnetlib.                                 8 days
       http://bugs.python.org/issue8707    georg.brandl                         
       patch, easy                                                             

mention explicitly Windows support for os.devnull                 8 days
       http://bugs.python.org/issue8709    georg.brandl                         
       patch                                                                   

bind_and_activate parameter is missed from directive              6 days
       http://bugs.python.org/issue8724    georg.brandl                         
                                                                               

The Mapping ABC's __eq__ method should return NotImplemented i    6 days
       http://bugs.python.org/issue8729    benjamin.peterson                    
       patch                                                                   

Cruft in object.c: PyObject_GenericGetAttr                        4 days
       http://bugs.python.org/issue8749    mark.dickinson                       
                                                                               

Make 'python -m sysconfig' print something useful                 5 days
       http://bugs.python.org/issue8770    tarek                                
                                                                               

py3k: child process don't inherit stdout / stderr on Windows      0 days
       http://bugs.python.org/issue8780    haypo                                
       patch                                                                   

inspect.getsource returns invalid source for non-newline-endin    1 days
       http://bugs.python.org/issue8782    benjamin.peterson                    
       patch                                                                   

The external link to a "Hash Collision FAQ" points to some com    0 days
       http://bugs.python.org/issue8783    georg.brandl                         
                                                                               

findall() and finditer() docs misleading                          0 days
       http://bugs.python.org/issue8785    georg.brandl                         
                                                                               

spam                                                              1 days
       http://bugs.python.org/issue8789    mark.dickinson                       
                                                                               

spam                                                              1 days
       http://bugs.python.org/issue8790    mark.dickinson                       
                                                                               

spam                                                              1 days
       http://bugs.python.org/issue8791    mark.dickinson                       
                                                                               

spam                                                              0 days
       http://bugs.python.org/issue8794    mark.dickinson                       
                                                                               

Error sending utf8 strings over syslog udp protocol               1 days
       http://bugs.python.org/issue8795    vinay.sajip                          
                                                                               

about oa module                                                   0 days
       http://bugs.python.org/issue8802    mark.dickinson                       
                                                                               

about oa module                                                   0 days
       http://bugs.python.org/issue8803    mark.dickinson                       
                                                                               

ftplib should support SSL contexts                                2 days
       http://bugs.python.org/issue8806    giampaolo.rodola                     
       patch                                                                   

Show package path in repr string for packages installed to use    0 days
       http://bugs.python.org/issue8812    srid                                 
                                                                               

65001                                                             0 days
       http://bugs.python.org/issue8815    haypo                                
                                                                               

Add test cases for builtins                                       1 days
       http://bugs.python.org/issue8816    mark.dickinson                       
       patch                                                                   

Expose round-to-nearest division algorithm in Objects/longobje    1 days
       http://bugs.python.org/issue8817    mark.dickinson                       
       patch                                                                   

variable resolution within exec() incorrect                       1 days
       http://bugs.python.org/issue8819    mark.dickinson                       
                                                                               

int("0",0) throws exception                                       2 days
       http://bugs.python.org/issue8825    mark.dickinson                       
                                                                               

poplib.POP3_SSL doesn't have an actual test suite                 0 days
       http://bugs.python.org/issue8827    giampaolo.rodola                     
                                                                               

Add nondestructive selection to sets                              0 days
       http://bugs.python.org/issue8830    eric.smith                           
                                                                               

Conflicting default values of parameter to __import__             0 days
       http://bugs.python.org/issue8836    benjamin.peterson                    
                                                                               

io.StringIO: truncate+print disabled in 3.1.2                     0 days
       http://bugs.python.org/issue8840    pitrou                               
                                                                               

sqlite3 library outdated in Windows builds                        0 days
       http://bugs.python.org/issue8842    brian.curtin                         
                                                                               

An inconsistency with nested scopes                            2142 days
       http://bugs.python.org/issue991196  mark.dickinson                       
                                                                               

Possible windows+python bug                                    1889 days
       http://bugs.python.org/issue1168427 tjreedy                              
                                                                               

yday in datetime module                                        1552 days
       http://bugs.python.org/issue1436346 georg.brandl                         
       patch, needs review                                                     

clean up Solaris port and allow C99 extension modules          1040 days
       http://bugs.python.org/issue1759169 loewis                               
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 21 datetime lacks concrete tzinfo impl. for UTC                     485 days
open        http://bugs.python.org/issue5094   

 18 Remove codecs.readbuffer_encode() and codecs.charbuffer_encode(    1 days
open        http://bugs.python.org/issue8838   

 11 urlsplit and urlparse add extra slash when using scheme            3 days
open        http://bugs.python.org/issue8818   

 11 timedelta multiply and divide by floating point                 1719 days
open        http://bugs.python.org/issue1289118

 10 int("0",0) throws exception                                        2 days
closed      http://bugs.python.org/issue8825   

  9 Atomic function to rename a file                                   2 days
open        http://bugs.python.org/issue8828   

  9 xz compressor support                                            284 days
open        http://bugs.python.org/issue6715   

  9 please support lzma compression as an extension and in	the tarf  419 days
open        http://bugs.python.org/issue5689   

  8 Too narrow platform check in test_datetime                       110 days
open        http://bugs.python.org/issue7879   

  7 ftplib should support SSL contexts                                 2 days
closed      http://bugs.python.org/issue8806   




From vinay_sajip at yahoo.co.uk  Sat May 29 00:28:46 2010
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Fri, 28 May 2010 22:28:46 +0000 (UTC)
Subject: [Python-Dev] PEP 3148 ready for pronouncement
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>
	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>
	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>
	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>
	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>
	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>
	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>
	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>
	<4BFC66F1.3060107@gmail.com>
	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>
	<4BFCFE8B.1080205@canterbury.ac.nz>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
Message-ID: <loom.20100529T001224-694@post.gmane.org>

Brian Quinlan <brian <at> sweetapp.com> writes:

> "future" is a computing science term of art, like "thread". Anyway,  
> this has been discussed in the past and Guido was happy with the name.

I've not seen this mentioned, but on such a long thread I might have missed it:
we already have a "__future__" module, as in

from __future__ import with_statement

and to my mind, this is a potential point of confusion with the term "future".

> > * It seems unnecessarily verbose to tack "Executor"
> > onto the end of every Executor subclass. They could
> > simply be called ThreadPool and ProcessPool without
> > losing anything.
> 
> You could have general thread pools that aren't related to executors  
> (actually, it would be great if Python had a good built-in thread pool  
> implementation) and I'd like to avoid using an overly generic name.
> 

Aren't executors and pools both housekeepers around a bunch of threads which
execute code as a service for other threads? A thread is useless unless it
executes code, isn't it? I'm not sure I understand the fine distinction between
an executor and a pool. Having Executor as a suffix will give a point of
familiarity to those who already know java.util.concurrent. And ProcessPool
might cause confusion with multiprocessing.Pool because at first glance they
seem to be the same thing.

> > * I don't see a strong reason to put this module
> > inside a newly-created namespace. If there were a
> > namespace called "concurrent", I would expect to find
> > other existing concurrency-related modules there as
> > well, such as threading and multiprocessing. But we
> > can't move them there without breaking existing code.
> 
> I think that Jesse was planning to add some functionality to this  
> namespace. I don't really have an opinion on this.
> 

I'm not sure of the benefit of a "concurrent" namespace, since it wouldn't
contain the existing concurrency stuff. I think it's something to consider only
for a big reorg which would break backward compatibility. IMO it would make more
sense to leave this module as a top-level module for now (a sibling to
"threading", "multiprocessing").

Regards,

Vinay Sajip


From tjreedy at udel.edu  Sat May 29 01:03:37 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 28 May 2010 19:03:37 -0400
Subject: [Python-Dev] Bugfix releases should not change APIs
Message-ID: <htpi4a$t2i$1@dough.gmane.org>

I had the strong impression that there was a policy that x.y.z bugfix 
releases should only fix bugs and not add new features and revise 
current ones. The rationale, as I understood it, is that x.y.z releases 
should be increasingly better implementations of a stable x.y feature 
set. Adding features is 'bad' because code using a new feature will not 
work in previous releases of the same x.y version. Changing features is 
even worse because it may also break working code going forward.

Because of this policy, an x.y.z Windows installer (I do not know about 
others) deletes any earlier release of the same version. Also, there is 
no What's New in Python x.y.z (relative to x.y.(z-1) since such should 
be empty. Consequently, violations of the policy are pretty much silent 
and well hidden.

Yesterday, I spent two hours puzzling over the failure of my previously 
'green' test sequence that tested a custom test function. I finally 
realized that the change was not due to anything I did (or undid), but a 
change in 3.1.2 in the interaction of StringIO.truncate, 
StringIO.getvalue, and print(x, StringIO()). (I should note the it is 
the usual stability and quality of Python that made me slow to blame 
Python rather than myself.)

After filing
http://bugs.python.org/issue8840
I was rather shocked to be told that the code-breaking and 
policy-violating change was intentional because being 'more consistent 
with other file-handling APIs out there' was somehow more important than 
consistency within version releases. It also seems to me that discussion 
of code-breaking API changes like this should involve more than one user 
and one developer. See
http://bugs.python.org/issue6939

I have fixed my tests so they works in 3.1.2 and should work in other 
3.1 releases, but that would be a nuisance to test. Of course, I should 
not have to worry about API changes within a version series.

I think issue 8840 illustrates another reason for the bugfix-only 
policy. New x.y features and docs are (nearly always) added before the 
first beta. They can then be tested, discussed, and improved. This 
'cooking' does not occur for bugfix releases. For reasons I give in my 
response on the tracker, I think the new behavior is buggy and the doc 
deficient.

So, I think
1) the supposed bugfix-only policy should really be the policy;
2) the violation in 3.1.2 should be reverted in 3.1.3, and the API 
change reviewed in the normal 3.2 alpha/beta process.
I am curious as to what others think on both points.

Terry Jan Reedy


From steve at pearwood.info  Sat May 29 02:12:26 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 29 May 2010 10:12:26 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <loom.20100529T001224-694@post.gmane.org>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<loom.20100529T001224-694@post.gmane.org>
Message-ID: <201005291012.26842.steve@pearwood.info>

On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote:

> I've not seen this mentioned, but on such a long thread I might have
> missed it: we already have a "__future__" module, as in
>
> from __future__ import with_statement
>
> and to my mind, this is a potential point of confusion with the term
> "future".
[...]
> I'm not sure of the benefit of a "concurrent" namespace, since it
> wouldn't contain the existing concurrency stuff. I think it's
> something to consider only for a big reorg which would break backward
> compatibility. IMO it would make more sense to leave this module as a
> top-level module for now (a sibling to "threading",
> "multiprocessing").

I have suggested a way to move the existing concurrency stuff without 
breaking backwards compatibility, and Terry Reedy asked if it would 
work. I haven't seen any responses, either positive or negative.

For the record, my suggestion was:

for each concurrency modules:
  move it into the concurrency package
  add a top level module with the same name containing:
    # e.g. for threading
    from concurrency.threading import *

Then in some future Python version, each top level module gets a 
PendingDeprecation warning, followed by a Deprecation warning some time 
later, and eventually in the indefinite future removal of the top level 
module.


Leaving the futures module in the top level of the std lib is far more 
likely to confuse users than adding it to its own namespace. Compare:

import __future__
import futures 

with:

import __future__
import concurrency.futures 


In my opinion, it is high time for the std lib to pay more attention to 
the final Zen:

Namespaces are one honking great idea -- let's do more of those!



-- 
Steven D'Aprano

From steve at holdenweb.com  Sat May 29 02:16:04 2010
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 28 May 2010 20:16:04 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <loom.20100529T001224-694@post.gmane.org>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<E4F20EAE-5082-431D-A2E4-ECD23243CFF4@sweetapp.com>	<AANLkTinTXKdmgP-Dv_yzEUDoKJkZZsPE4y6jr8p0-MYE@mail.gmail.com>	<76DC964B-F936-4E37-9EBF-44030BEA5257@sweetapp.com>	<AANLkTikb3cwA-G_mV46hr8_BMVSkYMdNO6uh7uDyWrmX@mail.gmail.com>	<161466AF-E7F0-4960-A5E5-964ED11B62F6@sweetapp.com>	<AANLkTilWSMP39rcCAm_vqZV2Wi0GFbZgV5t5IyOXLvgx@mail.gmail.com>	<E8C07098-20D2-4712-B685-3067DF7C54E2@sweetapp.com>	<AANLkTikM4Y5GfRCU5DWA4OyQlD8wyoEjLstruLijRKak@mail.gmail.com>	<4BFC66F1.3060107@gmail.com>	<AANLkTim8O9pjokHYEDxZ2lBYWAfV4kqGMKPp6j7fHQ4G@mail.gmail.com>	<4BFCFE8B.1080205@canterbury.ac.nz>	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<loom.20100529T001224-694@post.gmane.org>
Message-ID: <4C005CC4.2040205@holdenweb.com>

Vinay Sajip wrote:
> Brian Quinlan <brian <at> sweetapp.com> writes:
> 
>> "future" is a computing science term of art, like "thread". Anyway,  
>> this has been discussed in the past and Guido was happy with the name.
> 
> I've not seen this mentioned, but on such a long thread I might have missed it:
> we already have a "__future__" module, as in
> 
> from __future__ import with_statement
> 
> and to my mind, this is a potential point of confusion with the term "future".
> 
>>> * It seems unnecessarily verbose to tack "Executor"
>>> onto the end of every Executor subclass. They could
>>> simply be called ThreadPool and ProcessPool without
>>> losing anything.
>> You could have general thread pools that aren't related to executors  
>> (actually, it would be great if Python had a good built-in thread pool  
>> implementation) and I'd like to avoid using an overly generic name.
>>
> 
> Aren't executors and pools both housekeepers around a bunch of threads which
> execute code as a service for other threads? A thread is useless unless it
> executes code, isn't it? I'm not sure I understand the fine distinction between
> an executor and a pool. Having Executor as a suffix will give a point of
> familiarity to those who already know java.util.concurrent. And ProcessPool
> might cause confusion with multiprocessing.Pool because at first glance they
> seem to be the same thing.
> 
>>> * I don't see a strong reason to put this module
>>> inside a newly-created namespace. If there were a
>>> namespace called "concurrent", I would expect to find
>>> other existing concurrency-related modules there as
>>> well, such as threading and multiprocessing. But we
>>> can't move them there without breaking existing code.
>> I think that Jesse was planning to add some functionality to this  
>> namespace. I don't really have an opinion on this.
>>
> 
> I'm not sure of the benefit of a "concurrent" namespace, since it wouldn't
> contain the existing concurrency stuff. I think it's something to consider only
> for a big reorg which would break backward compatibility. IMO it would make more
> sense to leave this module as a top-level module for now (a sibling to
> "threading", "multiprocessing").
> 
Unless there's some way of having the two namespaces (existing and
concurrent-oriented) simultaneously coexist. A single implementation
with two separate namespace mappings?

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See Python Video!       http://python.mirocommunity.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/
"All I want for my birthday is another birthday" -
                                     Ian Dury, 1942-2000


From jnoller at gmail.com  Sat May 29 02:19:09 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 28 May 2010 20:19:09 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <201005291012.26842.steve@pearwood.info>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<loom.20100529T001224-694@post.gmane.org>
	<201005291012.26842.steve@pearwood.info>
Message-ID: <2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com>



On May 28, 2010, at 8:12 PM, Steven D'Aprano <steve at pearwood.info>  
wrote:

> On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote:
>
>> I've not seen this mentioned, but on such a long thread I might have
>> missed it: we already have a "__future__" module, as in
>>
>> from __future__ import with_statement
>>
>> and to my mind, this is a potential point of confusion with the term
>> "future".
> [...]
>> I'm not sure of the benefit of a "concurrent" namespace, since it
>> wouldn't contain the existing concurrency stuff. I think it's
>> something to consider only for a big reorg which would break backward
>> compatibility. IMO it would make more sense to leave this module as a
>> top-level module for now (a sibling to "threading",
>> "multiprocessing").
>
> I have suggested a way to move the existing concurrency stuff without
> breaking backwards compatibility, and Terry Reedy asked if it would
> work. I haven't seen any responses, either positive or negative.
>
> For the record, my suggestion was:
>
> for each concurrency modules:
>  move it into the concurrency package
>  add a top level module with the same name containing:
>    # e.g. for threading
>    from concurrency.threading import *
>
> Then in some future Python version, each top level module gets a
> PendingDeprecation warning, followed by a Deprecation warning some  
> time
> later, and eventually in the indefinite future removal of the top  
> level
> module.
>
>
> Leaving the futures module in the top level of the std lib is far more
> likely to confuse users than adding it to its own namespace. Compare:
>
> import __future__
> import futures
>
> with:
>
> import __future__
> import concurrency.futures
>
>
> In my opinion, it is high time for the std lib to pay more attention  
> to
> the final Zen:
>
> Namespaces are one honking great idea -- let's do more of those!
>
>
>
Yes, your suggestion for how to move things is the way we would want  
to do it, and in the back of my head, what we should do long term -  
just not right now.




From tseaver at palladion.com  Sat May 29 02:31:05 2010
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 28 May 2010 20:31:05 -0400
Subject: [Python-Dev] Bugfix releases should not change APIs
In-Reply-To: <htpi4a$t2i$1@dough.gmane.org>
References: <htpi4a$t2i$1@dough.gmane.org>
Message-ID: <htpn89$8qc$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Terry Reedy wrote:
> I had the strong impression that there was a policy that x.y.z bugfix 
> releases should only fix bugs and not add new features and revise 
> current ones. The rationale, as I understood it, is that x.y.z releases 
> should be increasingly better implementations of a stable x.y feature 
> set. Adding features is 'bad' because code using a new feature will not 
> work in previous releases of the same x.y version. Changing features is 
> even worse because it may also break working code going forward.
> 
> Because of this policy, an x.y.z Windows installer (I do not know about 
> others) deletes any earlier release of the same version. Also, there is 
> no What's New in Python x.y.z (relative to x.y.(z-1) since such should 
> be empty. Consequently, violations of the policy are pretty much silent 
> and well hidden.
> 
> Yesterday, I spent two hours puzzling over the failure of my previously 
> 'green' test sequence that tested a custom test function. I finally 
> realized that the change was not due to anything I did (or undid), but a 
> change in 3.1.2 in the interaction of StringIO.truncate, 
> StringIO.getvalue, and print(x, StringIO()). (I should note the it is 
> the usual stability and quality of Python that made me slow to blame 
> Python rather than myself.)
> 
> After filing
> http://bugs.python.org/issue8840
> I was rather shocked to be told that the code-breaking and 
> policy-violating change was intentional because being 'more consistent 
> with other file-handling APIs out there' was somehow more important than 
> consistency within version releases. It also seems to me that discussion 
> of code-breaking API changes like this should involve more than one user 
> and one developer. See
> http://bugs.python.org/issue6939
> 
> I have fixed my tests so they works in 3.1.2 and should work in other 
> 3.1 releases, but that would be a nuisance to test. Of course, I should 
> not have to worry about API changes within a version series.
> 
> I think issue 8840 illustrates another reason for the bugfix-only 
> policy. New x.y features and docs are (nearly always) added before the 
> first beta. They can then be tested, discussed, and improved. This 
> 'cooking' does not occur for bugfix releases. For reasons I give in my 
> response on the tracker, I think the new behavior is buggy and the doc 
> deficient.
> 
> So, I think
> 1) the supposed bugfix-only policy should really be the policy;
> 2) the violation in 3.1.2 should be reverted in 3.1.3, and the API 
> change reviewed in the normal 3.2 alpha/beta process.
> I am curious as to what others think on both points.

+1 on #1 as the general policy.

I don't have enough skin in the game of the 3.1.x world to have an
opinion about this specific breakage, but I have certainly seen other
examples in the 2.x releases, where such a resolution was the
appropriate outcome.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkwAYEkACgkQ+gerLs4ltQ6AuQCfTk4mAl3ClpE1uu6nBRNrNjBc
g54AoI2SObUNn/d0RvIbYj/vl7HDQbbj
=U8z2
-----END PGP SIGNATURE-----


From steve at holdenweb.com  Sat May 29 02:31:02 2010
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 28 May 2010 20:31:02 -0400
Subject: [Python-Dev] Bugfix releases should not change APIs
In-Reply-To: <htpi4a$t2i$1@dough.gmane.org>
References: <htpi4a$t2i$1@dough.gmane.org>
Message-ID: <htpn8h$9td$1@dough.gmane.org>

Terry Reedy wrote:
> I had the strong impression that there was a policy that x.y.z bugfix
> releases should only fix bugs and not add new features and revise
> current ones. The rationale, as I understood it, is that x.y.z releases
> should be increasingly better implementations of a stable x.y feature
> set. Adding features is 'bad' because code using a new feature will not
> work in previous releases of the same x.y version. Changing features is
> even worse because it may also break working code going forward.
> 
> Because of this policy, an x.y.z Windows installer (I do not know about
> others) deletes any earlier release of the same version. Also, there is
> no What's New in Python x.y.z (relative to x.y.(z-1) since such should
> be empty. Consequently, violations of the policy are pretty much silent
> and well hidden.
> 
> Yesterday, I spent two hours puzzling over the failure of my previously
> 'green' test sequence that tested a custom test function. I finally
> realized that the change was not due to anything I did (or undid), but a
> change in 3.1.2 in the interaction of StringIO.truncate,
> StringIO.getvalue, and print(x, StringIO()). (I should note the it is
> the usual stability and quality of Python that made me slow to blame
> Python rather than myself.)
> 
> After filing
> http://bugs.python.org/issue8840
> I was rather shocked to be told that the code-breaking and
> policy-violating change was intentional because being 'more consistent
> with other file-handling APIs out there' was somehow more important than
> consistency within version releases. It also seems to me that discussion
> of code-breaking API changes like this should involve more than one user
> and one developer. See
> http://bugs.python.org/issue6939
> 
> I have fixed my tests so they works in 3.1.2 and should work in other
> 3.1 releases, but that would be a nuisance to test. Of course, I should
> not have to worry about API changes within a version series.
> 
> I think issue 8840 illustrates another reason for the bugfix-only
> policy. New x.y features and docs are (nearly always) added before the
> first beta. They can then be tested, discussed, and improved. This
> 'cooking' does not occur for bugfix releases. For reasons I give in my
> response on the tracker, I think the new behavior is buggy and the doc
> deficient.
> 
> So, I think
> 1) the supposed bugfix-only policy should really be the policy;
> 2) the violation in 3.1.2 should be reverted in 3.1.3, and the API
> change reviewed in the normal 3.2 alpha/beta process.
> I am curious as to what others think on both points.
> 
I think it shows how developers can "get worked over" if they are
insufficiently vigilant.

1) I completely agree, and adduce as evidence the fact that something
like this always seems to happen when the rule is broken;

2) But we may wish to release 3.1.2.1(?) which backports fixes from the
3.2 line but retains the file store semantics (which I am assured will
be easy in the glorious reign of Hg).

Surely some compatible "shim" layer could have been introduced?

What is the process for ensuring that such API changes do *not* creep
into maintenance releases?

regards
 Steve


-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See Python Video!       http://python.mirocommunity.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/
"All I want for my birthday is another birthday" -
                                     Ian Dury, 1942-2000


From benjamin at python.org  Sat May 29 03:23:37 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 28 May 2010 20:23:37 -0500
Subject: [Python-Dev] Bugfix releases should not change APIs
In-Reply-To: <htpn8h$9td$1@dough.gmane.org>
References: <htpi4a$t2i$1@dough.gmane.org>
	<htpn8h$9td$1@dough.gmane.org>
Message-ID: <AANLkTikbg5SXNQpcrtBeUBKjJY8lqFQbkLFA1vJ4X_o2@mail.gmail.com>

2010/5/28 Steve Holden <steve at holdenweb.com>:
> Terry Reedy wrote:
>> http://bugs.python.org/issue8840
>> I was rather shocked to be told that the code-breaking and
>> policy-violating change was intentional because being 'more consistent
>> with other file-handling APIs out there' was somehow more important than
>> consistency within version releases. It also seems to me that discussion
>> of code-breaking API changes like this should involve more than one user
>> and one developer.

This was discussed on the mailing list and this was produced:
http://mail.python.org/pipermail/python-dev/2009-September/092247.html

> What is the process for ensuring that such API changes do *not* creep
> into maintenance releases?

Generally developers are good about checking with the wider developer
community about possibly incompatible changes. And given the link I
posted above I don't consider this change to have "crept" in.


-- 
Regards,
Benjamin

From ncoghlan at gmail.com  Sat May 29 05:31:07 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 May 2010 13:31:07 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>	<loom.20100529T001224-694@post.gmane.org>	<201005291012.26842.steve@pearwood.info>
	<2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com>
Message-ID: <4C008A7B.1090907@gmail.com>

On 29/05/10 10:19, Jesse Noller wrote:
>> In my opinion, it is high time for the std lib to pay more attention to
>> the final Zen:
>>
>> Namespaces are one honking great idea -- let's do more of those!
>>
>>
>>
> Yes, your suggestion for how to move things is the way we would want to
> do it, and in the back of my head, what we should do long term - just
> not right now.

Yep, this is what I have been saying as well.

1. Using concurrency.futures rather than a top level futures module 
resolves the potential confusion with __future__ and stock market 
futures without inventing our own name for a well established computer 
science concept.

2. With the concurrency package in place following PEP 3148, we can 
separately consider the question of if/when/how to move other 
concurrency related modules (e.g. threading, multiprocessing, Queue) 
into that package at a later date.

Since this topic keeps coming up, some reasoning along these lines 
should go into PEP 3148.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sat May 29 05:41:33 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 May 2010 13:41:33 +1000
Subject: [Python-Dev] Bugfix releases should not change APIs
In-Reply-To: <htpi4a$t2i$1@dough.gmane.org>
References: <htpi4a$t2i$1@dough.gmane.org>
Message-ID: <4C008CED.8080406@gmail.com>

On 29/05/10 09:03, Terry Reedy wrote:
> After filing
> http://bugs.python.org/issue8840
> I was rather shocked to be told that the code-breaking and
> policy-violating change was intentional because being 'more consistent
> with other file-handling APIs out there' was somehow more important than
> consistency within version releases. It also seems to me that discussion
> of code-breaking API changes like this should involve more than one user
> and one developer. See
> http://bugs.python.org/issue6939

As Benjamin noted, that change was discussed on python-dev and actually 
made at Guido's direction.

> So, I think
> 1) the supposed bugfix-only policy should really be the policy;
> 2) the violation in 3.1.2 should be reverted in 3.1.3, and the API
> change reviewed in the normal 3.2 alpha/beta process.
> I am curious as to what others think on both points.

The bugfix-only policy remains in place, but there are corner cases 
(such as this one) where the definition of what is and isn't a bug gets 
fuzzy and we need to make a judgment call as to whether to fix them or 
not. In particular, fixing bugs often runs the risk of breaking existing 
workarounds for those bugs.

The decimal module, for example, has a standing policy to treat 
deviations from the standard as bugs, and this may lead to changes in 
the module if the standard is updated between two releases.

This specific change was discussed on python-dev and the chance of 
breaking existing 3.1 code was deemed preferable to retaining semantics 
that were seen as broken, so I would be against reverting it.

However, it may be worth modifying the policy to ensure that such 
exceptional bug fixes be mentioned prominently in the release notes and 
on the download page for that maintenance release.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sat May 29 05:47:30 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 May 2010 13:47:30 +1000
Subject: [Python-Dev] Bugfix releases should not change APIs
In-Reply-To: <htpi4a$t2i$1@dough.gmane.org>
References: <htpi4a$t2i$1@dough.gmane.org>
Message-ID: <4C008E52.8070106@gmail.com>

On 29/05/10 09:03, Terry Reedy wrote:
> After filing
> http://bugs.python.org/issue8840
> I was rather shocked to be told that the code-breaking and
> policy-violating change was intentional because being 'more consistent
> with other file-handling APIs out there' was somehow more important than
> consistency within version releases. It also seems to me that discussion
> of code-breaking API changes like this should involve more than one user
> and one developer. See
> http://bugs.python.org/issue6939

For the record, I have added links to Guido's python-dev post to both of 
these tracker issues.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From eric at trueblade.com  Sat May 29 06:53:23 2010
From: eric at trueblade.com (Eric Smith)
Date: Sat, 29 May 2010 00:53:23 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <201005291012.26842.steve@pearwood.info>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>	<loom.20100529T001224-694@post.gmane.org>
	<201005291012.26842.steve@pearwood.info>
Message-ID: <4C009DC3.3060006@trueblade.com>

Steven D'Aprano wrote:
> I have suggested a way to move the existing concurrency stuff without 
> breaking backwards compatibility, and Terry Reedy asked if it would 
> work. I haven't seen any responses, either positive or negative.
> 
> For the record, my suggestion was:
> 
> for each concurrency modules:
>   move it into the concurrency package
>   add a top level module with the same name containing:
>     # e.g. for threading
>     from concurrency.threading import *

In the past the problem identified with this approach has been that 
pickles produced with new pythons would not be readable by older pythons.

I think this was the main reason that Brett's 3.0 library reorganization 
wasn't more radical. Theres a discussion if this here:
http://mail.python.org/pipermail/python-dev/2008-May/079535.html
http://mail.python.org/pipermail/stdlib-sig/2008-May/000303.html
and a little more here:
http://bugs.python.org/issue2775

Eric.

From hawkett at gmail.com  Sat May 29 12:20:29 2010
From: hawkett at gmail.com (Colin H)
Date: Sat, 29 May 2010 11:20:29 +0100
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTimUiP7GGPRM5hTyf50S4Z8yGubzpWvwX3P_Zlvz@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com> 
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com> 
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com> 
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com> 
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com> 
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com> 
	<4BFE92D9.1060306@gmail.com>
	<AANLkTimUiP7GGPRM5hTyf50S4Z8yGubzpWvwX3P_Zlvz@mail.gmail.com>
Message-ID: <AANLkTinj2liqSAWTFVgDbkiaZMpFZLJpYTuGSyhk4KMg@mail.gmail.com>

Perhaps the next step is to re-open the issue? If it is seen as a bug,
it would be great to see a fix in 2.6+ - a number of options which
will not break backward compatibility have been put forward - cheers,

Colin

On Thu, May 27, 2010 at 9:05 PM, Reid Kleckner <rnk at mit.edu> wrote:
> On Thu, May 27, 2010 at 11:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> However, attaining the (sensible) behaviour Colin is requesting when such
>> top level variable references exist would actually be somewhat tricky.
>> Considering Guido's suggestion to treat two argument exec like a function
>> rather than a class and generate a closure with full lexical scoping a
>> little further, I don't believe this could be done in exec itself without
>> breaking code that expects the current behaviour.
>
> Just to give a concrete example, here is code that would break if exec
> were to execute code in a function scope instead of a class scope:
>
> exec """
> def len(xs):
> ? ?return -1
> def foo():
> ? ?return len([])
> print foo()
> """ in globals(), {}
>
> Currently, the call to 'len' inside 'foo' skips the outer scope
> (because it's a class scope) and goes straight to globals and
> builtins. ?If it were switched to a local scope, a cell would be
> created for the broken definition of 'len', and the call would resolve
> to it.
>
> Honestly, to me, the fact that the above code ever worked (ie prints
> "0", not "-1") seems like a bug, so I wouldn't worry about backwards
> compatibility.
>
> Reid
>

From solipsis at pitrou.net  Sat May 29 12:39:37 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 29 May 2010 12:39:37 +0200
Subject: [Python-Dev] Bugfix releases should not change APIs
References: <htpi4a$t2i$1@dough.gmane.org>
	<htpn8h$9td$1@dough.gmane.org>
Message-ID: <20100529123937.222a4c0d@pitrou.net>

On Fri, 28 May 2010 20:31:02 -0400
Steve Holden <steve at holdenweb.com> wrote:
> I think it shows how developers can "get worked over" if they are
> insufficiently vigilant.
> 
> 1) I completely agree, and adduce as evidence the fact that something
> like this always seems to happen when the rule is broken;

Well it's true that the change may have been unfortunate (this is the
second bug report we get about it, the first one was from Holger Krekel
IIRC), its exceptional nature had also been discussed on this
mailing-list, and supported by Guido. It is not the product of
oversight.

What it does teach us is that Python 3.1 sees some real use, and
we have entered a phase where backwards compatibility will become as
important as it was in the 2.x line.

> 2) But we may wish to release 3.1.2.1(?) which backports fixes from the
> 3.2 line but retains the file store semantics (which I am assured will
> be easy in the glorious reign of Hg).

I think this would be worse, as in "even more confusing". We would have
a 3.1.2 with changed behaviour, a 3.1.2.1 with reverted behaviour, and a
3.2 with changed behaviour again. Now that we have inflicted this
pain on our users, let's not inflict more pain on them.

Regards

Antoine.



From ncoghlan at gmail.com  Sat May 29 14:13:24 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 May 2010 22:13:24 +1000
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTinj2liqSAWTFVgDbkiaZMpFZLJpYTuGSyhk4KMg@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>
	<4BFDB8EE.4010307@gmail.com>
	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
	<4BFE92D9.1060306@gmail.com>
	<AANLkTimUiP7GGPRM5hTyf50S4Z8yGubzpWvwX3P_Zlvz@mail.gmail.com>
	<AANLkTinj2liqSAWTFVgDbkiaZMpFZLJpYTuGSyhk4KMg@mail.gmail.com>
Message-ID: <4C0104E4.3080802@gmail.com>

On 29/05/10 20:20, Colin H wrote:
> Perhaps the next step is to re-open the issue? If it is seen as a bug,
> it would be great to see a fix in 2.6+ - a number of options which
> will not break backward compatibility have been put forward - cheers,

A new feature request requesting a "closure" mode for compile() in 3.2 
would probably be the best way forward. Once that is done, then the 
question of if or when to change the default behaviour for auto-compiled 
code in exec and/or dis can be considered.

It definitely isn't a bug fix though - it's worked this way for years, 
and while the existing semantics can certainly be surprising, they're 
far from being buggy (as Thomas said, prior to the introduction of 
lexical scoping all Python namespaces worked this way).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From jnoller at gmail.com  Sat May 29 14:46:28 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 29 May 2010 08:46:28 -0400
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <4C008A7B.1090907@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<loom.20100529T001224-694@post.gmane.org>
	<201005291012.26842.steve@pearwood.info>
	<2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com>
	<4C008A7B.1090907@gmail.com>
Message-ID: <80D222EA-EAFD-4EDB-BECF-0FB7F0567A95@gmail.com>



On May 28, 2010, at 11:31 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 29/05/10 10:19, Jesse Noller wrote:
>>> In my opinion, it is high time for the std lib to pay more  
>>> attention to
>>> the final Zen:
>>>
>>> Namespaces are one honking great idea -- let's do more of those!
>>>
>>>
>>>
>> Yes, your suggestion for how to move things is the way we would  
>> want to
>> do it, and in the back of my head, what we should do long term - just
>> not right now.
>
> Yep, this is what I have been saying as well.
>
> 1. Using concurrency.futures rather than a top level futures module  
> resolves the potential confusion with __future__ and stock market  
> futures without inventing our own name for a well established  
> computer science concept.
>
> 2. With the concurrency package in place following PEP 3148, we can  
> separately consider the question of if/when/how to move other  
> concurrency related modules (e.g. threading, multiprocessing, Queue)  
> into that package at a later date.
>
> Since this topic keeps coming up, some reasoning along these lines  
> should go into PEP 3148.
>

I'll type something up this weekend and shoot it to Brian for  
inclusion. I was hoping to be able to keep it out of the futures pep  
itself, but it seems that won't work :)

Jesse

From ncoghlan at gmail.com  Sat May 29 17:51:52 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 30 May 2010 01:51:52 +1000
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <80D222EA-EAFD-4EDB-BECF-0FB7F0567A95@gmail.com>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com>
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<loom.20100529T001224-694@post.gmane.org>
	<201005291012.26842.steve@pearwood.info>
	<2F64E36B-5849-4FEB-8ACD-F81AE2417910@gmail.com>
	<4C008A7B.1090907@gmail.com>
	<80D222EA-EAFD-4EDB-BECF-0FB7F0567A95@gmail.com>
Message-ID: <4C013818.3090000@gmail.com>

On 29/05/10 22:46, Jesse Noller wrote:
> On May 28, 2010, at 11:31 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Since this topic keeps coming up, some reasoning along these lines
>> should go into PEP 3148.
>
> I'll type something up this weekend and shoot it to Brian for inclusion.
> I was hoping to be able to keep it out of the futures pep itself, but it
> seems that won't work :)

Well, punting on whether or not we actually *do* part 2 is still fine. 
As Eric pointed out, there are issues with unpickling that make the 
wisdom of following through with renaming any existing modules fairly 
questionable.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From eric at trueblade.com  Sat May 29 15:52:02 2010
From: eric at trueblade.com (Eric Smith)
Date: Sat, 29 May 2010 09:52:02 -0400
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
Message-ID: <4C011C02.4010605@trueblade.com>

Last night Barry Warsaw, Jason Coombs, and I met to work on implementing 
PEP 382. As part of my research, I came across this email from Martin:
http://mail.python.org/pipermail/python-dev/2009-May/089316.html

In it he says that PEP 382 is being deferred until it can address PEP 
302 loaders. I can't find any follow-up to this. I don't see any 
discussion in PEP 382 about PEP 302 loaders, so I assume this issue was 
never resolved. Does it need to be before PEP 382 is implemented? Are we 
wasting our time by designing and (eventually) coding before this issue 
is resolved?

Eric.

From martin at v.loewis.de  Sat May 29 20:45:39 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 29 May 2010 20:45:39 +0200
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <4C011C02.4010605@trueblade.com>
References: <4C011C02.4010605@trueblade.com>
Message-ID: <4C0160D3.70304@v.loewis.de>

> In it he says that PEP 382 is being deferred until it can address PEP
> 302 loaders. I can't find any follow-up to this. I don't see any
> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was
> never resolved. Does it need to be before PEP 382 is implemented? Are we
> wasting our time by designing and (eventually) coding before this issue
> is resolved?

Yes, and yes.

Regards,
Martin

From martin at v.loewis.de  Sat May 29 21:29:33 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 29 May 2010 21:29:33 +0200
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <20100529190623.299C83A405F@sparrow.telecommunity.com>
References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de>
	<20100529190623.299C83A405F@sparrow.telecommunity.com>
Message-ID: <4C016B1D.8010604@v.loewis.de>

Am 29.05.2010 21:06, schrieb P.J. Eby:
> At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote:
>>> In it he says that PEP 382 is being deferred until it can address PEP
>>> 302 loaders. I can't find any follow-up to this. I don't see any
>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was
>>> never resolved. Does it need to be before PEP 382 is implemented? Are we
>>> wasting our time by designing and (eventually) coding before this issue
>>> is resolved?
>>
>> Yes, and yes.
>
> Is there anything we can do to help regarding that?

You could comment on the proposal I made back then, or propose a 
different solution.

Regards,
Martin

From brett at python.org  Sat May 29 22:46:25 2010
From: brett at python.org (Brett Cannon)
Date: Sat, 29 May 2010 13:46:25 -0700
Subject: [Python-Dev] PEP 3148 ready for pronouncement
In-Reply-To: <201005291012.26842.steve@pearwood.info>
References: <7138B5F1-09C5-42DE-A5D1-CF9F8B02F12F@sweetapp.com> 
	<4A43D53C-54D2-4BE6-8ED8-1D58723B1448@sweetapp.com>
	<loom.20100529T001224-694@post.gmane.org> 
	<201005291012.26842.steve@pearwood.info>
Message-ID: <AANLkTimf8mHkzKrtZndvpUYT2v-a9oQy-lVwmyIDv8wp@mail.gmail.com>

On Fri, May 28, 2010 at 17:12, Steven D'Aprano <steve at pearwood.info> wrote:
> On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote:
>
>> I've not seen this mentioned, but on such a long thread I might have
>> missed it: we already have a "__future__" module, as in
>>
>> from __future__ import with_statement
>>
>> and to my mind, this is a potential point of confusion with the term
>> "future".
> [...]
>> I'm not sure of the benefit of a "concurrent" namespace, since it
>> wouldn't contain the existing concurrency stuff. I think it's
>> something to consider only for a big reorg which would break backward
>> compatibility. IMO it would make more sense to leave this module as a
>> top-level module for now (a sibling to "threading",
>> "multiprocessing").
>
> I have suggested a way to move the existing concurrency stuff without
> breaking backwards compatibility, and Terry Reedy asked if it would
> work. I haven't seen any responses, either positive or negative.
>
> For the record, my suggestion was:
>
> for each concurrency modules:
> ?move it into the concurrency package
> ?add a top level module with the same name containing:
> ? ?# e.g. for threading
> ? ?from concurrency.threading import *
>
> Then in some future Python version, each top level module gets a
> PendingDeprecation warning, followed by a Deprecation warning some time
> later, and eventually in the indefinite future removal of the top level
> module.

This was the procedure we used for about a month for Python 2.6 in
order to help renamed modules migrate to their new names in Python 3.
The issue that came up (and forced use to revert this approach and
fully rely on 2to3) was anything pickled by the older interpreters is
not going to be happy with that shift. Luckily the stuff being moved
most likely does not contain things that have been pickled and stored
to disk for ages and thus would break in a transition.

From brett at python.org  Sun May 30 00:44:28 2010
From: brett at python.org (Brett Cannon)
Date: Sat, 29 May 2010 15:44:28 -0700
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <4C016B1D.8010604@v.loewis.de>
References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> 
	<20100529190623.299C83A405F@sparrow.telecommunity.com>
	<4C016B1D.8010604@v.loewis.de>
Message-ID: <AANLkTikReDF7mMCZBTCOO7Ky7WwsjwJMTNw3rplmiM8q@mail.gmail.com>

On Sat, May 29, 2010 at 12:29, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Am 29.05.2010 21:06, schrieb P.J. Eby:
>>
>> At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote:
>>>>
>>>> In it he says that PEP 382 is being deferred until it can address PEP
>>>> 302 loaders. I can't find any follow-up to this. I don't see any
>>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was
>>>> never resolved. Does it need to be before PEP 382 is implemented? Are we
>>>> wasting our time by designing and (eventually) coding before this issue
>>>> is resolved?
>>>
>>> Yes, and yes.
>>
>> Is there anything we can do to help regarding that?
>
> You could comment on the proposal I made back then, or propose a different
> solution.

[sorry for the fundamental PEP questions, but I think PEP 382 came
about while I was on my python-dev sabbatical last year]

I have some questions about the PEP which might help clarify how to
handle the API changes.

For finders, their search algorithm is changed in a couple of ways.
One is that modules are given priority over packages (is that
intentional, Martin, or just an oversight?). Two, the package search
requires checking for a .pth file on top of an __init__.py. This will
change finders that could before simply do an existence check on an
__init__ "file" (or whatever the storage back-end happened to be) and
make it into a list-and-search which one would hope wasn't costly, but
in same cases might be if the paths to files is not stored in a
hierarchical fashion (e.g. zip files list entire files paths in their
TOC or a sqlite3 DB which uses a path for keys will have to list
**all** keys, sort them to just the relevant directory, and then look
for .pth or some such approach). Are we worried about possible
performance implications of this search? I say no, but I just want to
make sure people we are not and people are aware about the design
shift required in finders. This entire worry would be alleviated if
only .pth files named after the package were supported, much like
*.pkg files in pkgutil.

And then the search for the __init__.py begins on the newly modified
__path__, which I assume ends with the first __init__ found on
__path__, but if no file is found it's okay and essentially an empty
module with just module-specific attributes is used? In other words,
can a .pth file replace an __init__ file in delineating a package? Or
is it purely additive? I assume the latter for compatibility reasons,
but the PEP says "a directory is considered a package if it **either**
contains a file named __init__.py, **or** a file whose name ends with
".pth"" (emphasis mine). Otherwise I assume that the search will be
done simply with ``os.path.isdir(os.path.join(sys_path_entry,
top_level_package_name)`` and all existing paths will be added to
__path__. Will they come before or after the directory where the *.pth
was found? And will any subsequent *.pth files found in other
directories also be executed?

As for how "*" works, is this limited to top-level packages, or will
sub-packages participate as well? I assume the former, but it is not
directly stated in the PEP. If the latter, is a dotted package name
changed to ``os.sep.join(sy_path_entry, package_name.replace('".",
os.sep)``?

For sys.path_hooks, I am assuming import will simply skip over passing
that as it is a marker that __path__ represents a namsepace package
and not in any way functional. Although with sys.namespace_packages,
is leaving the "*" in __path__ truly necessary?

For the search of paths to use to extend, are we limiting ourselves to
actual file system entries on sys.path (as pkgutil does), or do we
want to support other storage back-ends? To do the latter I would
suggest having a successful path discovery be when a finder can be
created for the hypothetical directory from sys.path_hooks.

OK, I *think* that's all of my clarification questions when it comes
to the PEP. =) Now, on to API discussion.

The PEP (seems to) ask finders to look for a .pth file(s), calculate
__path__, and then get a loader for the __init__. You could have
finders grow a find_namespace method which returns the contents of the
requisite .pth file(s). Import could then take that, calculate
__path__, and then use that new search path to find a loader for the
__init__ (I am assuming there is an __init__ file somewhere). That's
straight-forward and makes supporting .pth files additive for finders.

The trick then becomes how the heck you get the new __path__ value
into the module through the loader as up to this point it has
calculated __path__ on its own. You could slightly abuse load_module's
semantics for reloading and stick the namespace module into
sys.modules before calling the loader for __init__ and change the
semantics definition such that if __path__ is already defined you
don't change it. Unfortunately that seems rather messy in the face of
reloads that want a fresh __path__.

Another possibility is to have the loader add the new paths, but to
provide the calculated value of __path__ be stored on
sys.namespace_packages. That way the loader can simply calculate its
own version and extend it with what the dictionary provides. This
allows loaders get what import thinks __path__ should be and they
still have a chance to tweak things. If you want even more abstraction
I would change is_package to return what __path__ should be when it is
a package and provide an ABC that does the proper calculation of the
extended __path__ value for is_package() so they can do ``return
[extras].extend(super().is_package())`` for packages.

But unfortunately, because load_module is overloaded with
responsibilities, there is no way to dynamically add support for any
of this to existing loaders like there is with finders (unless we
factor out the responsibilities of load_module so it isn't so
overworked and is entirely optional to implement, but that goes beyond
this PEP's scope). There is also the issue of reloading with this
delineation of work since finders are not necessarily called by
imp.reload. Otherwise the loaders will have to recalculate everything
import calculates in order to find the __init__ module to begin with.

The only other option I can think of is to tweak find_module to always
take a path argument, not just meta path finders. Then the calculated
__path__ value can be passed in through find_module and thus passed on
to the loader through a constructor or some such. That doesn't
duplicate the work of calculating the extended __path__ value in both
the finder or loader, nor having to cache it somewhere outside of the
importer's reach where it might go stale. The finder simply passed the
__path__ value on to the loader however it wants (most likely through
a constructor call or internal caching). This also acts as a
performance perk when searching for the __init__ module as having the
'path' argument set can act as a flag to not look for a module but
only a package. This would make it no longer an additive feature to
finders, but wouldn't require anything to change in loaders directly.

I'll shut up now and stop causing trouble. =)

From pje at telecommunity.com  Sun May 30 00:56:10 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 29 May 2010 18:56:10 -0400
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <4C016B1D.8010604@v.loewis.de>
References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de>
	<20100529190623.299C83A405F@sparrow.telecommunity.com>
	<4C016B1D.8010604@v.loewis.de>
Message-ID: <20100529225613.2DD0A3A405F@sparrow.telecommunity.com>

At 09:29 PM 5/29/2010 +0200, Martin v. L?wis wrote:
>Am 29.05.2010 21:06, schrieb P.J. Eby:
>>At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote:
>>>>In it he says that PEP 382 is being deferred until it can address PEP
>>>>302 loaders. I can't find any follow-up to this. I don't see any
>>>>discussion in PEP 382 about PEP 302 loaders, so I assume this issue was
>>>>never resolved. Does it need to be before PEP 382 is implemented? Are we
>>>>wasting our time by designing and (eventually) coding before this issue
>>>>is resolved?
>>>
>>>Yes, and yes.
>>
>>Is there anything we can do to help regarding that?
>
>You could comment on the proposal I made back then, or propose a 
>different solution.

Looking at that proposal, I don't follow how changing *loaders* (vs. 
importers) would help.  If an importer's find_module doesn't natively 
support PEP 382, then there's no way to get a loader for the package 
in the first place.  Today, namespace packages work fine with PEP 302 
loaders, because the namespace-ness is really only about setting up 
the __path__, and detecting that you need to do this in the first place.

In the PEP 302 scheme, then, it's either importers that have to 
change, or the process that invokes them.  Being able to ask an 
importer the equivalents of os.path.join, listdir, and get_data would 
suffice to make an import process that could do the trick.

Essentially, you'd ask each importer to first attempt to find the 
module, and then asking it (or the loader, if the find worked) 
whether packagename/*.pth exists, and then processing their contents.

I don't think there's a need to have a special method for executing a 
package __init__, since what you'd do in the case where there are 
.pth but no __init__, is to simply continue the search to the end of 
sys.path (or the parent package __path__), and *then* create the 
module with an appropriate __path__.

If at any point the find_module() call succeeds, then subsequent 
importers will just be asked for .pth files, which can then be 
processed into the __path__ of the now-loaded module.

IOW, something like this (very rough draft):

     pth_contents = []
     module = None

     for pathitem in syspath_or_parent__path__:

         importer = pkgutil.get_importer(pathitem)
         if importer is None:
             continue

         if module is None:
             try:
                 loader = importer.find_module(fullname)
             except ImportError:
                 pass
             else:
                 # errors here should propagate
                 module = loader.load_module(fullname)
                 if not hasattr(module, '__path__'):
                     # found, but not a package
                     return module

         pc = get_pth_contents(importer)
         if pc is not None:
             subpath = os.path.join(pathitem, modulebasename)
             pth_contents.append(subpath)
             pth_contents.extend(pc)
             if '*' not in pth_contents:
                 # got a package, but not a namespace
                 break

     if pth_contents:
         if module is None:
             # No __init__, but we have paths, so make an empty package
             module = # new module object w/empty __path__
         modify__path__(module, pth_contents)

     return module

Obviously, the details are all in the 'get_pth_contents()', and 
'modify__path__()' functions, and the above process would do extra 
work in the case where an individual importer implements PEP 382 on 
its own (although why would it?).

It's also the case that this algorithm will be slow to fail imports 
when implemented as a meta_path hook, since it will be doing an extra 
pass over sys.path or the parent __path__, in addition to the one 
that's done by the normal __import__ machinery.  (Though that's not 
an issue for Python 3.x, since this can be built into the core __import__).

(Technically, the 3.x version should probably ask meta_path hooks for 
their .pth files as well, but I'm not entirely sure that that's a 
meaningful thing to ask.)

The PEP 302 questions all boil down to how get_pth_contents() is 
implemented, and whether 'subpath' really should be created with 
os.path.join.  Simply adding a get_pth_contents() method to the 
importer protocol (that returns None or a list of lines), and maybe a 
get_subpath(modulename) method that returns the path string that 
should be used for a subdirectory importer (i.e. __path__ entry), or 
None if no such subpath exists.

Adding finer-grained methods is probably a waste of time, as there 
aren't likely to be many use cases for asking an *importer* to fetch 
files (vs. a loader).

(In my case, of course, I'd use the pkgutil-style approach of 
augmenting importers or loaders that don't natively implement a 
needed method, that still allows third parties to register their own 
support for a fourth party's loader or importer type.)


From pje at telecommunity.com  Sun May 30 09:40:37 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 30 May 2010 03:40:37 -0400
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
Message-ID: <20100530074041.2279D3A405F@sparrow.telecommunity.com>

At 03:44 PM 5/29/2010 -0700, Brett Cannon wrote:
>On Sat, May 29, 2010 at 12:29, "Martin v. L??wis" <martin at v.loewis.de> wrote:
> > Am 29.05.2010 21:06, schrieb P.J. Eby:
> >>
> >> At 08:45 PM 5/29/2010 +0200, Martin v. L??wis wrote:
> >>>>
> >>>> In it he says that PEP 382 is being deferred until it can address PEP
> >>>> 302 loaders. I can't find any follow-up to this. I don't see any
> >>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was
> >>>> never resolved. Does it need to be before PEP 382 is implemented? Are we
> >>>> wasting our time by designing and (eventually) coding before this issue
> >>>> is resolved?
> >>>
> >>> Yes, and yes.
> >>
> >> Is there anything we can do to help regarding that?
> >
> > You could comment on the proposal I made back then, or propose a different
> > solution.
>
>[sorry for the fundamental PEP questions, but I think PEP 382 came
>about while I was on my python-dev sabbatical last year]
>
>I have some questions about the PEP which might help clarify how to
>handle the API changes.
>
>For finders, their search algorithm is changed in a couple of ways.
>One is that modules are given priority over packages (is that
>intentional, Martin, or just an oversight?). Two, the package search
>requires checking for a .pth file on top of an __init__.py. This will
>change finders that could before simply do an existence check on an
>__init__ "file" (or whatever the storage back-end happened to be) and
>make it into a list-and-search which one would hope wasn't costly, but
>in same cases might be if the paths to files is not stored in a
>hierarchical fashion (e.g. zip files list entire files paths in their
>TOC or a sqlite3 DB which uses a path for keys will have to list
>**all** keys, sort them to just the relevant directory, and then look
>for .pth or some such approach). Are we worried about possible
>performance implications of this search?

No.  First, an importer would not be required to implement it in a 
precisely analagous way; you could have database entries or a special 
consolidated index in a zipfile, if you wanted to do it like 
that.  (In practice, Python's zipimporter has a memory cache of the 
TOC, and a simple database index on paths would make a search for 
.pth's in a subdirectory trivial for the database case.)


>  I say no, but I just want to
>make sure people we are not and people are aware about the design
>shift required in finders. This entire worry would be alleviated if
>only .pth files named after the package were supported, much like
>*.pkg files in pkgutil.

Which would completely break one of the major use cases of the PEP, 
which is precisely to ensure that you can install two pieces of code 
to the same namespace without either one overwriting the other's files.


>And then the search for the __init__.py begins on the newly modified
>__path__, which I assume ends with the first __init__ found on
>__path__, but if no file is found it's okay and essentially an empty
>module with just module-specific attributes is used? In other words,
>can a .pth file replace an __init__ file in delineating a package?

Yes.


>Or is it purely additive? I assume the latter for compatibility reasons,

Nope.  The idea is specifically to allow separately installed 
projects to create a package without overwriting any files (causing 
conflicts for system installers).


>but the PEP says "a directory is considered a package if it **either**
>contains a file named __init__.py, **or** a file whose name ends with
>".pth"" (emphasis mine). Otherwise I assume that the search will be
>done simply with ``os.path.isdir(os.path.join(sys_path_entry,
>top_level_package_name)`` and all existing paths will be added to
>__path__. Will they come before or after the directory where the *.pth
>was found? And will any subsequent *.pth files found in other
>directories also be executed?
>
>As for how "*" works, is this limited to top-level packages, or will
>sub-packages participate as well?

Sub-packages as well.


>  I assume the former, but it is not
>directly stated in the PEP. If the latter, is a dotted package name
>changed to ``os.sep.join(sy_path_entry, package_name.replace('".",
>os.sep)``?
>
>For sys.path_hooks, I am assuming import will simply skip over passing
>that as it is a marker that __path__ represents a namsepace package
>and not in any way functional. Although with sys.namespace_packages,
>is leaving the "*" in __path__ truly necessary?

I'm going to leave these to Martin to answer.


>For the search of paths to use to extend, are we limiting ourselves to
>actual file system entries on sys.path (as pkgutil does),

pkgutil doesn't have such a limitation, except in the case 
extend_path, and that limitation is one that PEP 382 intends to remove.


>or do we
>want to support other storage back-ends? To do the latter I would
>suggest having a successful path discovery be when a finder can be
>created for the hypothetical directory from sys.path_hooks.

The downside to that is that NullImporter is the default importer, so 
you'd still have to special case it.  It would make more sense to add 
to the PEP 302 protocols directly.


>I'll shut up now and stop causing trouble. =)

May I suggest you take a look at the implementation draft in my other 
email?  I realize in retrospect it doesn't handle __init__ searching 
in precisely the order proposed by the PEP, but I'm not sure it would 
be that difficult to add.  (It also needs to split the operation into 
find/load pieces, but that's also a straightforward mod: just defer 
the module loading until the end, and return a wrapper around the 
loader that finishes the process.)


From tjreedy at udel.edu  Mon May 31 01:22:11 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 30 May 2010 19:22:11 -0400
Subject: [Python-Dev] Bugfix releases should not change APIs
In-Reply-To: <20100529123937.222a4c0d@pitrou.net>
References: <htpi4a$t2i$1@dough.gmane.org>	<htpn8h$9td$1@dough.gmane.org>
	<20100529123937.222a4c0d@pitrou.net>
Message-ID: <hturv4$s5s$1@dough.gmane.org>

On 5/29/2010 6:39 AM, Antoine Pitrou wrote:
> It is not the product of oversight.

I am actually glad, in a sense, that it was not casual whim. ;-)
I do not like the change, since it moves streams back further away from 
Python's sequence model, but I withdraw the request for reversion in 3.1.3.

I will add further comments on the docs to the issue.

> What it does teach us is that Python 3.1 sees some real use,

It is an odd 'coincidence' that the method changed was one of the only 
two stdlib methods I have used so far used directly. But with enough 
users, such happens.

What it teaches *me* is that before I install another release, I should, 
as planned, automate the running of all module tests together so I can 
easily test everything before and after a new installation.

When I do release sample chapters and code, I will try to remember to 
specify the version and platform I tested with.

> we have entered a phase where backwards compatibility will become as
> important as it was in the 2.x line.

I have assumed that there might be a few stdlib API tweeks in 3.2 -- and 
that they would be well announced.

Terry Jan Reedy


From tjreedy at udel.edu  Mon May 31 01:22:55 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 30 May 2010 19:22:55 -0400
Subject: [Python-Dev] Bugfix releases should not change APIs
In-Reply-To: <4C008CED.8080406@gmail.com>
References: <htpi4a$t2i$1@dough.gmane.org> <4C008CED.8080406@gmail.com>
Message-ID: <htus0e$s5s$2@dough.gmane.org>

On 5/28/2010 11:41 PM, Nick Coghlan wrote:

> However, it may be worth modifying the policy to ensure that such
> exceptional bug fixes be mentioned prominently in the release notes and
> on the download page for that maintenance release.

A sentence like "The behavior of it.X.truncate has been intentionally 
changed from ... to ... .", if I read and cognized it, would have helped 
me, in this case, to the problem and fix much more quickly.

Is it possible with svn or hg to get a list of the commits that changed 
version x to version y?

Would is not be possible to get a diff between at least the .rst 
versions of the docs for version x and version y?

Terry Jan Reedy



From tjreedy at udel.edu  Mon May 31 01:55:31 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 30 May 2010 19:55:31 -0400
Subject: [Python-Dev] variable name resolution in exec is incorrect
In-Reply-To: <AANLkTinj2liqSAWTFVgDbkiaZMpFZLJpYTuGSyhk4KMg@mail.gmail.com>
References: <AANLkTilUBbivySNgHM22_8lixOq5se9l05OL9ORUTUE9@mail.gmail.com>
	<AANLkTinhi-RFphaGZi7Wn58CEGP6fafSR9PmHY_mnIyt@mail.gmail.com>
	<AANLkTilo8uTiRF9XH3KDJjNBhyJ_dN8meBV6Ilwk2Rz9@mail.gmail.com>
	<AANLkTikj_lrEsA3g6QDA7hBVxRCGYvY1GO0Vm-3AKifQ@mail.gmail.com>
	<AANLkTilmboNnesgbFkoajrY-kPQ4EQBhml-hDotwEL7g@mail.gmail.com>
	<4BFDB8EE.4010307@gmail.com>	<AANLkTimJ6KmCtLZ19moTDtw6VUQtaXYwmvacVpoXwmYC@mail.gmail.com>
	<4BFE92D9.1060306@gmail.com>	<AANLkTimUiP7GGPRM5hTyf50S4Z8yGubzpWvwX3P_Zlvz@mail.gmail.com>
	<AANLkTinj2liqSAWTFVgDbkiaZMpFZLJpYTuGSyhk4KMg@mail.gmail.com>
Message-ID: <htuttl$gt$1@dough.gmane.org>

On 5/29/2010 6:20 AM, Colin H wrote:
> Perhaps the next step is to re-open the issue? If it is seen as a bug,
> it would be great to see a fix in 2.6+ -

For the purpose of bugfix releases, a 'bug' is a discrepancy between doc 
and behavior. Every new feature is seen as a 'design bug' by someone.

 > a number of options which
> will not break backward compatibility have been put forward - cheers,

Code that uses a new x.y.z feature does not work in previous x.y 
versions. Problems with such micro-release additions lead to the current 
policy.

The 3.2 feature addition deadline is about 5 months away. It will 
probably take 1 or more people at least a couple of months to write a 
PEP listing the rationale for a change, the options and possible pros 
and cons, possibly test one or more patches, solicit opinions on which 
is best, select one, write new test cases and docs, and get the final 
patch committed.

Terry Jan Reedy


From brett at python.org  Mon May 31 02:59:57 2010
From: brett at python.org (Brett Cannon)
Date: Sun, 30 May 2010 17:59:57 -0700
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <20100529225613.2DD0A3A405F@sparrow.telecommunity.com>
References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> 
	<20100529190623.299C83A405F@sparrow.telecommunity.com>
	<4C016B1D.8010604@v.loewis.de> 
	<20100529225613.2DD0A3A405F@sparrow.telecommunity.com>
Message-ID: <AANLkTikrvAowoPyIvSCS1jTdp8jofEB_5P0Sw_O-nKdY@mail.gmail.com>

On Sat, May 29, 2010 at 15:56, P.J. Eby <pje at telecommunity.com> wrote:
> At 09:29 PM 5/29/2010 +0200, Martin v. L?wis wrote:
>>
>> Am 29.05.2010 21:06, schrieb P.J. Eby:
>>>
>>> At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote:
>>>>>
>>>>> In it he says that PEP 382 is being deferred until it can address PEP
>>>>> 302 loaders. I can't find any follow-up to this. I don't see any
>>>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue was
>>>>> never resolved. Does it need to be before PEP 382 is implemented? Are
>>>>> we
>>>>> wasting our time by designing and (eventually) coding before this issue
>>>>> is resolved?
>>>>
>>>> Yes, and yes.
>>>
>>> Is there anything we can do to help regarding that?
>>
>> You could comment on the proposal I made back then, or propose a different
>> solution.
>
> Looking at that proposal, I don't follow how changing *loaders* (vs.
> importers) would help. ?If an importer's find_module doesn't natively
> support PEP 382, then there's no way to get a loader for the package in the
> first place. ?Today, namespace packages work fine with PEP 302 loaders,
> because the namespace-ness is really only about setting up the __path__, and
> detecting that you need to do this in the first place.
>
> In the PEP 302 scheme, then, it's either importers that have to change, or
> the process that invokes them. ?Being able to ask an importer the
> equivalents of os.path.join, listdir, and get_data would suffice to make an
> import process that could do the trick.
>
> Essentially, you'd ask each importer to first attempt to find the module,
> and then asking it (or the loader, if the find worked) whether
> packagename/*.pth exists, and then processing their contents.
>
> I don't think there's a need to have a special method for executing a
> package __init__, since what you'd do in the case where there are .pth but
> no __init__, is to simply continue the search to the end of sys.path (or the
> parent package __path__), and *then* create the module with an appropriate
> __path__.
>
> If at any point the find_module() call succeeds, then subsequent importers
> will just be asked for .pth files, which can then be processed into the
> __path__ of the now-loaded module.
>
> IOW, something like this (very rough draft):
>
> ? ?pth_contents = []
> ? ?module = None
>
> ? ?for pathitem in syspath_or_parent__path__:
>
> ? ? ? ?importer = pkgutil.get_importer(pathitem)
> ? ? ? ?if importer is None:
> ? ? ? ? ? ?continue
>
> ? ? ? ?if module is None:
> ? ? ? ? ? ?try:
> ? ? ? ? ? ? ? ?loader = importer.find_module(fullname)
> ? ? ? ? ? ?except ImportError:
> ? ? ? ? ? ? ? ?pass
> ? ? ? ? ? ?else:
> ? ? ? ? ? ? ? ?# errors here should propagate
> ? ? ? ? ? ? ? ?module = loader.load_module(fullname)
> ? ? ? ? ? ? ? ?if not hasattr(module, '__path__'):
> ? ? ? ? ? ? ? ? ? ?# found, but not a package
> ? ? ? ? ? ? ? ? ? ?return module
>
> ? ? ? ?pc = get_pth_contents(importer)
> ? ? ? ?if pc is not None:
> ? ? ? ? ? ?subpath = os.path.join(pathitem, modulebasename)
> ? ? ? ? ? ?pth_contents.append(subpath)
> ? ? ? ? ? ?pth_contents.extend(pc)
> ? ? ? ? ? ?if '*' not in pth_contents:
> ? ? ? ? ? ? ? ?# got a package, but not a namespace
> ? ? ? ? ? ? ? ?break
>
> ? ?if pth_contents:
> ? ? ? ?if module is None:
> ? ? ? ? ? ?# No __init__, but we have paths, so make an empty package
> ? ? ? ? ? ?module = # new module object w/empty __path__
> ? ? ? ?modify__path__(module, pth_contents)
>
> ? ?return module
>

Is it wise to modify __path__ post-import? Today people can make sure
that __path__ is set to what they want before potentially reading it
in their __init__ module by making the pkgutil.extend_path() call
first. This would actually defer to after the import and thus not
allow any __init__ code to rely on what __path__ eventually becomes.

> Obviously, the details are all in the 'get_pth_contents()', and
> 'modify__path__()' functions, and the above process would do extra work in
> the case where an individual importer implements PEP 382 on its own
> (although why would it?).
>
> It's also the case that this algorithm will be slow to fail imports when
> implemented as a meta_path hook, since it will be doing an extra pass over
> sys.path or the parent __path__, in addition to the one that's done by the
> normal __import__ machinery. ?(Though that's not an issue for Python 3.x,
> since this can be built into the core __import__).
>
> (Technically, the 3.x version should probably ask meta_path hooks for their
> .pth files as well, but I'm not entirely sure that that's a meaningful thing
> to ask.)
>
> The PEP 302 questions all boil down to how get_pth_contents() is
> implemented, and whether 'subpath' really should be created with
> os.path.join. ?Simply adding a get_pth_contents() method to the importer
> protocol (that returns None or a list of lines), and maybe a
> get_subpath(modulename) method that returns the path string that should be
> used for a subdirectory importer (i.e. __path__ entry), or None if no such
> subpath exists.

Code already out there uses os.path.join() to extend __path__ (e.g.
Django), so I would stick with that unless we want to start
transitioning to '/' only.

From brett at python.org  Mon May 31 03:18:07 2010
From: brett at python.org (Brett Cannon)
Date: Sun, 30 May 2010 18:18:07 -0700
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <20100530074041.2279D3A405F@sparrow.telecommunity.com>
References: <20100530074041.2279D3A405F@sparrow.telecommunity.com>
Message-ID: <AANLkTinjHXpfLtmwpxouqdqJixQSXKzcYVUi0py1a4hT@mail.gmail.com>

On Sun, May 30, 2010 at 00:40, P.J. Eby <pje at telecommunity.com> wrote:
> At 03:44 PM 5/29/2010 -0700, Brett Cannon wrote:
>>
>> On Sat, May 29, 2010 at 12:29, "Martin v. L?wis" <martin at v.loewis.de>
>> wrote:
>> > Am 29.05.2010 21:06, schrieb P.J. Eby:
>> >>
>> >> At 08:45 PM 5/29/2010 +0200, Martin v. L?wis wrote:
>> >>>>
>> >>>> In it he says that PEP 382 is being deferred until it can address PEP
>> >>>> 302 loaders. I can't find any follow-up to this. I don't see any
>> >>>> discussion in PEP 382 about PEP 302 loaders, so I assume this issue
>> >>>> was
>> >>>> never resolved. Does it need to be before PEP 382 is implemented? Are
>> >>>> we
>> >>>> wasting our time by designing and (eventually) coding before this
>> >>>> issue
>> >>>> is resolved?
>> >>>
>> >>> Yes, and yes.
>> >>
>> >> Is there anything we can do to help regarding that?
>> >
>> > You could comment on the proposal I made back then, or propose a
>> > different
>> > solution.
>>
>> [sorry for the fundamental PEP questions, but I think PEP 382 came
>> about while I was on my python-dev sabbatical last year]
>>
>> I have some questions about the PEP which might help clarify how to
>> handle the API changes.
>>
>> For finders, their search algorithm is changed in a couple of ways.
>> One is that modules are given priority over packages (is that
>> intentional, Martin, or just an oversight?). Two, the package search
>> requires checking for a .pth file on top of an __init__.py. This will
>> change finders that could before simply do an existence check on an
>> __init__ "file" (or whatever the storage back-end happened to be) and
>> make it into a list-and-search which one would hope wasn't costly, but
>> in same cases might be if the paths to files is not stored in a
>> hierarchical fashion (e.g. zip files list entire files paths in their
>> TOC or a sqlite3 DB which uses a path for keys will have to list
>> **all** keys, sort them to just the relevant directory, and then look
>> for .pth or some such approach). Are we worried about possible
>> performance implications of this search?
>
> No. ?First, an importer would not be required to implement it in a precisely
> analagous way; you could have database entries or a special consolidated
> index in a zipfile, if you wanted to do it like that. ?(In practice,
> Python's zipimporter has a memory cache of the TOC, and a simple database
> index on paths would make a search for .pth's in a subdirectory trivial for
> the database case.)

Sure, for the two examples this works, but who knows about other odd
back-ends people might be using. Granted, this is all hypothetical and
why I figured we wouldn't worry about it.

>
>
>> ?I say no, but I just want to
>> make sure people we are not and people are aware about the design
>> shift required in finders. This entire worry would be alleviated if
>> only .pth files named after the package were supported, much like
>> *.pkg files in pkgutil.
>
> Which would completely break one of the major use cases of the PEP, which is
> precisely to ensure that you can install two pieces of code to the same
> namespace without either one overwriting the other's files.

The PEP says the goal is to span packages across directories. If you
split something like zope into multiple directories, does having a
separate zope.pth file in each of those directories really cause a
problem here? You are not importing them so it isn't like you are
worrying about precedence. If you specify that all .pth files found
are run then using the same file name in all package directories isn't
an issue. But I guess packages that do this want to keep unique files
per directory separation that they support and not have to fix the
file names at distribution time.

>
>
>> And then the search for the __init__.py begins on the newly modified
>> __path__, which I assume ends with the first __init__ found on
>> __path__, but if no file is found it's okay and essentially an empty
>> module with just module-specific attributes is used? In other words,
>> can a .pth file replace an __init__ file in delineating a package?
>
> Yes.
>
>
>> Or is it purely additive? I assume the latter for compatibility reasons,
>
> Nope. ?The idea is specifically to allow separately installed projects to
> create a package without overwriting any files (causing conflicts for system
> installers).
>
>
>> but the PEP says "a directory is considered a package if it **either**
>> contains a file named __init__.py, **or** a file whose name ends with
>> ".pth"" (emphasis mine). Otherwise I assume that the search will be
>> done simply with ``os.path.isdir(os.path.join(sys_path_entry,
>> top_level_package_name)`` and all existing paths will be added to
>> __path__. Will they come before or after the directory where the *.pth
>> was found? And will any subsequent *.pth files found in other
>> directories also be executed?
>>
>> As for how "*" works, is this limited to top-level packages, or will
>> sub-packages participate as well?
>
> Sub-packages as well.
>
>
>> ?I assume the former, but it is not
>> directly stated in the PEP. If the latter, is a dotted package name
>> changed to ``os.sep.join(sy_path_entry, package_name.replace('".",
>> os.sep)``?
>>
>> For sys.path_hooks, I am assuming import will simply skip over passing
>> that as it is a marker that __path__ represents a namsepace package
>> and not in any way functional. Although with sys.namespace_packages,
>> is leaving the "*" in __path__ truly necessary?
>
> I'm going to leave these to Martin to answer.
>
>
>> For the search of paths to use to extend, are we limiting ourselves to
>> actual file system entries on sys.path (as pkgutil does),
>
> pkgutil doesn't have such a limitation, except in the case extend_path, and
> that limitation is one that PEP 382 intends to remove.

It's because pkgutil.extend_path has that limitation I am asking as
that's what the PEP refers to. If the PEP wants to remove the
limitation it should clearly state how it is going to do that.

>
>
>> or do we
>> want to support other storage back-ends? To do the latter I would
>> suggest having a successful path discovery be when a finder can be
>> created for the hypothetical directory from sys.path_hooks.
>
> The downside to that is that NullImporter is the default importer, so you'd
> still have to special case it. ?It would make more sense to add to the PEP
> 302 protocols directly.

But import is the one that adds NullImporter. And if import is the one
figuring out what paths do and don't work, then it doesn't matter
about NullImporter as it will know what does and does not fail. And as
you said, you can special-case it.

As for adding to the PEP 302 protocols, it's a question of how much we
want importer implementors to have control over this versus us. I
personally would rather keep any protocol extensions simple and have
import handle as many of the details as possible. I think the PEP 3147
has shown the benefits of letting import details be under our control
as much as possible can be beneficial as it doesn't put pre-existing
importers at a disadvantage.

>
>
>> I'll shut up now and stop causing trouble. =)
>
> May I suggest you take a look at the implementation draft in my other email?
> ?I realize in retrospect it doesn't handle __init__ searching in precisely
> the order proposed by the PEP, but I'm not sure it would be that difficult
> to add. ?(It also needs to split the operation into find/load pieces, but
> that's also a straightforward mod: just defer the module loading until the
> end, and return a wrapper around the loader that finishes the process.)

I replied separately to that email.

From pje at telecommunity.com  Mon May 31 06:50:39 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 31 May 2010 00:50:39 -0400
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <AANLkTikrvAowoPyIvSCS1jTdp8jofEB_5P0Sw_O-nKdY@mail.gmail.c
 om>
References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de>
	<20100529190623.299C83A405F@sparrow.telecommunity.com>
	<4C016B1D.8010604@v.loewis.de>
	<20100529225613.2DD0A3A405F@sparrow.telecommunity.com>
	<AANLkTikrvAowoPyIvSCS1jTdp8jofEB_5P0Sw_O-nKdY@mail.gmail.com>
Message-ID: <20100531045042.2A3DC3A402D@sparrow.telecommunity.com>

At 05:59 PM 5/30/2010 -0700, Brett Cannon wrote:
>Is it wise to modify __path__ post-import? Today people can make sure
>that __path__ is set to what they want before potentially reading it
>in their __init__ module by making the pkgutil.extend_path() call
>first. This would actually defer to after the import and thus not
>allow any __init__ code to rely on what __path__ eventually becomes.

Well, that's what the other lines in the .pth files are for.  Keep in 
mind that only *one* project can contain the namespace package's 
__init__ module, so it's only sane for that __init__ to import things 
that are bundled with the __init__ module.

AFAIK, most uses of namespace packages today are via setuptools' API, 
which doesn't support having a non-empty __init__.py at all (apart 
from the namespace declaration), so this limitation is unlikely to 
cause problems in practice.

When the code I gave is refactored into a proper importer/loader 
pair, it can actually be structured such that the full __path__ is 
set *before* the low-level loader is called; however, if the loader 
itself chooses to overwrite __path__ at that point, there's little 
that can be done about it.

In the Python 3.x case, the loader protocol could be revised to 
require only *adding* a non-duplicate entry to __path__ if it's 
present, and the stdlib loaders updated accordingly.  For my 
backport, OTOH, I'd have to do some sort of workaround to wrap the 
regular importers, so I'd just as soon leave it undefined by PEP 382 
what an __init__ module sees in __path__ during its execution.  (And 
for a backport whose sole purpose is to cut down on setuptools' funky 
.pth manipulations, that might suffice anyway.)


From pje at telecommunity.com  Mon May 31 07:03:26 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 31 May 2010 01:03:26 -0400
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <AANLkTinjHXpfLtmwpxouqdqJixQSXKzcYVUi0py1a4hT@mail.gmail.c
 om>
References: <20100530074041.2279D3A405F@sparrow.telecommunity.com>
	<AANLkTinjHXpfLtmwpxouqdqJixQSXKzcYVUi0py1a4hT@mail.gmail.com>
Message-ID: <20100531050328.40B073A402D@sparrow.telecommunity.com>

At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote:
>On Sun, May 30, 2010 at 00:40, P.J. Eby <pje at telecommunity.com> wrote:
> >
> > Which would completely break one of the major use cases of the 
> PEP, which is
> > precisely to ensure that you can install two pieces of code to the same
> > namespace without either one overwriting the other's files.
>
>The PEP says the goal is to span packages across directories.

The goal of namespace packages is to allow separately-distributed 
pieces of code to live in the same package namespace.  That this is 
sometimes achieved by installing them to different paths is an 
implementation detail.

In the case of e.g. Linux distributions and other system packaging 
scenarios, the code will all be installed to the *same* directory -- 
so there cannot be any filename collisions among the 
separately-distributed modules.  That's why we want to get rid of the 
need for an __init__.py to mark the directory as a package: it's a 
collision point for system package management tools.


> > pkgutil doesn't have such a limitation, except in the case extend_path, and
> > that limitation is one that PEP 382 intends to remove.
>
>It's because pkgutil.extend_path has that limitation I am asking as
>that's what the PEP refers to. If the PEP wants to remove the
>limitation it should clearly state how it is going to do that.

I'm flexible on it either way.  The only other importer I know of 
that does anything else is one that actually allows (unsafely) 
importing from URLs.

If we allow for other things, then we need to extend the PEP 302 
protocol to have a way to ask an importer for a subpath string.



>As for adding to the PEP 302 protocols, it's a question of how much we
>want importer implementors to have control over this versus us. I
>personally would rather keep any protocol extensions simple and have
>import handle as many of the details as possible.

I lean the other way a bit, in that the more of the importer 
internals you expose, the harder you make it for an importer to be 
anything other than a mere virtual file system.  (As it is, I think 
there is too much "file-ness" coupling in the protocol already, what 
with file extensions and the like.)

Indeed, now that I'm thinking about it, it actually seems to make 
more sense to just require the importers to implement PEP 382, and 
provide some common machinery in imp or pkgutil for reading .pth 
strings, setting up __path__, and hunting down all the other directories.


From martin at v.loewis.de  Mon May 31 09:24:22 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 31 May 2010 09:24:22 +0200
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <20100529225613.2DD0A3A405F@sparrow.telecommunity.com>
References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de>
	<20100529190623.299C83A405F@sparrow.telecommunity.com>
	<4C016B1D.8010604@v.loewis.de>
	<20100529225613.2DD0A3A405F@sparrow.telecommunity.com>
Message-ID: <4C036426.5050907@v.loewis.de>

> Looking at that proposal, I don't follow how changing *loaders* (vs.
> importers) would help. If an importer's find_module doesn't natively
> support PEP 382, then there's no way to get a loader for the package in
> the first place.

True. However, this requires no changes to the API, AFAICT. The *finder*
(there are no importers, right?) will need to accept a folder with just 
a pth file as a package, but then still return a loader.

The issue is then how to make the loader scan the folder for .pth files.
One option would be to ask it "give me the contents of all the pth 
files", the other would be to have a method "load all the pth files".

> In the PEP 302 scheme, then, it's either importers that have to change,
> or the process that invokes them. Being able to ask an importer the
> equivalents of os.path.join, listdir, and get_data would suffice to make
> an import process that could do the trick.

That would also work, but it would make a fairly wide interface. IIRC, 
MAL complained that doing so would break importers which can't do listdir.

> Essentially, you'd ask each importer to first attempt to find the
> module, and then asking it (or the loader, if the find worked) whether
> packagename/*.pth exists, and then processing their contents.

The latter is my proposal, yes: ask the loader to process all pth files.

> If at any point the find_module() call succeeds, then subsequent
> importers will just be asked for .pth files, which can then be processed
> into the __path__ of the now-loaded module.
>
> IOW, something like this (very rough draft):


 >    pth_contents = []
 >    module = None
 >
 >    for pathitem in syspath_or_parent__path__:
 >
 >        importer = pkgutil.get_importer(pathitem)
 >        if importer is None:
 >            continue
 >
 >        if module is None:
 >            try:
 >                loader = importer.find_module(fullname)
 >            except ImportError:
 >                pass

Is this really how it works today? Shouldn't it abort here
if there is an ImportError?

 >            else:
 >                # errors here should propagate
 >                module = loader.load_module(fullname)
 >                if not hasattr(module, '__path__'):
 >                    # found, but not a package
 >                    return module
 >
 >        pc = get_pth_contents(importer)

Assuming we always get here with a loader, I'd rather call this
on the loader.

Regards,
Martin

From martin at v.loewis.de  Mon May 31 09:53:19 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 31 May 2010 09:53:19 +0200
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <AANLkTikReDF7mMCZBTCOO7Ky7WwsjwJMTNw3rplmiM8q@mail.gmail.com>
References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de>
	<20100529190623.299C83A405F@sparrow.telecommunity.com>
	<4C016B1D.8010604@v.loewis.de>
	<AANLkTikReDF7mMCZBTCOO7Ky7WwsjwJMTNw3rplmiM8q@mail.gmail.com>
Message-ID: <4C036AEF.6040504@v.loewis.de>

> For finders, their search algorithm is changed in a couple of ways.
> One is that modules are given priority over packages (is that
> intentional, Martin, or just an oversight?).

That's an oversight. Notice, however, that it's really not the case that
currently directories have precedence over modules, either: if a 
directory is later on the __path__ than a module, it's still the module 
that gets imported. So the precedence takes place only when a module and 
a directory exist in the same directory.

In any case, I have now fixed it.

> Two, the package search
> requires checking for a .pth file on top of an __init__.py. This will
> change finders that could before simply do an existence check on an
> __init__ "file"

You are reading something into the PEP that isn't there yet. PEP 302 
currently isn't considered, and the question of this discussion is 
precisely how the loaders API should be changed.

> (or whatever the storage back-end happened to be) and
> make it into a list-and-search which one would hope wasn't costly, but
> in same cases might be if the paths to files is not stored in a
> hierarchical fashion (e.g. zip files list entire files paths in their
> TOC or a sqlite3 DB which uses a path for keys will have to list
> **all** keys, sort them to just the relevant directory, and then look
> for .pth or some such approach).

First, I think it's up to the specific loader mechanism whether
PEP 382 should be supported at all. It should be possible to implement
it if desired, but if it's not feasible (e.g. for URL loaders), pth
files just may not get considered. The loader may well provide a
different mechanism to support namespace packages.

 > Are we worried about possible
> performance implications of this search?

For the specific case of zip files, I'm not. I don't think performance 
will suffer at all.

> And then the search for the __init__.py begins on the newly modified
> __path__, which I assume ends with the first __init__ found on
> __path__, but if no file is found it's okay and essentially an empty
> module with just module-specific attributes is used?

Correct.

> In other words,
> can a .pth file replace an __init__ file in delineating a package?

That's what it means by '''a directory is considered a package if it 
either contains a file named __init__.py, or a file whose name ends with 
".pth".'''

> Or
> is it purely additive? I assume the latter for compatibility reasons,
> but the PEP says "a directory is considered a package if it **either**
> contains a file named __init__.py, **or** a file whose name ends with
> ".pth"" (emphasis mine).

Why do you think this causes an incompatibility?

> Otherwise I assume that the search will be
> done simply with ``os.path.isdir(os.path.join(sys_path_entry,
> top_level_package_name)`` and all existing paths will be added to
> __path__. Will they come before or after the directory where the *.pth
> was found? And will any subsequent *.pth files found in other
> directories also be executed?

I may misremember, but from reading the text, it seems to say "no".
It should work like the current pth mechanism (plus *, minus import).

> As for how "*" works, is this limited to top-level packages, or will
> sub-packages participate as well? I assume the former, but it is not
> directly stated in the PEP.

And indeed, the latter is intended. You should be able to create 
namespace packages on all levels.

> If the latter, is a dotted package name
> changed to ``os.sep.join(sy_path_entry, package_name.replace('".",
> os.sep)``?

No. Instead, the parent package's __path__ is being searched for 
directories; sys.path is not considered anymore. I have fixed the text.

> For sys.path_hooks, I am assuming import will simply skip over passing
> that as it is a marker that __path__ represents a namsepace package
> and not in any way functional. Although with sys.namespace_packages,
> is leaving the "*" in __path__ truly necessary?

It would help with efficiency, no?

> For the search of paths to use to extend, are we limiting ourselves to
> actual file system entries on sys.path (as pkgutil does), or do we
> want to support other storage back-ends? To do the latter I would
> suggest having a successful path discovery be when a finder can be
> created for the hypothetical directory from sys.path_hooks.

Again: PEP 302 isn't really considered yet. Proposals are welcome.

> The PEP (seems to) ask finders to look for a .pth file(s), calculate
> __path__, and then get a loader for the __init__. You could have
> finders grow a find_namespace method which returns the contents of the
> requisite .pth file(s).

I must be misunderstanding the concept of finders. Why is it that it 
would be their function to process the pth files, and not the function 
of the loader?

Regards,
Martin

From martin at v.loewis.de  Mon May 31 09:59:40 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 31 May 2010 09:59:40 +0200
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <AANLkTinjHXpfLtmwpxouqdqJixQSXKzcYVUi0py1a4hT@mail.gmail.com>
References: <20100530074041.2279D3A405F@sparrow.telecommunity.com>
	<AANLkTinjHXpfLtmwpxouqdqJixQSXKzcYVUi0py1a4hT@mail.gmail.com>
Message-ID: <4C036C6C.2080606@v.loewis.de>

> The PEP says the goal is to span packages across directories. If you
> split something like zope into multiple directories, does having a
> separate zope.pth file in each of those directories really cause a
> problem here?

I think pje already answered this: yes, you do split zope into multiple 
packages. But then, you may install all of them into the same folder.
This has caused pain for Linux package management tools in particular,
because they dislike if to packages install the same file. So if you
can arrange the pth files to have non-overlapping names, you can still 
install them into the same directory, and dpkg is happy.

Regards,
Martin

From martin at v.loewis.de  Mon May 31 10:03:00 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 31 May 2010 10:03:00 +0200
Subject: [Python-Dev] Bugfix releases should not change APIs
In-Reply-To: <htus0e$s5s$2@dough.gmane.org>
References: <htpi4a$t2i$1@dough.gmane.org> <4C008CED.8080406@gmail.com>
	<htus0e$s5s$2@dough.gmane.org>
Message-ID: <4C036D34.7080401@v.loewis.de>

> Is it possible with svn or hg to get a list of the commits that changed
> version x to version y?

A regular "svn log" on the maintenance branch will give you all the 
changes. You'll recognize from the checkin messages when the previous 
release was.

> Would is not be possible to get a diff between at least the .rst
> versions of the docs for version x and version y?

That's most certainly possible, using "svn diff -rX:Y".

Regards,
Martin

From pje at telecommunity.com  Mon May 31 16:51:45 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 31 May 2010 10:51:45 -0400
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
Message-ID: <20100531145147.9BEF83A402D@sparrow.telecommunity.com>

At 09:24 AM 5/31/2010 +0200, Martin v. L?wis wrote:
>Is this really how it works today? Shouldn't it abort here
>if there is an ImportError?

Oops.  You're right - I was confusing find_module with the path_hooks 
protoocol.


> >            else:
> >                # errors here should propagate
> >                module = loader.load_module(fullname)
> >                if not hasattr(module, '__path__'):
> >                    # found, but not a package
> >                    return module
> >
> >        pc = get_pth_contents(importer)
>
>Assuming we always get here with a loader, I'd rather call this
>on the loader.

We don't, unless the finder natively supports PEP 382.


From smarv at gmx.net  Mon May 31 20:45:22 2010
From: smarv at gmx.net (smarv at gmx.net)
Date: Mon, 31 May 2010 20:45:22 +0200
Subject: [Python-Dev] tp_dealloc
Message-ID: <20100531184522.173170@gmx.net>

Hi,
when embedding python 3.1, I have my own free-method in tp_dealloc.
The allocated memory is in host-memory, not in python (dll). Now, the problem is, Python appears to read-access the deallocated memory still after tp_dealloc. After tp_dealloc, I get an access violation if the pyobject-header fields have been modified inside tp_dealloc. If I leave the header unmodified, then no access violation occurs. Accessing deallocated memory sounds like a bug, or is this intended design?

Thank you
Marvin


-- 
GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

From brett at python.org  Mon May 31 22:10:45 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 31 May 2010 13:10:45 -0700
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <4C036AEF.6040504@v.loewis.de>
References: <4C011C02.4010605@trueblade.com> <4C0160D3.70304@v.loewis.de> 
	<20100529190623.299C83A405F@sparrow.telecommunity.com>
	<4C016B1D.8010604@v.loewis.de> 
	<AANLkTikReDF7mMCZBTCOO7Ky7WwsjwJMTNw3rplmiM8q@mail.gmail.com> 
	<4C036AEF.6040504@v.loewis.de>
Message-ID: <AANLkTika8NsFNFmmI60tzBWkf6VXAUuao_e5RrqeFXdE@mail.gmail.com>

On Mon, May 31, 2010 at 00:53, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> For finders, their search algorithm is changed in a couple of ways.
>> One is that modules are given priority over packages (is that
>> intentional, Martin, or just an oversight?).
>
> That's an oversight. Notice, however, that it's really not the case that
> currently directories have precedence over modules, either: if a directory
> is later on the __path__ than a module, it's still the module that gets
> imported. So the precedence takes place only when a module and a directory
> exist in the same directory.
>
> In any case, I have now fixed it.
>
>> Two, the package search
>> requires checking for a .pth file on top of an __init__.py. This will
>> change finders that could before simply do an existence check on an
>> __init__ "file"
>
> You are reading something into the PEP that isn't there yet. PEP 302
> currently isn't considered, and the question of this discussion is precisely
> how the loaders API should be changed.
>
>> (or whatever the storage back-end happened to be) and
>> make it into a list-and-search which one would hope wasn't costly, but
>> in same cases might be if the paths to files is not stored in a
>> hierarchical fashion (e.g. zip files list entire files paths in their
>> TOC or a sqlite3 DB which uses a path for keys will have to list
>> **all** keys, sort them to just the relevant directory, and then look
>> for .pth or some such approach).
>
> First, I think it's up to the specific loader mechanism whether
> PEP 382 should be supported at all. It should be possible to implement
> it if desired, but if it's not feasible (e.g. for URL loaders), pth
> files just may not get considered. The loader may well provide a
> different mechanism to support namespace packages.
>
>> Are we worried about possible
>>
>> performance implications of this search?
>
> For the specific case of zip files, I'm not. I don't think performance will
> suffer at all.
>
>> And then the search for the __init__.py begins on the newly modified
>> __path__, which I assume ends with the first __init__ found on
>> __path__, but if no file is found it's okay and essentially an empty
>> module with just module-specific attributes is used?
>
> Correct.
>
>> In other words,
>> can a .pth file replace an __init__ file in delineating a package?
>
> That's what it means by '''a directory is considered a package if it either
> contains a file named __init__.py, or a file whose name ends with ".pth".'''
>
>> Or
>> is it purely additive? I assume the latter for compatibility reasons,
>> but the PEP says "a directory is considered a package if it **either**
>> contains a file named __init__.py, **or** a file whose name ends with
>> ".pth"" (emphasis mine).
>
> Why do you think this causes an incompatibility?

It's just if a project has no __init__ older finders won't process it,
that's all. But it looks like they are going to have to change
somewhat anyway so that's not an issue.

>
>> Otherwise I assume that the search will be
>> done simply with ``os.path.isdir(os.path.join(sys_path_entry,
>> top_level_package_name)`` and all existing paths will be added to
>> __path__. Will they come before or after the directory where the *.pth
>> was found? And will any subsequent *.pth files found in other
>> directories also be executed?
>
> I may misremember, but from reading the text, it seems to say "no".
> It should work like the current pth mechanism (plus *, minus import).
>
>> As for how "*" works, is this limited to top-level packages, or will
>> sub-packages participate as well? I assume the former, but it is not
>> directly stated in the PEP.
>
> And indeed, the latter is intended. You should be able to create namespace
> packages on all levels.
>
>> If the latter, is a dotted package name
>> changed to ``os.sep.join(sy_path_entry, package_name.replace('".",
>> os.sep)``?
>
> No. Instead, the parent package's __path__ is being searched for
> directories; sys.path is not considered anymore. I have fixed the text.
>
>> For sys.path_hooks, I am assuming import will simply skip over passing
>> that as it is a marker that __path__ represents a namsepace package
>> and not in any way functional. Although with sys.namespace_packages,
>> is leaving the "*" in __path__ truly necessary?
>
> It would help with efficiency, no?

Not sure how having "*" in __path__ helps with efficiency. What are
you thinking it will help with specifically?

>
>> For the search of paths to use to extend, are we limiting ourselves to
>> actual file system entries on sys.path (as pkgutil does), or do we
>> want to support other storage back-ends? To do the latter I would
>> suggest having a successful path discovery be when a finder can be
>> created for the hypothetical directory from sys.path_hooks.
>
> Again: PEP 302 isn't really considered yet. Proposals are welcome.
>
>> The PEP (seems to) ask finders to look for a .pth file(s), calculate
>> __path__, and then get a loader for the __init__. You could have
>> finders grow a find_namespace method which returns the contents of the
>> requisite .pth file(s).
>
> I must be misunderstanding the concept of finders. Why is it that it would
> be their function to process the pth files, and not the function of the
> loader?

I'm thinking from the perspective of finding an __init__ module that
exists somewhere else than where the .pth file was discovered. Someone
has to find the .pth files and provide their contents. Someone else
needs to find the __init__ module for the package (if it exists). Then
someone needs to load a namespace package, potentially from the
__init__ module. It's that second step -- find the __init__ module --
that makes me think the finder is more involved. It doesn't have to be
by definition, but seeing the word "find" just makes me think
"finder".

From brett at python.org  Mon May 31 22:19:11 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 31 May 2010 13:19:11 -0700
Subject: [Python-Dev] Implementing PEP 382, Namespace Packages
In-Reply-To: <20100531050328.40B073A402D@sparrow.telecommunity.com>
References: <20100530074041.2279D3A405F@sparrow.telecommunity.com> 
	<AANLkTinjHXpfLtmwpxouqdqJixQSXKzcYVUi0py1a4hT@mail.gmail.com> 
	<20100531050328.40B073A402D@sparrow.telecommunity.com>
Message-ID: <AANLkTimGzZ34Gu6cc3-_aSKLtOrqKyXgluiGbhn7vHJm@mail.gmail.com>

On Sun, May 30, 2010 at 22:03, P.J. Eby <pje at telecommunity.com> wrote:
> At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote:
>>
>> On Sun, May 30, 2010 at 00:40, P.J. Eby <pje at telecommunity.com> wrote:
>> >
>> > Which would completely break one of the major use cases of the PEP,
>> > which is
>> > precisely to ensure that you can install two pieces of code to the same
>> > namespace without either one overwriting the other's files.
>>
>> The PEP says the goal is to span packages across directories.
>
> The goal of namespace packages is to allow separately-distributed pieces of
> code to live in the same package namespace. ?That this is sometimes achieved
> by installing them to different paths is an implementation detail.
>
> In the case of e.g. Linux distributions and other system packaging
> scenarios, the code will all be installed to the *same* directory -- so
> there cannot be any filename collisions among the separately-distributed
> modules. ?That's why we want to get rid of the need for an __init__.py to
> mark the directory as a package: it's a collision point for system package
> management tools.
>
>
>> > pkgutil doesn't have such a limitation, except in the case extend_path,
>> > and
>> > that limitation is one that PEP 382 intends to remove.
>>
>> It's because pkgutil.extend_path has that limitation I am asking as
>> that's what the PEP refers to. If the PEP wants to remove the
>> limitation it should clearly state how it is going to do that.
>
> I'm flexible on it either way. ?The only other importer I know of that does
> anything else is one that actually allows (unsafely) importing from URLs.
>
> If we allow for other things, then we need to extend the PEP 302 protocol to
> have a way to ask an importer for a subpath string.
>
>
>
>> As for adding to the PEP 302 protocols, it's a question of how much we
>> want importer implementors to have control over this versus us. I
>> personally would rather keep any protocol extensions simple and have
>> import handle as many of the details as possible.
>
> I lean the other way a bit, in that the more of the importer internals you
> expose, the harder you make it for an importer to be anything other than a
> mere virtual file system. ?(As it is, I think there is too much "file-ness"
> coupling in the protocol already, what with file extensions and the like.)
>

Yes, there is a VERY strong file path connection in loaders and they
have essentially become simple virtual file systems. But that does
make implementation of alternative back-ends easier which seems to be
the common case. Plus pre-existing code already mutates __path__,
__file__, etc. as if they are file paths using os.path.join. And imp's
new source_from_cache and cache_from_source are not weakening the tie
to file paths either.

But as long as whatever mechanism gets exposed allows people to work
from a module name that will be enough. The path connection is not
required as load_module is the end-all-be-all method. If we have a
similar API added for .pth files that works off of module names then
those loaders that don't want to work from file paths don't have to.