From cool-rr at cool-rr.com  Sat Jan  9 13:31:18 2010
From: cool-rr at cool-rr.com (cool-RR)
Date: Sat, 9 Jan 2010 14:31:18 +0200
Subject: [Python-ideas] Having a messageless exception show its docstring
Message-ID: <a422b72d1001090431n7d2c9a47kd2277374f9126460@mail.gmail.com>

I propose that when raising an exception without providing a message, the
first line of the docstring will be shown. (I mean shown in the traceback
where the message would normally be.

Example session:

####################################

>>> class MyExcpetion(Exception):
'''Trying to fit a square piece into a round hole.'''

>>> raise MyExcpetion('Problem bla bla')

Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    raise MyExcpetion('Problem bla bla')
MyExcpetion: Problem bla bla

>>> raise MyExcpetion()

Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    raise MyExcpetion()
MyExcpetion: Trying to fit a square piece into a round hole.

####################################

This will be useful for me because I often raise exceptions and have nothing
to say besides repeating the exception's docstring, and I have many places
in the code where I'd want to conditionally raise it. I'm faced with two
choices: Type the exception's docstring in every place, which is lame, and
breaks DRY. Or just raise the exception without any message, which may
potentially confuse users.

This is the motivation for this suggestion, so I could be succinct without
confusing users.

What do you think?

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100109/fd5bb147/attachment.html>

From fuzzyman at gmail.com  Sat Jan  9 13:35:10 2010
From: fuzzyman at gmail.com (Michael Foord)
Date: Sat, 9 Jan 2010 12:35:10 +0000
Subject: [Python-ideas] Having a messageless exception show its docstring
In-Reply-To: <a422b72d1001090431n7d2c9a47kd2277374f9126460@mail.gmail.com>
References: <a422b72d1001090431n7d2c9a47kd2277374f9126460@mail.gmail.com>
Message-ID: <6f4025011001090435t66cf7144r3827c36fd6f0013a@mail.gmail.com>

2010/1/9 cool-RR <cool-rr at cool-rr.com>

> I propose that when raising an exception without providing a message, the
> first line of the docstring will be shown. (I mean shown in the traceback
> where the message would normally be.
>
> Example session:
>
> ####################################
>
> >>> class MyExcpetion(Exception):
> '''Trying to fit a square piece into a round hole.'''
>
>
class MyException(Exception):
     "Some docstring"
    def __init__(self, msg=None):
         if msg == None:
             msg = self.__doc__
         Exception.__init__(self, msg)

All the best,

Michael



> >>> raise MyExcpetion('Problem bla bla')
>
> Traceback (most recent call last):
>   File "<pyshell#3>", line 1, in <module>
>     raise MyExcpetion('Problem bla bla')
> MyExcpetion: Problem bla bla
>
> >>> raise MyExcpetion()
>
> Traceback (most recent call last):
>   File "<pyshell#4>", line 1, in <module>
>     raise MyExcpetion()
> MyExcpetion: Trying to fit a square piece into a round hole.
>
> ####################################
>
> This will be useful for me because I often raise exceptions and have
> nothing to say besides repeating the exception's docstring, and I have many
> places in the code where I'd want to conditionally raise it. I'm faced with
> two choices: Type the exception's docstring in every place, which is lame,
> and breaks DRY. Or just raise the exception without any message, which may
> potentially confuse users.
>
> This is the motivation for this suggestion, so I could be succinct without
> confusing users.
>
> What do you think?
>
> Ram.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


-- 
http://www.ironpythoninaction.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100109/a1fcd0be/attachment.html>

From cool-rr at cool-rr.com  Tue Jan 19 21:29:18 2010
From: cool-rr at cool-rr.com (cool-RR)
Date: Tue, 19 Jan 2010 22:29:18 +0200
Subject: [Python-ideas] `__iter__` for queues?
Message-ID: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>

Hello,

Is there a reason that queues don't have an `__iter__` method? I mean both
`Queue.Queue` and `multiprocessing.Queue`.

I had to write up my own "iterate on queue" function for use in my project.
Do you think that it should be a built-in method on the Queue class?

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100119/14f9dd0e/attachment.html>

From simon at brunningonline.net  Tue Jan 19 22:10:58 2010
From: simon at brunningonline.net (Simon Brunning)
Date: Tue, 19 Jan 2010 21:10:58 +0000
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
Message-ID: <8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>

2010/1/19 cool-RR <cool-rr at cool-rr.com>:
> Is there a reason that queues don't have an `__iter__` method? I mean both
> `Queue.Queue` and `multiprocessing.Queue`.

Could it be made threadsafe?

-- 
Cheers,
Simon B.


From cool-rr at cool-rr.com  Tue Jan 19 22:20:46 2010
From: cool-rr at cool-rr.com (cool-RR)
Date: Tue, 19 Jan 2010 23:20:46 +0200
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
Message-ID: <a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>

On Tue, Jan 19, 2010 at 11:10 PM, Simon Brunning
<simon at brunningonline.net>wrote:

> 2010/1/19 cool-RR <cool-rr at cool-rr.com>:
> > Is there a reason that queues don't have an `__iter__` method? I mean
> both
> > `Queue.Queue` and `multiprocessing.Queue`.
>
> Could it be made threadsafe?
>
> --
> Cheers,
> Simon B.
>

For me, iterating on the queue means just calling `get` repeatedly until
it's empty. Now that I think about it, maybe this is not the most obvious
meaning? I'm not sure now.

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100119/3bda2b16/attachment.html>

From g.brandl at gmx.net  Tue Jan 19 23:47:42 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 19 Jan 2010 22:47:42 +0000
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
Message-ID: <hj5cqa$q8f$1@ger.gmane.org>

Am 19.01.2010 21:20, schrieb cool-RR:
> 
> 
> On Tue, Jan 19, 2010 at 11:10 PM, Simon Brunning
> <simon at brunningonline.net
> <mailto:simon at brunningonline.net>> wrote:
> 
>     2010/1/19 cool-RR <cool-rr at cool-rr.com
>     <mailto:cool-rr at cool-rr.com>>:
>     > Is there a reason that queues don't have an `__iter__` method? I
>     mean both
>     > `Queue.Queue` and `multiprocessing.Queue`.
> 
>     Could it be made threadsafe?
> 
>     --
>     Cheers,
>     Simon B.
> 
> 
> For me, iterating on the queue means just calling `get` repeatedly until
> it's empty. Now that I think about it, maybe this is not the most
> obvious meaning? I'm not sure now.

Your obvious queue iterator would call get(block=False) and stop on Empty.
The other obvious meaning is be to call get(block=True) forever.  IMO they
are both too "obvious" to make a call -- an explicit while loop is better.

Georg



From python at mrabarnett.plus.com  Wed Jan 20 00:01:31 2010
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 19 Jan 2010 23:01:31 +0000
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <hj5cqa$q8f$1@ger.gmane.org>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
	<hj5cqa$q8f$1@ger.gmane.org>
Message-ID: <4B5639CB.1040908@mrabarnett.plus.com>

Georg Brandl wrote:
> Am 19.01.2010 21:20, schrieb cool-RR:
>>
>> On Tue, Jan 19, 2010 at 11:10 PM, Simon Brunning
>> <simon at brunningonline.net
>> <mailto:simon at brunningonline.net>> wrote:
>>
>>     2010/1/19 cool-RR <cool-rr at cool-rr.com
>>     <mailto:cool-rr at cool-rr.com>>:
>>     > Is there a reason that queues don't have an `__iter__` method? I
>>     mean both
>>     > `Queue.Queue` and `multiprocessing.Queue`.
>>
>>     Could it be made threadsafe?
>>
>>     --
>>     Cheers,
>>     Simon B.
>>
>>
>> For me, iterating on the queue means just calling `get` repeatedly until
>> it's empty. Now that I think about it, maybe this is not the most
>> obvious meaning? I'm not sure now.
> 
> Your obvious queue iterator would call get(block=False) and stop on Empty.
> The other obvious meaning is be to call get(block=True) forever.  IMO they
> are both too "obvious" to make a call -- an explicit while loop is better.
> 
To me the 'obvious' meaning is to call get(block=True) and have it raise
Empty (actually, StopIteration) when the queue is empty and the 'sender'
has somehow signalled that no more items will be put into the queue
(q.finished()?). This would also eliminate the need for a sentinel!


From cs at zip.com.au  Wed Jan 20 00:09:38 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Wed, 20 Jan 2010 10:09:38 +1100
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <4B5639CB.1040908@mrabarnett.plus.com>
References: <4B5639CB.1040908@mrabarnett.plus.com>
Message-ID: <20100119230938.GA14779@cskk.homeip.net>

On 19Jan2010 23:01, MRAB <python at mrabarnett.plus.com> wrote:
| Georg Brandl wrote:
| >Am 19.01.2010 21:20, schrieb cool-RR:
| >>On Tue, Jan 19, 2010 at 11:10 PM, Simon Brunning <simon at brunningonline.net>:
| >>    2010/1/19 cool-RR <cool-rr at cool-rr.com>:
| >>    > Is there a reason that queues don't have an `__iter__` method? I
| >>    > mean both `Queue.Queue` and `multiprocessing.Queue`.
| >>
| >>    Could it be made threadsafe?
| >>
| >>For me, iterating on the queue means just calling `get` repeatedly until
| >>it's empty. Now that I think about it, maybe this is not the most
| >>obvious meaning? I'm not sure now.
|
| >Your obvious queue iterator would call get(block=False) and stop on Empty.
| >The other obvious meaning is be to call get(block=True) forever.  IMO they
| >are both too "obvious" to make a call -- an explicit while loop is better.
| >
| To me the 'obvious' meaning is to call get(block=True) and have it raise
| Empty (actually, StopIteration) when the queue is empty and the 'sender'
| has somehow signalled that no more items will be put into the queue
| (q.finished()?). This would also eliminate the need for a sentinel!

Personally, I have long had an IterableQueue subclass, but it uses a
tunable sentinel (None by default).

But it adds a .close() method, and doesn't iterate until empty, it
iterates until closed. This is because I want to write a handler like
so:

  for item in Q:
    ...do stuff...

and have it block if the queue is empty.

So clearly there are two reasonable approaches to the end-of-iteration
idea; extending Queue to do iteration probably would want to choose one.

So maybe two helper iteration methods might be the go: it's easy enough
to write a generator to iterate-until-empty or iterate-until-sentinel.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

... It beeped and said "Countdown initiated." Is that bad?


From python at mrabarnett.plus.com  Wed Jan 20 00:52:42 2010
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 19 Jan 2010 23:52:42 +0000
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <20100119230938.GA14779@cskk.homeip.net>
References: <4B5639CB.1040908@mrabarnett.plus.com>
	<20100119230938.GA14779@cskk.homeip.net>
Message-ID: <4B5645CA.5080905@mrabarnett.plus.com>

Cameron Simpson wrote:
> On 19Jan2010 23:01, MRAB <python at mrabarnett.plus.com> wrote:
> | Georg Brandl wrote:
> | >Am 19.01.2010 21:20, schrieb cool-RR:
> | >>On Tue, Jan 19, 2010 at 11:10 PM, Simon Brunning <simon at brunningonline.net>:
> | >>    2010/1/19 cool-RR <cool-rr at cool-rr.com>:
> | >>    > Is there a reason that queues don't have an `__iter__` method? I
> | >>    > mean both `Queue.Queue` and `multiprocessing.Queue`.
> | >>
> | >>    Could it be made threadsafe?
> | >>
> | >>For me, iterating on the queue means just calling `get` repeatedly until
> | >>it's empty. Now that I think about it, maybe this is not the most
> | >>obvious meaning? I'm not sure now.
> |
> | >Your obvious queue iterator would call get(block=False) and stop on Empty.
> | >The other obvious meaning is be to call get(block=True) forever.  IMO they
> | >are both too "obvious" to make a call -- an explicit while loop is better.
> | >
> | To me the 'obvious' meaning is to call get(block=True) and have it raise
> | Empty (actually, StopIteration) when the queue is empty and the 'sender'
> | has somehow signalled that no more items will be put into the queue
> | (q.finished()?). This would also eliminate the need for a sentinel!
> 
> Personally, I have long had an IterableQueue subclass, but it uses a
> tunable sentinel (None by default).
> 
> But it adds a .close() method, and doesn't iterate until empty, it
> iterates until closed. This is because I want to write a handler like
> so:
> 
>   for item in Q:
>     ...do stuff...
> 
> and have it block if the queue is empty.
> 
> So clearly there are two reasonable approaches to the end-of-iteration
> idea; extending Queue to do iteration probably would want to choose one.
> 
> So maybe two helper iteration methods might be the go: it's easy enough
> to write a generator to iterate-until-empty or iterate-until-sentinel.
> 
Ah, yes, .close() would be better. :-)

As for iterate-until-empty or iterate-until-sentinel, perhaps only a
keyword argument is needed when the queue is created.

Another possible feature (although this might be going too far) is a
keyword argument for the number of 'producers' (default=1). Each can
close the queue when it's finished, and StopIteration will be raised
only when all have closed and the queue is empty.


From raymond.hettinger at gmail.com  Wed Jan 20 01:08:29 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 19 Jan 2010 16:08:29 -0800
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
Message-ID: <07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>


On Jan 19, 2010, at 1:10 PM, Simon Brunning wrote:

> 2010/1/19 cool-RR <cool-rr at cool-rr.com>:
>> Is there a reason that queues don't have an `__iter__` method? I mean both
>> `Queue.Queue` and `multiprocessing.Queue`.
> 
> Could it be made threadsafe?

To me, this is exactly the right line of questioning.

Is there a use case for iterating through a queue
that is being mutated by consumers and producers?
If so, should the consumers and producers be locked
out during iteration or should the iterator fail if it detects
a mutation (like we currently do for dictionaries that
change during iteration).

The semantics should be dictated by use cases.
When someone writes "for t in q: action(q)" what
is the most useful thing to happen when another
thread does a get or put?  

My personal opinion is that queues are better-off without
an iterator.  They serve mainly to buffer consumers and
producers and iteration does fit in well.  In other words,
adding __iter__ to queues in threaded environment
may do more harm than good. 


Raymond




From cool-rr at cool-rr.com  Wed Jan 20 01:24:47 2010
From: cool-rr at cool-rr.com (cool-RR)
Date: Wed, 20 Jan 2010 02:24:47 +0200
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>
Message-ID: <a422b72d1001191624g2c015a0ate63e3587aeb5decc@mail.gmail.com>

On Wed, Jan 20, 2010 at 2:08 AM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> > Could it be made threadsafe?
>
> To me, this is exactly the right line of questioning.
>
> Is there a use case for iterating through a queue
> that is being mutated by consumers and producers?
> If so, should the consumers and producers be locked
> out during iteration or should the iterator fail if it detects
> a mutation (like we currently do for dictionaries that
> change during iteration).
>

In my program, I have one thread putting data into a queue, and the main
thread periodically iterates over the whole queue to integrate all the data
into a central data structure.

The general use case is when you want a thread to take items from a queue
and process them until the queue's empty. Like:

for work_item in queue:
    process(work_item)



> The semantics should be dictated by use cases.
> When someone writes "for t in q: action(q)" what
> is the most useful thing to happen when another
> thread does a get or put?
>

(I think you did a typo in your code sample, I think you meant `action(t)`)

What I'm suggesting is that the iteration won't care about what other
threads do with the queue. It'll just `get` until it's empty.



> My personal opinion is that queues are better-off without
> an iterator.  They serve mainly to buffer consumers and
> producers and iteration does fit in well.  In other words,
> adding __iter__ to queues in threaded environment
> may do more harm than good.
>
> Raymond
>

Okay. I don't have much experience in this, so unless someone else will find
this suggestion useful, I'll stick with my custom function.

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100120/17adbc2d/attachment.html>

From python at mrabarnett.plus.com  Wed Jan 20 01:41:49 2010
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 20 Jan 2010 00:41:49 +0000
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <a422b72d1001191624g2c015a0ate63e3587aeb5decc@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>	<07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>
	<a422b72d1001191624g2c015a0ate63e3587aeb5decc@mail.gmail.com>
Message-ID: <4B56514D.5000405@mrabarnett.plus.com>

cool-RR wrote:
> On Wed, Jan 20, 2010 at 2:08 AM, Raymond Hettinger 
> <raymond.hettinger at gmail.com <mailto:raymond.hettinger at gmail.com>> wrote:
> 
> 
>      > Could it be made threadsafe?
> 
>     To me, this is exactly the right line of questioning.
> 
>     Is there a use case for iterating through a queue
>     that is being mutated by consumers and producers?
>     If so, should the consumers and producers be locked
>     out during iteration or should the iterator fail if it detects
>     a mutation (like we currently do for dictionaries that
>     change during iteration).
> 
> 
> In my program, I have one thread putting data into a queue, and the main 
> thread periodically iterates over the whole queue to integrate all the 
> data into a central data structure.
> 
> The general use case is when you want a thread to take items from a 
> queue and process them until the queue's empty. Like:
> 
> for work_item in queue:
>     process(work_item)
> 
>  
> 
>     The semantics should be dictated by use cases.
>     When someone writes "for t in q: action(q)" what
>     is the most useful thing to happen when another
>     thread does a get or put?
> 
> 
> (I think you did a typo in your code sample, I think you meant `action(t)`)
> 
> What I'm suggesting is that the iteration won't care about what other 
> threads do with the queue. It'll just `get` until it's empty.
> 
I agree. I don't see why __iter__ wouldn't work when .get does.

'Empty' could mean either a non-blocking .get and stopping when there
are no more items in the queue, or a blocking .get and stopping when 
there are no more items in the queue _and_ the queue has been closed by
the producer(s).
>  
> 
>     My personal opinion is that queues are better-off without
>     an iterator.  They serve mainly to buffer consumers and
>     producers and iteration does fit in well.  In other words,
>     adding __iter__ to queues in threaded environment
>     may do more harm than good.
> 
>     Raymond
> 
> 
> Okay. I don't have much experience in this, so unless someone else will 
> find this suggestion useful, I'll stick with my custom function.
> 


From jnoller at gmail.com  Wed Jan 20 02:11:13 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Tue, 19 Jan 2010 20:11:13 -0500
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>
Message-ID: <4222a8491001191711w36c06863kef788ce6dcb054d4@mail.gmail.com>

On Tue, Jan 19, 2010 at 7:08 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Jan 19, 2010, at 1:10 PM, Simon Brunning wrote:
>
>> 2010/1/19 cool-RR <cool-rr at cool-rr.com>:
>>> Is there a reason that queues don't have an `__iter__` method? I mean both
>>> `Queue.Queue` and `multiprocessing.Queue`.
>>
>> Could it be made threadsafe?
>
> To me, this is exactly the right line of questioning.
>
> Is there a use case for iterating through a queue
> that is being mutated by consumers and producers?
> If so, should the consumers and producers be locked
> out during iteration or should the iterator fail if it detects
> a mutation (like we currently do for dictionaries that
> change during iteration).
>
> The semantics should be dictated by use cases.
> When someone writes "for t in q: action(q)" what
> is the most useful thing to happen when another
> thread does a get or put?
>
> My personal opinion is that queues are better-off without
> an iterator. ?They serve mainly to buffer consumers and
> producers and iteration does fit in well. ?In other words,
> adding __iter__ to queues in threaded environment
> may do more harm than good.
>
>
> Raymond

+1


From tjreedy at udel.edu  Wed Jan 20 03:06:16 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 19 Jan 2010 21:06:16 -0500
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
Message-ID: <hj5oem$nt9$1@ger.gmane.org>

On 1/19/2010 4:20 PM, cool-RR wrote:
>
>
> On Tue, Jan 19, 2010 at 11:10 PM, Simon Brunning
> <simon at brunningonline.net
> <mailto:simon at brunningonline.net>> wrote:
>
>     2010/1/19 cool-RR <cool-rr at cool-rr.com
>     <mailto:cool-rr at cool-rr.com>>:
>      > Is there a reason that queues don't have an `__iter__` method? I
>     mean both
>      > `Queue.Queue` and `multiprocessing.Queue`.
>
>     Could it be made threadsafe?
>
>     --
>     Cheers,
>     Simon B.
>
>
> For me, iterating on the queue means just calling `get` repeatedly until
> it's empty.

Explicit .get()s release the queue between getting each item so other 
code can also put and get. A mutable collection iterator may or may not 
lock the collection or monitor mutations. Dict iteration (and, I 
presume, set iteration, but I see nothing in the doc) monitors mutation. 
List iteration does not. While modifying a list during iteration can 
have uses, it is also leads to bugs and newby confusion. On the 
otherhand, one can write explicitly iterate dicts with

while d:
   k,v = dict.getitem()
   <go ahead and modify dict>

and the same with set.pop and have no problem.

So iterating with an iterator is not quite the same as repeated one-item 
fetches.

Terry Jan Reedy



From raymond.hettinger at gmail.com  Wed Jan 20 03:54:01 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 19 Jan 2010 18:54:01 -0800
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <hj5oem$nt9$1@ger.gmane.org>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
	<hj5oem$nt9$1@ger.gmane.org>
Message-ID: <5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com>


On Jan 19, 2010, at 6:06 PM, Terry Reedy wrote:

> On 1/19/2010 4:20 PM, cool-RR wrote:
>> 
>> For me, iterating on the queue means just calling `get` repeatedly until
>> it's empty.
> 
  
If that is all you're looking for, then just do that.  Write a short generator to dump the queue.  You can do that without blocking your producer thread.

The problem with proposing an change to the API for Queue.Queue and MultiProcessing.Queue is that it presents non-obvious pitfalls for other common use cases.  It is better to leave it out of the API and let users explicitly craft solutions that fit their problem (perhaps using two queues:  Producer --->  DatabaseUpdater --> Consumer or some solution involving locking).  For many uses of Queue, it doesn't make sense to block a producer for the time it takes to iterate, nor does it make sense to have a non-blocking iterator (leading to a race condition and inconsistent results).

It is better to use the API as designed than to propose a new method than may be unsuitable for many use cases.


> 
> So iterating with an iterator is not quite the same as repeated one-item fetches.
> 


Right.  The two ideas should not be conflated.  A Queue object is all about buffering producers and consumers running in different threads.  Anything API that is at odds with that notion is probably not a good idea (i.e. iterating over a dynamically updating queue buffer).


Raymond



From python at mrabarnett.plus.com  Wed Jan 20 04:47:15 2010
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 20 Jan 2010 03:47:15 +0000
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>	<hj5oem$nt9$1@ger.gmane.org>
	<5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com>
Message-ID: <4B567CC3.9020408@mrabarnett.plus.com>

Raymond Hettinger wrote:
> On Jan 19, 2010, at 6:06 PM, Terry Reedy wrote:
> 
>> On 1/19/2010 4:20 PM, cool-RR wrote:
>>> For me, iterating on the queue means just calling `get`
>>> repeatedly until it's empty.
> 
> If that is all you're looking for, then just do that.  Write a short
> generator to dump the queue.  You can do that without blocking your
> producer thread.
> 
> The problem with proposing an change to the API for Queue.Queue and
> MultiProcessing.Queue is that it presents non-obvious pitfalls for
> other common use cases.  It is better to leave it out of the API and
> let users explicitly craft solutions that fit their problem (perhaps
> using two queues:  Producer --->  DatabaseUpdater --> Consumer or
> some solution involving locking).  For many uses of Queue, it doesn't
> make sense to block a producer for the time it takes to iterate, nor
> does it make sense to have a non-blocking iterator (leading to a race
> condition and inconsistent results).
> 
> It is better to use the API as designed than to propose a new method
> than may be unsuitable for many use cases.
> 
> 
>> So iterating with an iterator is not quite the same as repeated
>> one-item fetches.
>> 
> 
> 
> Right.  The two ideas should not be conflated.  A Queue object is all
> about buffering producers and consumers running in different threads.
> Anything API that is at odds with that notion is probably not a good
> idea (i.e. iterating over a dynamically updating queue buffer).
> 
I was thinking of methods like these, for example:

def close(self):
     self.put(sentinel)
def __iter__(self):
     while True:
         item = self.get()
         if item is sentinel:
             break
         yield item
     raise StopIteration

How would they not be threadsafe, or block a producer?



From anh.hai.trinh at gmail.com  Wed Jan 20 06:24:06 2010
From: anh.hai.trinh at gmail.com (Anh Hai Trinh)
Date: Wed, 20 Jan 2010 12:24:06 +0700
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <4B567CC3.9020408@mrabarnett.plus.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com> 
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com> 
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com> 
	<hj5oem$nt9$1@ger.gmane.org>
	<5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com> 
	<4B567CC3.9020408@mrabarnett.plus.com>
Message-ID: <f23bc8651001192124o663e5383q511fd813bc14060c@mail.gmail.com>

> I was thinking of methods like these, for example:
>
> def close(self):
> ? ?self.put(sentinel)
> def __iter__(self):
> ? ?while True:
> ? ? ? ?item = self.get()
> ? ? ? ?if item is sentinel:
> ? ? ? ? ? ?break
> ? ? ? ?yield item
> ? ?raise StopIteration

That would not be thread-safe because your sentinel is only seen once
by a single lucky consumer, other listeners who called iter() will
still execute self.get() and block forever.

The solution I found is to put the sentinel back in the queue so that
other consumers will see it (this use StopIteration as the sentinel):

def _iterqueue(queue):
  # Turn a either a threading.Queue or a multiprocessing.queues.SimpleQueue
  # into an thread-safe iterator which will exhaust when StopIteration is
  # put into it.
  while 1:
    item = queue.get()
    if item is StopIteration:
      # Re-broadcast, in case there is another listener blocking on
      # queue.get().  That listener will receive StopIteration and
      # re-broadcast to the next one in line.
      queue.put(StopIteration)
      break
    else:
      yield item

I agree, however, that this should be left out of the stdlib API.

-- 
// aht
http://blog.onideas.ws


From ncoghlan at gmail.com  Wed Jan 20 12:49:19 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 20 Jan 2010 21:49:19 +1000
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <4B56514D.5000405@mrabarnett.plus.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>	<07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>	<a422b72d1001191624g2c015a0ate63e3587aeb5decc@mail.gmail.com>
	<4B56514D.5000405@mrabarnett.plus.com>
Message-ID: <4B56EDBF.7090106@gmail.com>

MRAB wrote:
> 'Empty' could mean either a non-blocking .get and stopping when there
> are no more items in the queue, or a blocking .get and stopping when
> there are no more items in the queue _and_ the queue has been closed by
> the producer(s).

When there are multiple valid approaches to iterating over a queue, and
none of them are more obviously fundamental than any of the others, does
it make sense for us to bless one by implementing it as the __iter__ method?

Now, an iteration API (separate from __iter__) that helped developers
easily avoid some of the pitfalls of rolling your own iteration
techniques (such as the reinsertion of sentinel values) may provide some
value, but I suspect it would be difficult to produce an effective API
even with function parameters to tweak the behaviour.

I will note that the check-and-reinsert example given elsewhere in this
thread contains a race condition in the multiple producer use case or in
cases where a single producer may place additional items in the queue
after the shutdown sentinel:

def _iterqueue(queue):
  while 1:
    item = queue.get()
    # Queue is not locked here, producers may insert more items
    # and other consumers may process items that were in the queue
    # after the sentinel
    if item is StopIteration:
      # We put StopIteration into the queue again, but it may not be
      # at the front if a producer inserted something after the original
      # insertion of the sentinel value
      queue.put(StopIteration)
      break
    else:
      yield item

This approach also doesn't work for sentinel values that are intended to
mean "stop processing the queue, go do something else, then come back
and start processing this queue again" since the queue can no longer be
processed at all after the sentinel value has been inserted once.

Cheers,
Nick.

P.S. An easy way to iterate over a queue in the simple "single consumer
with sentinel value" use case:

  for obj in iter(q.get, sentinel):
    pass

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------


From cool-rr at cool-rr.com  Wed Jan 20 12:59:40 2010
From: cool-rr at cool-rr.com (cool-RR)
Date: Wed, 20 Jan 2010 13:59:40 +0200
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <f23bc8651001192124o663e5383q511fd813bc14060c@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com> 
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com> 
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com> 
	<hj5oem$nt9$1@ger.gmane.org>
	<5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com> 
	<4B567CC3.9020408@mrabarnett.plus.com>
	<f23bc8651001192124o663e5383q511fd813bc14060c@mail.gmail.com>
Message-ID: <a422b72d1001200359l7b559185v2633503f89f36543@mail.gmail.com>

On Wed, Jan 20, 2010 at 7:24 AM, Anh Hai Trinh <anh.hai.trinh at gmail.com>wrote:

> [...]
>
> I agree, however, that this should be left out of the stdlib API.
>
> --
> // aht


Okay, but what about having an object like `Queue.dump` (or some other name
instead of `dump`), which can be iterated? This will be similar to
`dict.keys` and `dict.values`.

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100120/c43122b3/attachment.html>

From anh.hai.trinh at gmail.com  Wed Jan 20 14:54:52 2010
From: anh.hai.trinh at gmail.com (Anh Hai Trinh)
Date: Wed, 20 Jan 2010 20:54:52 +0700
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <4B56EDBF.7090106@gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com> 
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com> 
	<07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>
	<a422b72d1001191624g2c015a0ate63e3587aeb5decc@mail.gmail.com> 
	<4B56514D.5000405@mrabarnett.plus.com> <4B56EDBF.7090106@gmail.com>
Message-ID: <f23bc8651001200554m52b7cd7boe6ee6a28b85cf84a@mail.gmail.com>

> I will note that the check-and-reinsert example given elsewhere in this
> thread contains a race condition in the multiple producer use case or in
> cases where a single producer may place additional items in the queue
> after the shutdown sentinel:
>
> def _iterqueue(queue):
> ?while 1:
> ? ?item = queue.get()
> ? ?# Queue is not locked here, producers may insert more items
> ? ?# and other consumers may process items that were in the queue
> ? ?# after the sentinel
> ? ?if item is StopIteration:
> ? ? ?# We put StopIteration into the queue again, but it may not be
> ? ? ?# at the front if a producer inserted something after the original
> ? ? ?# insertion of the sentinel value
> ? ? ?queue.put(StopIteration)
> ? ? ?break
> ? ?else:
> ? ? ?yield item

You are quite right. The assumption is that the StopIteration
singleton correctly marks the end of the queue however that is
achieved (e.g. have the producer-spawning thread join() with all
producers, then put StopIteration).

Cheers,

-- 
// aht
http://blog.onideas.ws


From python at mrabarnett.plus.com  Wed Jan 20 18:02:08 2010
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 20 Jan 2010 17:02:08 +0000
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <f23bc8651001200554m52b7cd7boe6ee6a28b85cf84a@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<07FB3744-70AE-4FC4-BA9B-527BF3AABACE@gmail.com>	<a422b72d1001191624g2c015a0ate63e3587aeb5decc@mail.gmail.com>
	<4B56514D.5000405@mrabarnett.plus.com> <4B56EDBF.7090106@gmail.com>
	<f23bc8651001200554m52b7cd7boe6ee6a28b85cf84a@mail.gmail.com>
Message-ID: <4B573710.6000008@mrabarnett.plus.com>

Anh Hai Trinh wrote:
>> I will note that the check-and-reinsert example given elsewhere in this
>> thread contains a race condition in the multiple producer use case or in
>> cases where a single producer may place additional items in the queue
>> after the shutdown sentinel:
>>
>> def _iterqueue(queue):
>>  while 1:
>>    item = queue.get()
>>    # Queue is not locked here, producers may insert more items
>>    # and other consumers may process items that were in the queue
>>    # after the sentinel
>>    if item is StopIteration:
>>      # We put StopIteration into the queue again, but it may not be
>>      # at the front if a producer inserted something after the original
>>      # insertion of the sentinel value
>>      queue.put(StopIteration)
>>      break
>>    else:
>>      yield item
> 
> You are quite right. The assumption is that the StopIteration
> singleton correctly marks the end of the queue however that is
> achieved (e.g. have the producer-spawning thread join() with all
> producers, then put StopIteration).
> 
I realised that my suggestion wasn't threadsafe when there was more than
one consumer some time after I posted. :-(

OK, so how about this:

Add a new method close(), which sets a 'closed' flag in the queue.

Currently, if the queue is empty then a blocking get() will wait and a
non-blocking get() will raise an Empty exception.

In future, if the queue is empty and the 'closed' flag is set then a
get(), whether blocking or non-blocking, will raise a Closed exception.

(The Closed exception is only ever raised if close() is called, so the
addition won't break existing code.)

An additional enhancement for multiple producers is to add a keyword
argument that specifies the number of producers and count the number of
times that close() has been called (each producer would call close()
once). The Closed exception would be raised only if the queue is empty
and the 'closed' count has reached the number of producers.

__iter__ can now be:

def __iter__(self):
     try:
         while True:
             yield self.get()
     except Closed:
         raise StopIteration


From stefan_ml at behnel.de  Wed Jan 20 20:03:58 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 20 Jan 2010 20:03:58 +0100
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <4B5639CB.1040908@mrabarnett.plus.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>	<hj5cqa$q8f$1@ger.gmane.org>
	<4B5639CB.1040908@mrabarnett.plus.com>
Message-ID: <hj7k2t$9tt$1@ger.gmane.org>

MRAB, 20.01.2010 00:01:
> Georg Brandl wrote:
>> Your obvious queue iterator would call get(block=False) and stop on
>> Empty. The other obvious meaning is be to call get(block=True) forever.
>> IMO they are both too "obvious" to make a call -- an explicit while loop
>> is better.

I would have expected the behaviour to be infinite/blocking, so I guess I'm
+1 on the "non-obvious" bit.


> To me the 'obvious' meaning is to call get(block=True) and have it raise
> Empty (actually, StopIteration) when the queue is empty and the 'sender'
> has somehow signalled that no more items will be put into the queue
> (q.finished()?). This would also eliminate the need for a sentinel!

You could always use

    for item in iter(my_queue, THE_SENTINEL):
        ...

I think that's simple enough.

Stefan



From raymond.hettinger at gmail.com  Wed Jan 20 20:36:39 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Jan 2010 11:36:39 -0800
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <a422b72d1001200359l7b559185v2633503f89f36543@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
	<hj5oem$nt9$1@ger.gmane.org>
	<5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com>
	<4B567CC3.9020408@mrabarnett.plus.com>
	<f23bc8651001192124o663e5383q511fd813bc14060c@mail.gmail.com>
	<a422b72d1001200359l7b559185v2633503f89f36543@mail.gmail.com>
Message-ID: <A6D0FA30-3CC1-4DB8-A0FB-D279BACA10FD@gmail.com>


On Jan 20, 2010, at 3:59 AM, cool-RR wrote:

> On Wed, Jan 20, 2010 at 7:24 AM, Anh Hai Trinh <anh.hai.trinh at gmail.com> wrote:
> [...]
> 
> I agree, however, that this should be left out of the stdlib API.
> 
> --
> // aht
> 
> Okay, but what about having an object like `Queue.dump` (or some other name instead of `dump`), which can be iterated? This will be similar to `dict.keys` and `dict.values`.
> 


I think you need to abandon the notion of iteration
and instead focus on your original use case of
(destructively) pulling all of the items out of a queue.

Taking a cue (no pun intended) from the DB-API's
fetchone() and fetchall(), perhaps you're looking for
a getall() method?


    def get_all(self, block=True, timeout=None):
        """Remove and return all items from the queue.

        If optional args 'block' is true and 'timeout' is None (the default),
        block if necessary until an item is available. If 'timeout' is
        a positive number, it blocks at most 'timeout' seconds and raises
        the Empty exception if no item was available within that time.
        Otherwise ('block' is false), return an item if one is immediately
        available, else raise the Empty exception ('timeout' is ignored
        in that case).
        """
        self.not_empty.acquire()
        try:
            if not block:
                if not self._qsize():
                    raise Empty
            elif timeout is None:
                while not self._qsize():
                    self.not_empty.wait()
            elif timeout < 0:
                raise ValueError("'timeout' must be a positive number")
            else:
                endtime = _time() + timeout
                while not self._qsize():
                    remaining = endtime - _time()
                    if remaining <= 0.0:
                        raise Empty
                    self.not_empty.wait(remaining)
            result = []
            while self._qsize():
                result.append(self._get())
            self.not_full.notify()
            return result
        finally:
            self.not_empty.release()

Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100120/c4130bd7/attachment.html>

From cool-rr at cool-rr.com  Wed Jan 20 21:28:45 2010
From: cool-rr at cool-rr.com (cool-RR)
Date: Wed, 20 Jan 2010 22:28:45 +0200
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <A6D0FA30-3CC1-4DB8-A0FB-D279BACA10FD@gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com> 
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com> 
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com> 
	<hj5oem$nt9$1@ger.gmane.org>
	<5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com> 
	<4B567CC3.9020408@mrabarnett.plus.com>
	<f23bc8651001192124o663e5383q511fd813bc14060c@mail.gmail.com> 
	<a422b72d1001200359l7b559185v2633503f89f36543@mail.gmail.com> 
	<A6D0FA30-3CC1-4DB8-A0FB-D279BACA10FD@gmail.com>
Message-ID: <a422b72d1001201228p178f0c9eyd8a7d79f1619396b@mail.gmail.com>

Thanks for the code Raymond. However, what I use is actually simpler:

def iterate(queue, block=False):
    '''Iterate over the items in the queue.'''
    while True:
        try:
            yield queue.get(block=block)
        except Queue.Empty:
            raise StopIteration

Since it's a generator, the flow of the program is (get item) -> (process
it) -> (get item) -> (process it) instead of getting all the items, putting
them in a list, and then beginning to process. (Because by the time we're
done processing, there might be more work in the queue.)

Ram.

On Wed, Jan 20, 2010 at 9:36 PM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> On Jan 20, 2010, at 3:59 AM, cool-RR wrote:
>
> On Wed, Jan 20, 2010 at 7:24 AM, Anh Hai Trinh <anh.hai.trinh at gmail.com>wrote:
>
>> [...]
>>
>> I agree, however, that this should be left out of the stdlib API.
>>
>> --
>> // aht
>
>
> Okay, but what about having an object like `Queue.dump` (or some other name
> instead of `dump`), which can be iterated? This will be similar to
> `dict.keys` and `dict.values`.
>
>
> I think you need to abandon the notion of iteration
> and instead focus on your original use case of
> (destructively) pulling all of the items out of a queue.
>
> Taking a cue (no pun intended) from the DB-API's
> fetchone() and fetchall(), perhaps you're looking for
> a getall() method?
>
>
>     def get_all(self, block=True, timeout=None):
>         """Remove and return all items from the queue.
>
>         If optional args 'block' is true and 'timeout' is None (the
> default),
>         block if necessary until an item is available. If 'timeout' is
>         a positive number, it blocks at most 'timeout' seconds and raises
>         the Empty exception if no item was available within that time.
>         Otherwise ('block' is false), return an item if one is immediately
>         available, else raise the Empty exception ('timeout' is ignored
>         in that case).
>         """
>         self.not_empty.acquire()
>         try:
>             if not block:
>                 if not self._qsize():
>                     raise Empty
>             elif timeout is None:
>                 while not self._qsize():
>                     self.not_empty.wait()
>             elif timeout < 0:
>                 raise ValueError("'timeout' must be a positive number")
>             else:
>                 endtime = _time() + timeout
>                 while not self._qsize():
>                     remaining = endtime - _time()
>                     if remaining <= 0.0:
>                         raise Empty
>                     self.not_empty.wait(remaining)
>             result = []
>             while self._qsize():
>                 result.append(self._get())
>             self.not_full.notify()
>             return result
>         finally:
>             self.not_empty.release()
>
> Raymond
>



-- 
Sincerely,
Ram Rachum
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100120/c5de3dd6/attachment.html>

From george.sakkis at gmail.com  Wed Jan 20 21:56:38 2010
From: george.sakkis at gmail.com (George Sakkis)
Date: Wed, 20 Jan 2010 22:56:38 +0200
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <A6D0FA30-3CC1-4DB8-A0FB-D279BACA10FD@gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
	<hj5oem$nt9$1@ger.gmane.org>
	<5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com>
	<4B567CC3.9020408@mrabarnett.plus.com>
	<f23bc8651001192124o663e5383q511fd813bc14060c@mail.gmail.com>
	<a422b72d1001200359l7b559185v2633503f89f36543@mail.gmail.com>
	<A6D0FA30-3CC1-4DB8-A0FB-D279BACA10FD@gmail.com>
Message-ID: <91ad5bf81001201256m1304053evd65862f2dcd97356@mail.gmail.com>

On Wed, Jan 20, 2010 at 9:36 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Jan 20, 2010, at 3:59 AM, cool-RR wrote:
>
> On Wed, Jan 20, 2010 at 7:24 AM, Anh Hai Trinh <anh.hai.trinh at gmail.com>
> wrote:
>>
>> [...]
>>
>> I agree, however, that this should be left out of the stdlib API.
>>
>> --
>> // aht
>
> Okay, but what about having an object like `Queue.dump` (or some other name
> instead of `dump`), which can be iterated? This will be similar to
> `dict.keys` and `dict.values`.
>
> I think you need to abandon the notion of iteration
> and instead focus on your original use case of
> (destructively) pulling all of the items out of a queue.
> Taking a cue (no pun intended) from the DB-API's
> fetchone() and fetchall(), perhaps you're looking for
> a getall() method?
>
> ?? ?def get_all(self, block=True, timeout=None):
> ?? ? ? ?"""Remove and return all items from the queue.
> ?? ? ? ?If optional args 'block' is true and 'timeout' is None (the
> default),
> ?? ? ? ?block if necessary until an item is available. If 'timeout' is
> ?? ? ? ?a positive number, it blocks at most 'timeout' seconds and raises
> ?? ? ? ?the Empty exception if no item was available within that time.
> ?? ? ? ?Otherwise ('block' is false), return an item if one is immediately
> ?? ? ? ?available, else raise the Empty exception ('timeout' is ignored
> ?? ? ? ?in that case).
> ?? ? ? ?"""
> ?? ? ? ?self.not_empty.acquire()
> ?? ? ? ?try:
> ?? ? ? ? ? ?if not block:
> ?? ? ? ? ? ? ? ?if not self._qsize():
> ?? ? ? ? ? ? ? ? ? ?raise Empty
> ?? ? ? ? ? ?elif timeout is None:
> ?? ? ? ? ? ? ? ?while not self._qsize():
> ?? ? ? ? ? ? ? ? ? ?self.not_empty.wait()
> ?? ? ? ? ? ?elif timeout < 0:
> ?? ? ? ? ? ? ? ?raise ValueError("'timeout' must be a positive number")
> ?? ? ? ? ? ?else:
> ?? ? ? ? ? ? ? ?endtime = _time() + timeout
> ?? ? ? ? ? ? ? ?while not self._qsize():
> ?? ? ? ? ? ? ? ? ? ?remaining = endtime - _time()
> ?? ? ? ? ? ? ? ? ? ?if remaining <= 0.0:
> ?? ? ? ? ? ? ? ? ? ? ? ?raise Empty
> ?? ? ? ? ? ? ? ? ? ?self.not_empty.wait(remaining)
> ?? ? ? ? ? ?result = []
> ?? ? ? ? ? ?while self._qsize():
> ?? ? ? ? ? ? ? ?result.append(self._get())
> ?? ? ? ? ? ?self.not_full.notify()
> ?? ? ? ? ? ?return result
> ?? ? ? ?finally:
> ?? ? ? ? ? ?self.not_empty.release()

It's not obvious to me what "all" should mean, or even if it makes
sense, in the default case (`block=True, timeout=None`) when there are
no items. IIUC, the implementation above makes it effectively
equivalent to get() in this case, i.e. a list with a single-item is
returned. Although that's reasonable, it seems arbitrary. I'd find
more intuitive a method with signature `get_all(self, timeout=0.0)`,
which by default is equivalent to the `block=False` above, and
equivalent to `block=True, timeout=timeout` if timeout>0.

George


From raymond.hettinger at gmail.com  Wed Jan 20 23:48:13 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 20 Jan 2010 14:48:13 -0800
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <a422b72d1001201228p178f0c9eyd8a7d79f1619396b@mail.gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<8c7f10c61001191310y5ee46759h888671ffcfc6f0d2@mail.gmail.com>
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
	<hj5oem$nt9$1@ger.gmane.org>
	<5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com>
	<4B567CC3.9020408@mrabarnett.plus.com>
	<f23bc8651001192124o663e5383q511fd813bc14060c@mail.gmail.com>
	<a422b72d1001200359l7b559185v2633503f89f36543@mail.gmail.com>
	<A6D0FA30-3CC1-4DB8-A0FB-D279BACA10FD@gmail.com>
	<a422b72d1001201228p178f0c9eyd8a7d79f1619396b@mail.gmail.com>
Message-ID: <0FEC2D3B-781A-4533-98B2-BC7699E8C98E@gmail.com>


On Jan 20, 2010, at 12:28 PM, cool-RR wrote:

> Thanks for the code Raymond. However, what I use is actually simpler:
> 
> def iterate(queue, block=False):
>     '''Iterate over the items in the queue.'''
>     while True:
>         try:
>             yield queue.get(block=block)
>         except Queue.Empty:
>             raise StopIteration

Then it sounds like the existing API is more than adequate for your needs.
Dump the above fragment in your utils module and be done with it.


Raymond


From cool-rr at cool-rr.com  Thu Jan 21 00:09:06 2010
From: cool-rr at cool-rr.com (cool-RR)
Date: Thu, 21 Jan 2010 01:09:06 +0200
Subject: [Python-ideas] `__iter__` for queues?
In-Reply-To: <0FEC2D3B-781A-4533-98B2-BC7699E8C98E@gmail.com>
References: <a422b72d1001191229x130ca944x71a8f71873ec5385@mail.gmail.com>
	<a422b72d1001191320r1ada54cbyd07417074adad543@mail.gmail.com>
	<hj5oem$nt9$1@ger.gmane.org>
	<5F9102B2-F01C-4F7E-89DC-AA496A564EC0@gmail.com>
	<4B567CC3.9020408@mrabarnett.plus.com>
	<f23bc8651001192124o663e5383q511fd813bc14060c@mail.gmail.com>
	<a422b72d1001200359l7b559185v2633503f89f36543@mail.gmail.com>
	<A6D0FA30-3CC1-4DB8-A0FB-D279BACA10FD@gmail.com>
	<a422b72d1001201228p178f0c9eyd8a7d79f1619396b@mail.gmail.com>
	<0FEC2D3B-781A-4533-98B2-BC7699E8C98E@gmail.com>
Message-ID: <a422b72d1001201509u5a91dbf0x7b21d4197d7cb963@mail.gmail.com>

Okay Raymond. Thanks for your patience.

Ram.

On Thu, Jan 21, 2010 at 12:48 AM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> On Jan 20, 2010, at 12:28 PM, cool-RR wrote:
>
> > Thanks for the code Raymond. However, what I use is actually simpler:
> >
> > def iterate(queue, block=False):
> >     '''Iterate over the items in the queue.'''
> >     while True:
> >         try:
> >             yield queue.get(block=block)
> >         except Queue.Empty:
> >             raise StopIteration
>
> Then it sounds like the existing API is more than adequate for your needs.
> Dump the above fragment in your utils module and be done with it.
>
>
> Raymond
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100121/2c01bd19/attachment.html>

From markroddy at gmail.com  Thu Jan 21 01:38:29 2010
From: markroddy at gmail.com (Mark Roddy)
Date: Wed, 20 Jan 2010 19:38:29 -0500
Subject: [Python-ideas] Test Class setup and teardown in unittest
Message-ID: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>

Earlier this week on the Testing In Python list, there was a
discussion on how to execute a setup and/or teardown for a single test
class instead of for each test fixture on the class (see the 'setUp
and tearDown behavior' thread).  I have had to deal with situation
myself before, and I am obviously not the only one (since I did not
initiate the thread).  As such I'd like to propose adding a class
level setup and tear down method the unittest TestCase class.

Rationale:
Test cases can at times require the setup of expensive resources.
This is often the case when implementing integration testing.  Having
to perform this setup for each fixture can prohibited for large number
of fixtures and/or for resources that are expensive to setup.  For
example, I have several hundred integration tests that hit a live
database.  If I create the connection object for each fixture, most of
the tests fail due to the maximum connection limit being reached.  As
a work around, I create a connection object once for each test case.
Without such functionality built in, the common idiom runs along these
lines:

class MyTest(TestCase):

    def setUp(self):
        if not self.ClassIsSetup:
            self.setupClass()
            self.ClassIsSetup=True

While this achieves the desired functionality, it is unclear due to
conditional setup code and is also error prone as the same code
segment would need to appear in every TestCase requiring the
functionality.  Having a class wide setup and teardown function that
implementers of test cases can override would make the code/intent
more clear and alleviate the need to implement test case functionality
when the user should be focusing on writing tests.

I emailed Michael Foord about some of his comments in the TIP thread
and to ask if he would be interested in a patch adding this
functionality, and I have included his response below.  I would like
to hear people's comments/suggestions/ideas before I start working on
said patch.

Thanks,
-Mark


Michael Foord's Email:
=======================================
I would certainly be interested in adding this to unittest.

It needs a discussion of the API and the semantics:

* What should the methods be called? setup_class and teardown_class or
setupClass and teardownClass? For consistency with existing methods
the camelCase should probably be used.
* If the setupClass fails how should the error be reported? The
*easiest* way is for the failure to be reported as part of the first
test
* Ditto for teardownClass - again the easiest way is for it to be
reported as a failure in the last test
* If setupClass fails then should all the tests in that class be
skipped? I think yes.

Also details like ensuring that even if just a single test method from
a class is run the setupClass and teardownClass are still run. It
probably needs to go to python-dev or python-ideas for discussion.

All the best,

Michael


From robertc at robertcollins.net  Thu Jan 21 02:37:05 2010
From: robertc at robertcollins.net (Robert Collins)
Date: Thu, 21 Jan 2010 12:37:05 +1100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
Message-ID: <1264037825.4254.12.camel@lifeless-64>

On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
> Earlier this week on the Testing In Python list, there was a
> discussion on how to execute a setup and/or teardown for a single test
> class instead of for each test fixture on the class (see the 'setUp
> and tearDown behavior' thread).  I have had to deal with situation
> myself before, and I am obviously not the only one (since I did not
> initiate the thread).  As such I'd like to propose adding a class
> level setup and tear down method the unittest TestCase class.

I think that this is the wrong approach to the problem:
 - class scope for such fixtures drives large test classes, reduces
flexability
 - doing it the rough way you suggest interacts in non obvious ways with
test order randomisation
 - it also interacts with concurrent testing by having shared objects
(classes) hold state in a way that is not introspectable by the test
running framework.

I'd much much much rather see e.g. testresources integrated into the
core allowing fixtures to be shared across test instances in a way that
doesn't prohibit their use with concurrent testing, doesn't make it
awkward to do it across multiple classes. I'm happy to make any
[reasonable] changes (including license) to testresources to make it
includable in the stdlib if that's of interest.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100121/afa495e6/attachment.pgp>

From jml at mumak.net  Thu Jan 21 03:04:15 2010
From: jml at mumak.net (Jonathan Lange)
Date: Thu, 21 Jan 2010 02:04:15 +0000 (UTC)
Subject: [Python-ideas] Test Class setup and teardown in unittest
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
Message-ID: <loom.20100121T022121-895@post.gmane.org>

Mark Roddy <markroddy at ...> writes:
> 
> Earlier this week on the Testing In Python list, there was a
> discussion on how to execute a setup and/or teardown for a single test
> class instead of for each test fixture on the class (see the 'setUp
> and tearDown behavior' thread).  I have had to deal with situation
> myself before, and I am obviously not the only one (since I did not
> initiate the thread).  As such I'd like to propose adding a class
> level setup and tear down method the unittest TestCase class.
> 
> Rationale:
> Test cases can at times require the setup of expensive resources.
> This is often the case when implementing integration testing.  Having
> to perform this setup for each fixture can prohibited for large number
> of fixtures and/or for resources that are expensive to setup.  For
> example, I have several hundred integration tests that hit a live
> database.  If I create the connection object for each fixture, most of
> the tests fail due to the maximum connection limit being reached.  As
> a work around, I create a connection object once for each test case.
> Without such functionality built in, the common idiom runs along these
> lines:
> 
> class MyTest(TestCase):
> 
>     def setUp(self):
>         if not self.ClassIsSetup:
>             self.setupClass()
>             self.ClassIsSetup=True
> 
> While this achieves the desired functionality, it is unclear due to
> conditional setup code and is also error prone as the same code
> segment would need to appear in every TestCase requiring the
> functionality. 

I agree that this is a common problem, and that the Python community would 
benefit from a well-known, well-understood and widely applicable solution.

> Having a class wide setup and teardown function that
> implementers of test cases can override would make the code/intent
> more clear and alleviate the need to implement test case functionality
> when the user should be focusing on writing tests.
>

I'd take issue with the argument that this makes the intent clearer. In your 
motivating example, what you mean is "the test needs a connection" not "I want 
to do a set up that spans the scope of this class". A better approach would be 
to _declare_ the dependency in the test and let something else figure out how to 
best provide the dependency.

Also, setUpClass / tearDownClass is probably a bad idea. We implemented such 
behaviour in Twisted's testing framework some time ago, and have only recently 
managed to undo the damage.[1]

If you do this in such a way that setUpClass methods are encouraged to set 
attributes on 'self', then you are compelled to share the TestCase instance 
between tests. This runs contrary to the design of unittest, and breaks code 
that relies on a test object represent a single test.

It turns out that if you are building extensions to testing frameworks (like any 
big app does), it helps a lot to have an object per runnable test. In 
particular, it makes distributing tests across multiple processors & multiple 
computers much, much easier.

It also poses a difficult challenge for test runners that provide features such 
as running the tests in a random order. It's very hard to know if the class is 
actually "done" and ready to be torn down.

Finally, we found that it's use often lead to hard-to-debug failures due to test 
isolation issues.

There are already alternatives for this in the Python unit testing world. 
zope.testing provides a facility called 'layers' which solves this problem. I 
don't like it[2], but if we are talking about changing the standard library then 
we should consult existing practice.

Another solution is testresources[3]. It takes a declarative approach and works 
with all the code that's already in the Python standard library.

I'm not deeply familiar with xUnit implementations in other languages, but the 
problem isn't unique to Python. I imagine it would be worth doing some research 
on what Nunit, JUnit etc do.
 
> I emailed Michael Foord about some of his comments in the TIP thread
> and to ask if he would be interested in a patch adding this
> functionality, and I have included his response below.  I would like
> to hear people's comments/suggestions/ideas before I start working on
> said patch.
> 

Replies below.

...
> Michael Foord's Email:
> =======================================
> I would certainly be interested in adding this to unittest.
> 
> It needs a discussion of the API and the semantics:
> 
> * What should the methods be called? setup_class and teardown_class or
> setupClass and teardownClass? For consistency with existing methods
> the camelCase should probably be used.

In Twisted, we called them setUpClass and tearDownClass.

> * If the setupClass fails how should the error be reported? The
> *easiest* way is for the failure to be reported as part of the first
> test

Of course, it actually represents a failure in all of the tests. Another way of 
doing it is to construct a test-like object representing the entire class and 
record it as a failure in that.

> * Ditto for teardownClass - again the easiest way is for it to be
> reported as a failure in the last test

Ditto.

> * If setupClass fails then should all the tests in that class be
> skipped? I think yes.
> 

They should all be failed.

> Also details like ensuring that even if just a single test method from
> a class is run the setupClass and teardownClass are still run. It
> probably needs to go to python-dev or python-ideas for discussion.
> 

That's really really hard, and not a detail at all. See above.

I know this is a mostly negative response, but I really do hope it helps.

jml

[1] http://twistedmatrix.com/trac/ticket/4175
[2] http://code.mumak.net/2009/09/layers-are-terrible.html
[3] http://pypi.python.org/pypi/testresources/



From markroddy at gmail.com  Thu Jan 21 04:32:49 2010
From: markroddy at gmail.com (Mark Roddy)
Date: Wed, 20 Jan 2010 22:32:49 -0500
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <1264037825.4254.12.camel@lifeless-64>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<1264037825.4254.12.camel@lifeless-64>
Message-ID: <7930f1201001201932o4b98a031i18d80a3265929fbc@mail.gmail.com>

On Wed, Jan 20, 2010 at 8:37 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
>> Earlier this week on the Testing In Python list, there was a
>> discussion on how to execute a setup and/or teardown for a single test
>> class instead of for each test fixture on the class (see the 'setUp
>> and tearDown behavior' thread). ?I have had to deal with situation
>> myself before, and I am obviously not the only one (since I did not
>> initiate the thread). ?As such I'd like to propose adding a class
>> level setup and tear down method the unittest TestCase class.
>
> I think that this is the wrong approach to the problem:
> ?- class scope for such fixtures drives large test classes, reduces
> flexability
I can see how this could arise, but it has not been my experience (I
have a class setup system I've been using for a little while).  A
setUp method alone can be abused to create large inflexible tests.  I
think the answer to this is good documentation that clearly states
that abusing the functionality can lead to shooting oneself in the
foot.

> ?- doing it the rough way you suggest interacts in non obvious ways with
> test order randomisation
This is dependent on how randomization is implemented.  Something
simple such as pick a random test of those available shouldn't affect
the ability to keep track of whether a class's setup method hasn't
been implemented, but imitadly I'm not familiar with the internals of
any existing randomization system so I will look into these to see
what issues that would arise from this change.

> ?- it also interacts with concurrent testing by having shared objects
> (classes) hold state in a way that is not introspectable by the test
> running framework.
Very true, but it seems counter intuitive to expect to be able to have
shared state in this context.
At the same time, it wouldn't be acceptable to change in a way that
breaks frameworks with such functionality.

>
> I'd much much much rather see e.g. testresources integrated into the
> core allowing fixtures to be shared across test instances in a way that
> doesn't prohibit their use with concurrent testing, doesn't make it
> awkward to do it across multiple classes. I'm happy to make any
> [reasonable] changes (including license) to testresources to make it
> includable in the stdlib if that's of interest.
A pretty good approach for a complicated setup, but in a simple case
where you only need a line or two it seems like a lot of boiler plate
to get the job done.  Though I'll look into this library a little more
as I am not intimately familiar with it at the moment.

>
> -Rob
>


From markroddy at gmail.com  Thu Jan 21 04:33:05 2010
From: markroddy at gmail.com (Mark Roddy)
Date: Wed, 20 Jan 2010 22:33:05 -0500
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <loom.20100121T022121-895@post.gmane.org>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<loom.20100121T022121-895@post.gmane.org>
Message-ID: <7930f1201001201933s5fe2802cj3e7baaa4a8a290b7@mail.gmail.com>

On Wed, Jan 20, 2010 at 9:04 PM, Jonathan Lange <jml at mumak.net> wrote:
> Mark Roddy <markroddy at ...> writes:
>>
>> Earlier this week on the Testing In Python list, there was a
>> discussion on how to execute a setup and/or teardown for a single test
>> class instead of for each test fixture on the class (see the 'setUp
>> and tearDown behavior' thread). ?I have had to deal with situation
>> myself before, and I am obviously not the only one (since I did not
>> initiate the thread). ?As such I'd like to propose adding a class
>> level setup and tear down method the unittest TestCase class.
>>
>> Rationale:
>> Test cases can at times require the setup of expensive resources.
>> This is often the case when implementing integration testing. ?Having
>> to perform this setup for each fixture can prohibited for large number
>> of fixtures and/or for resources that are expensive to setup. ?For
>> example, I have several hundred integration tests that hit a live
>> database. ?If I create the connection object for each fixture, most of
>> the tests fail due to the maximum connection limit being reached. ?As
>> a work around, I create a connection object once for each test case.
>> Without such functionality built in, the common idiom runs along these
>> lines:
>>
>> class MyTest(TestCase):
>>
>> ? ? def setUp(self):
>> ? ? ? ? if not self.ClassIsSetup:
>> ? ? ? ? ? ? self.setupClass()
>> ? ? ? ? ? ? self.ClassIsSetup=True
>>
>> While this achieves the desired functionality, it is unclear due to
>> conditional setup code and is also error prone as the same code
>> segment would need to appear in every TestCase requiring the
>> functionality.
>
> I agree that this is a common problem, and that the Python community would
> benefit from a well-known, well-understood and widely applicable solution.
>
>> Having a class wide setup and teardown function that
>> implementers of test cases can override would make the code/intent
>> more clear and alleviate the need to implement test case functionality
>> when the user should be focusing on writing tests.
>>
>


> I'd take issue with the argument that this makes the intent clearer. In your
> motivating example, what you mean is "the test needs a connection" not "I want
> to do a set up that spans the scope of this class". A better approach would be
> to _declare_ the dependency in the test and let something else figure out how to
> best provide the dependency.
I think having the class setup is clearer than the 'if not
self.CaseIsSetup' idiom, but I wouldn't claim that it's definitely the
clearest way to achieve such functionality.

I like the class setup semantics as the closely follow the existing
fixture setup semantics.  Having to declare and configure dependencies
(while possibly being clearer as far as expressing intent) introduces
a completely different set of semantics that have to be grasped in
order to be used.  I'm going to with hold forming an opinion as to
whether or not this presents any meaningful barrier to entry until
after I've had a chance to review Rob's resources library.

>
> Also, setUpClass / tearDownClass is probably a bad idea. We implemented such
> behaviour in Twisted's testing framework some time ago, and have only recently
> managed to undo the damage.[1]
Thanks, I was not aware of this.  Do you have an references as to the
particular problems it caused?  The ticket only seems to describe it
being removed (with much excitement :), but doesn't seem to mention
the motivation.

>
> If you do this in such a way that setUpClass methods are encouraged to set
> attributes on 'self', then you are compelled to share the TestCase instance
> between tests. This runs contrary to the design of unittest, and breaks code
> that relies on a test object represent a single test.
I was actually thinking that a setupclass method would be a
classmethod since the use case is sharing across fixtures within a
class.  Wouldn't that be enough to cover this issue?

>
> It turns out that if you are building extensions to testing frameworks (like any
> big app does), it helps a lot to have an object per runnable test. In
> particular, it makes distributing tests across multiple processors & multiple
> computers much, much easier.
True, and clearly the case for highly focused unit tests.  However,
for integration tests (or whatever we could call tests that are
explicitly designed to work with an expensive resource) it can be cost
prohibitive to have this type of isolation (or flat out impossible in
the case that I gave).  I'll look into the implementation of some of
the testing frameworks that support distributed testing and see if
there isn't a way that this can be supported in both contexts (is it
possible to implement in a way that the case setup would get run in
each process/machine?).

>
> It also poses a difficult challenge for test runners that provide features such
> as running the tests in a random order. It's very hard to know if the class is
> actually "done" and ready to be torn down.
My initial (off the top of my head) thinking was to count the number
of test fixtures to be run via a class attribute set in the test case
constructor, and then the test runner would either decrement this
after the test is complete and call the tear down method once the
counter reached zero.  This doesn't seem like it would be affected by
randomized testing ordering, but I'll look into some existing
implementations to see how this could be affected.  Any obvious issues
I'm missing?

>
> Finally, we found that it's use often lead to hard-to-debug failures due to test
> isolation issues.
I think there's a distinction between "can lead to bad situations" and
"encourages bad situations".  The former is almost impossible to avoid
without becoming java :).  That latter is much subtler, but can be
addressed.  Do you have any suggestions for altering the semantics to
discourage abuse without reducing flexibility.  With a similar feature
I use, we have a rule to not use the case setup unless explicitly
writing integration tests, though there is no functional way to
enforce this, only communicating the idea (via documentation and code
reviews).

>
> There are already alternatives for this in the Python unit testing world.
> zope.testing provides a facility called 'layers' which solves this problem. I
> don't like it[2], but if we are talking about changing the standard library then
> we should consult existing practice.
Thanks, I will look into this and try to enumerate some pros and cons.
 Are there any specifics about it that you don't like?

>
> Another solution is testresources[3]. It takes a declarative approach and works
> with all the code that's already in the Python standard library.
Will be looking into this as well as previously stated.

>
> I'm not deeply familiar with xUnit implementations in other languages, but the
> problem isn't unique to Python. I imagine it would be worth doing some research
> on what Nunit, JUnit etc do.
Both JUnit and Nunit have a class setup and teardown functionality:
http://www.junit.org/apidocs/org/junit/BeforeClass.html
http://www.junit.org/apidocs/org/junit/AfterClass.html
http://www.nunit.org/index.php?p=fixtureSetup&r=2.5.3
http://www.nunit.org/index.php?p=fixtureTeardown&r=2.5.3
(The Nunit implementation I found a little confusing as they
apparently refer to a TestCase as a Fixture).

>
>> I emailed Michael Foord about some of his comments in the TIP thread
>> and to ask if he would be interested in a patch adding this
>> functionality, and I have included his response below. ?I would like
>> to hear people's comments/suggestions/ideas before I start working on
>> said patch.
>>
>
> Replies below.
>
> ...
>> Michael Foord's Email:
>> =======================================
>> I would certainly be interested in adding this to unittest.
>>
>> It needs a discussion of the API and the semantics:
>>
>> * What should the methods be called? setup_class and teardown_class or
>> setupClass and teardownClass? For consistency with existing methods
>> the camelCase should probably be used.
>
> In Twisted, we called them setUpClass and tearDownClass.
>
>> * If the setupClass fails how should the error be reported? The
>> *easiest* way is for the failure to be reported as part of the first
>> test
>
> Of course, it actually represents a failure in all of the tests. Another way of
> doing it is to construct a test-like object representing the entire class and
> record it as a failure in that.
>
>> * Ditto for teardownClass - again the easiest way is for it to be
>> reported as a failure in the last test
>
> Ditto.
>
>> * If setupClass fails then should all the tests in that class be
>> skipped? I think yes.
>>
>
> They should all be failed.
>
>> Also details like ensuring that even if just a single test method from
>> a class is run the setupClass and teardownClass are still run. It
>> probably needs to go to python-dev or python-ideas for discussion.
>>
>
> That's really really hard, and not a detail at all. See above.
>
> I know this is a mostly negative response, but I really do hope it helps.
>
> jml
>
> [1] http://twistedmatrix.com/trac/ticket/4175
> [2] http://code.mumak.net/2009/09/layers-are-terrible.html
> [3] http://pypi.python.org/pypi/testresources/
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From robertc at robertcollins.net  Thu Jan 21 05:03:49 2010
From: robertc at robertcollins.net (Robert Collins)
Date: Thu, 21 Jan 2010 15:03:49 +1100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <7930f1201001201932o4b98a031i18d80a3265929fbc@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<1264037825.4254.12.camel@lifeless-64>
	<7930f1201001201932o4b98a031i18d80a3265929fbc@mail.gmail.com>
Message-ID: <1264046629.11747.17.camel@lifeless-64>

On Wed, 2010-01-20 at 22:32 -0500, Mark Roddy wrote:

> > I think that this is the wrong approach to the problem:
> >  - class scope for such fixtures drives large test classes, reduces
> > flexability

> I can see how this could arise, but it has not been my experience (I
> have a class setup system I've been using for a little while).  A
> setUp method alone can be abused to create large inflexible tests.  I
> think the answer to this is good documentation that clearly states
> that abusing the functionality can lead to shooting oneself in the
> foot.

Yes, setUp and tearDown can be problematic as well. Junit which has has
setUp and setUpClass a long time has recently
(http://kentbeck.github.com/junit/doc/ReleaseNotes4.7.html) added a
system called Rules which is similar to testresources (though the fine
detail is different). The key elements are the same though:
 - rules are their own hierarchy
 - rules are added to tests not part of the test specific code
 - framework takes care of bringing up and taking down rules.

> >  - doing it the rough way you suggest interacts in non obvious ways with
> > test order randomisation

> This is dependent on how randomization is implemented.  Something
> simple such as pick a random test of those available shouldn't affect
> the ability to keep track of whether a class's setup method hasn't
> been implemented, but imitadly I'm not familiar with the internals of
> any existing randomization system so I will look into these to see
> what issues that would arise from this change.

It can be made to work: the non-obviousness is a simple side effect of
having dependencies between multiple tests.

> >  - it also interacts with concurrent testing by having shared objects
> > (classes) hold state in a way that is not introspectable by the test
> > running framework.

> Very true, but it seems counter intuitive to expect to be able to have
> shared state in this context.
> At the same time, it wouldn't be acceptable to change in a way that
> breaks frameworks with such functionality.

I don't think shared state here is good: but setting attributes *on the
class* *is* shared state: Its why I don't like setUpClass :).

> > I'd much much much rather see e.g. testresources integrated into the
> > core allowing fixtures to be shared across test instances in a way that
> > doesn't prohibit their use with concurrent testing, doesn't make it
> > awkward to do it across multiple classes. I'm happy to make any
> > [reasonable] changes (including license) to testresources to make it
> > includable in the stdlib if that's of interest.

> A pretty good approach for a complicated setup, but in a simple case
> where you only need a line or two it seems like a lot of boiler plate
> to get the job done.  Though I'll look into this library a little more
> as I am not intimately familiar with it at the moment.

If you only need a line or two, its hard to justify setUpClass being
used :).

Anyhow, its really not much boilerplate:
class MyResource(TestResource):
    def make(self, deps):
        # line or two
        return thing

I am considering adding a helper to testresources:

def simple_resource(maker, cleaner):
    class MyResource(TestResource):
        def make(self, deps):
            return maker(deps)
        def clean(self, resource):
            cleaner(resource)
    return MyResource

which would make the boilerplate smaller still.

Cheers,
Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100121/f9b807e6/attachment.pgp>

From robertc at robertcollins.net  Thu Jan 21 05:16:17 2010
From: robertc at robertcollins.net (Robert Collins)
Date: Thu, 21 Jan 2010 15:16:17 +1100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <7930f1201001201933s5fe2802cj3e7baaa4a8a290b7@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<loom.20100121T022121-895@post.gmane.org>
	<7930f1201001201933s5fe2802cj3e7baaa4a8a290b7@mail.gmail.com>
Message-ID: <1264047377.11747.30.camel@lifeless-64>

On Wed, 2010-01-20 at 22:33 -0500, Mark Roddy wrote:


> > I'd take issue with the argument that this makes the intent clearer. In your
> > motivating example, what you mean is "the test needs a connection" not "I want
> > to do a set up that spans the scope of this class". A better approach would be
> > to _declare_ the dependency in the test and let something else figure out how to
> > best provide the dependency.
> I think having the class setup is clearer than the 'if not
> self.CaseIsSetup' idiom, but I wouldn't claim that it's definitely the
> clearest way to achieve such functionality.
> 
> I like the class setup semantics as the closely follow the existing
> fixture setup semantics.  Having to declare and configure dependencies
> (while possibly being clearer as far as expressing intent) introduces
> a completely different set of semantics that have to be grasped in
> order to be used.  I'm going to with hold forming an opinion as to
> whether or not this presents any meaningful barrier to entry until
> after I've had a chance to review Rob's resources library.

A data point here is that the setUp/tearDown/cleanUp semantics are not
trivial themselves: here is a self check (check the source after coming
up with answers :P)
 - does tearDown run if setUp raises?
 - do cleanUps?
 - do cleanUps run before or after tearDown?

Also would you add classCleanUps or something, if you add a class setUp?
clean ups are _much_ nicer than tearDown, so I'd really want to see
that.

> > It turns out that if you are building extensions to testing frameworks (like any
> > big app does), it helps a lot to have an object per runnable test. In
> > particular, it makes distributing tests across multiple processors & multiple
> > computers much, much easier.

> True, and clearly the case for highly focused unit tests.  However,
> for integration tests (or whatever we could call tests that are
> explicitly designed to work with an expensive resource) it can be cost
> prohibitive to have this type of isolation (or flat out impossible in
> the case that I gave).

Its been trimmed, but here was your example:
"For
example, I have several hundred integration tests that hit a live
database.  If I create the connection object for each fixture, most of
the tests fail due to the maximum connection limit being reached.  As
a work around, I create a connection object once for each test case."

Database connections can be dirtied - they need to be in a clean state
with no outstanding requests, and outside a transaction, to be valid for
the start of a test. I do appreciate that if you have a buggy TCP stack,
or are not closing your connections, you can certainly hit connection
limits. 

I'd be inclined to have a pool of connections, rather than one
connection per class. That would under ideal circumstances give you one
connection for an entire test run (without concurrency), but also mean
that a broken connection would at most affect only one test.

>   I'll look into the implementation of some of
> the testing frameworks that support distributed testing and see if
> there isn't a way that this can be supported in both contexts (is it
> possible to implement in a way that the case setup would get run in
> each process/machine?).

Its about partitioning the state: e.g. (ugly) deep copy the class object
per thread.

> > It also poses a difficult challenge for test runners that provide features such
> > as running the tests in a random order. It's very hard to know if the class is
> > actually "done" and ready to be torn down.
> My initial (off the top of my head) thinking was to count the number
> of test fixtures to be run via a class attribute set in the test case
> constructor, and then the test runner would either decrement this
> after the test is complete and call the tear down method once the
> counter reached zero.  This doesn't seem like it would be affected by
> randomized testing ordering, but I'll look into some existing
> implementations to see how this could be affected.  Any obvious issues
> I'm missing?

Well, this has a few undesirable facets:
 - its complex
 - you'll need to honour tests that don't have that attribute set
 - and tests that already have that attribute set
 - it breaks the abstraction of 'a test knows how to run itself'.
 - you can't set that attribute in test case constructors because tests
can be constructed while the suite is running
 - using a global like that will mean that a single classes class setup
stuff would *stay setup* while other classes run, with a randomised
order : thats bad because things that are expensive and need to be
optimised also tend to be *large*, either in memory or external
resources (like TCP connections in your example).

> > Finally, we found that it's use often lead to hard-to-debug failures due to test
> > isolation issues.

> I think there's a distinction between "can lead to bad situations" and
> "encourages bad situations".  The former is almost impossible to avoid
> without becoming java :).  That latter is much subtler, but can be
> addressed.  Do you have any suggestions for altering the semantics to
> discourage abuse without reducing flexibility.  With a similar feature
> I use, we have a rule to not use the case setup unless explicitly
> writing integration tests, though there is no functional way to
> enforce this, only communicating the idea (via documentation and code
> reviews).

Have you seen Rusty Russell's interface usability scale? Having to have
a rule strongly suggests that the API is prone to misuse :).

> > I'm not deeply familiar with xUnit implementations in other languages, but the
> > problem isn't unique to Python. I imagine it would be worth doing some research
> > on what Nunit, JUnit etc do.
> Both JUnit and Nunit have a class setup and teardown functionality:
> http://www.junit.org/apidocs/org/junit/BeforeClass.html
> http://www.junit.org/apidocs/org/junit/AfterClass.html
> http://www.nunit.org/index.php?p=fixtureSetup&r=2.5.3
> http://www.nunit.org/index.php?p=fixtureTeardown&r=2.5.3
> (The Nunit implementation I found a little confusing as they
> apparently refer to a TestCase as a Fixture).

See my other mail for a link to JUnit's Rules.

Cheers,
Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100121/e14b4c05/attachment.pgp>

From fuzzyman at voidspace.org.uk  Thu Jan 21 12:42:38 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 21 Jan 2010 11:42:38 +0000
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <1264037825.4254.12.camel@lifeless-64>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<1264037825.4254.12.camel@lifeless-64>
Message-ID: <4B583DAE.3060209@voidspace.org.uk>

On 21/01/2010 01:37, Robert Collins wrote:
> On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
>
>> Earlier this week on the Testing In Python list, there was a
>> discussion on how to execute a setup and/or teardown for a single test
>> class instead of for each test fixture on the class (see the 'setUp
>> and tearDown behavior' thread).  I have had to deal with situation
>> myself before, and I am obviously not the only one (since I did not
>> initiate the thread).  As such I'd like to propose adding a class
>> level setup and tear down method the unittest TestCase class.
>>
> I think that this is the wrong approach to the problem:
>   - class scope for such fixtures drives large test classes, reduces
> flexability
>

Agreed.

>   - doing it the rough way you suggest interacts in non obvious ways with
> test order randomisation
>

Not currently in unittest, so not really an issue.
>   - it also interacts with concurrent testing by having shared objects
> (classes) hold state in a way that is not introspectable by the test
> running framework.
>

Again, unittest doesn't do concurrent testing out of the box.

How are shared fixtures like this not introspectable?

As I see it.

Advantages of setupClass and teardownClass:

* Simple to implement
* Simple to explain
* A commonly requested feature
* Provided by other test frameworks in and outside of Python

Disadvantages:

* Can encourage a poor style of testing, including monolithic test classes

> I'd much much much rather see e.g. testresources integrated into the
>

How is testresources superior? Can you demonstrate this - in particular 
what is the simplest example? The simplest example of setupClass would be:

class SomeTest(unittest.TestCase):
     def setupClass(self):
         ... # setup shared fixture

> core allowing fixtures to be shared across test instances in a way that
> doesn't prohibit their use with concurrent testing, doesn't make it
> awkward to do it across multiple classes.

Can you demonstrate how testresources solves these problems.

> I'm happy to make any
> [reasonable] changes (including license) to testresources to make it
> includable in the stdlib if that's of interest.
>

That's not how contributing code to Python works. You need to provide a 
signed contributor agreement to the PSF and then you specifically 
license to the PSF any code you are contributing using one of a few 
specific licenses. For new modules in the standard library you also need 
to be willing to maintain the code.

All the best,

Michael Foord


> -Rob
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf 
of your employer, to release me from all obligations and waivers arising 
from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, 
shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, 
non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have 
entered into with your employer, its partners, licensors, agents and 
assigns, in perpetuity, without prejudice to my ongoing rights and 
privileges. You further represent that you have the authority to release 
me from any BOGUS AGREEMENTS on behalf of your employer.




From fuzzyman at voidspace.org.uk  Thu Jan 21 15:16:00 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 21 Jan 2010 14:16:00 +0000
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <1264046629.11747.17.camel@lifeless-64>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>	
	<1264037825.4254.12.camel@lifeless-64>	
	<7930f1201001201932o4b98a031i18d80a3265929fbc@mail.gmail.com>
	<1264046629.11747.17.camel@lifeless-64>
Message-ID: <4B5861A0.60700@voidspace.org.uk>

On 21/01/2010 04:03, Robert Collins wrote:
> On Wed, 2010-01-20 at 22:32 -0500, Mark Roddy wrote:
>
>    
>>> I think that this is the wrong approach to the problem:
>>>   - class scope for such fixtures drives large test classes, reduces
>>> flexability
>>>        
>    
>> I can see how this could arise, but it has not been my experience (I
>> have a class setup system I've been using for a little while).  A
>> setUp method alone can be abused to create large inflexible tests.  I
>> think the answer to this is good documentation that clearly states
>> that abusing the functionality can lead to shooting oneself in the
>> foot.
>>      
> Yes, setUp and tearDown can be problematic as well. Junit which has has
> setUp and setUpClass a long time has recently
> (http://kentbeck.github.com/junit/doc/ReleaseNotes4.7.html) added a
> system called Rules which is similar to testresources (though the fine
> detail is different). The key elements are the same though:
>   - rules are their own hierarchy
>   - rules are added to tests not part of the test specific code
>   - framework takes care of bringing up and taking down rules.
>    

Right, they *can* be problematic but they can also be extremely useful 
if not abused. My feeling is that the same is true of setupClass and 
teardownClass.

Replacing them with a more complex mechanism is not necessarily a win, 
although a more complex mechanism for more complex use cases is 
justified (although it doesn't necessarily *need* to be in the standard 
library).

> [snip...]
>>> I'd much much much rather see e.g. testresources integrated into the
>>> core allowing fixtures to be shared across test instances in a way that
>>> doesn't prohibit their use with concurrent testing, doesn't make it
>>> awkward to do it across multiple classes. I'm happy to make any
>>> [reasonable] changes (including license) to testresources to make it
>>> includable in the stdlib if that's of interest.
>>>        
>    
>> A pretty good approach for a complicated setup, but in a simple case
>> where you only need a line or two it seems like a lot of boiler plate
>> to get the job done.  Though I'll look into this library a little more
>> as I am not intimately familiar with it at the moment.
>>      
> If you only need a line or two, its hard to justify setUpClass being
> used :).
>
> Anyhow, its really not much boilerplate:
> class MyResource(TestResource):
>      def make(self, deps):
>          # line or two
>          return thing
>
> I am considering adding a helper to testresources:
>
> def simple_resource(maker, cleaner):
>      class MyResource(TestResource):
>          def make(self, deps):
>              return maker(deps)
>          def clean(self, resource):
>              cleaner(resource)
>      return MyResource
>
> which would make the boilerplate smaller still.
>    

This doesn't show how testresources is used in conjunction with unittest 
- can you give a minimal *self contained* example please.

Thanks

Michael

> Cheers,
> Rob
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.




From Nikolaus at rath.org  Thu Jan 21 03:49:56 2010
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Wed, 20 Jan 2010 21:49:56 -0500
Subject: [Python-ideas] Test Class setup and teardown in unittest
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
Message-ID: <871vhk894b.fsf@vostro.rath.org>

Mark Roddy <markroddy at gmail.com> writes:
> Earlier this week on the Testing In Python list, there was a
> discussion on how to execute a setup and/or teardown for a single test
> class instead of for each test fixture on the class (see the 'setUp
> and tearDown behavior' thread).  I have had to deal with situation
> myself before, and I am obviously not the only one (since I did not
> initiate the thread).  As such I'd like to propose adding a class
> level setup and tear down method the unittest TestCase class.
>
> Rationale:
> Test cases can at times require the setup of expensive resources.
> This is often the case when implementing integration testing.  Having

I think this is a great idea, I ran into this problem too.

> * What should the methods be called? setup_class and teardown_class or
> setupClass and teardownClass? For consistency with existing methods
> the camelCase should probably be used.
>
> * If the setupClass fails how should the error be reported? The
> *easiest* way is for the failure to be reported as part of the first
> test
> * Ditto for teardownClass - again the easiest way is for it to be
> reported as a failure in the last test
> * If setupClass fails then should all the tests in that class be
> skipped? I think yes.

No strong feelings about this, I'm happy as long as some solution gets
implemented.



   -Nikolaus

-- 
 ?Time flies like an arrow, fruit flies like a Banana.?

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C



From guido at python.org  Thu Jan 21 18:09:54 2010
From: guido at python.org (Guido van Rossum)
Date: Thu, 21 Jan 2010 09:09:54 -0800
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <871vhk894b.fsf@vostro.rath.org>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com> 
	<871vhk894b.fsf@vostro.rath.org>
Message-ID: <ca471dc21001210909v13e264b8w655d5dbd927c6445@mail.gmail.com>

Ignoring many of the finer points brought up here, and putting
practicality before purity, I think having setUpClass and
tearDownClass methods is a great idea.

While we're at it I would also recommend adding module-level setUp and
tearDown function -- Google's extension of pyunit implements these and
they are often handy for a variety of use cases.
-- 
--Guido van Rossum (python.org/~guido)


From brett at python.org  Thu Jan 21 20:12:07 2010
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Jan 2010 11:12:07 -0800
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
Message-ID: <bbaeab101001211112i2081305ev63772c6054e1bac5@mail.gmail.com>

On Wed, Jan 20, 2010 at 16:38, Mark Roddy <markroddy at gmail.com> wrote:

> Earlier this week on the Testing In Python list, there was a
> discussion on how to execute a setup and/or teardown for a single test
> class instead of for each test fixture on the class (see the 'setUp
> and tearDown behavior' thread).  I have had to deal with situation
> myself before, and I am obviously not the only one (since I did not
> initiate the thread).  As such I'd like to propose adding a class
> level setup and tear down method the unittest TestCase class.
>
>
Maybe I'm missing something, but why can't you simply use __init__ and
__del__ for this purpose?

-Brett



> Rationale:
> Test cases can at times require the setup of expensive resources.
> This is often the case when implementing integration testing.  Having
> to perform this setup for each fixture can prohibited for large number
> of fixtures and/or for resources that are expensive to setup.  For
> example, I have several hundred integration tests that hit a live
> database.  If I create the connection object for each fixture, most of
> the tests fail due to the maximum connection limit being reached.  As
> a work around, I create a connection object once for each test case.
> Without such functionality built in, the common idiom runs along these
> lines:
>
> class MyTest(TestCase):
>
>    def setUp(self):
>        if not self.ClassIsSetup:
>            self.setupClass()
>            self.ClassIsSetup=True
>
> While this achieves the desired functionality, it is unclear due to
> conditional setup code and is also error prone as the same code
> segment would need to appear in every TestCase requiring the
> functionality.  Having a class wide setup and teardown function that
> implementers of test cases can override would make the code/intent
> more clear and alleviate the need to implement test case functionality
> when the user should be focusing on writing tests.
>
> I emailed Michael Foord about some of his comments in the TIP thread
> and to ask if he would be interested in a patch adding this
> functionality, and I have included his response below.  I would like
> to hear people's comments/suggestions/ideas before I start working on
> said patch.
>
> Thanks,
> -Mark
>
>
> Michael Foord's Email:
> =======================================
> I would certainly be interested in adding this to unittest.
>
> It needs a discussion of the API and the semantics:
>
> * What should the methods be called? setup_class and teardown_class or
> setupClass and teardownClass? For consistency with existing methods
> the camelCase should probably be used.
> * If the setupClass fails how should the error be reported? The
> *easiest* way is for the failure to be reported as part of the first
> test
> * Ditto for teardownClass - again the easiest way is for it to be
> reported as a failure in the last test
> * If setupClass fails then should all the tests in that class be
> skipped? I think yes.
>
> Also details like ensuring that even if just a single test method from
> a class is run the setupClass and teardownClass are still run. It
> probably needs to go to python-dev or python-ideas for discussion.
>
> All the best,
>
> Michael
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100121/f526c8f2/attachment.html>

From fuzzyman at voidspace.org.uk  Thu Jan 21 20:15:17 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 21 Jan 2010 19:15:17 +0000
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <bbaeab101001211112i2081305ev63772c6054e1bac5@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<bbaeab101001211112i2081305ev63772c6054e1bac5@mail.gmail.com>
Message-ID: <4B58A7C5.80100@voidspace.org.uk>

On 21/01/2010 19:12, Brett Cannon wrote:
>
>
> On Wed, Jan 20, 2010 at 16:38, Mark Roddy <markroddy at gmail.com 
> <mailto:markroddy at gmail.com>> wrote:
>
>     Earlier this week on the Testing In Python list, there was a
>     discussion on how to execute a setup and/or teardown for a single test
>     class instead of for each test fixture on the class (see the 'setUp
>     and tearDown behavior' thread).  I have had to deal with situation
>     myself before, and I am obviously not the only one (since I did not
>     initiate the thread).  As such I'd like to propose adding a class
>     level setup and tear down method the unittest TestCase class.
>
>
> Maybe I'm missing something, but why can't you simply use __init__ and 
> __del__ for this purpose?

unittest creates one instance *per test method*. That way tests are 
isolated from each other in separate instances. __init__ and __del__ are 
(or would be) run per test method.

Michael


>
> -Brett
>
>     Rationale:
>     Test cases can at times require the setup of expensive resources.
>     This is often the case when implementing integration testing.  Having
>     to perform this setup for each fixture can prohibited for large number
>     of fixtures and/or for resources that are expensive to setup.  For
>     example, I have several hundred integration tests that hit a live
>     database.  If I create the connection object for each fixture, most of
>     the tests fail due to the maximum connection limit being reached.  As
>     a work around, I create a connection object once for each test case.
>     Without such functionality built in, the common idiom runs along these
>     lines:
>
>     class MyTest(TestCase):
>
>        def setUp(self):
>            if not self.ClassIsSetup:
>                self.setupClass()
>                self.ClassIsSetup=True
>
>     While this achieves the desired functionality, it is unclear due to
>     conditional setup code and is also error prone as the same code
>     segment would need to appear in every TestCase requiring the
>     functionality.  Having a class wide setup and teardown function that
>     implementers of test cases can override would make the code/intent
>     more clear and alleviate the need to implement test case functionality
>     when the user should be focusing on writing tests.
>
>     I emailed Michael Foord about some of his comments in the TIP thread
>     and to ask if he would be interested in a patch adding this
>     functionality, and I have included his response below.  I would like
>     to hear people's comments/suggestions/ideas before I start working on
>     said patch.
>
>     Thanks,
>     -Mark
>
>
>     Michael Foord's Email:
>     =======================================
>     I would certainly be interested in adding this to unittest.
>
>     It needs a discussion of the API and the semantics:
>
>     * What should the methods be called? setup_class and teardown_class or
>     setupClass and teardownClass? For consistency with existing methods
>     the camelCase should probably be used.
>     * If the setupClass fails how should the error be reported? The
>     *easiest* way is for the failure to be reported as part of the first
>     test
>     * Ditto for teardownClass - again the easiest way is for it to be
>     reported as a failure in the last test
>     * If setupClass fails then should all the tests in that class be
>     skipped? I think yes.
>
>     Also details like ensuring that even if just a single test method from
>     a class is run the setupClass and teardownClass are still run. It
>     probably needs to go to python-dev or python-ideas for discussion.
>
>     All the best,
>
>     Michael
>     _______________________________________________
>     Python-ideas mailing list
>     Python-ideas at python.org <mailto:Python-ideas at python.org>
>     http://mail.python.org/mailman/listinfo/python-ideas
>
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100121/2b460b25/attachment.html>

From markroddy at gmail.com  Thu Jan 21 20:44:54 2010
From: markroddy at gmail.com (Mark Roddy)
Date: Thu, 21 Jan 2010 14:44:54 -0500
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <4B58A7C5.80100@voidspace.org.uk>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<bbaeab101001211112i2081305ev63772c6054e1bac5@mail.gmail.com>
	<4B58A7C5.80100@voidspace.org.uk>
Message-ID: <7930f1201001211144i596aab19q91f6e015af7d8c25@mail.gmail.com>

On Thu, Jan 21, 2010 at 2:15 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> On 21/01/2010 19:12, Brett Cannon wrote:
>
> On Wed, Jan 20, 2010 at 16:38, Mark Roddy <markroddy at gmail.com> wrote:
>>
>> Earlier this week on the Testing In Python list, there was a
>> discussion on how to execute a setup and/or teardown for a single test
>> class instead of for each test fixture on the class (see the 'setUp
>> and tearDown behavior' thread). ?I have had to deal with situation
>> myself before, and I am obviously not the only one (since I did not
>> initiate the thread). ?As such I'd like to propose adding a class
>> level setup and tear down method the unittest TestCase class.
>>
>
> Maybe I'm missing something, but why can't you simply use __init__ and
> __del__ for this purpose?
>
> unittest creates one instance *per test method*. That way tests are isolated
> from each other in separate instances. __init__ and __del__ are (or would
> be) run per test method.
>
> Michael
>
>
>
> -Brett
>
>>
>> Rationale:
>> Test cases can at times require the setup of expensive resources.
>> This is often the case when implementing integration testing. ?Having
>> to perform this setup for each fixture can prohibited for large number
>> of fixtures and/or for resources that are expensive to setup. ?For
>> example, I have several hundred integration tests that hit a live
>> database. ?If I create the connection object for each fixture, most of
>> the tests fail due to the maximum connection limit being reached. ?As
>> a work around, I create a connection object once for each test case.
>> Without such functionality built in, the common idiom runs along these
>> lines:
>>
>> class MyTest(TestCase):
>>
>> ? ?def setUp(self):
>> ? ? ? ?if not self.ClassIsSetup:
>> ? ? ? ? ? ?self.setupClass()
>> ? ? ? ? ? ?self.ClassIsSetup=True
>>
>> While this achieves the desired functionality, it is unclear due to
>> conditional setup code and is also error prone as the same code
>> segment would need to appear in every TestCase requiring the
>> functionality. ?Having a class wide setup and teardown function that
>> implementers of test cases can override would make the code/intent
>> more clear and alleviate the need to implement test case functionality
>> when the user should be focusing on writing tests.
>>
>> I emailed Michael Foord about some of his comments in the TIP thread
>> and to ask if he would be interested in a patch adding this
>> functionality, and I have included his response below. ?I would like
>> to hear people's comments/suggestions/ideas before I start working on
>> said patch.
>>
>> Thanks,
>> -Mark
>>
>>
>> Michael Foord's Email:
>> =======================================
>> I would certainly be interested in adding this to unittest.
>>
>> It needs a discussion of the API and the semantics:
>>
>> * What should the methods be called? setup_class and teardown_class or
>> setupClass and teardownClass? For consistency with existing methods
>> the camelCase should probably be used.
>> * If the setupClass fails how should the error be reported? The
>> *easiest* way is for the failure to be reported as part of the first
>> test
>> * Ditto for teardownClass - again the easiest way is for it to be
>> reported as a failure in the last test
>> * If setupClass fails then should all the tests in that class be
>> skipped? I think yes.
>>
>> Also details like ensuring that even if just a single test method from
>> a class is run the setupClass and teardownClass are still run. It
>> probably needs to go to python-dev or python-ideas for discussion.
>>
>> All the best,
>>
>> Michael
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>
>
>
> --
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
> READ CAREFULLY. By accepting and reading this email you agree, on behalf of
> your employer, to release me from all obligations and waivers arising from
> any and all NON-NEGOTIATED agreements, licenses, terms-of-service,
> shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure,
> non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have
> entered into with your employer, its partners, licensors, agents and
> assigns, in perpetuity, without prejudice to my ongoing rights and
> privileges. You further represent that you have the authority to release me
> from any BOGUS AGREEMENTS on behalf of your employer.
>
>

Also, if you did a teardown (of any type/scope) in __del__ you
wouldn't be able report errors occurring in the method as __del__
isn't called until the object is no longer referenced (possibly not
occurring until the interpreter shuts down).

-Mark


From robertc at robertcollins.net  Thu Jan 21 20:47:13 2010
From: robertc at robertcollins.net (Robert Collins)
Date: Fri, 22 Jan 2010 06:47:13 +1100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <4B583DAE.3060209@voidspace.org.uk>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<1264037825.4254.12.camel@lifeless-64>
	<4B583DAE.3060209@voidspace.org.uk>
Message-ID: <1264103233.11747.91.camel@lifeless-64>

On Thu, 2010-01-21 at 11:42 +0000, Michael Foord wrote:
> On 21/01/2010 01:37, Robert Collins wrote:
> > On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
> >
> >> Earlier this week on the Testing In Python list, there was a
> >> discussion on how to execute a setup and/or teardown for a single test
> >> class instead of for each test fixture on the class (see the 'setUp
> >> and tearDown behavior' thread).  I have had to deal with situation
> >> myself before, and I am obviously not the only one (since I did not
> >> initiate the thread).  As such I'd like to propose adding a class
> >> level setup and tear down method the unittest TestCase class.
> >>
> > I think that this is the wrong approach to the problem:
> >   - class scope for such fixtures drives large test classes, reduces
> > flexability
> >
> 
> Agreed.
> 
> >   - doing it the rough way you suggest interacts in non obvious ways with
> > test order randomisation
> >
> 
> Not currently in unittest, so not really an issue.

Done by nearly every extended UI out there. I would hope that messing
them up would count as an issue.

> >   - it also interacts with concurrent testing by having shared objects
> > (classes) hold state in a way that is not introspectable by the test
> > running framework.
> >
> 
> Again, unittest doesn't do concurrent testing out of the box.

testtools includes a concurrent TestResult which buffers activities from
different executing test cases and outputs them to a single regular
TestResult. Using that run() on TestSuite can be redefined to start some
threads, giving each one a helper TestResult forwarding to the caller
supplied TestResult, and finally treat self.__iter__ as a queue to
dispatch to each worker thread.

> How are shared fixtures like this not introspectable?

How can you tell if there is shared state when setUpClass is being used?

> As I see it.
> 
> Advantages of setupClass and teardownClass:
> 
> * Simple to implement
> * Simple to explain
> * A commonly requested feature
> * Provided by other test frameworks in and outside of Python

Mmm. It is commonly requested, but I'm not sure I entirely agree about
the simple claims.

> Disadvantages:
> 
> * Can encourage a poor style of testing, including monolithic test classes

* Can limit concurrency

> > I'd much much much rather see e.g. testresources integrated into the
> >
> 
> How is testresources superior? 

It separates concerns, which allows introspection for test runners. This
is significantly more flexible, particularly when combined with test
parameterisation.

> Can you demonstrate this - in particular 
> what is the simplest example? The simplest example of setupClass would be:
> 
> class SomeTest(unittest.TestCase):
>      def setupClass(self):
>          ... # setup shared fixture

                     ^ this is buggy :)
Its buggy because if it really does write to self, you now have what
twisted did where all the test cases will be sharing one test object
instance - which very frequently confuses and makes isolation between
unrelated test state fragile.

If its not writing to self, then you don't know what class object to
write to (because you might be in a subclass) - that really should be a
class method.

And the fact that you have this confusion, as demonstrated here, is
precisely why I don't agree about easy to implement and easy to
understand.

> > core allowing fixtures to be shared across test instances in a way that
> > doesn't prohibit their use with concurrent testing, doesn't make it
> > awkward to do it across multiple classes.
> 
> Can you demonstrate how testresources solves these problems.

Yes. resources are declared in a list, which allows the test framework,
when parallelising, to group together tests that use the same resources
when partitioning. Compare this with setUpClass where all tests in an
inheritance hierarchy have to be grouped together (after excluding your
base TestCase).

You ask in another mail for a SSCE: here is one attached to this mail
('foo.py'). Run with 'python -m unittest foo'.

> > I'm happy to make any
> > [reasonable] changes (including license) to testresources to make it
> > includable in the stdlib if that's of interest.
> >
> 
> That's not how contributing code to Python works. You need to provide a 
> signed contributor agreement to the PSF and then you specifically 
> license to the PSF any code you are contributing using one of a few 
> specific licenses. For new modules in the standard library you also need 
> to be willing to maintain the code.

There is other code of mine in the standard library unittest module,
done without a contributor agreement: so this is at best inconsistent. I
mention license changes because that would be required to include it in
the standard library (its currently a copyleft license) - and I'm not
the sole author. Regardless, whatever process and actions are needed, I
will do them, or arrange with other people where needed.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.py
Type: text/x-python
Size: 546 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100122/5658864a/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100122/5658864a/attachment.pgp>

From holger.krekel at gmail.com  Thu Jan 21 20:52:44 2010
From: holger.krekel at gmail.com (Holger Krekel)
Date: Thu, 21 Jan 2010 20:52:44 +0100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <bbaeab101001211112i2081305ev63772c6054e1bac5@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com> 
	<bbaeab101001211112i2081305ev63772c6054e1bac5@mail.gmail.com>
Message-ID: <d235cdc41001211152x45e863e4kefbad9d864a6d6da@mail.gmail.com>

On Thu, Jan 21, 2010 at 8:12 PM, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, Jan 20, 2010 at 16:38, Mark Roddy <markroddy at gmail.com> wrote:
>>
>> Earlier this week on the Testing In Python list, there was a
>> discussion on how to execute a setup and/or teardown for a single test
>> class instead of for each test fixture on the class (see the 'setUp
>> and tearDown behavior' thread). ?I have had to deal with situation
>> myself before, and I am obviously not the only one (since I did not
>> initiate the thread). ?As such I'd like to propose adding a class
>> level setup and tear down method the unittest TestCase class.
>>
>
> Maybe I'm missing something, but why can't you simply use __init__ and
> __del__ for this purpose?

Test resource finalization needs strict semantics and predictable
timing i'd say - and at least '__del__' doesn't provide that on some
python interpreters.  Moreover, unitttest (as well as py.test and
other test runners) instantiate fresh instances for a test method exec
to prevent hidden dependencies between tests.

cheers,
holger


> -Brett
>
>>
>> Rationale:
>> Test cases can at times require the setup of expensive resources.
>> This is often the case when implementing integration testing. ?Having
>> to perform this setup for each fixture can prohibited for large number
>> of fixtures and/or for resources that are expensive to setup. ?For
>> example, I have several hundred integration tests that hit a live
>> database. ?If I create the connection object for each fixture, most of
>> the tests fail due to the maximum connection limit being reached. ?As
>> a work around, I create a connection object once for each test case.
>> Without such functionality built in, the common idiom runs along these
>> lines:
>>
>> class MyTest(TestCase):
>>
>> ? ?def setUp(self):
>> ? ? ? ?if not self.ClassIsSetup:
>> ? ? ? ? ? ?self.setupClass()
>> ? ? ? ? ? ?self.ClassIsSetup=True
>>
>> While this achieves the desired functionality, it is unclear due to
>> conditional setup code and is also error prone as the same code
>> segment would need to appear in every TestCase requiring the
>> functionality. ?Having a class wide setup and teardown function that
>> implementers of test cases can override would make the code/intent
>> more clear and alleviate the need to implement test case functionality
>> when the user should be focusing on writing tests.
>>
>> I emailed Michael Foord about some of his comments in the TIP thread
>> and to ask if he would be interested in a patch adding this
>> functionality, and I have included his response below. ?I would like
>> to hear people's comments/suggestions/ideas before I start working on
>> said patch.
>>
>> Thanks,
>> -Mark
>>
>>
>> Michael Foord's Email:
>> =======================================
>> I would certainly be interested in adding this to unittest.
>>
>> It needs a discussion of the API and the semantics:
>>
>> * What should the methods be called? setup_class and teardown_class or
>> setupClass and teardownClass? For consistency with existing methods
>> the camelCase should probably be used.
>> * If the setupClass fails how should the error be reported? The
>> *easiest* way is for the failure to be reported as part of the first
>> test
>> * Ditto for teardownClass - again the easiest way is for it to be
>> reported as a failure in the last test
>> * If setupClass fails then should all the tests in that class be
>> skipped? I think yes.
>>
>> Also details like ensuring that even if just a single test method from
>> a class is run the setupClass and teardownClass are still run. It
>> probably needs to go to python-dev or python-ideas for discussion.
>>
>> All the best,
>>
>> Michael
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


From holger.krekel at gmail.com  Thu Jan 21 21:18:45 2010
From: holger.krekel at gmail.com (Holger Krekel)
Date: Thu, 21 Jan 2010 21:18:45 +0100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <ca471dc21001210909v13e264b8w655d5dbd927c6445@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com> 
	<871vhk894b.fsf@vostro.rath.org>
	<ca471dc21001210909v13e264b8w655d5dbd927c6445@mail.gmail.com>
Message-ID: <d235cdc41001211218m5fa70a0dw8b9bf44520f277ec@mail.gmail.com>

On Thu, Jan 21, 2010 at 6:09 PM, Guido van Rossum <guido at python.org> wrote:
> Ignoring many of the finer points brought up here, and putting
> practicality before purity, I think having setUpClass and
> tearDownClass methods is a great idea.
>
> While we're at it I would also recommend adding module-level setUp and
> tearDown function -- Google's extension of pyunit implements these and
> they are often handy for a variety of use cases.

If going for that i'd rather like to see those named
setup_class/teardown_class and setup_module/teardown_module like
py.test and nose do for a long time now. When i first went for those i
actually did so because i wanted to follow PEP8 ... But the stronger
argument now is that it would be cool seeing some tool/approach
convergence and unittest is the new kid on the setup-block there :)

cheers,
holger

> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From holger.krekel at gmail.com  Thu Jan 21 21:46:35 2010
From: holger.krekel at gmail.com (Holger Krekel)
Date: Thu, 21 Jan 2010 21:46:35 +0100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <1264103233.11747.91.camel@lifeless-64>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com> 
	<1264037825.4254.12.camel@lifeless-64>
	<4B583DAE.3060209@voidspace.org.uk> 
	<1264103233.11747.91.camel@lifeless-64>
Message-ID: <d235cdc41001211246t2dc31048jecc20ab18c644c41@mail.gmail.com>

On Thu, Jan 21, 2010 at 8:47 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On Thu, 2010-01-21 at 11:42 +0000, Michael Foord wrote:
>> On 21/01/2010 01:37, Robert Collins wrote:
>> > core allowing fixtures to be shared across test instances in a way that
>> > doesn't prohibit their use with concurrent testing, doesn't make it
>> > awkward to do it across multiple classes.
>>
>> Can you demonstrate how testresources solves these problems.
>
> Yes. resources are declared in a list, which allows the test framework,
> when parallelising, to group together tests that use the same resources
> when partitioning. Compare this with setUpClass where all tests in an
> inheritance hierarchy have to be grouped together (after excluding your
> base TestCase).

FWIW I'd like to add that after a number of years of having py.test
support setup_class
and setup_module i came to the conclusion some time ago that this
"grouping" requirement is bad for functional tests which usually
require more setup management and parametrization than classic
unittests.  It's also not easy to have expensive setup easily shared
between test modules and it's easy to get this wrong and to in the end
hinder refactorings rather than to further it.

At least that's my and some of my users experience with it. And is the
reason why I came up with a new method to do test fixtures that allows
to directly manipulate setup state in a single place instead of all
across test suites. Basically test functions receive objects
("funcargs") that are created/handled in factories. Loose coupling
here allows to have the same setup used across many test modules
efficiently.  Obviously different from the testresources approach
although i share a number of related design considerations and goals.

cheers,
holger

> You ask in another mail for a SSCE: here is one attached to this mail
> ('foo.py'). Run with 'python -m unittest foo'.
>
>> > I'm happy to make any
>> > [reasonable] changes (including license) to testresources to make it
>> > includable in the stdlib if that's of interest.
>> >
>>
>> That's not how contributing code to Python works. You need to provide a
>> signed contributor agreement to the PSF and then you specifically
>> license to the PSF any code you are contributing using one of a few
>> specific licenses. For new modules in the standard library you also need
>> to be willing to maintain the code.
>
> There is other code of mine in the standard library unittest module,
> done without a contributor agreement: so this is at best inconsistent. I
> mention license changes because that would be required to include it in
> the standard library (its currently a copyleft license) - and I'm not
> the sole author. Regardless, whatever process and actions are needed, I
> will do them, or arrange with other people where needed.
>
> -Rob
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


From robertc at robertcollins.net  Thu Jan 21 21:58:01 2010
From: robertc at robertcollins.net (Robert Collins)
Date: Fri, 22 Jan 2010 07:58:01 +1100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <d235cdc41001211246t2dc31048jecc20ab18c644c41@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<1264037825.4254.12.camel@lifeless-64>
	<4B583DAE.3060209@voidspace.org.uk>
	<1264103233.11747.91.camel@lifeless-64>
	<d235cdc41001211246t2dc31048jecc20ab18c644c41@mail.gmail.com>
Message-ID: <1264107481.11747.94.camel@lifeless-64>

On Thu, 2010-01-21 at 21:46 +0100, Holger Krekel wrote:
> 
> At least that's my and some of my users experience with it. And is the
> reason why I came up with a new method to do test fixtures that allows
> to directly manipulate setup state in a single place instead of all
> across test suites. Basically test functions receive objects
> ("funcargs") that are created/handled in factories. Loose coupling
> here allows to have the same setup used across many test modules
> efficiently.  Obviously different from the testresources approach
> although i share a number of related design considerations and goals. 

Yup, I quite liked the funcargs approach you have. Its fitting into a
dramatically different unittest framework but we seem to have otherwise
generated very similar solutions.

-Rob

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100122/5cc5d7d4/attachment.pgp>

From guido at python.org  Thu Jan 21 22:24:46 2010
From: guido at python.org (Guido van Rossum)
Date: Thu, 21 Jan 2010 13:24:46 -0800
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <d235cdc41001211218m5fa70a0dw8b9bf44520f277ec@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com> 
	<871vhk894b.fsf@vostro.rath.org>
	<ca471dc21001210909v13e264b8w655d5dbd927c6445@mail.gmail.com> 
	<d235cdc41001211218m5fa70a0dw8b9bf44520f277ec@mail.gmail.com>
Message-ID: <ca471dc21001211324u3ed6b110tc051e41f59da17ad@mail.gmail.com>

On Thu, Jan 21, 2010 at 12:18 PM, Holger Krekel <holger.krekel at gmail.com> wrote:
> On Thu, Jan 21, 2010 at 6:09 PM, Guido van Rossum <guido at python.org> wrote:
>> Ignoring many of the finer points brought up here, and putting
>> practicality before purity, I think having setUpClass and
>> tearDownClass methods is a great idea.
>>
>> While we're at it I would also recommend adding module-level setUp and
>> tearDown function -- Google's extension of pyunit implements these and
>> they are often handy for a variety of use cases.
>
> If going for that i'd rather like to see those named
> setup_class/teardown_class and setup_module/teardown_module like
> py.test and nose do for a long time now. When i first went for those i
> actually did so because i wanted to follow PEP8 ... But the stronger
> argument now is that it would be cool seeing some tool/approach
> convergence and unittest is the new kid on the setup-block there :)

Even PEP 8 admits that consistency within a module trumps theany
global style requirements. It's already setUp and tearDown, and any
new methods added should follow that camel-case convention.

-- 
--Guido van Rossum (python.org/~guido)


From markroddy at gmail.com  Thu Jan 21 22:30:02 2010
From: markroddy at gmail.com (Mark Roddy)
Date: Thu, 21 Jan 2010 16:30:02 -0500
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <1264103233.11747.91.camel@lifeless-64>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<1264037825.4254.12.camel@lifeless-64>
	<4B583DAE.3060209@voidspace.org.uk>
	<1264103233.11747.91.camel@lifeless-64>
Message-ID: <7930f1201001211330l583be743n2c8b9113c42bc732@mail.gmail.com>

On Thu, Jan 21, 2010 at 2:47 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On Thu, 2010-01-21 at 11:42 +0000, Michael Foord wrote:
>> On 21/01/2010 01:37, Robert Collins wrote:
>> > On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
>> >
>> >> Earlier this week on the Testing In Python list, there was a
>> >> discussion on how to execute a setup and/or teardown for a single test
>> >> class instead of for each test fixture on the class (see the 'setUp
>> >> and tearDown behavior' thread). ?I have had to deal with situation
>> >> myself before, and I am obviously not the only one (since I did not
>> >> initiate the thread). ?As such I'd like to propose adding a class
>> >> level setup and tear down method the unittest TestCase class.
>> >>
>> > I think that this is the wrong approach to the problem:
>> > ? - class scope for such fixtures drives large test classes, reduces
>> > flexability
>> >
>>
>> Agreed.
>>
>> > ? - doing it the rough way you suggest interacts in non obvious ways with
>> > test order randomisation
>> >
>>
>> Not currently in unittest, so not really an issue.
>
> Done by nearly every extended UI out there. I would hope that messing
> them up would count as an issue.
Agreed, messing/breaking with major unittest extensions should be
taken into serious consideration.

>
>> > ? - it also interacts with concurrent testing by having shared objects
>> > (classes) hold state in a way that is not introspectable by the test
>> > running framework.
>> >
>>
>> Again, unittest doesn't do concurrent testing out of the box.
>
> testtools includes a concurrent TestResult which buffers activities from
> different executing test cases and outputs them to a single regular
> TestResult. Using that run() on TestSuite can be redefined to start some
> threads, giving each one a helper TestResult forwarding to the caller
> supplied TestResult, and finally treat self.__iter__ as a queue to
> dispatch to each worker thread.
Again, this should be taken into account to be easy to accommodate.
For in process parallelization, I think the best approach would be to
make it easy to extend to add the locking that would be necessary
(thread extensions must already be doing a decent amount of locking to
accomplish this).  Multi-process/machine distribute testing will take
some more analysis to determine how to accomidate.

>
>> How are shared fixtures like this not introspectable?
>
> How can you tell if there is shared state when setUpClass is being used?
>
>> As I see it.
>>
>> Advantages of setupClass and teardownClass:
>>
>> * Simple to implement
>> * Simple to explain
>> * A commonly requested feature
>> * Provided by other test frameworks in and outside of Python
>
> Mmm. It is commonly requested, but I'm not sure I entirely agree about
> the simple claims.
Since the discussion thus far has focused almost entirely on the
merits of the feature (and not on how to implement) I don't think it's
fair to make an assumption either way on how easy or not it would be
to implement.  I already have most of the implementation mapped out in
my head (of which I'm sure there will be issues to be resolved) and
could have a first try patch before the end of the weekend, but I
thought there should be more input before I go running off and doing
so.

>
>> Disadvantages:
>>
>> * Can encourage a poor style of testing, including monolithic test classes
>
> * Can limit concurrency
>
>> > I'd much much much rather see e.g. testresources integrated into the
>> >
>>
>> How is testresources superior?
>
> It separates concerns, which allows introspection for test runners. This
> is significantly more flexible, particularly when combined with test
> parameterisation.
This is a definite upside (especially for for complicated resources)
but it also increases the complexity reducing the ease of
understanding the test case by introducing a host of new semamtics
people will not (at first) be familiar with.  By the fact that people
have frequently requested a class setup/teardown illustrates that this
has a semantic they can easily understand (since it's understood
without even existing at the moment).

For complex resources that require complicated tear down and/or the
ability to invalidate to signal a need to recreate the resource (I
believe this is refered to being 'dirty' in the framework) the loss of
ease of understanding clearly out ways the additional layer, but for
simple resources (of which I think the attached is a great example) I
don't see this trade off being paid off.

So I don't really see this as having to be an either or situation.  I
don't think people should have to understand a whole additional API in
order to achieve this behavior, and I also wouldn't advocate people
being force to fit a round peg in a square whole for extremely large
resources.

The way I see it we don't have two competing implementations, but two
possible new features: class oriented setup/teardown functionality (as
already found in JUnit and Nunit), and a resource abstraction layer
(possible testresources itself, possibly something inspired by the
JUnit Rules previously mentioned, or a combination of the two).



>
>> Can you demonstrate this - in particular
>> what is the simplest example? The simplest example of setupClass would be:
>>
>> class SomeTest(unittest.TestCase):
>> ? ? ?def setupClass(self):
>> ? ? ? ? ?... # setup shared fixture
>
> ? ? ? ? ? ? ? ? ? ? ^ this is buggy :)
> Its buggy because if it really does write to self, you now have what
> twisted did where all the test cases will be sharing one test object
> instance - which very frequently confuses and makes isolation between
> unrelated test state fragile.
>
> If its not writing to self, then you don't know what class object to
> write to (because you might be in a subclass) - that really should be a
> class method.
>
> And the fact that you have this confusion, as demonstrated here, is
> precisely why I don't agree about easy to implement and easy to
> understand.
>
>> > core allowing fixtures to be shared across test instances in a way that
>> > doesn't prohibit their use with concurrent testing, doesn't make it
>> > awkward to do it across multiple classes.
>>
>> Can you demonstrate how testresources solves these problems.
>
> Yes. resources are declared in a list, which allows the test framework,
> when parallelising, to group together tests that use the same resources
> when partitioning. Compare this with setUpClass where all tests in an
> inheritance hierarchy have to be grouped together (after excluding your
> base TestCase).
>
> You ask in another mail for a SSCE: here is one attached to this mail
> ('foo.py'). Run with 'python -m unittest foo'.
>
>> > I'm happy to make any
>> > [reasonable] changes (including license) to testresources to make it
>> > includable in the stdlib if that's of interest.
>> >
>>
>> That's not how contributing code to Python works. You need to provide a
>> signed contributor agreement to the PSF and then you specifically
>> license to the PSF any code you are contributing using one of a few
>> specific licenses. For new modules in the standard library you also need
>> to be willing to maintain the code.
>
> There is other code of mine in the standard library unittest module,
> done without a contributor agreement: so this is at best inconsistent. I
> mention license changes because that would be required to include it in
> the standard library (its currently a copyleft license) - and I'm not
> the sole author. Regardless, whatever process and actions are needed, I
> will do them, or arrange with other people where needed.
>
> -Rob
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


From holger.krekel at gmail.com  Thu Jan 21 23:02:45 2010
From: holger.krekel at gmail.com (Holger Krekel)
Date: Thu, 21 Jan 2010 23:02:45 +0100
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <ca471dc21001211324u3ed6b110tc051e41f59da17ad@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com> 
	<871vhk894b.fsf@vostro.rath.org>
	<ca471dc21001210909v13e264b8w655d5dbd927c6445@mail.gmail.com> 
	<d235cdc41001211218m5fa70a0dw8b9bf44520f277ec@mail.gmail.com> 
	<ca471dc21001211324u3ed6b110tc051e41f59da17ad@mail.gmail.com>
Message-ID: <d235cdc41001211402t20485302pa39acd347430c899@mail.gmail.com>

On Thu, Jan 21, 2010 at 10:24 PM, Guido van Rossum <guido at python.org> wrote:
> On Thu, Jan 21, 2010 at 12:18 PM, Holger Krekel <holger.krekel at gmail.com> wrote:
>> On Thu, Jan 21, 2010 at 6:09 PM, Guido van Rossum <guido at python.org> wrote:
>>> Ignoring many of the finer points brought up here, and putting
>>> practicality before purity, I think having setUpClass and
>>> tearDownClass methods is a great idea.
>>>
>>> While we're at it I would also recommend adding module-level setUp and
>>> tearDown function -- Google's extension of pyunit implements these and
>>> they are often handy for a variety of use cases.
>>
>> If going for that i'd rather like to see those named
>> setup_class/teardown_class and setup_module/teardown_module like
>> py.test and nose do for a long time now. When i first went for those i
>> actually did so because i wanted to follow PEP8 ... But the stronger
>> argument now is that it would be cool seeing some tool/approach
>> convergence and unittest is the new kid on the setup-block there :)
>
> Even PEP 8 admits that consistency within a module trumps theany
> global style requirements. It's already setUp and tearDown, and any
> new methods added should follow that camel-case convention.

I see the module consistency argument.  It'd still confuse a lot of
existing test suites and test writers out there which already use the
idiom. The PEP8 mention was for reference that the naming wasn't
chosen arbitrarily.  A more fundamental question maybe is if/how how
we want to work towards python test tool convergence on concepts and
naming.

best,
holger


> --
> --Guido van Rossum (python.org/~guido)
>


From ncoghlan at gmail.com  Thu Jan 21 23:05:04 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2010 08:05:04 +1000
Subject: [Python-ideas] Test Class setup and teardown in unittest
In-Reply-To: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
Message-ID: <4B58CF90.2030609@gmail.com>

On a slight tangent, I'd be happier if we could move away from the
setUp/tearDown model and move towards __enter__/__exit__. The existing
convention comes from a time before the with statement - extending it
further doesn't seem like it would be taking things in the right direction.

By allowing test suites to identify context managers that are invoked by
the test framework at various points, it allows the setup/teardown code
to be cleanly separated from the test classes themselves. E.g. the test
framework might promise to do the following for each test:

  with module_cm(test_instance): # However identified
    with class_cm(test_instance): # However identified
      with test_instance: # **
        test_instance.test_method()

(** Assuming the addition of "def __enter__(self): self.setUp()" and
"def __exit__(self, *args): self.tearDown()" to unittest.TestCase for
backwards compatibility)

Caching of expensive state on the module and class context managers
would then be the prerogative of those context managers rather than the
responsibility of the test cases themselves.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------


From jml at mumak.net  Fri Jan 22 01:31:32 2010
From: jml at mumak.net (Jonathan Lange)
Date: Fri, 22 Jan 2010 00:31:32 +0000 (UTC)
Subject: [Python-ideas] Test Class setup and teardown in unittest
References: <7930f1201001201638m6e2173ecs20b5a35c71051262@mail.gmail.com>
	<1264037825.4254.12.camel@lifeless-64>
	<7930f1201001201932o4b98a031i18d80a3265929fbc@mail.gmail.com>
Message-ID: <loom.20100122T012622-683@post.gmane.org>

Mark Roddy <markroddy at ...
> writes:
> On Wed, Jan 20, 2010 at 8:37 PM, Robert Collins
> <robertc at ...> wrote:
...
> > I'd much much much rather see e.g. testresources integrated into the
> > core allowing fixtures to be shared across test instances in a way that
> > doesn't prohibit their use with concurrent testing, doesn't make it
> > awkward to do it across multiple classes. I'm happy to make any
> > [reasonable] changes (including license) to testresources to make it
> > includable in the stdlib if that's of interest.
>
> A pretty good approach for a complicated setup, but in a simple case
> where you only need a line or two it seems like a lot of boiler plate
> to get the job done.  Though I'll look into this library a little more
> as I am not intimately familiar with it at the moment.

Yeah, it is a lot of boiler plate when you need only a couple of lines of code. 
There should be some convenience APIs on top of testresources. That way, you'll 
get the "nice and easy" along with support for test isolation, randomization and 
a declarative interface for those who prefer it.

jml



From smizrahi at redhat.com  Mon Jan 25 13:38:09 2010
From: smizrahi at redhat.com (Saggi Mizrahi)
Date: Mon, 25 Jan 2010 07:38:09 -0500 (EST)
Subject: [Python-ideas] Timout in subprocess
In-Reply-To: <470241977.56301264422802737.JavaMail.root@zmail05.collab.prod.int.phx2.redhat.com>
Message-ID: <575768102.56421264423089901.JavaMail.root@zmail05.collab.prod.int.phx2.redhat.com>

I have a need for a timeout feature in some of the subprocess.Popen methods.
I looked around and found this bug
http://bugs.python.org/issue5673

As it seems it the activity on the issue just stopped.

I would like to take the reins on the issue and tie the loose ends. The current state is "test needed".
If it is updated, what tests are missing for it to qualify?

What other comments do you have on the subject?


From redcomet87 at gmail.com  Tue Jan 26 22:22:35 2010
From: redcomet87 at gmail.com (Charles Solar)
Date: Tue, 26 Jan 2010 15:22:35 -0600
Subject: [Python-ideas] getpath differences Linux vs Windows
Message-ID: <708326ee1001261322pebc24fcp1feb695129b14449@mail.gmail.com>

I have been integrating python scripting support into my project and have
been impressed at how easy it has been with boost::python and the excelant
documentation.  I ran into a technical 'problem' today while writing up
Makefiles for my project.  It is not really a problem, just unexpected
behavior that I want to propose a change for.  But first, a little
background info..

I have been doing most of my development so far in windows and just started
porting my project to linux yesterday.  Because I want to make using this
project as easy as possible and because I wanted a python debug build, I
integrated python into my building process in windows.  This involved
creating a new visual studio project for both pythonlib, and python exe, as
well as a few of the libraries such as _ctypes, _elementtree,
_multiprocessing, and _socket.  All these projects compile with my project
and the python Lib directory from the install is kept in a folder that I
then copy into the final destination folder if necessary.  I did all this
work so that my users could check out my project, build it, and run it with
python without installing python.  Also I wanted a debug build, as mentioned
earlier.

Anyway, enough background, this works perfectly on windows because in
PC/getpathp.c when everything goes to hell and you cant figure out a path
you default to ./Lib
Because of this behavior, I can copy the Lib directory into the bin dir and
run python no problem.  I throw all the .py into $(BIN_DIR)/Lib and the C
libs in $(BIN_DIR)/DLLs and python is happy as a clam.

My trouble, and the reason for this email, is on linux the behavior is
different, and Modules/getpath.c does not default to ./Lib when everything
goes to hell, and I get
> ./python
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: can't initialize sys standard streams
ImportError: No module named encodings.utf_8
Aborted

I am going to spend the rest of my afternoon editing Modules/getpath.c and I
thought I would propose updating the non-windows getpath.c file to be more
in line with the windows one.  Or at least, have it take some pointers from
windows.

Thanks for your time
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100126/69156831/attachment.html>

From ncoghlan at gmail.com  Tue Jan 26 22:39:37 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Jan 2010 07:39:37 +1000
Subject: [Python-ideas] getpath differences Linux vs Windows
In-Reply-To: <708326ee1001261322pebc24fcp1feb695129b14449@mail.gmail.com>
References: <708326ee1001261322pebc24fcp1feb695129b14449@mail.gmail.com>
Message-ID: <4B5F6119.6000303@gmail.com>

Charles Solar wrote:
> I am going to spend the rest of my afternoon editing Modules/getpath.c
> and I thought I would propose updating the non-windows getpath.c file to
> be more in line with the windows one.  Or at least, have it take some
> pointers from windows.

Probably not going to fly - the current default behaviour follows
standard Linux idioms, which differ significantly from those of Windows.

The error message tells you how to fix the problem though: make sure
PYTHONHOME points to the right place in the environment seen by the
embedded interpreter.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------


From redcomet87 at gmail.com  Tue Jan 26 22:54:03 2010
From: redcomet87 at gmail.com (Charles Solar)
Date: Tue, 26 Jan 2010 15:54:03 -0600
Subject: [Python-ideas] getpath differences Linux vs Windows
In-Reply-To: <4B5F6119.6000303@gmail.com>
References: <708326ee1001261322pebc24fcp1feb695129b14449@mail.gmail.com> 
	<4B5F6119.6000303@gmail.com>
Message-ID: <708326ee1001261354j522beabfl97e5f956be49e2d9@mail.gmail.com>

Yea it most definitely works when I set PYTHONHOME and PYTHONPATH correctly.
But my goal is 0 environment monkeying.
I do not want my users to have to remember to define those environment
variables to get it running. My trouble is that the framework I am writing
up is going to be hard enough to convince these guys to use at all, and I
want no tricks or gotchas when I present the final solution.

This is just a suggestion since I found the windows behavior extremely
convenient.

Charles

On Tue, Jan 26, 2010 at 3:39 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Charles Solar wrote:
> > I am going to spend the rest of my afternoon editing Modules/getpath.c
> > and I thought I would propose updating the non-windows getpath.c file to
> > be more in line with the windows one.  Or at least, have it take some
> > pointers from windows.
>
> Probably not going to fly - the current default behaviour follows
> standard Linux idioms, which differ significantly from those of Windows.
>
> The error message tells you how to fix the problem though: make sure
> PYTHONHOME points to the right place in the environment seen by the
> embedded interpreter.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20100126/bb76f27a/attachment.html>

From fdrake at acm.org  Tue Jan 26 23:03:22 2010
From: fdrake at acm.org (Fred Drake)
Date: Tue, 26 Jan 2010 17:03:22 -0500
Subject: [Python-ideas] getpath differences Linux vs Windows
In-Reply-To: <708326ee1001261354j522beabfl97e5f956be49e2d9@mail.gmail.com>
References: <708326ee1001261322pebc24fcp1feb695129b14449@mail.gmail.com> 
	<4B5F6119.6000303@gmail.com>
	<708326ee1001261354j522beabfl97e5f956be49e2d9@mail.gmail.com>
Message-ID: <9cee7ab81001261403q72f77bd7xb9fc4a398aea1129@mail.gmail.com>

On Tue, Jan 26, 2010 at 4:54 PM, Charles Solar <redcomet87 at gmail.com> wrote:
> Yea it most definitely works when I set PYTHONHOME and PYTHONPATH correctly.
> But my goal is 0 environment monkeying.

Sounds like you could solve the problem by providing your own getpath.c.


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller


From solipsis at pitrou.net  Thu Jan 28 16:12:45 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Jan 2010 15:12:45 +0000 (UTC)
Subject: [Python-ideas] Timout in subprocess
References: <470241977.56301264422802737.JavaMail.root@zmail05.collab.prod.int.phx2.redhat.com>
	<575768102.56421264423089901.JavaMail.root@zmail05.collab.prod.int.phx2.redhat.com>
Message-ID: <loom.20100128T161006-928@post.gmane.org>

Saggi Mizrahi <smizrahi at ...> writes:
> 
> I have a need for a timeout feature in some of the subprocess.Popen methods.
> I looked around and found this bug
> http://bugs.python.org/issue5673
> 
> As it seems it the activity on the issue just stopped.
> 
> I would like to take the reins on the issue and tie the loose ends. The
current 
> state is "test needed".

This looks misleading, since there /are/ unit tests in the proposed patch.
I've switched the state to "patch review".
It merely needs someone to review the patch for errors/issues/etc., and then a
committer to ok it (but the reviewer needn't be a committer, though). Don't
hesitate to chime in on the issue.

Regards

Antoine.