From brian at sweetapp.com  Fri Nov  6 23:35:35 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 7 Nov 2009 09:35:35 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
Message-ID: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>

Hey all,

I'd like to propose adding a module/package to Python that makes it  
easy to parallelize arbitrary function calls.

I recently wrote a solution for the use case of parallelizing network  
copies and RPC using threads without forcing the user to explicitly  
creating thread pools, work queues, etc.

I have a concrete implementation that I'll describe below but I'd be  
happy to hear about other strategies!

The basic idea is to implement an asynchronous execution method  
patterned heavily on java.util.concurrent (but less lame because  
Python has functions as first-class objects).  Here is a fairly  
advanced example:

import futures
import functools
import urllib.request

URLS = [
    'http://www.foxnews.com/',
    'http://www.cnn.com/',
    'http://europe.wsj.com/',
    'http://www.bbc.co.uk/',
    'http://some-made-up-domain.com/']

def load_url(url, timeout):
    return urllib.request.urlopen(url, timeout=timeout).read()

# Use a thread pool with 5 threads to download the URLs. Using a pool
# of processes would involve changing the initialization to:
#   with futures.ProcessPoolExecutor(max_processes=5) as executor
with futures.ThreadPoolExecutor(max_threads=5) as executor:
    future_list = executor.run_to_futures(
        [functools.partial(load_url, url, 30) for url in URLS])

# Check the results of each future.
for url, future in zip(URLS, future_list):
    if future.exception() is not None:
        print('%r generated an exception: %s' % (url,  
future.exception()))
    else:
        print('%r page is %d bytes' % (url, len(future.result())))

In this example, executor.run_to_futures() returns only when every url  
has been retrieved but it is possible to return immediately, on the  
first completion or on the first failure depending on the desired work  
pattern.

The complete docs are here:
http://sweetapp.com/futures/

A draft PEP is here:
http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt

And the code is here:
http://pypi.python.org/pypi/futures3/

All feedback appreciated!

Cheers,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091107/0e11aa5b/attachment.htm>

From benjamin at python.org  Sat Nov  7 00:00:53 2009
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 6 Nov 2009 17:00:53 -0600
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
Message-ID: <1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>

2009/11/6 Brian Quinlan <brian at sweetapp.com>:
> Hey all,
>
> I'd like to propose adding a module/package to Python that makes it easy to
> parallelize arbitrary function calls.

Your package is less than a year old. It's at version 0.1. We do not
normally accept packages into the stdlib until they are considered
mature and best of breed in the community. And then only if we feel a
need for its functionality in the stdlib.



-- 
Regards,
Benjamin

From guido at python.org  Sat Nov  7 00:10:57 2009
From: guido at python.org (Guido van Rossum)
Date: Fri, 6 Nov 2009 15:10:57 -0800
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com> 
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
Message-ID: <ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>

On Fri, Nov 6, 2009 at 3:00 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2009/11/6 Brian Quinlan <brian at sweetapp.com>:
>> Hey all,
>>
>> I'd like to propose adding a module/package to Python that makes it easy to
>> parallelize arbitrary function calls.
>
> Your package is less than a year old. It's at version 0.1. We do not
> normally accept packages into the stdlib until they are considered
> mature and best of breed in the community. And then only if we feel a
> need for its functionality in the stdlib.

In Brian's defense, Brett Cannon referred him here (wearing his PEP
editor hat -- and I stood by and watched :-). But yeah, it's probably
a little early to start proposing this as a stdlib addition. Maybe
python-ideas? I do think that having a standard API for this kind of
stuff makes some kind of sense. (And no, I don't have any experiencing
using Brian's code either. :-)

-- 
--Guido van Rossum (python.org/~guido)

From jnoller at gmail.com  Sat Nov  7 00:57:35 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 6 Nov 2009 18:57:35 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
Message-ID: <4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>

On Fri, Nov 6, 2009 at 6:10 PM, Guido van Rossum <guido at python.org> wrote:
> On Fri, Nov 6, 2009 at 3:00 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> 2009/11/6 Brian Quinlan <brian at sweetapp.com>:
>>> Hey all,
>>>
>>> I'd like to propose adding a module/package to Python that makes it easy to
>>> parallelize arbitrary function calls.
>>
>> Your package is less than a year old. It's at version 0.1. We do not
>> normally accept packages into the stdlib until they are considered
>> mature and best of breed in the community. And then only if we feel a
>> need for its functionality in the stdlib.
>
> In Brian's defense, Brett Cannon referred him here (wearing his PEP
> editor hat -- and I stood by and watched :-). But yeah, it's probably
> a little early to start proposing this as a stdlib addition. Maybe
> python-ideas? I do think that having a standard API for this kind of
> stuff makes some kind of sense. (And no, I don't have any experiencing
> using Brian's code either. :-)
>

I'm +1 adding something which provides this functionality to the
standard library, in fact it's been on my todo list since adding
multiprocessing. I actually suggested something like this in private
discussions around possible additions to jython for additional
concurrency constructs.

That being said, I'll try to carve time off soon to dig into the pep
and the implementation.

jesse

From solipsis at pitrou.net  Sat Nov  7 00:59:26 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 00:59:26 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
Message-ID: <1257551966.12371.0.camel@localhost>


> I'm +1 adding something which provides this functionality to the
> standard library, in fact it's been on my todo list since adding
> multiprocessing. I actually suggested something like this in private
> discussions around possible additions to jython for additional
> concurrency constructs.
> 
> That being said, I'll try to carve time off soon to dig into the pep
> and the implementation.

Have you thought at looking at Twisted and Deferred objects?
What's the point of starting from scratch when an existing project has a
well-known and well-established API?



From jnoller at gmail.com  Sat Nov  7 01:09:20 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 6 Nov 2009 19:09:20 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257551966.12371.0.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
Message-ID: <4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>

On Fri, Nov 6, 2009 at 6:59 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> I'm +1 adding something which provides this functionality to the
>> standard library, in fact it's been on my todo list since adding
>> multiprocessing. I actually suggested something like this in private
>> discussions around possible additions to jython for additional
>> concurrency constructs.
>>
>> That being said, I'll try to carve time off soon to dig into the pep
>> and the implementation.
>
> Have you thought at looking at Twisted and Deferred objects?
> What's the point of starting from scratch when an existing project has a
> well-known and well-established API?

Personally; I have. I don't prefer them, but that's my personal taste.

From fwierzbicki at gmail.com  Sat Nov  7 01:09:53 2009
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Fri, 6 Nov 2009 19:09:53 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
Message-ID: <4dab5f760911061609n48799a62n9ece3c64f4a60fc3@mail.gmail.com>

On Fri, Nov 6, 2009 at 5:35 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
>
> I'd like to propose adding a module/package to Python that makes it easy to
> parallelize arbitrary function calls.
> I recently wrote a solution for the use case of parallelizing network copies
> and RPC using threads without forcing the user to explicitly creating thread
> pools, work queues, etc.
> I have a concrete implementation that I'll describe below but I'd be happy
> to hear about other strategies!
> The basic idea is to implement an asynchronous execution method patterned
> heavily on java.util.concurrent (but less lame because Python has functions
> as first-class objects). ?Here is a fairly advanced example:
> import futures
> import functools
> import urllib.request
>
> URLS = [
> ???'http://www.foxnews.com/',
> ???'http://www.cnn.com/',
> ???'http://europe.wsj.com/',
> ???'http://www.bbc.co.uk/',
> ???'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
> ???return urllib.request.urlopen(url, timeout=timeout).read()
>
> # Use a thread pool with 5 threads to download the URLs. Using a pool
> # of processes would involve changing the initialization to:
> # ??with futures.ProcessPoolExecutor(max_processes=5) as executor
> with futures.ThreadPoolExecutor(max_threads=5) as executor:
> ???future_list = executor.run_to_futures(
> ???????[functools.partial(load_url, url, 30) for url in URLS])
>
> # Check the results of each future.
> for url, future in zip(URLS, future_list):
> ???if future.exception() is not None:
> ???????print('%r generated an exception: %s' % (url, future.exception()))
> ???else:
> ???????print('%r page is %d bytes' % (url, len(future.result())))
>
> In this example, executor.run_to_futures() returns only when every url has
> been retrieved but it is possible to return immediately, on the first
> completion or on the first failure depending on the desired work pattern.
>
> The complete docs are here:
> http://sweetapp.com/futures/
>
> A draft PEP is here:
> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>
> And the code is here:
> http://pypi.python.org/pypi/futures3/
Since this is modeled on java.util.concurrent Futures, I bet it would
be pretty straightforward to implement in Jython... I'm going to be a
little swamped with job hunt related stuff for a bit, but after that I
love to have a talk about this.

-Frank

From solipsis at pitrou.net  Sat Nov  7 01:23:53 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 01:23:53 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
Message-ID: <1257553433.13956.5.camel@localhost>

Le vendredi 06 novembre 2009 ? 19:09 -0500, Jesse Noller a ?crit :
> >
> > Have you thought at looking at Twisted and Deferred objects?
> > What's the point of starting from scratch when an existing project has a
> > well-known and well-established API?
> 
> Personally; I have. I don't prefer them, but that's my personal taste.

The Twisted Deferred object is a quasi-standard when doing asynchronous
programming in Python.
The only seriously competing approach AFAICT is generator-based syntaxes
(which Twisted also provides on top of Deferreds by the way).

Inventing a third idiom just for taste reasons doesn't sound very
reasonable. Of course, if the third idiom turns out *really* better in
terms of expressiveness then why not.



From jnoller at gmail.com  Sat Nov  7 01:32:04 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 6 Nov 2009 19:32:04 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257553433.13956.5.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
Message-ID: <4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>

On Fri, Nov 6, 2009 at 7:23 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le vendredi 06 novembre 2009 ? 19:09 -0500, Jesse Noller a ?crit :
>> >
>> > Have you thought at looking at Twisted and Deferred objects?
>> > What's the point of starting from scratch when an existing project has a
>> > well-known and well-established API?
>>
>> Personally; I have. I don't prefer them, but that's my personal taste.
>
> The Twisted Deferred object is a quasi-standard when doing asynchronous
> programming in Python.
> The only seriously competing approach AFAICT is generator-based syntaxes
> (which Twisted also provides on top of Deferreds by the way).
>
> Inventing a third idiom just for taste reasons doesn't sound very
> reasonable. Of course, if the third idiom turns out *really* better in
> terms of expressiveness then why not.

I just looked at the API and idioms outlined and they seem clean and
simpler than other approaches. I also don't personally consider this
to be a "module" in it's own right, it's a light(ish) API on top of
threading and multiprocessing.

I'd gladly add this API (or some flavor of it) into multiprocessing,
as I do see it as a logical extension to the API itself. Just like
multiprocessing has pool/etc

jesse

From solipsis at pitrou.net  Sat Nov  7 01:35:45 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 01:35:45 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
Message-ID: <1257554145.13956.7.camel@localhost>


> I'd gladly add this API (or some flavor of it) into multiprocessing,
> as I do see it as a logical extension to the API itself. Just like
> multiprocessing has pool/etc

I agree that it can make sense as an additional facility to run parallel
tasks (like the shortcut functions in the subprocess or commands
modules). It should probably stay minimal then.

Regards

Antoine.



From jnoller at gmail.com  Sat Nov  7 01:41:30 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 6 Nov 2009 19:41:30 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257554145.13956.7.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
Message-ID: <4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>

On Fri, Nov 6, 2009 at 7:35 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> I'd gladly add this API (or some flavor of it) into multiprocessing,
>> as I do see it as a logical extension to the API itself. Just like
>> multiprocessing has pool/etc
>
> I agree that it can make sense as an additional facility to run parallel
> tasks (like the shortcut functions in the subprocess or commands
> modules). It should probably stay minimal then.
>
> Regards
>
> Antoine.
>

Looking at the code dist; processing.py is 338 lines, _base.py is 558
and thread.py is 153; it really is a light API on top of
threading/multiprocessing. I think most of the line count are doc
strings. It's very clean/simple at first glance.

Could I also point out that this would be a might make a nice py3k carrot? ;)

jesse

From fwierzbicki at gmail.com  Sat Nov  7 01:42:14 2009
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Fri, 6 Nov 2009 19:42:14 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
Message-ID: <4dab5f760911061642k7989197dva4fa9a33eb955df@mail.gmail.com>

On Fri, Nov 6, 2009 at 7:32 PM, Jesse Noller <jnoller at gmail.com> wrote:
> I'd gladly add this API (or some flavor of it) into multiprocessing,
> as I do see it as a logical extension to the API itself. Just like
> multiprocessing has pool/etc
We've spoken before about how the main part of multiprocessing (that
is using multiple processes to emulate threading) doesn't make sense
with Jython (since there is no GIL to work around, and Jython
processes are *much* more expensive vs. CPython) -- but some of these
added APIs probably do make sense on Jython. Do you think they need a
different package name in these cases - or should Jython just
partially implement multiprocessing... or maybe you have another
suggestion?

-Frank

From fwierzbicki at gmail.com  Sat Nov  7 01:43:37 2009
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Fri, 6 Nov 2009 19:43:37 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
Message-ID: <4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>

On Fri, Nov 6, 2009 at 7:41 PM, Jesse Noller <jnoller at gmail.com> wrote:

> Looking at the code dist; processing.py is 338 lines, _base.py is 558
> and thread.py is 153; it really is a light API on top of
> threading/multiprocessing. I think most of the line count are doc
> strings. It's very clean/simple at first glance.
Ah this probably answers my question, Jython would just support the
threading side.

-Frank

From jnoller at gmail.com  Sat Nov  7 01:44:46 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 6 Nov 2009 19:44:46 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
Message-ID: <4222a8490911061644u72d5f3fcl8dbbf394e0c0ba0d@mail.gmail.com>

On Fri, Nov 6, 2009 at 5:35 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
>
> I'd like to propose adding a module/package to Python that makes it easy to
> parallelize arbitrary function calls.
> I recently wrote a solution for the use case of parallelizing network copies
> and RPC using threads without forcing the user to explicitly creating thread
> pools, work queues, etc.
> I have a concrete implementation that I'll describe below but I'd be happy
> to hear about other strategies!
> The basic idea is to implement an asynchronous execution method patterned
> heavily on java.util.concurrent (but less lame because Python has functions
> as first-class objects). ?Here is a fairly advanced example:
> import futures
> import functools
> import urllib.request
>
> URLS = [
> ???'http://www.foxnews.com/',
> ???'http://www.cnn.com/',
> ???'http://europe.wsj.com/',
> ???'http://www.bbc.co.uk/',
> ???'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
> ???return urllib.request.urlopen(url, timeout=timeout).read()
>
> # Use a thread pool with 5 threads to download the URLs. Using a pool
> # of processes would involve changing the initialization to:
> # ??with futures.ProcessPoolExecutor(max_processes=5) as executor
> with futures.ThreadPoolExecutor(max_threads=5) as executor:
> ???future_list = executor.run_to_futures(
> ???????[functools.partial(load_url, url, 30) for url in URLS])
>
> # Check the results of each future.
> for url, future in zip(URLS, future_list):
> ???if future.exception() is not None:
> ???????print('%r generated an exception: %s' % (url, future.exception()))
> ???else:
> ???????print('%r page is %d bytes' % (url, len(future.result())))
>
> In this example, executor.run_to_futures() returns only when every url has
> been retrieved but it is possible to return immediately, on the first
> completion or on the first failure depending on the desired work pattern.
>
> The complete docs are here:
> http://sweetapp.com/futures/
>
> A draft PEP is here:
> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>
> And the code is here:
> http://pypi.python.org/pypi/futures3/
>
> All feedback appreciated!
>
> Cheers,
> Brian
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>
>

Hey brian; a few things - I think the code looks good, and your docs
are really good so far; but I'd personally like to see tests and more
examples within the docs. I obviously like the concept/idea, but tests
are a must, and more examples in the docs would make it a lot better.

jesse

From jnoller at gmail.com  Sat Nov  7 01:45:50 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 6 Nov 2009 19:45:50 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
Message-ID: <4222a8490911061645l5694a367h420b048c4914d7ee@mail.gmail.com>

On Fri, Nov 6, 2009 at 7:43 PM, Frank Wierzbicki <fwierzbicki at gmail.com> wrote:
> On Fri, Nov 6, 2009 at 7:41 PM, Jesse Noller <jnoller at gmail.com> wrote:
>
>> Looking at the code dist; processing.py is 338 lines, _base.py is 558
>> and thread.py is 153; it really is a light API on top of
>> threading/multiprocessing. I think most of the line count are doc
>> strings. It's very clean/simple at first glance.
> Ah this probably answers my question, Jython would just support the
> threading side.
>
> -Frank
>

Yeah, jython would just support the threading side. And it should
"just work" (faster! :))

From brett at python.org  Sat Nov  7 02:12:00 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 6 Nov 2009 17:12:00 -0800
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
Message-ID: <bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>

[I am going to be lazy and mass reply to people with a top-post; you
can burn an effigy of me later]

In response to Guido, yes, I sent Brian here with my PEP editor hat
on. Unfortunately the hat was on rather firmly and I totally forgot to
check to see how old the code is. Yet another reason I need to get the
Hg conversion done so I can start writing a "Adding to the Stdlib"
PEP.

To Antoine's Twisted comment, I don't see a direct comparison. From my
understanding Twisted's Deferred objects are ways to have callbacks
executed once an async event occurs, not to help execute code
concurrently. So I don't see enough similarity to discount the idea
Brian is pushing forward.

As for Benjamin's one year reminder, since I don't see this happening
in time for Python 3.2a1 it isn't a major worry right now. That means
this won't land until Python 3.3, which gives everyone time until that
alpha which would probably be June 2011. So that would give Brian (and
Jesse if he gets involved like it sounds he will) time to work out the
API, get public feedback, and get the code checked in. The only way I
would feel comfortable letting this in past 3.2a1 would be if it
landed before 3.2a4, Jesse personally shuttled it through, and started
work in it now and REALLY pushed it. But as he knows from personal
experience, rushing a module into the stdlib can bite you in the ass.
=)

But I really do like the idea. With java.util.concurrent and Grand
Central Dispatch out there, I think it shows some demand for a way to
easily abstract out concurrency management stuff and leave it up to a
library.

-Brett

On Fri, Nov 6, 2009 at 14:35, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
>
> I'd like to propose adding a module/package to Python that makes it easy to
> parallelize arbitrary function calls.
> I recently wrote a solution for the use case of parallelizing network copies
> and RPC using threads without forcing the user to explicitly creating thread
> pools, work queues, etc.
> I have a concrete implementation that I'll describe below but I'd be happy
> to hear about other strategies!
> The basic idea is to implement an asynchronous execution method patterned
> heavily on java.util.concurrent (but less lame because Python has functions
> as first-class objects). ?Here is a fairly advanced example:
> import futures
> import functools
> import urllib.request
>
> URLS = [
> ???'http://www.foxnews.com/',
> ???'http://www.cnn.com/',
> ???'http://europe.wsj.com/',
> ???'http://www.bbc.co.uk/',
> ???'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
> ???return urllib.request.urlopen(url, timeout=timeout).read()
>
> # Use a thread pool with 5 threads to download the URLs. Using a pool
> # of processes would involve changing the initialization to:
> # ??with futures.ProcessPoolExecutor(max_processes=5) as executor
> with futures.ThreadPoolExecutor(max_threads=5) as executor:
> ???future_list = executor.run_to_futures(
> ???????[functools.partial(load_url, url, 30) for url in URLS])
>
> # Check the results of each future.
> for url, future in zip(URLS, future_list):
> ???if future.exception() is not None:
> ???????print('%r generated an exception: %s' % (url, future.exception()))
> ???else:
> ???????print('%r page is %d bytes' % (url, len(future.result())))
>
> In this example, executor.run_to_futures() returns only when every url has
> been retrieved but it is possible to return immediately, on the first
> completion or on the first failure depending on the desired work pattern.
>
> The complete docs are here:
> http://sweetapp.com/futures/
>
> A draft PEP is here:
> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>
> And the code is here:
> http://pypi.python.org/pypi/futures3/
>
> All feedback appreciated!
>
> Cheers,
> Brian
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>
>

From solipsis at pitrou.net  Sat Nov  7 02:40:04 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 02:40:04 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
Message-ID: <1257558004.13956.24.camel@localhost>


> To Antoine's Twisted comment, I don't see a direct comparison. From my
> understanding Twisted's Deferred objects are ways to have callbacks
> executed once an async event occurs, not to help execute code
> concurrently.

Well, waiting for concurrently executing code to terminate *is* a case
of waiting for an async event to happen.

Deferred objects are a generic mechanism to chain reactions to
termination or failure of code. Whether the event your Deferred reacts
to is "async" or not is really a matter of how you use it (and of how
you define "async" -- perhaps you meant "I/O" but Deferreds are not
specialized for I/O).

Regards

Antoine.



From brian at sweetapp.com  Sat Nov  7 02:42:37 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 7 Nov 2009 12:42:37 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
Message-ID: <4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>

On Nov 7, 2009, at 10:00 AM, Benjamin Peterson wrote:

> 2009/11/6 Brian Quinlan <brian at sweetapp.com>:
>> Hey all,
>>
>> I'd like to propose adding a module/package to Python that makes it  
>> easy to
>> parallelize arbitrary function calls.
>
> Your package is less than a year old. It's at version 0.1. We do not
> normally accept packages into the stdlib until they are considered
> mature and best of breed in the community. And then only if we feel a
> need for its functionality in the stdlib.

Hey Benjamin,

You were probably looking at an old version - the latest official  
release is 1.0. But that is, of course, totally arbitrary :-)

I think that building a community around this package would be fairly  
hard because it is fairly easy to replicate its functionality to solve  
particular problems.

I don't think that should be a barrier for inclusion of new utility  
modules though. Consider urlparse, StringIO, etc. They are very useful  
but I doubt that they would have had much usage if they had been  
released as third-party modules - everyone would have simply coded the  
sub-set of the functionality that they needed to address the problem  
at hand.

Cheers,
Brian


From jnoller at gmail.com  Sat Nov  7 03:12:14 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 6 Nov 2009 21:12:14 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
Message-ID: <4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>

On Fri, Nov 6, 2009 at 8:42 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> On Nov 7, 2009, at 10:00 AM, Benjamin Peterson wrote:
>
>> 2009/11/6 Brian Quinlan <brian at sweetapp.com>:
>>>
>>> Hey all,
>>>
>>> I'd like to propose adding a module/package to Python that makes it easy
>>> to
>>> parallelize arbitrary function calls.
>>
>> Your package is less than a year old. It's at version 0.1. We do not
>> normally accept packages into the stdlib until they are considered
>> mature and best of breed in the community. And then only if we feel a
>> need for its functionality in the stdlib.
>
> Hey Benjamin,
>
> You were probably looking at an old version - the latest official release is
> 1.0. But that is, of course, totally arbitrary :-)
>
> I think that building a community around this package would be fairly hard
> because it is fairly easy to replicate its functionality to solve particular
> problems.
>
> I don't think that should be a barrier for inclusion of new utility modules
> though. Consider urlparse, StringIO, etc. They are very useful but I doubt
> that they would have had much usage if they had been released as third-party
> modules - everyone would have simply coded the sub-set of the functionality
> that they needed to address the problem at hand.
>
> Cheers,
> Brian

I obviously tend to agree with Brian; I know I've personally had to
implement things like this plenty of times, it's relatively simple
once you do it once or twice. This is a nice bit of syntactic sugar on
top of the threading/multiprocessing modules.

jesse

From jnoller at gmail.com  Sat Nov  7 03:20:43 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 6 Nov 2009 21:20:43 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
Message-ID: <4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>

On Fri, Nov 6, 2009 at 8:12 PM, Brett Cannon <brett at python.org> wrote:
> [I am going to be lazy and mass reply to people with a top-post; you
> can burn an effigy of me later]
>
> In response to Guido, yes, I sent Brian here with my PEP editor hat
> on. Unfortunately the hat was on rather firmly and I totally forgot to
> check to see how old the code is. Yet another reason I need to get the
> Hg conversion done so I can start writing a "Adding to the Stdlib"
> PEP.

Want to start the skeleton and we can fill in the blanks as needed? :)
I vote for "The Getting Your Pony into Python" PEP.

> To Antoine's Twisted comment, I don't see a direct comparison. From my
> understanding Twisted's Deferred objects are ways to have callbacks
> executed once an async event occurs, not to help execute code
> concurrently. So I don't see enough similarity to discount the idea
> Brian is pushing forward.

It's also significantly less code than many existing solutions; it's
really light when compared to those as well.

> As for Benjamin's one year reminder, since I don't see this happening
> in time for Python 3.2a1 it isn't a major worry right now. That means
> this won't land until Python 3.3, which gives everyone time until that
> alpha which would probably be June 2011. So that would give Brian (and
> Jesse if he gets involved like it sounds he will) time to work out the
> API, get public feedback, and get the code checked in. The only way I
> would feel comfortable letting this in past 3.2a1 would be if it
> landed before 3.2a4, Jesse personally shuttled it through, and started
> work in it now and REALLY pushed it. But as he knows from personal
> experience, rushing a module into the stdlib can bite you in the ass.
> =)

I don't want to rush it; I think with tests and more examples and a
run through the python-ideas and -dev spin cycles, it's small enough
(and consumable enough) to get into the 3.2 stream. But just to
stress, I'm really not interested in rushing it, I just don't see it
as something as "big" and potentially as big a disruption as
multiprocessing was.

Again, I consider this more akin to the addition of the same API just
added to the threading or multiprocessing modules. My pony list for
both of those have been built in producer/consumer pools, things like
this, more context manager things, etc. Build on the modules to
implement common constructs.

> But I really do like the idea. With java.util.concurrent and Grand
> Central Dispatch out there, I think it shows some demand for a way to
> easily abstract out concurrency management stuff and leave it up to a
> library.

Making it eas(y)(ier), safer and simple would be nice. There's plenty
of tried and true patterns out there that just make sense and that
plenty of people implement so frequently that they just sort of scream
(to me, I might be in the minority) for inclusion in the relevant
modules.

Who knows, this could be the start of python.concurrent :)

jesse

From brian at sweetapp.com  Sat Nov  7 06:01:25 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 7 Nov 2009 16:01:25 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257551966.12371.0.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
Message-ID: <0F9BB6A5-4DD4-4FC7-BC94-C23873B0EAD1@sweetapp.com>


On Nov 7, 2009, at 10:59 AM, Antoine Pitrou wrote:

>
>> I'm +1 adding something which provides this functionality to the
>> standard library, in fact it's been on my todo list since adding
>> multiprocessing. I actually suggested something like this in private
>> discussions around possible additions to jython for additional
>> concurrency constructs.
>>
>> That being said, I'll try to carve time off soon to dig into the pep
>> and the implementation.
>
> Have you thought at looking at Twisted and Deferred objects?
> What's the point of starting from scratch when an existing project  
> has a
> well-known and well-established API?

Hey Antoine,

I'm not an expert in Twisted but I think that Deferreds are looking to  
solve a slightly different problem.

I'm trying to solve the problem of easily parallelizing *synchronous*  
tasks (using threads of processes). My original use case looked  
something like this:

for source_path in source_paths:
     for destination_path in destination_paths:
         DoBlockingNetworkCopy(source_path, destination_path)

Which, in my proposed package, could be written as:

copies = []
for source_path in source_paths:
     for destination_path in destination_paths:
         copies.append(partial(DoBlockingNetworkCopy, source_path,  
destination_path))

with ThreadPoolExecutor(20) as e:
     e.run_to_futures(copies, return_when=FIRST_EXCEPTION)

Twisted Deferreds represent a fundamentally *asynchronous* operation  
(e.g. non-blocking I/O).

Cheers,
Brian

From brian at sweetapp.com  Sat Nov  7 06:11:27 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 7 Nov 2009 16:11:27 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911061644u72d5f3fcl8dbbf394e0c0ba0d@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<4222a8490911061644u72d5f3fcl8dbbf394e0c0ba0d@mail.gmail.com>
Message-ID: <B4973353-893C-4314-ACE4-145A9BDCA1E9@sweetapp.com>


On Nov 7, 2009, at 11:44 AM, Jesse Noller wrote:

> On Fri, Nov 6, 2009 at 5:35 PM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> Hey all,
>>
>> I'd like to propose adding a module/package to Python that makes it  
>> easy to
>> parallelize arbitrary function calls.
>> I recently wrote a solution for the use case of parallelizing  
>> network copies
>> and RPC using threads without forcing the user to explicitly  
>> creating thread
>> pools, work queues, etc.
>> I have a concrete implementation that I'll describe below but I'd  
>> be happy
>> to hear about other strategies!
>> The basic idea is to implement an asynchronous execution method  
>> patterned
>> heavily on java.util.concurrent (but less lame because Python has  
>> functions
>> as first-class objects).  Here is a fairly advanced example:
>> import futures
>> import functools
>> import urllib.request
>>
>> URLS = [
>>    'http://www.foxnews.com/',
>>    'http://www.cnn.com/',
>>    'http://europe.wsj.com/',
>>    'http://www.bbc.co.uk/',
>>    'http://some-made-up-domain.com/']
>>
>> def load_url(url, timeout):
>>    return urllib.request.urlopen(url, timeout=timeout).read()
>>
>> # Use a thread pool with 5 threads to download the URLs. Using a pool
>> # of processes would involve changing the initialization to:
>> #   with futures.ProcessPoolExecutor(max_processes=5) as executor
>> with futures.ThreadPoolExecutor(max_threads=5) as executor:
>>    future_list = executor.run_to_futures(
>>        [functools.partial(load_url, url, 30) for url in URLS])
>>
>> # Check the results of each future.
>> for url, future in zip(URLS, future_list):
>>    if future.exception() is not None:
>>        print('%r generated an exception: %s' % (url,  
>> future.exception()))
>>    else:
>>        print('%r page is %d bytes' % (url, len(future.result())))
>>
>> In this example, executor.run_to_futures() returns only when every  
>> url has
>> been retrieved but it is possible to return immediately, on the first
>> completion or on the first failure depending on the desired work  
>> pattern.
>>
>> The complete docs are here:
>> http://sweetapp.com/futures/
>>
>> A draft PEP is here:
>> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>>
>> And the code is here:
>> http://pypi.python.org/pypi/futures3/
>>
>> All feedback appreciated!
>>
>> Cheers,
>> Brian
>> _______________________________________________
>> stdlib-sig mailing list
>> stdlib-sig at python.org
>> http://mail.python.org/mailman/listinfo/stdlib-sig
>>
>>
>
> Hey brian; a few things - I think the code looks good, and your docs
> are really good so far;

Cool, thanks.

> but I'd personally like to see tests

The tests live here:
http://code.google.com/p/pythonfutures/source/browse/trunk/python3/test_futures.py

Last time I measured, there was 100% test coverage (it's a crappy  
metric but an easy one) but I'm not completely happy with them because  
they use time.sleep() in several places to try to provoke deadlock.

> and more
> examples within the docs. I obviously like the concept/idea, but tests
> are a must, and more examples in the docs would make it a lot better.


More examples in the docs sound good, I'll work on that.

Cheers,
Brian


From brian at sweetapp.com  Sat Nov  7 06:29:43 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 7 Nov 2009 16:29:43 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<ca471dc20911061510g1f7a3391j5be2c1dc6f7859ac@mail.gmail.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
Message-ID: <EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>

On Nov 7, 2009, at 11:43 AM, Frank Wierzbicki wrote:

> On Fri, Nov 6, 2009 at 7:41 PM, Jesse Noller <jnoller at gmail.com>  
> wrote:
>
>> Looking at the code dist; processing.py is 338 lines, _base.py is 558
>> and thread.py is 153; it really is a light API on top of
>> threading/multiprocessing. I think most of the line count are doc
>> strings. It's very clean/simple at first glance.
> Ah this probably answers my question, Jython would just support the
> threading side.

Consolidating the interfaces between threading and multiprocessing was  
a secondary design goal.

Right now multiprocessing is ahead of threading in terms of features.  
Pool.map() in particular is a pretty powerful idiom that has no  
equivalent in threading.

The Executor ABC provides a consistent interface to both  
(with .map() :-)).

Cheers,
Brian


From brian at sweetapp.com  Sat Nov  7 08:57:05 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 7 Nov 2009 18:57:05 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <da776a8c0911062337p4ed5a744wed662ba4d3069c58@mail.gmail.com>
References: <F543DEDB-3099-4A81-9DE0-5D842A0B2B3D@sweetapp.com>
	<da776a8c0911062337p4ed5a744wed662ba4d3069c58@mail.gmail.com>
Message-ID: <77998188-FAA3-448E-BE5F-D82E7B7B30AE@sweetapp.com>

On 7 Nov 2009, at 18:37, inhahe wrote:

> i don't understand this at all
> i hope you will provide a basic explanation of what we're doing for  
> us simpletons :P
>

Sure, but next time could you ask a more precise question? ;-)

# Create a pool of threads to execute calls.
with futures.ThreadPoolExecutor(max_threads=5) as executor:
   # Schedule the given calls to run using the thread pool created  
above and
   # return a FutureList (a list of Futures + some convenience  
methods). Called
   # without the "return_when" argument, waits until all calls are  
complete.
   futures_list = executor.run_to_futures(...)

# Iterate through every Future. A Future represents one of the  
asynchronous
# calls.
for url, future in zip(URLS, future_list):
   # Check if the call raised an exception.
   if future.exception() is not None:
     # Print the exception
     print('%r generated an exception: %s' % (url, future.exception()))
   else:
     # The call returned successfully so future.result() contains the  
return
     # value.
     print('%r page is %d bytes' % (url, len(future.result())))

Cheers,
Brian

> import futures
> import functools
> import urllib.request
>
> URLS = [
>   'http://www.foxnews.com/',
>   'http://www.cnn.com/',
>   'http://europe.wsj.com/',
>   'http://www.bbc.co.uk/',
>   'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
>   return urllib.request.urlopen(url, timeout=timeout).read()
>
> # Use a thread pool with 5 threads to download the URLs. Using a pool
> # of processes would involve changing the initialization to:
> #   with futures.ProcessPoolExecutor(max_processes=5) as executor
> with futures.ThreadPoolExecutor(max_threads=5) as executor:

>   future_list = executor.run_to_futures(
>       [functools.partial(load_url, url, 30) for url in URLS])
>
> # Check the results of each future.
> for url, future in zip(URLS, future_list):
>   if future.exception() is not None:
>       print('%r generated an exception: %s' % (url,  
> future.exception()))
>   else:
>       print('%r page is %d bytes' % (url, len(future.result())))
>
> In this example, executor.run_to_futures() returns only when every  
> url has been retrieved but it is possible to return immediately, on  
> the first completion or on the first failure depending on the  
> desired work pattern.
>
> The complete docs are here:
> http://sweetapp.com/futures/
>


From p.f.moore at gmail.com  Sat Nov  7 12:06:20 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 7 Nov 2009 11:06:20 +0000
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
	<4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>
Message-ID: <79990c6b0911070306y1ed637d9v167b8c3b6bce8e3e@mail.gmail.com>

2009/11/7 Jesse Noller <jnoller at gmail.com>:
> I obviously tend to agree with Brian; I know I've personally had to
> implement things like this plenty of times, it's relatively simple
> once you do it once or twice. This is a nice bit of syntactic sugar on
> top of the threading/multiprocessing modules.

I agree. I've implemented futures a few times, and I'd be very glad
not to have to again. I'll certainly check out the package, but I'd
like to see the functionality in the stdlib.

I'm not convinced it should go in multiprocessing, though. After all,
it uses threading rather than multiple processes.

Paul.

From brian at sweetapp.com  Sat Nov  7 12:46:46 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 7 Nov 2009 22:46:46 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <79990c6b0911070306y1ed637d9v167b8c3b6bce8e3e@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
	<4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>
	<79990c6b0911070306y1ed637d9v167b8c3b6bce8e3e@mail.gmail.com>
Message-ID: <52997465-6031-456A-9374-E85050BA2AAD@sweetapp.com>


On 7 Nov 2009, at 22:06, Paul Moore wrote:

> 2009/11/7 Jesse Noller <jnoller at gmail.com>:
>> I obviously tend to agree with Brian; I know I've personally had to
>> implement things like this plenty of times, it's relatively simple
>> once you do it once or twice. This is a nice bit of syntactic sugar  
>> on
>> top of the threading/multiprocessing modules.
>
> I agree. I've implemented futures a few times, and I'd be very glad
> not to have to again. I'll certainly check out the package, but I'd
> like to see the functionality in the stdlib.
>
> I'm not convinced it should go in multiprocessing, though. After all,
> it uses threading rather than multiple processes.


Actually, you can choose weather to use threads or processes. The  
current implementation includes a ThreadPoolExecutor and a  
ProcessPoolExecutor (which is an argument to making it a separate  
package) and should be abstract enough to accommodate other strategies  
in the future.

Java, for example,
Cheers,
Brian

From p.f.moore at gmail.com  Sat Nov  7 15:03:34 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 7 Nov 2009 14:03:34 +0000
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <52997465-6031-456A-9374-E85050BA2AAD@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
	<4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>
	<79990c6b0911070306y1ed637d9v167b8c3b6bce8e3e@mail.gmail.com>
	<52997465-6031-456A-9374-E85050BA2AAD@sweetapp.com>
Message-ID: <79990c6b0911070603n31282b94h649d889ff4a716e3@mail.gmail.com>

2009/11/7 Brian Quinlan <brian at sweetapp.com>:
>
> On 7 Nov 2009, at 22:06, Paul Moore wrote:
>> I'm not convinced it should go in multiprocessing, though. After all,
>> it uses threading rather than multiple processes.
>
>
> Actually, you can choose weather to use threads or processes. The current
> implementation includes a ThreadPoolExecutor and a ProcessPoolExecutor
> (which is an argument to making it a separate package) and should be
> abstract enough to accommodate other strategies in the future.

That's my point. Multiprocessing is about just that - multiprocessing.
I wouldn't use it (or even think of looking in it) if I wanted to
write a single-process multithreaded program (which is what I usually
do on Windows). I was responding to the suggestion that your futures
module would work as a component of the multiprocessing package.

Actually, it's a pity that things like the Pool class only exist in
multiprocessing. A threaded version of that would be very useful to me
as well.

Paul.

From jnoller at gmail.com  Sat Nov  7 15:08:26 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 09:08:26 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <79990c6b0911070603n31282b94h649d889ff4a716e3@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
	<4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>
	<79990c6b0911070306y1ed637d9v167b8c3b6bce8e3e@mail.gmail.com>
	<52997465-6031-456A-9374-E85050BA2AAD@sweetapp.com>
	<79990c6b0911070603n31282b94h649d889ff4a716e3@mail.gmail.com>
Message-ID: <69086482-6314-4336-9FEC-FF3D63D933A6@gmail.com>



On Nov 7, 2009, at 9:03 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> 2009/11/7 Brian Quinlan <brian at sweetapp.com>:
>>
>> On 7 Nov 2009, at 22:06, Paul Moore wrote:
>>> I'm not convinced it should go in multiprocessing, though. After  
>>> all,
>>> it uses threading rather than multiple processes.
>>
>>
>> Actually, you can choose weather to use threads or processes. The  
>> current
>> implementation includes a ThreadPoolExecutor and a  
>> ProcessPoolExecutor
>> (which is an argument to making it a separate package) and should be
>> abstract enough to accommodate other strategies in the future.
>
> That's my point. Multiprocessing is about just that - multiprocessing.
> I wouldn't use it (or even think of looking in it) if I wanted to
> write a single-process multithreaded program (which is what I usually
> do on Windows). I was responding to the suggestion that your futures
> module would work as a component of the multiprocessing package.
>
> Actually, it's a pity that things like the Pool class only exist in
> multiprocessing. A threaded version of that would be very useful to me
> as well.
>
> Paul.
> ______________________________

It's an easily rectified pity. Also, your not the only use case  
addressed by multiprocessing, which is why stuff you wouldn't use is  
in there.

Jesse

From fwierzbicki at gmail.com  Sat Nov  7 15:42:28 2009
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Sat, 7 Nov 2009 09:42:28 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
	<EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
Message-ID: <4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>

On Sat, Nov 7, 2009 at 12:29 AM, Brian Quinlan <brian at sweetapp.com> wrote:

> Right now multiprocessing is ahead of threading in terms of features.
> Pool.map() in particular is a pretty powerful idiom that has no equivalent
> in threading.
This is the area where I am most worried. Though multiprocessing is a
drop in replacement for threading, threading is not currently a drop
in replacement for multiprocessing. If multiprocessing doesn't make
sense for Jython and we need to tell our users that they should just
use threading, threading needs to do everything that multiprocessing
does... or maybe there needs to be a higher level package?

-Frank

From brian at sweetapp.com  Sat Nov  7 15:46:18 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 8 Nov 2009 01:46:18 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257558004.13956.24.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<1257558004.13956.24.camel@localhost>
Message-ID: <0191DFF2-BE7C-42F5-B77D-6AA2A3EBFA90@sweetapp.com>


On 7 Nov 2009, at 12:40, Antoine Pitrou wrote:

>
>> To Antoine's Twisted comment, I don't see a direct comparison. From  
>> my
>> understanding Twisted's Deferred objects are ways to have callbacks
>> executed once an async event occurs, not to help execute code
>> concurrently.
>
> Well, waiting for concurrently executing code to terminate *is* a case
> of waiting for an async event to happen.
>
> Deferred objects are a generic mechanism to chain reactions to
> termination or failure of code. Whether the event your Deferred reacts
> to is "async" or not is really a matter of how you use it (and of how
> you define "async" -- perhaps you meant "I/O" but Deferreds are not
> specialized for I/O).

They do seem specialized for continuation-passing style programming  
though. As far as I can tell from the docs (http://python.net/crew/mwh/apidocs/twisted.internet.defer.Deferred.html 
), the only way to process the results of a Deferred is my installing  
a callback.

Maybe you could outline (at a super-high-level) how you would  
implement my URL-downloading example using a Deferred-based API? Maybe  
something like:

def print_success(result, url):
    print('%r page is %d bytes' % (url, len(result)))

def print_failure(exception, url):
   print('%r generated an exception: %s' % (url, exception))

with ThreadedDeferredMaker(max_threads=5) as dm
   deferreds = []
   for url in URLS:
     deferred = dm.defer(load_url, url)
     deferred. addCallbacks(print_success, print_failure, url=url)
     deferred.unpause()
     deferreds.append(deferred)
   dm.wait_for_all_to_complete(deferreds)

The semantics aren't quite the same because the order of the output  
would be non-deterministic in this case. OTOH, you are going to get  
intermediate results as they become available, which is cool.


Cheers,
Brian


From p.f.moore at gmail.com  Sat Nov  7 15:48:06 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 7 Nov 2009 14:48:06 +0000
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <69086482-6314-4336-9FEC-FF3D63D933A6@gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
	<4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>
	<79990c6b0911070306y1ed637d9v167b8c3b6bce8e3e@mail.gmail.com>
	<52997465-6031-456A-9374-E85050BA2AAD@sweetapp.com>
	<79990c6b0911070603n31282b94h649d889ff4a716e3@mail.gmail.com>
	<69086482-6314-4336-9FEC-FF3D63D933A6@gmail.com>
Message-ID: <79990c6b0911070648h1ede221dm49e255867ad39e28@mail.gmail.com>

2009/11/7 Jesse Noller <jnoller at gmail.com>:
>> Actually, it's a pity that things like the Pool class only exist in
>> multiprocessing. A threaded version of that would be very useful to me
>> as well.
>
> It's an easily rectified pity. Also, your not the only use case addressed by
> multiprocessing, which is why stuff you wouldn't use is in there.

I'm not quite sure what you mean there. Are you suggesting that there
could be a threading.Pool which mirrors multiprocessing.Pool? If so,
then yes I agree - but of course, it wouldn't be available till
2.7/3.2.

What I suppose I was thinking of as a "pity" was that it wasn't
already added to threading. I thought multiprocessing was "just" the
threading API using multiple processes - but it looks like it's more
than that.

2009/11/7 Frank Wierzbicki <fwierzbicki at gmail.com>:
> On Sat, Nov 7, 2009 at 12:29 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>
>> Right now multiprocessing is ahead of threading in terms of features.
>> Pool.map() in particular is a pretty powerful idiom that has no equivalent
>> in threading.
> This is the area where I am most worried. Though multiprocessing is a
> drop in replacement for threading, threading is not currently a drop
> in replacement for multiprocessing. If multiprocessing doesn't make
> sense for Jython and we need to tell our users that they should just
> use threading, threading needs to do everything that multiprocessing
> does... or maybe there needs to be a higher level package?

Yes, *that's* my point.

Paul.

From solipsis at pitrou.net  Sat Nov  7 15:49:36 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 15:49:36 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<4222a8490911061557y48e25d65ibae653a1c432444d@mail.gmail.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
	<EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
	<4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
Message-ID: <1257605376.3437.0.camel@localhost>


> This is the area where I am most worried. Though multiprocessing is a
> drop in replacement for threading, threading is not currently a drop
> in replacement for multiprocessing. If multiprocessing doesn't make
> sense for Jython and we need to tell our users that they should just
> use threading, threading needs to do everything that multiprocessing
> does...

Well, feel free to propose a patch for threading.py.
I'm not sure this has anything to do with the discussion about futures
anyway.

Regards

Antoine.



From solipsis at pitrou.net  Sat Nov  7 15:53:03 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 15:53:03 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
Message-ID: <1257605583.3437.3.camel@localhost>

Le vendredi 06 novembre 2009 ? 21:20 -0500, Jesse Noller a ?crit :
> > But I really do like the idea. With java.util.concurrent and Grand
> > Central Dispatch out there, I think it shows some demand for a way to
> > easily abstract out concurrency management stuff and leave it up to a
> > library.
> 
> Making it eas(y)(ier), safer and simple would be nice.

I agree with that. From a quick look at the API it seems it deserves
simplifying and polishing.

Regards

Antoine.



From jnoller at gmail.com  Sat Nov  7 15:55:14 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 09:55:14 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1257551966.12371.0.camel@localhost>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
	<EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
	<4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
Message-ID: <4222a8490911070655u7c941eb1gaa867e360385cf04@mail.gmail.com>

On Sat, Nov 7, 2009 at 9:42 AM, Frank Wierzbicki <fwierzbicki at gmail.com> wrote:
> On Sat, Nov 7, 2009 at 12:29 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>
>> Right now multiprocessing is ahead of threading in terms of features.
>> Pool.map() in particular is a pretty powerful idiom that has no equivalent
>> in threading.
> This is the area where I am most worried. Though multiprocessing is a
> drop in replacement for threading, threading is not currently a drop
> in replacement for multiprocessing. If multiprocessing doesn't make
> sense for Jython and we need to tell our users that they should just
> use threading, threading needs to do everything that multiprocessing
> does... or maybe there needs to be a higher level package?

The only reason in my view that this is not the case is because no one
has submitted a patch, myself included, it's been on my wish list for
some time.

There is nothing blocking that, AFAIK.

From jnoller at gmail.com  Sat Nov  7 15:58:35 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 09:58:35 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <79990c6b0911070648h1ede221dm49e255867ad39e28@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
	<4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>
	<79990c6b0911070306y1ed637d9v167b8c3b6bce8e3e@mail.gmail.com>
	<52997465-6031-456A-9374-E85050BA2AAD@sweetapp.com>
	<79990c6b0911070603n31282b94h649d889ff4a716e3@mail.gmail.com>
	<69086482-6314-4336-9FEC-FF3D63D933A6@gmail.com>
	<79990c6b0911070648h1ede221dm49e255867ad39e28@mail.gmail.com>
Message-ID: <4222a8490911070658h4184cd54ma3d816ba5c6fae3a@mail.gmail.com>

On Sat, Nov 7, 2009 at 9:48 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 2009/11/7 Jesse Noller <jnoller at gmail.com>:
>>> Actually, it's a pity that things like the Pool class only exist in
>>> multiprocessing. A threaded version of that would be very useful to me
>>> as well.
>>
>> It's an easily rectified pity. Also, your not the only use case addressed by
>> multiprocessing, which is why stuff you wouldn't use is in there.
>
> I'm not quite sure what you mean there. Are you suggesting that there
> could be a threading.Pool which mirrors multiprocessing.Pool? If so,
> then yes I agree - but of course, it wouldn't be available till
> 2.7/3.2.
>
> What I suppose I was thinking of as a "pity" was that it wasn't
> already added to threading. I thought multiprocessing was "just" the
> threading API using multiple processes - but it looks like it's more
> than that.
>

See my response to frank: There's nothing blocking this except for:

1> A patch
2> Tests
3> Docs

It's been on my wish list for ~2 years now, I might get it done in the
next decade. Also, multiprocessing has never been "just" the threading
API on top of processes. Part of the PEP for it's inclusion was that
it had other items in it of value.

jesse

From jnoller at gmail.com  Sat Nov  7 16:05:18 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 10:05:18 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257605376.3437.0.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
	<EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
	<4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
	<1257605376.3437.0.camel@localhost>
Message-ID: <4222a8490911070705l1b3ed098od2505e152268a3b3@mail.gmail.com>

On Sat, Nov 7, 2009 at 9:49 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> This is the area where I am most worried. Though multiprocessing is a
>> drop in replacement for threading, threading is not currently a drop
>> in replacement for multiprocessing. If multiprocessing doesn't make
>> sense for Jython and we need to tell our users that they should just
>> use threading, threading needs to do everything that multiprocessing
>> does...
>
> Well, feel free to propose a patch for threading.py.
> I'm not sure this has anything to do with the discussion about futures
> anyway.
>
> Regards
>
> Antoine.
>

It may smell off topic Antoine - but it fundamentally isn't.
Multiprocessing exposes a lot more "goodies" than the threading
module. Threading lacks parity (and actually can't have it for some
things) with multiprocessing.

The futures package actually adds a nice API for a given set of tasks
on top of both threading and multiprocessing, and so begs the question
"how do alternative implementations which don't have multiprocessing"
deal with a new package which offers access to an API of which part
builds on multiprocessing.

I don't think that question is going to be solved in the context of
this particular discussion, given that any solution to that question
lacks something Brian's futures package has - working code.

On the other hand, possibly pushing futures into a concurrent.*
namespace within the standard library means that you could have
concurrent.futures, concurrent.map, concurrent.apply and so on, and
pull the things which multiprocessing does and threading can as well
into that concurrent package.

jesse

From jnoller at gmail.com  Sat Nov  7 16:07:07 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 10:07:07 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257605583.3437.3.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
Message-ID: <4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>

On Sat, Nov 7, 2009 at 9:53 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le vendredi 06 novembre 2009 ? 21:20 -0500, Jesse Noller a ?crit :
>> > But I really do like the idea. With java.util.concurrent and Grand
>> > Central Dispatch out there, I think it shows some demand for a way to
>> > easily abstract out concurrency management stuff and leave it up to a
>> > library.
>>
>> Making it eas(y)(ier), safer and simple would be nice.
>
> I agree with that. From a quick look at the API it seems it deserves
> simplifying and polishing.
>
> Regards
>
> Antoine.

Which API? My comment wasn't aimed at the API of the package - in the
time I got to scan it last night nothing jumped out at me as overly
offensive API-wise.

From solipsis at pitrou.net  Sat Nov  7 16:21:13 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 16:21:13 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
Message-ID: <1257607273.3437.13.camel@localhost>


> Which API? My comment wasn't aimed at the API of the package - in the
> time I got to scan it last night nothing jumped out at me as overly
> offensive API-wise.

Not offensive, but probably too complicated if it's meant to be a simple
helper. Anyway, let's wait for the PEP.



From jnoller at gmail.com  Sat Nov  7 16:32:09 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 10:32:09 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257607273.3437.13.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
Message-ID: <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>

On Sat, Nov 7, 2009 at 10:21 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> Which API? My comment wasn't aimed at the API of the package - in the
>> time I got to scan it last night nothing jumped out at me as overly
>> offensive API-wise.
>
> Not offensive, but probably too complicated if it's meant to be a simple
> helper. Anyway, let's wait for the PEP.


The PEP is right here:

http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt

I'm interested in hearing specific complaints about the API in the
context of what it's trying to *do*. The only thing which jumped out
at me was the number of methods on FutureList; but then again, each
one of those makes conceptual sense, even if they are verbose -
they're explicit on what's being done.

jesse

From solipsis at pitrou.net  Sat Nov  7 16:33:34 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 16:33:34 +0100
Subject: [stdlib-sig] futures and deferreds
In-Reply-To: <0191DFF2-BE7C-42F5-B77D-6AA2A3EBFA90@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<1257558004.13956.24.camel@localhost>
	<0191DFF2-BE7C-42F5-B77D-6AA2A3EBFA90@sweetapp.com>
Message-ID: <1257608014.3437.21.camel@localhost>


Hello,

> They do seem specialized for continuation-passing style programming  
> though. As far as I can tell from the docs (http://python.net/crew/mwh/apidocs/twisted.internet.defer.Deferred.html 
> ), the only way to process the results of a Deferred is my installing  
> a callback.

Yes, but the last callback in the chain could trigger an Event, a Queue
or any other synchronization object to which you could wait on, if you
want something "synchronous" (despite the title of your original
message: "asynchronous execution" :-)).

> with ThreadedDeferredMaker(max_threads=5) as dm
>    deferreds = []
>    for url in URLS:
>      deferred = dm.defer(load_url, url)
>      deferred. addCallbacks(print_success, print_failure, url=url)
>      deferred.unpause()
>      deferreds.append(deferred)

If you have a bunch of deferreds and want your callback to trigger when
all deferreds are finished/failed, you can use a DeferredList. A
DeferredList is itself a Deferred so you can add callbacks to it
(including one which makes things synchronous as explained above).

http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.DeferredList.html

(note that, as the doc says, "that you can still use a Deferred after
putting it in a DeferredList". That is, you can react individually to
each result to implement e.g. a progress indicator, and also react to
the completion of all deferreds)

Of course an all-synchronous API is still simpler to use for the use
cases it is meant for.

Regards

Antoine.



From solipsis at pitrou.net  Sat Nov  7 17:13:24 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 17:13:24 +0100
Subject: [stdlib-sig] futures - PEP and API
In-Reply-To: <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
Message-ID: <1257610404.3437.59.camel@localhost>


> The PEP is right here:
> 
> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
> 
> I'm interested in hearing specific complaints about the API in the
> context of what it's trying to *do*. The only thing which jumped out
> at me was the number of methods on FutureList;

Yes that would be the first complaint. Then many of those methods are as
(un)trustable as, say, Queue.qsize().

An example :
        """`done_futures()`
        
        Return an iterator over all `Future` instances that completed or
        were cancelled."""
        
First, it claims to return an iterator but the internal container could
mutate while iterating (since it can be mutated when a task terminates
in another thread). So the API looks broken with respect to what the
semantics dictate. It should probably return a distinct container (list
or set) instead.

Second, by the time the result is processed by the caller, there's no
way to know if the information is still valid or not. It's entirely
speculative, which makes it potentially deceiving -- and should be
mentioned in the doc.

        """`has_done_futures()`
        
        Return `True` if any `Future` in the list has completed or was
        successfully cancelled."""
        
Same problem. Please note that it can be removed if `done_futures()`
returns a container, since you then just have to do a boolean check on
the container (that would remove 5 methods :-)).


Then about the Future API itself. I would argue that if we want it to be
a simple helper, it should be as simple to use as a weakref.

That is, rather than :

        """`result(timeout=None)`
        
        Return the value returned by the call.
        [...]
        
        `exception(timeout=None)`
        
        Return the exception raised by the call."""
        
Make the object callable, such as `future(timeout=None)` either returns
the computed result (if successful), raises an exception (if failed) or
raises a TimeoutError.


Then about the Executor API. I don't understand why we have the
possibility to wait on a FutureList *and* on the Executor's
run_to_results() method. I think all wait-type methods should be folded
in to the Future or FutureList API, and the Executor should only
generate that Future(List).

Practically, there should be two ways to wait for multiple results,
depending on whether you need the results ordered or not. In the web
crawling situation given as example, it is silly to wait for the results
in order rather than process each result as soon as it gets available.
(*)

I don't understand why the Executor seems to be used as a context
manager in the examples. Its resources are still alive after the "with"
since the tasks are still executin, so it can't possibly have cleaned up
anything, has it?


(*) And, of course, you start to understand why a callback-based API
such as Deferreds makes a lot of sense...

Regards

Antoine.



From g.brandl at gmx.net  Sat Nov  7 18:26:38 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 07 Nov 2009 18:26:38 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
Message-ID: <hd479g$k5s$1@ger.gmane.org>

Brett Cannon schrieb:
> [I am going to be lazy and mass reply to people with a top-post; you
> can burn an effigy of me later]
> 
> In response to Guido, yes, I sent Brian here with my PEP editor hat
> on. Unfortunately the hat was on rather firmly and I totally forgot to
> check to see how old the code is. Yet another reason I need to get the
> Hg conversion done so I can start writing a "Adding to the Stdlib"
> PEP.

I think it isn't entirely wrong to post to stdlib-sig about an interesting
area that could use a battery, and to present code that may become that
battery given enough time.  That way, we who need to accept the code later
can suggest API changes or point out problems now, instead of just before
inclusion when incompatible changes will only upset the (by then hopefully
many) users of the existing package.

Now if I could remember where I put the matches...

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From fwierzbicki at gmail.com  Sat Nov  7 17:37:22 2009
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Sat, 7 Nov 2009 11:37:22 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257605376.3437.0.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
	<EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
	<4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
	<1257605376.3437.0.camel@localhost>
Message-ID: <4dab5f760911070837n498472b4x878f11181370c8c1@mail.gmail.com>

On Sat, Nov 7, 2009 at 9:49 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> This is the area where I am most worried. Though multiprocessing is a
>> drop in replacement for threading, threading is not currently a drop
>> in replacement for multiprocessing. If multiprocessing doesn't make
>> sense for Jython and we need to tell our users that they should just
>> use threading, threading needs to do everything that multiprocessing
>> does...
>
> Well, feel free to propose a patch for threading.py.
> I'm not sure this has anything to do with the discussion about futures
> anyway.
If it can be done in pure Python I'd certainly be up for taking a a
crack at such a patch.  If it involves significant work with C and
threading it might be a little out of my scope.  If pure python is
out, I may end up implementing those parts missing in threading.py in
Java for Jython, and then circling back to see if doing it in C for
CPython makes sense.

-Frank

From fwierzbicki at gmail.com  Sat Nov  7 17:41:20 2009
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Sat, 7 Nov 2009 11:41:20 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911070705l1b3ed098od2505e152268a3b3@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
	<EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
	<4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
	<1257605376.3437.0.camel@localhost>
	<4222a8490911070705l1b3ed098od2505e152268a3b3@mail.gmail.com>
Message-ID: <4dab5f760911070841i7616c9ag44c8b0d02393c3bb@mail.gmail.com>

On Sat, Nov 7, 2009 at 10:05 AM, Jesse Noller <jnoller at gmail.com> wrote:
> On the other hand, possibly pushing futures into a concurrent.*
> namespace within the standard library means that you could have
> concurrent.futures, concurrent.map, concurrent.apply and so on, and
> pull the things which multiprocessing does and threading can as well
> into that concurrent package.
I really like the idea of a concurrent package over adding these
things to multiprocessing - I might even be able to get it implemented
before  CPython since I have some obvious implementation advantages ;)

-Frank

From p.f.moore at gmail.com  Sat Nov  7 17:45:36 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 7 Nov 2009 16:45:36 +0000
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911070658h4184cd54ma3d816ba5c6fae3a@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1afaf6160911061500m695806edufe034ab8ac89591a@mail.gmail.com>
	<4D8A508D-0E50-4EE0-826C-79684070C0A1@sweetapp.com>
	<4222a8490911061812i50a98cf0gab5a1ba15df7afbe@mail.gmail.com>
	<79990c6b0911070306y1ed637d9v167b8c3b6bce8e3e@mail.gmail.com>
	<52997465-6031-456A-9374-E85050BA2AAD@sweetapp.com>
	<79990c6b0911070603n31282b94h649d889ff4a716e3@mail.gmail.com>
	<69086482-6314-4336-9FEC-FF3D63D933A6@gmail.com>
	<79990c6b0911070648h1ede221dm49e255867ad39e28@mail.gmail.com>
	<4222a8490911070658h4184cd54ma3d816ba5c6fae3a@mail.gmail.com>
Message-ID: <79990c6b0911070845x5035f463yb30ef47ecbce0c39@mail.gmail.com>

2009/11/7 Jesse Noller <jnoller at gmail.com>:
> Also, multiprocessing has never been "just" the threading
> API on top of processes. Part of the PEP for it's inclusion was that
> it had other items in it of value.

I guess it's my fault then for not paying enough attention to the
multithreading PEP. I did think it was "just" a multiprocess version
of threading, and if I'd have realised, I'd have lobbied for parity of
implementation at the time. Ah well, none of that is a criticism of
multiprocessing itself, and it's certainly not too late to add this,
as you said.

Paul.

From p.f.moore at gmail.com  Sat Nov  7 17:47:11 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 7 Nov 2009 16:47:11 +0000
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257605376.3437.0.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<4222a8490911061609w1691b605k9816597f558ecf4d@mail.gmail.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
	<EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
	<4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
	<1257605376.3437.0.camel@localhost>
Message-ID: <79990c6b0911070847x3d94e7ffp26c7d4c93e3aebe0@mail.gmail.com>

2009/11/7 Antoine Pitrou <solipsis at pitrou.net>:
> I'm not sure this has anything to do with the discussion about futures
> anyway.

It's not - unless the suggestion that futures get added into
multiprocessing was serious.

Personally, I like the idea of a "concurrent" namespace -
concurrent.futures seems like an ideal place for the module.

Paul.

From jnoller at gmail.com  Sat Nov  7 19:23:49 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 13:23:49 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <79990c6b0911070847x3d94e7ffp26c7d4c93e3aebe0@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<1257553433.13956.5.camel@localhost>
	<4222a8490911061632i75feaa1cw73cd74d759c3294e@mail.gmail.com>
	<1257554145.13956.7.camel@localhost>
	<4222a8490911061641i4786dacbrb32ec8975a659445@mail.gmail.com>
	<4dab5f760911061643x258a83c5n4630a08019cd336b@mail.gmail.com>
	<EE9E20F3-24CF-4A39-91BC-BB3EFF889446@sweetapp.com>
	<4dab5f760911070642wd34a4buf7344da59c54201a@mail.gmail.com>
	<1257605376.3437.0.camel@localhost>
	<79990c6b0911070847x3d94e7ffp26c7d4c93e3aebe0@mail.gmail.com>
Message-ID: <4222a8490911071023h65570713xe5d2dfd69ab99e47@mail.gmail.com>

On Sat, Nov 7, 2009 at 11:47 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 2009/11/7 Antoine Pitrou <solipsis at pitrou.net>:
>> I'm not sure this has anything to do with the discussion about futures
>> anyway.
>
> It's not - unless the suggestion that futures get added into
> multiprocessing was serious.
>
> Personally, I like the idea of a "concurrent" namespace -
> concurrent.futures seems like an ideal place for the module.

My point in saying that was to note that I've wanted to add something
like this into multiprocessing for awhile. More expansive use of
context managers to control pools of processes, possibly decorators to
indicate a function should be run in a process, etc.

That all being said; I'm more closely aligned with the concept of
building out/starting a python.concurrent package (starting with the
futures package) and then refactoring some of the multiprocessing API
into that package than I am adding futures right into multiprocessing.

jesse

From jnoller at gmail.com  Sat Nov  7 19:40:26 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 13:40:26 -0500
Subject: [stdlib-sig] Concurrent/Jython (was: futures - a new package
	for asynchronous execution)
Message-ID: <4222a8490911071040k5c94e3d6gfb14d173c9dffdb2@mail.gmail.com>

On Sat, Nov 7, 2009 at 11:37 AM, Frank Wierzbicki <fwierzbicki at gmail.com> wrote:

> If it can be done in pure Python I'd certainly be up for taking a a
> crack at such a patch.  If it involves significant work with C and
> threading it might be a little out of my scope.  If pure python is
> out, I may end up implementing those parts missing in threading.py in
> Java for Jython, and then circling back to see if doing it in C for
> CPython makes sense.
>
> -Frank

Figured I'd start a new thread rather than overloading the existing one.

If we were to seriously go down the path of building out a concurrent
package for Python's stdlib, I think 99% of the work would be in pure
python. Right now multiprocessing and threading expose the primitives
needed to do most of the work without diving into C.

For example, the pool.py module [1] multiprocessing has could easily
have threads swapped in - this would probably change the API to look
like this:

Original:

class Pool(object):
    '''
    Class which supports an async version of the `apply()` builtin
    '''
    Process = Process

    def __init__(self, processes=None, initializer=None, initargs=()):

New:

class Pool(object):
    '''
    Class which supports an async version of the `apply()` builtin
    '''
    worker = threading.Thread

    def __init__(self, worker_type=multiprocessing.Process,
workers=None, initializer=None, initargs=()):

Theres some other internals which would need to be adjusted, but just
modifying it to support passing in the Worker type to use would make
it easy to swap it for threads vs. process. My person preference would
be to have the above hidden with _ and two classes:

class ThreadPool
class ProcessPool

Which just called the hidden _Pool with the worker type, but that's a
smell-thing. I think explicitness in APIs dealing with threads and
processes is preferable than highly flexible generic APIs (the
restriction on multiprocessing being "objects must be pickleable").

Then again, passing in the worker type means less work for
IronPython/Jython - users could just be told "just accept the default
thread worker type" - they would only need to change process-based
code to remove the worker_type argument. Or Jython could swap it under
the covers.

So, my pony list would look something like:

1> add concurrent
2> Put futures in concurrent
3> Refactor multiprocessing.pool into concurrent, add deprecation
notes to the multiprocessing APIs in the docs

A simple producer/consumer implementation could also be added in here
too (one day).

This work should only target Python 3, imho.

jesse

[1]: http://svn.python.org/view/python/trunk/Lib/multiprocessing/pool.py?revision=74023&view=markup

From lac at openend.se  Sat Nov  7 20:19:22 2009
From: lac at openend.se (Laura Creighton)
Date: Sat, 07 Nov 2009 20:19:22 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: Message from Jesse Noller <jnoller@gmail.com> of "Fri,
	06 Nov 2009 21:20:43 EST."
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com> 
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com> 
Message-ID: <200911071919.nA7JJM3R013864@theraft.openend.se>

Anybody here looked at Kamaelia? http://www.kamaelia.org/
I haven't, but wonder if we are considering re-inventing the
wheel.

Laura


From jnoller at gmail.com  Sat Nov  7 20:36:06 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 14:36:06 -0500
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <200911071919.nA7JJM3R013864@theraft.openend.se>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<200911071919.nA7JJM3R013864@theraft.openend.se>
Message-ID: <6ED7ACBB-B587-4166-8E2C-23279B07CAEF@gmail.com>



On Nov 7, 2009, at 2:19 PM, Laura Creighton <lac at openend.se> wrote:

> Anybody here looked at Kamaelia? http://www.kamaelia.org/
> I haven't, but wonder if we are considering re-inventing the
> wheel.
>
> Laura

I have, and no - it's not what we're discussing. Kamaelia is a  
framework, really. It's also GPL.
>

Inclusion of kamaelia, or twisted wholesale won't occur any time soon.

Jesse

From jyasskin at gmail.com  Sat Nov  7 20:37:09 2009
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sat, 7 Nov 2009 11:37:09 -0800
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
Message-ID: <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>

On Sat, Nov 7, 2009 at 7:32 AM, Jesse Noller <jnoller at gmail.com> wrote:
> On Sat, Nov 7, 2009 at 10:21 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>>> Which API? My comment wasn't aimed at the API of the package - in the
>>> time I got to scan it last night nothing jumped out at me as overly
>>> offensive API-wise.
>>
>> Not offensive, but probably too complicated if it's meant to be a simple
>> helper. Anyway, let's wait for the PEP.
>
>
> The PEP is right here:
>
> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>
> I'm interested in hearing specific complaints about the API in the
> context of what it's trying to *do*. The only thing which jumped out
> at me was the number of methods on FutureList; but then again, each
> one of those makes conceptual sense, even if they are verbose -
> they're explicit on what's being done.

Overall, I like the idea of having futures in the standard library,
and I like the idea of pulling common bits of multiprocessing and
threading into a concurrent.* package. Here's my
stream-of-consciousness review of the PEP. I'll try to ** things that
really affect the API.

The "Interface" section should start with a conceptual description of
what Executor, Future, and FutureList are. Something like "An Executor
is an object you can hand tasks to, which will run them for you,
usually in another thread or process. A Future represents a task that
may or may not have completed yet, and which can be waited for and its
value or exception queries. A FutureList is ... <haven't read that
far>."

** The Executor interface is pretty redundant, and it's missing the
most basic call. Fundamentally, all you need is an
Executor.execute(callable) method returning None, and all the
future-oriented methods can be built on top of that. I'd support using
Executor.submit(callable) for the simplest method instead, which
returns a Future, but there should be some way for implementers to
only implement execute() and get submit either for free or with a
1-line definition. (I'm using method names from
http://java.sun.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html
in case I'm unclear about the semantics here.) run_to_futures,
run_to_results, and map should be implementable on top of the Future
interface, and shouldn't need to be methods on Executor. I'd recommend
they be demoted to helper functions in the concurrent.futures module
unless there's a reason they need to be methods, and that reason
should be documented in the PEP.

** run_to_futures() shouldn't take a return_when argument. It should
be possible to wait for those conditions on any list of Futures.
(_not_ just a FutureList)

The code sample looks like Executor is a context manager. What does
its __exit__ do? shutdown()? shutdown&awaitTermination? I prefer
waiting in Executor.__exit__, since that makes it easier for users to
avoid having tasks run after they've cleaned up data those tasks
depend on. But that could be my C++ bias, where we have to be sure to
free memory in the right places. Frank, does Java run into any
problems with people cleaning things up that an Executor's tasks
depend on without awaiting for the Executor first?

shutdown should explain why it's important. Specifically, since the
Executor controls threads, and those threads hold a reference to the
Executor, nothing will get garbage collected without the explicit
call.

** What happens when FutureList.wait(FIRST_COMPLETED) is called twice?
Does it return immediately the second time? Does it wait for the
second task to finish? I'm inclined to think that FutureList should go
away and be replaced by functions that just take lists of Futures.

In general, I think the has_done_futures(), exception_futures(), etc.
are fine even though their results may be out of date by the time you
inspect them. That's because any individual Future goes monotonically
from not-started->running->(exception|value), so users can take
advantage of even an out-of-date done_futures() result. However, it's
dangerous to have several query functions, since users may think that
running_futures() `union` done_futures() `union` cancelled_futures()
covers the whole FutureList, but instead a Future can move between two
of the sets between two of those calls. Instead, perhaps an atomic
partition() function would be better, which returns a collection of
sub-lists that cover the whole original set.

I would rename result() to get() (or maybe Antoine's suggestion of
__call__) to match Java. I'm not sure exception() needs to exist.

--- More general points ---

** Java's Futures made a mistake in not supporting work stealing, and
this has caused deadlocks at Google. Specifically, in a bounded-size
thread or process pool, when a task in the pool can wait for work
running in the same pool, you can fill up the pool with tasks that are
waiting for tasks that haven't started running yet. To avoid this,
Future.get() should be able to steal the task it's waiting on out of
the pool's queue and run it immediately.

** I think both the Future-oriented blocking model and the
callback-based model Deferreds support are important for different
situations. Futures tend to be easier to program with, while Deferreds
use fewer threads and can have more predictable latencies. It should
be possible to create a Future from a Deferred or a Deferred from a
Future without taking up a thread.

From brett at python.org  Sat Nov  7 23:30:01 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 7 Nov 2009 14:30:01 -0800
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <6ED7ACBB-B587-4166-8E2C-23279B07CAEF@gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com> 
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com> 
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com> 
	<200911071919.nA7JJM3R013864@theraft.openend.se>
	<6ED7ACBB-B587-4166-8E2C-23279B07CAEF@gmail.com>
Message-ID: <bbaeab100911071430r233315bbr39bc33862b6205ab@mail.gmail.com>

On Sat, Nov 7, 2009 at 11:36, Jesse Noller <jnoller at gmail.com> wrote:
>
>
> On Nov 7, 2009, at 2:19 PM, Laura Creighton <lac at openend.se> wrote:
>
>> Anybody here looked at Kamaelia? http://www.kamaelia.org/
>> I haven't, but wonder if we are considering re-inventing the
>> wheel.
>>
>> Laura
>
> I have, and no - it's not what we're discussing. Kamaelia is a framework,
> really. It's also GPL.

They also have not stepped forward for inclusion. A point needs to be
made that Brian has working code and has come forward to contribute
it.

It also should be noted that the concepts are extremely simple here as
management pools for threads and processes is not hard to grasp (a
definitely knock against Twisted as their Deferred approach definitely
hurts some peoples' heads no matter how much Antoine loves them =).
And especially important, the proposed solution is in pure Python and
does not rely exclusively on multiprocessing, thus avoids alienating
other VMs beyond CPython.

>>
>
> Inclusion of kamaelia, or twisted wholesale won't occur any time soon.

I would replace "any time soon" with "ever". Both projects are massive
and probably don't want to be locked into our development cycle. Both
also have their own development teams and culture. Adding a single
module or package by a single author is enough challenge as it is, but
bringing in another team would be more than challenging.

-Brett

From brian at sweetapp.com  Sat Nov  7 23:33:02 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 8 Nov 2009 09:33:02 +1100
Subject: [stdlib-sig] futures and deferreds
In-Reply-To: <1257608014.3437.21.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<1257558004.13956.24.camel@localhost>
	<0191DFF2-BE7C-42F5-B77D-6AA2A3EBFA90@sweetapp.com>
	<1257608014.3437.21.camel@localhost>
Message-ID: <CBE2E577-CB44-4D23-9BFB-E2FCED361E3E@sweetapp.com>


On 8 Nov 2009, at 02:33, Antoine Pitrou wrote:

>
> Hello,
>
>> They do seem specialized for continuation-passing style programming
>> though. As far as I can tell from the docs (http://python.net/crew/mwh/apidocs/twisted.internet.defer.Deferred.html
>> ), the only way to process the results of a Deferred is my installing
>> a callback.
>
> Yes, but the last callback in the chain could trigger an Event, a  
> Queue
> or any other synchronization object to which you could wait on, if you
> want something "synchronous" (despite the title of your original
> message: "asynchronous execution" :-)).

Touch? ;-) But I was actually indenting this to be used for  
applications that are organized for synchronous but need some extra  
performance for a few method calls.

>> with ThreadedDeferredMaker(max_threads=5) as dm
>>   deferreds = []
>>   for url in URLS:
>>     deferred = dm.defer(load_url, url)
>>     deferred. addCallbacks(print_success, print_failure, url=url)
>>     deferred.unpause()
>>     deferreds.append(deferred)
>
> If you have a bunch of deferreds and want your callback to trigger  
> when
> all deferreds are finished/failed, you can use a DeferredList. A
> DeferredList is itself a Deferred so you can add callbacks to it
> (including one which makes things synchronous as explained above).
>
> http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.DeferredList.html
>
> (note that, as the doc says, "that you can still use a Deferred after
> putting it in a DeferredList". That is, you can react individually to
> each result to implement e.g. a progress indicator, and also react to
> the completion of all deferreds)
>

That's cool. A hybrid approach that allows you to use both callbacks  
and inspection might make sense.

Cheers,
Brian

> Of course an all-synchronous API is still simpler to use for the use
> cases it is meant for.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig


From brian at sweetapp.com  Sat Nov  7 23:41:56 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 8 Nov 2009 09:41:56 +1100
Subject: [stdlib-sig] futures - PEP and API
In-Reply-To: <1257610404.3437.59.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<1257610404.3437.59.camel@localhost>
Message-ID: <46C12EDD-A015-452A-9C05-21BFC3AD4038@sweetapp.com>


On 8 Nov 2009, at 03:13, Antoine Pitrou wrote:

>
>> The PEP is right here:
>>
>> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>>
>> I'm interested in hearing specific complaints about the API in the
>> context of what it's trying to *do*. The only thing which jumped out
>> at me was the number of methods on FutureList;
>
> Yes that would be the first complaint. Then many of those methods  
> are as
> (un)trustable as, say, Queue.qsize().
>
> An example :
>        """`done_futures()`
>
>        Return an iterator over all `Future` instances that completed  
> or
>        were cancelled."""
>
> First, it claims to return an iterator but the internal container  
> could
> mutate while iterating (since it can be mutated when a task terminates
> in another thread). So the API looks broken with respect to what the
> semantics dictate. It should probably return a distinct container  
> (list
> or set) instead.
> Second, by the time the result is processed by the caller, there's no
> way to know if the information is still valid or not. It's entirely
> speculative, which makes it potentially deceiving -- and should be
> mentioned in the doc.
>
>        """`has_done_futures()`
>
>        Return `True` if any `Future` in the list has completed or was
>        successfully cancelled."""
>
> Same problem. Please note that it can be removed if `done_futures()`
> returns a container, since you then just have to do a boolean check on
> the container (that would remove 5 methods :-)).
>
>
> Then about the Future API itself. I would argue that if we want it  
> to be
> a simple helper, it should be as simple to use as a weakref.
>
> That is, rather than :
>
>        """`result(timeout=None)`
>
>        Return the value returned by the call.
>        [...]
>
>        `exception(timeout=None)`
>
>        Return the exception raised by the call."""
>
> Make the object callable, such as `future(timeout=None)` either  
> returns
> the computed result (if successful), raises an exception (if failed)  
> or
> raises a TimeoutError.
>
>
> Then about the Executor API. I don't understand why we have the
> possibility to wait on a FutureList *and* on the Executor's
> run_to_results() method. I think all wait-type methods should be  
> folded
> in to the Future or FutureList API, and the Executor should only
> generate that Future(List).
>
> Practically, there should be two ways to wait for multiple results,
> depending on whether you need the results ordered or not. In the web
> crawling situation given as example, it is silly to wait for the  
> results
> in order rather than process each result as soon as it gets available.
> (*)
>
> I don't understand why the Executor seems to be used as a context
> manager in the examples. Its resources are still alive after the  
> "with"
> since the tasks are still executin, so it can't possibly have  
> cleaned up
> anything, has it?

The executor needs to know when no more futures will be scheduled so  
it can shutdown its thread/process pool after it has finished the  
current work items.

Let me think about your other comments for a while.

Cheers,
Brian



>
> (*) And, of course, you start to understand why a callback-based API
> such as Deferreds makes a lot of sense...
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig


From jimjjewett at gmail.com  Sat Nov  7 23:46:25 2009
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sat, 7 Nov 2009 17:46:25 -0500
Subject: [stdlib-sig] comments on PEP for futures - execute computations
	asynchronously
Message-ID: <fb6fbf560911071446o49e3f460ne1519b5621f075c7@mail.gmail.com>

(1)  It would be helpful to have the API classes named earlier, even
before the examples if the examples take more than three lines (as
they currently do).  The details of what methods they have can come
later, but naming the classes should serve as a useful landmark.

(2)  Within the FutureList methods definitions, please refer to the
FutureList, rather than "the list".  The first few times I came across
that, I was trying to figure out how you knew the container would be a
list, or whether this was a list that the caller had created.  I then
kept trying to view methods like has_exception_futures as global
functions that ought to take a list of futures as an argument.

(3)  Why even have a FutureList?  Can't the Executor object itself act
as a container for the tasks it is executing?  (And you might want a
way to remove tasks from the Executor, short of shutting down the
whole thing.)

(4)  Please add some sort of factory or global function so that users
can say "parallelize this however makes sense" and let the environment
pick.  (For example, Jython would presumably push threads, as would
windows, but Linux implementations might be happier with processes.)

-jJ

From solipsis at pitrou.net  Sat Nov  7 23:50:08 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 07 Nov 2009 23:50:08 +0100
Subject: [stdlib-sig] futures - PEP and API
In-Reply-To: <46C12EDD-A015-452A-9C05-21BFC3AD4038@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<1257610404.3437.59.camel@localhost>
	<46C12EDD-A015-452A-9C05-21BFC3AD4038@sweetapp.com>
Message-ID: <1257634208.3437.81.camel@localhost>


> The executor needs to know when no more futures will be scheduled so  
> it can shutdown its thread/process pool after it has finished the  
> current work items.

Ah, I see and it's useful indeed.
I think it makes things a bit counter-intuitive however. People are used
to "with" releasing things right at the end of the block, not scheduling
them for later release at an unknown point in time.

Regards

Antoine.



From solipsis at pitrou.net  Sun Nov  8 00:02:49 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 08 Nov 2009 00:02:49 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <bbaeab100911071430r233315bbr39bc33862b6205ab@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<200911071919.nA7JJM3R013864@theraft.openend.se>
	<6ED7ACBB-B587-4166-8E2C-23279B07CAEF@gmail.com>
	<bbaeab100911071430r233315bbr39bc33862b6205ab@mail.gmail.com>
Message-ID: <1257634969.3437.94.camel@localhost>


> They also have not stepped forward for inclusion. A point needs to be
> made that Brian has working code and has come forward to contribute
> it.

I do hope that the code and its API get exercised before being included,
though. Looking at the Google Code page, the project has only existed
since May 2009, and there isn't even a mailing-list.

Remember, once we release a Python version with that code in it, we
can't change the API very easily.



From brett at python.org  Sun Nov  8 00:26:01 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 7 Nov 2009 15:26:01 -0800
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1257634969.3437.94.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com> 
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com> 
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com> 
	<200911071919.nA7JJM3R013864@theraft.openend.se>
	<6ED7ACBB-B587-4166-8E2C-23279B07CAEF@gmail.com> 
	<bbaeab100911071430r233315bbr39bc33862b6205ab@mail.gmail.com> 
	<1257634969.3437.94.camel@localhost>
Message-ID: <bbaeab100911071526k337d8344i688e7c3b9c5636f1@mail.gmail.com>

On Sat, Nov 7, 2009 at 15:02, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> They also have not stepped forward for inclusion. A point needs to be
>> made that Brian has working code and has come forward to contribute
>> it.
>
> I do hope that the code and its API get exercised before being included,
> though. Looking at the Google Code page, the project has only existed
> since May 2009, and there isn't even a mailing-list.
>

That's why we are discussing it now. But as Jesse mentioned, this is
simple stuff that he was already planning to add to multiprocessing,
so it isn't that radical or controversial of an approach.

> Remember, once we release a Python version with that code in it, we
> can't change the API very easily.

As the author of PEP 3108, I am acutely aware of that fact. =)

-Brett

From brett at python.org  Sun Nov  8 22:26:59 2009
From: brett at python.org (Brett Cannon)
Date: Sun, 8 Nov 2009 13:26:59 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how noisy
	they are)
Message-ID: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>

During the moratorium PEP discussions Guido said he wanted to quiet
down deprecation warnings. I see there being two options on this.

One is to keep things as is, but to require two releases with
PendingDeprecationWarning so there are three years of
silent-by-default warnings to update your code. But that last release
before removal came would still be noisy.

The other option is to simply have all warnings filtered out by
default. We could alter -W so that when it is used w/o an argument it
turns to what is currently the default behaviour (or even turn all
warnings which is more than what happens now). This will require that
people proactively check for warnings when updating for compatibility,
else they will eventually use a Python release where there code will
simply break because something changed. This route means we do not
have to specify any deprecation policy right now (that would be a
separate discussion).

Channeling Guido he is after the latter, but a general discussion
would still be good since he didn't explicitly say what he was after
other than to quiet down warnings.

-Brett

From michael at voidspace.org.uk  Sun Nov  8 22:35:30 2009
From: michael at voidspace.org.uk (Michael Foord)
Date: Sun, 08 Nov 2009 22:35:30 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
Message-ID: <4AF739A2.3010601@voidspace.org.uk>

Brett Cannon wrote:
> During the moratorium PEP discussions Guido said he wanted to quiet
> down deprecation warnings. I see there being two options on this.
>
> One is to keep things as is, but to require two releases with
> PendingDeprecationWarning so there are three years of
> silent-by-default warnings to update your code. But that last release
> before removal came would still be noisy.
>   

This would be my preferred option. Noisy deprecations are annoying for a
*lot* of users, but are also useful to quite a lot of other users /
developers.

Almost no-one is ever going to run Python with PendingDeprecation
warnings switched on, so there should be at least one 'noisy' release in
my opinion.

All the best,

Michael Foord

> The other option is to simply have all warnings filtered out by
> default. We could alter -W so that when it is used w/o an argument it
> turns to what is currently the default behaviour (or even turn all
> warnings which is more than what happens now). This will require that
> people proactively check for warnings when updating for compatibility,
> else they will eventually use a Python release where there code will
> simply break because something changed. This route means we do not
> have to specify any deprecation policy right now (that would be a
> separate discussion).
>
> Channeling Guido he is after the latter, but a general discussion
> would still be good since he didn't explicitly say what he was after
> other than to quiet down warnings.
>
> -Brett
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>   



-- 
http://www.ironpythoninaction.com/

From ctb at msu.edu  Sun Nov  8 22:38:31 2009
From: ctb at msu.edu (C. Titus Brown)
Date: Sun, 8 Nov 2009 13:38:31 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and
	how	noisy they are)
In-Reply-To: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
Message-ID: <20091108213831.GA32503@idyll.org>

On Sun, Nov 08, 2009 at 01:26:59PM -0800, Brett Cannon wrote:
> During the moratorium PEP discussions Guido said he wanted to quiet
> down deprecation warnings. I see there being two options on this.
> 
> One is to keep things as is, but to require two releases with
> PendingDeprecationWarning so there are three years of
> silent-by-default warnings to update your code. But that last release
> before removal came would still be noisy.
> 
> The other option is to simply have all warnings filtered out by
> default. We could alter -W so that when it is used w/o an argument it
> turns to what is currently the default behaviour (or even turn all
> warnings which is more than what happens now). This will require that
> people proactively check for warnings when updating for compatibility,
> else they will eventually use a Python release where there code will
> simply break because something changed. This route means we do not
> have to specify any deprecation policy right now (that would be a
> separate discussion).

How about turning warnings on by default in unittest-like situations, and by
default have them off at other times?  Test running is when they would be most
useful, I think.

...then there's the problem of how to decide if we're "test running".

--titus
-- 
C. Titus Brown, ctb at msu.edu

From guido at python.org  Sun Nov  8 22:40:47 2009
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Nov 2009 13:40:47 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
Message-ID: <ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>

On Sun, Nov 8, 2009 at 1:26 PM, Brett Cannon <brett at python.org> wrote:
> During the moratorium PEP discussions Guido said he wanted to quiet
> down deprecation warnings. I see there being two options on this.
>
> One is to keep things as is, but to require two releases with
> PendingDeprecationWarning so there are three years of
> silent-by-default warnings to update your code. But that last release
> before removal came would still be noisy.
>
> The other option is to simply have all warnings filtered out by
> default. We could alter -W so that when it is used w/o an argument it
> turns to what is currently the default behaviour (or even turn all
> warnings which is more than what happens now). This will require that
> people proactively check for warnings when updating for compatibility,
> else they will eventually use a Python release where there code will
> simply break because something changed. This route means we do not
> have to specify any deprecation policy right now (that would be a
> separate discussion).
>
> Channeling Guido he is after the latter, but a general discussion
> would still be good since he didn't explicitly say what he was after
> other than to quiet down warnings.

I was specifically after all kinds of deprecation warnings, which
generally are telling you that you need to make a change in order to
remain compatible in the future. That's sometimes interesting but
often irrelevant noise. So I would agree with Gregory P Smith's
proposal to just treat all deprecation warnings as silent.

There are other kinds warnings which might be useful for other reasons
-- they typically point out code that does not do what you might think
it does. A good example is the warning added in 2.6 about "assert (x,
y)". This is something you ignore at your peril.

On Sun, Nov 8, 2009 at 1:35 PM, Michael Foord <michael at voidspace.org.uk> wrote:
> Almost no-one is ever going to run Python with PendingDeprecation
> warnings switched on, so there should be at least one 'noisy' release in
> my opinion.

I disagree. The -3 option is an example of a better approach: silent
by default, warnings enabled by a command line flag. If we can trust
developers to use -3 to check for Py3k incompatibilities, we should
also be able to trust them to check for deprecation warnings
explicitly.

(Another argument to think about: if you download and install some 3rd
party code, deprecation warnings about that code are *only noise*
since they are not yours to fix. Warnings in a dynamic language work
very different than warnings in a compiled language.)

-- 
--Guido van Rossum (python.org/~guido)

From michael at voidspace.org.uk  Sun Nov  8 22:51:12 2009
From: michael at voidspace.org.uk (Michael Foord)
Date: Sun, 08 Nov 2009 22:51:12 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
Message-ID: <4AF73D50.9080602@voidspace.org.uk>

Guido van Rossum wrote:
> [snip...]
> On Sun, Nov 8, 2009 at 1:35 PM, Michael Foord <michael at voidspace.org.uk> wrote:
>   
>> Almost no-one is ever going to run Python with PendingDeprecation
>> warnings switched on, so there should be at least one 'noisy' release in
>> my opinion.
>>     
>
> I disagree. The -3 option is an example of a better approach: silent
> by default, warnings enabled by a command line flag. If we can trust
> developers to use -3 to check for Py3k incompatibilities, we should
> also be able to trust them to check for deprecation warnings
> explicitly.
>
> (Another argument to think about: if you download and install some 3rd
> party code, deprecation warnings about that code are *only noise*
> since they are not yours to fix. Warnings in a dynamic language work
> very different than warnings in a compiled language.)
>
>   
Sounds like a good argument. The downside is that code 'just stops 
working' with no *apparent* warning.

Michael

-- 
http://www.ironpythoninaction.com/


From solipsis at pitrou.net  Sun Nov  8 23:01:26 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 08 Nov 2009 23:01:26 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
Message-ID: <1257717686.3411.2.camel@localhost>


> One is to keep things as is, but to require two releases with
> PendingDeprecationWarning so there are three years of
> silent-by-default warnings to update your code. But that last release
> before removal came would still be noisy.

While deprecation warnings are noisy, I also think it's the best
compromise we can think of.
Making them silent by default would be worse, except for those very few
programs/libraries which have absolutely 100% test coverage.

Regards

Antoine.



From ben+python at benfinney.id.au  Sun Nov  8 23:43:19 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 09 Nov 2009 09:43:19 +1100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
Message-ID: <874op44ozc.fsf@benfinney.id.au>

Guido van Rossum <guido at python.org> writes:

> On Sun, Nov 8, 2009 at 1:35 PM, Michael Foord <michael at voidspace.org.uk> wrote:
> > Almost no-one is ever going to run Python with PendingDeprecation
> > warnings switched on, so there should be at least one 'noisy'
> > release in my opinion.
>
> I disagree. The -3 option is an example of a better approach: silent
> by default, warnings enabled by a command line flag. If we can trust
> developers to use -3 to check for Py3k incompatibilities, we should
> also be able to trust them to check for deprecation warnings
> explicitly.

That doesn't seem to follow. The Python 2 ? Python 3 transition has
received a *huge* amount of attention (in the Python world), to the
point which pretty much the *only* thing many people know about Python 3
is ?it breaks backward compatibility?. So it's not surprising that
people are explicitly looking for ways to tell whether their code will
break in the transition, and consequently many people using the ?-3?
flag. I think this is a special case.

A transition between minor versions of Python, on the other hand, does
not inherently have the same clanging alarm bell associated with it:
indeed, by contrast with Python 2 ? Python 3, the message is that the
upgrade from Python N.M ? Python N.M+1 will be a *compatible* upgrade. I
would expect consequently far fewer developers go actively looking for
compatibility problems.

> (Another argument to think about: if you download and install some 3rd
> party code, deprecation warnings about that code are *only noise*
> since they are not yours to fix. Warnings in a dynamic language work
> very different than warnings in a compiled language.)

Right, so the users can then pressure the third party to fix their code
before it stops working altogether in a future version. When the user
complains to the Python community, we can publicly shame the third party
into testing and declaring their supported Python versions.

After all, they only need to see whether it runs without warnings; they
don't even need to remember to turn on any special flags.

-- 
 \           ?Let others praise ancient times; I am glad I was born in |
  `\                                      these.? ?Ovid (43 BCE?18 CE) |
_o__)                                                                  |
Ben Finney


From ubershmekel at gmail.com  Sun Nov  8 23:59:06 2009
From: ubershmekel at gmail.com (Yuvgoog Greenle)
Date: Mon, 9 Nov 2009 00:59:06 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <874op44ozc.fsf@benfinney.id.au>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<874op44ozc.fsf@benfinney.id.au>
Message-ID: <9d153b7c0911081459q2ccc819fme4a236cc9f46a117@mail.gmail.com>

On Mon, Nov 9, 2009 at 12:43 AM, Ben Finney
<ben+python at benfinney.id.au<ben%2Bpython at benfinney.id.au>
> wrote:

> Guido van Rossum <guido at python.org> writes:
>
> > On Sun, Nov 8, 2009 at 1:35 PM, Michael Foord <michael at voidspace.org.uk>
> wrote:
> > > Almost no-one is ever going to run Python with PendingDeprecation
> > > warnings switched on, so there should be at least one 'noisy'
> > > release in my opinion.
> >
> > I disagree. [snip]
>

I learned python from an old book that taught me to raise string exceptions,
noisy warnings about deprecation were the only thing that tipped me off that
something's wrong. As a newb I would have never executed python with any
option at all, no way I would have had enough mileage to hear about such a
thing.

For me back then, python would have just magically broken on a random day.


--yuv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091109/48298fb2/attachment.htm>

From g.brandl at gmx.net  Mon Nov  9 01:53:00 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 09 Nov 2009 01:53:00 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
Message-ID: <hd7lqc$59t$1@ger.gmane.org>

Brett Cannon schrieb:
> During the moratorium PEP discussions Guido said he wanted to quiet
> down deprecation warnings. I see there being two options on this.
> 
> One is to keep things as is, but to require two releases with
> PendingDeprecationWarning so there are three years of
> silent-by-default warnings to update your code. But that last release
> before removal came would still be noisy.
> 
> The other option is to simply have all warnings filtered out by
> default. We could alter -W so that when it is used w/o an argument it
> turns to what is currently the default behaviour (or even turn all
> warnings which is more than what happens now).

As a technical note, optional option arguments are a bad idea.

   python -W ignore

Am I calling the file "ignore" with all warnings enabled or running the
interactive interpreter ignoring all warnings?

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From brett at python.org  Mon Nov  9 02:44:48 2009
From: brett at python.org (Brett Cannon)
Date: Sun, 8 Nov 2009 17:44:48 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <hd7lqc$59t$1@ger.gmane.org>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<hd7lqc$59t$1@ger.gmane.org>
Message-ID: <bbaeab100911081744l276f2563xe548bd701e3ce405@mail.gmail.com>

On Sun, Nov 8, 2009 at 16:53, Georg Brandl <g.brandl at gmx.net> wrote:
> Brett Cannon schrieb:
>> During the moratorium PEP discussions Guido said he wanted to quiet
>> down deprecation warnings. I see there being two options on this.
>>
>> One is to keep things as is, but to require two releases with
>> PendingDeprecationWarning so there are three years of
>> silent-by-default warnings to update your code. But that last release
>> before removal came would still be noisy.
>>
>> The other option is to simply have all warnings filtered out by
>> default. We could alter -W so that when it is used w/o an argument it
>> turns to what is currently the default behaviour (or even turn all
>> warnings which is more than what happens now).
>
> As a technical note, optional option arguments are a bad idea.
>
> ? python -W ignore
>
> Am I calling the file "ignore" with all warnings enabled or running the
> interactive interpreter ignoring all warnings?

You are calling the file called "ignore"; the argument to -W must
contain colons to be valid.

-Brett

From brett at python.org  Mon Nov  9 03:05:51 2009
From: brett at python.org (Brett Cannon)
Date: Sun, 8 Nov 2009 18:05:51 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
Message-ID: <bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>

On Sun, Nov 8, 2009 at 13:40, Guido van Rossum <guido at python.org> wrote:
> On Sun, Nov 8, 2009 at 1:26 PM, Brett Cannon <brett at python.org> wrote:
>> During the moratorium PEP discussions Guido said he wanted to quiet
>> down deprecation warnings. I see there being two options on this.
>>
>> One is to keep things as is, but to require two releases with
>> PendingDeprecationWarning so there are three years of
>> silent-by-default warnings to update your code. But that last release
>> before removal came would still be noisy.
>>
>> The other option is to simply have all warnings filtered out by
>> default. We could alter -W so that when it is used w/o an argument it
>> turns to what is currently the default behaviour (or even turn all
>> warnings which is more than what happens now). This will require that
>> people proactively check for warnings when updating for compatibility,
>> else they will eventually use a Python release where there code will
>> simply break because something changed. This route means we do not
>> have to specify any deprecation policy right now (that would be a
>> separate discussion).
>>
>> Channeling Guido he is after the latter, but a general discussion
>> would still be good since he didn't explicitly say what he was after
>> other than to quiet down warnings.
>
> I was specifically after all kinds of deprecation warnings, which
> generally are telling you that you need to make a change in order to
> remain compatible in the future. That's sometimes interesting but
> often irrelevant noise. So I would agree with Gregory P Smith's
> proposal to just treat all deprecation warnings as silent.
>
> There are other kinds warnings which might be useful for other reasons
> -- they typically point out code that does not do what you might think
> it does. A good example is the warning added in 2.6 about "assert (x,
> y)". This is something you ignore at your peril.
>
> On Sun, Nov 8, 2009 at 1:35 PM, Michael Foord <michael at voidspace.org.uk> wrote:
>> Almost no-one is ever going to run Python with PendingDeprecation
>> warnings switched on, so there should be at least one 'noisy' release in
>> my opinion.
>
> I disagree. The -3 option is an example of a better approach: silent
> by default, warnings enabled by a command line flag. If we can trust
> developers to use -3 to check for Py3k incompatibilities, we should
> also be able to trust them to check for deprecation warnings
> explicitly.
>

There is also the argument that if people are told they have to
explicitly turn on warnings to get them and the switch turns them all
on then it might be the case that more people will see and deal with
PendingDeprecationWarnings than they do now. The arguments that people
will not switch them on is based on current practice. Guido's and
Greg's argument is that habits will change such that it is not going
to be an issue.

What does this mean for the warning filter? It will obviously tone
down its need, but would it be needed at all? If warnings are off by
default the need to actively filter them out goes down by a lot. And
you probably don't want to accidentally filter out a warning if you
have to explicitly turn them on. I would argue that if you have to
explicitly turn filters on then you don't have much of a need to
selectively silence them and thus the filter can just go away.

> (Another argument to think about: if you download and install some 3rd
> party code, deprecation warnings about that code are *only noise*
> since they are not yours to fix. Warnings in a dynamic language work
> very different than warnings in a compiled language.)

That's very true. Until Mercurial caught up to Python 2.6 the sha/md5
warnings were very annoying. I actually modified my copy to turn them
off myself.

-Brett

From nate at binkert.org  Mon Nov  9 06:25:48 2009
From: nate at binkert.org (nathan binkert)
Date: Sun, 8 Nov 2009 21:25:48 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
Message-ID: <217accd40911082125v6ea58e8fg8ce8d44a94863e77@mail.gmail.com>

>> (Another argument to think about: if you download and install some 3rd
>> party code, deprecation warnings about that code are *only noise*
>> since they are not yours to fix. Warnings in a dynamic language work
>> very different than warnings in a compiled language.)
>
> That's very true. Until Mercurial caught up to Python 2.6 the sha/md5
> warnings were very annoying. I actually modified my copy to turn them
> off myself.

I too find this sort of thing very annoying. Perhaps we should teach
people to install their programs with
#!/usr/bin/python -W ignore

This way pending deprecations are on by default, but people that
install programs will not have them.

Even better would be some sort of "release mode" argument (e.g. "-R")
to python.  That could turn off pending deprecation warnings and
perhaps other things that fall into a similar category (maybe to allow
an app developer to choose between tracebacks or not depending on the
value of this flag).    Still better if setuptools knew about the
release mode so developers would not use it and get the warnings, but
people installing it would.

I have no idea how this would be done in windows.

I'm not offering patches though, so I'll shut up now.

  Nate

From solipsis at pitrou.net  Mon Nov  9 07:10:08 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 09 Nov 2009 07:10:08 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
Message-ID: <1257747008.3507.5.camel@localhost>


> There is also the argument that if people are told they have to
> explicitly turn on warnings to get them and the switch turns them all
> on then it might be the case that more people will see and deal with
> PendingDeprecationWarnings than they do now. The arguments that people
> will not switch them on is based on current practice. Guido's and
> Greg's argument is that habits will change such that it is not going
> to be an issue.

Well as I said this helps only if the package has 100% test coverage,
such that the developer is sure to exercise all code paths when he/she
runs the test suite with warnings explicitly enabled.

Besides, as Yuvgoog pointed out, this would not help the casual hobbyist
programmer (or professional non-programmer, such as a scientist) who
doesn't really have a clue that those warnings exist and are important
to check for.

Regards

Antoine.



From lac at openend.se  Mon Nov  9 07:25:34 2009
From: lac at openend.se (Laura Creighton)
Date: Mon, 09 Nov 2009 07:25:34 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: Message from Antoine Pitrou <solipsis@pitrou.net> of "Mon,
	09 Nov 2009 07:10:08 +0100." <1257747008.3507.5.camel@localhost> 
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost> 
Message-ID: <200911090625.nA96PYfX009999@theraft.openend.se>

In a message of Mon, 09 Nov 2009 07:10:08 +0100, Antoine Pitrou writes:
>
>> There is also the argument that if people are told they have to
>> explicitly turn on warnings to get them and the switch turns them all
>> on then it might be the case that more people will see and deal with
>> PendingDeprecationWarnings than they do now. The arguments that people
>> will not switch them on is based on current practice. Guido's and
>> Greg's argument is that habits will change such that it is not going
>> to be an issue.
>
>Well as I said this helps only if the package has 100% test coverage,
>such that the developer is sure to exercise all code paths when he/she
>runs the test suite with warnings explicitly enabled.
>
>Besides, as Yuvgoog pointed out, this would not help the casual hobbyist
>programmer (or professional non-programmer, such as a scientist) who
>doesn't really have a clue that those warnings exist and are important
>to check for.
>
>Regards
>
>Antoine.

Experience has shown that when people get used to seeing 'a bunch of
warnings that don't really matter' they either a) turn them off or
b) ignore them, even when they are telling them valuable things that
they should be paying attention to.  So constantly spitting out
DeprecationWarnings as soon as something becomes deprecated is a
most excellent way to train people to ignore DeprecationWarnings.

I appreciate the desire to help the casual programmer, but it is
not clear to me that this will turn out to actually be helpful.  Teaching
people to run their code through some sort of code-checker every
so often strikes me as more likely to be so.

Laura

From ben+python at benfinney.id.au  Mon Nov  9 07:35:35 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 09 Nov 2009 17:35:35 +1100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
Message-ID: <87r5s82ojs.fsf@benfinney.id.au>

Laura Creighton <lac at openend.se> writes:

> So constantly spitting out DeprecationWarnings as soon as something
> becomes deprecated is a most excellent way to train people to ignore
> DeprecationWarnings.
[?]

> Teaching people to run their code through some sort of code-checker
> every so often strikes me as more likely to be [helpful].

Why is one of these helpful but the other one not so? Why can't the ?run
your code through a code-checker? remain as easy as running the code
with the next version of Python?

-- 
 \        ?The greatest tragedy in mankind's entire history may be the |
  `\       hijacking of morality by religion.? ?Arthur C. Clarke, 1991 |
_o__)                                                                  |
Ben Finney


From solipsis at pitrou.net  Mon Nov  9 07:39:30 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 09 Nov 2009 07:39:30 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <200911090625.nA96PYfX009999@theraft.openend.se>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
Message-ID: <1257748770.3507.12.camel@localhost>


> Experience has shown that when people get used to seeing 'a bunch of
> warnings that don't really matter' they either a) turn them off or
> b) ignore them, even when they are telling them valuable things that
> they should be paying attention to.  So constantly spitting out
> DeprecationWarnings as soon as something becomes deprecated is a
> most excellent way to train people to ignore DeprecationWarnings.

Well at least people get a chance to see them. If some people think the
warnings are useless (even though the messages warn about removal of a
construct), they won't run a code checker either.

If Mercurial users and developers hadn't seen those warnings at all,
perhaps Mercurial would have continued using deprecated constructs, and
ended up broken when the N+1 Python version had been released. If even
an established FLOSS project such as Mercurial is vulnerable to this
kind of risk, then any in-house or one-man project will be even more
vulnerable.

Besides, do we have such a code checker that is able to find out
deprecated constructs (not talking about 2to3 here) ?



From kevin at bud.ca  Mon Nov  9 07:44:45 2009
From: kevin at bud.ca (Kevin Teague)
Date: Sun, 8 Nov 2009 22:44:45 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <217accd40911082125v6ea58e8fg8ce8d44a94863e77@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<217accd40911082125v6ea58e8fg8ce8d44a94863e77@mail.gmail.com>
Message-ID: <03F48080-54E8-400D-BB0E-45B4DDC1A6E2@bud.ca>


On Nov 8, 2009, at 9:25 PM, nathan binkert wrote:

>>> (Another argument to think about: if you download and install some  
>>> 3rd
>>> party code, deprecation warnings about that code are *only noise*
>>> since they are not yours to fix. Warnings in a dynamic language work
>>> very different than warnings in a compiled language.)
>>
>> That's very true. Until Mercurial caught up to Python 2.6 the sha/md5
>> warnings were very annoying. I actually modified my copy to turn them
>> off myself.
>

+1 for making warnings silent unless asked for.

There is a time when I'm in "migration and maintenance mode" when I  
want to see these warnings and fix them. When I'm not in this mode,  
deprecation warnings are like some annoying co-worker who stops by  
every 15 minutes with a super trivial questions to which you respond,  
"Go away, do a Google search for the answer!" The nagging of seeing  
deprecations all the time doesn't make me want to fix the deprecations  
even more, but tends to have the reverse effect. Maybe a flaw in  
myself, but quite a common one in humans I think :P

> I too find this sort of thing very annoying. Perhaps we should teach
> people to install their programs with
> #!/usr/bin/python -W ignore
>
> This way pending deprecations are on by default, but people that
> install programs will not have them.
>


Hey, I'd never thought of this before. I'd still rather reverse it,  
and teach people how to install with warnings enabled.

For those installing scripts with Buildout, you can use  
"initialization = sys.warnoptions = ['ignore']" to silence warnings.

Here's a Buildout with the necessary configuration for "silent"  
Mercurial install (not that a silent install is necessary ATM, but it  
would "protect" you if a future release re-introduced new deprecation  
warnings ...):

[buildout]
parts = mercurial

[mercurial]
recipe = zc.recipe.egg
eggs = mercurial
entry-points = hg=mercurial.dispatch:run
initialization =
     sys.warnoptions = ['ignore']

Speaking of, I need to submit a patch to mercurial so that the command- 
line entry point is specified. But like seeing deprecation warnings in  
third-party code, it's always a big chasm to approach some project  
which you have no involvement in (and need to determine if fixes are  
done on a mailing list, personal e-mail, issue tracker, etc.) for such  
a trivial fixes as entry points and deprecation warnings ...


From lac at openend.se  Mon Nov  9 08:31:39 2009
From: lac at openend.se (Laura Creighton)
Date: Mon, 09 Nov 2009 08:31:39 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: Message from Ben Finney <ben+python@benfinney.id.au> 
	of "Mon, 09 Nov 2009 17:35:35 +1100." <87r5s82ojs.fsf@benfinney.id.au> 
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<87r5s82ojs.fsf@benfinney.id.au> 
Message-ID: <200911090731.nA97Vdj4014042@theraft.openend.se>

In a message of Mon, 09 Nov 2009 17:35:35 +1100, Ben Finney writes:
>Laura Creighton <lac at openend.se> writes:
>
>> So constantly spitting out DeprecationWarnings as soon as something
>> becomes deprecated is a most excellent way to train people to ignore
>> DeprecationWarnings.
>[???]
>
>> Teaching people to run their code through some sort of code-checker
>> every so often strikes me as more likely to be [helpful].
>
>Why is one of these helpful but the other one not so? Why can't the run
>your code through a code-checker remain as easy as running the code
>with the next version of Python?
>

>Ben Finney

Because casual programmers are not motivated to go after Deprecation
Warnings and modify their code to make such things go away. They're
coding to a different standard, one where you don't go off and change
things unless you absolutely have to.  So lots of DeprecationWarnings
will only train them to ignore DeprecationWarnings, or all Warnings.

Laura

From brian at sweetapp.com  Mon Nov  9 08:41:20 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Mon, 9 Nov 2009 18:41:20 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com>
	<5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com>
Message-ID: <F516D09C-3615-4B95-8B86-144B5E1DFD8C@sweetapp.com>


On Nov 8, 2009, at 7:01 PM, Jeffrey Yasskin wrote:

> Did you mean to drop the list? Feel free to cc them back in when you  
> reply.

No, that was a brain malfunction. Redirecting the discussion to the  
list.

> On Sat, Nov 7, 2009 at 3:31 PM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>>
>> On 8 Nov 2009, at 06:37, Jeffrey Yasskin wrote:
>>
>>> On Sat, Nov 7, 2009 at 7:32 AM, Jesse Noller <jnoller at gmail.com>  
>>> wrote:
>>>>
>>>> On Sat, Nov 7, 2009 at 10:21 AM, Antoine Pitrou <solipsis at pitrou.net 
>>>> >
>>>> wrote:
>>>>>
>>>>>> Which API? My comment wasn't aimed at the API of the package -  
>>>>>> in the
>>>>>> time I got to scan it last night nothing jumped out at me as  
>>>>>> overly
>>>>>> offensive API-wise.
>>>>>
>>>>> Not offensive, but probably too complicated if it's meant to be  
>>>>> a simple
>>>>> helper. Anyway, let's wait for the PEP.
>>>>
>>>>
>>>> The PEP is right here:
>>>>
>>>> http://code.google.com/p/pythonfutures/source/browse/trunk/PEP.txt
>>>>
>>>> I'm interested in hearing specific complaints about the API in the
>>>> context of what it's trying to *do*. The only thing which jumped  
>>>> out
>>>> at me was the number of methods on FutureList; but then again, each
>>>> one of those makes conceptual sense, even if they are verbose -
>>>> they're explicit on what's being done.
>>>
>>> Overall, I like the idea of having futures in the standard library,
>>> and I like the idea of pulling common bits of multiprocessing and
>>> threading into a concurrent.* package. Here's my
>>> stream-of-consciousness review of the PEP. I'll try to ** things  
>>> that
>>> really affect the API.
>>>
>>> The "Interface" section should start with a conceptual description  
>>> of
>>> what Executor, Future, and FutureList are. Something like "An  
>>> Executor
>>> is an object you can hand tasks to, which will run them for you,
>>> usually in another thread or process. A Future represents a task  
>>> that
>>> may or may not have completed yet, and which can be waited for and  
>>> its
>>> value or exception queries. A FutureList is ... <haven't read that
>>> far>."
>>>
>>> ** The Executor interface is pretty redundant, and it's missing the
>>> most basic call. Fundamentally, all you need is an
>>> Executor.execute(callable) method returning None,
>>
>> How do you extract the results?
>
> To implement submit in terms of execute, you write something like:
>
> def submit(executor, callable):
>  future = Future()
>  def worker():
>    try:
>      result = callable()
>    except:
>      future.set_exception(sys.exc_info())
>    else:
>      future.set_value(result)
>  executor.execute(worker)
>  return future

I see. I'm not sure if that abstraction is useful but I get it now.

>>> and all the
>>> future-oriented methods can be built on top of that. I'd support  
>>> using
>>> Executor.submit(callable) for the simplest method instead, which
>>> returns a Future, but there should be some way for implementers to
>>> only implement execute() and get submit either for free or with a
>>> 1-line definition. (I'm using method names from
>>>
>>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html
>>> in case I'm unclear about the semantics here.) run_to_futures,
>>> run_to_results, and map should be implementable on top of the Future
>>> interface, and shouldn't need to be methods on Executor. I'd  
>>> recommend
>>> they be demoted to helper functions in the concurrent.futures module
>>> unless there's a reason they need to be methods, and that reason
>>> should be documented in the PEP.
>>>
>>> ** run_to_futures() shouldn't take a return_when argument. It should
>>> be possible to wait for those conditions on any list of Futures.
>>> (_not_ just a FutureList)
>>
>> I packaged up Futures into FutureLists to fix an annoyance that I  
>> have with
>> the Java implementation - you have all of these Future objects but no
>> convenient way of operating over them.
>
> Yep, I totally agree with that annoyance. Note, though, that Java has
> the CompletionService to support nearly same use cases as
> run_to_futures.

CompletionService's use case is handling results as they finish (just  
like the callbacks do in Deferreds).

The FutureList use case is querying e.g. which callables raised, which  
returned, which are still running?

>> I made the FutureList the unit of waiting because:
>> 1. I couldn't think of a use case where this wasn't sufficient
>
> Take your webcrawl example. In a couple years, when Futures are widely
> accepted, it's quite possible that urllib.request.urlopen() will
> return a Future instead of a file. Then I'd like to request a bunch of
> URLs and process each as they come back. With the run_to_futures (or
> CompletionService) API, urllib would instead have to take a set of
> requests to open at once, which makes its API much harder to design.
> With a wait-for-any function, urllib could continue to return a single
> Future and let its users combine several results.

If we go down this road then we should just switch to Twisted :-)

Seriously, the idea is that no one would ever change their API to  
accommodate futures - they are a way of making a library with no  
notion of concurrency concurrent.

But I am starting to be convinced that individual futures are a good  
idea because it makes the run/submit method easier to use.

> Alternately, say you have an RPC system returning Futures. You've sent
> off RPCs A, B, and C. Now you need two separate subsystems D and E to
> do something with the results, except that D can continue when either
> A or B finishes, but E can continue when either B or C finishes. Can D
> and E express just what they need to express, or do they have to deal
> with futures they don't really care about?
>
>> 2. It makes the internal locking semantics a bit easier and faster  
>> (if you
>> can wait on any future then the wait has to acquire a lock for  
>> every future
>> [in a consistent order to prevent deadlocks when other threads are  
>> doing the
>> same thing with an intersecting set of futures], add a result  
>> listener for
>> each and then great some sort of object to aggregate their state)
>
> Yep. I suspect the extra overhead isn't significant compared to the
> cost of scheduling threads.
>
>> But I am thinking that maybe FutureLists aren't the right  
>> abstraction.
>>
>>> The code sample looks like Executor is a context manager. What does
>>> its __exit__ do? shutdown()? shutdown&awaitTermination? I prefer
>>> waiting in Executor.__exit__, since that makes it easier for users  
>>> to
>>> avoid having tasks run after they've cleaned up data those tasks
>>> depend on. But that could be my C++ bias, where we have to be sure  
>>> to
>>> free memory in the right places. Frank, does Java run into any
>>> problems with people cleaning things up that an Executor's tasks
>>> depend on without awaiting for the Executor first?
>>>
>>> shutdown should explain why it's important. Specifically, since the
>>> Executor controls threads, and those threads hold a reference to the
>>> Executor, nothing will get garbage collected without the explicit
>>> call.
>>
>> Actually, the threads hold a weakref to the Executor so they can  
>> exit (when
>> the work queue is empty) if the Executor is collected. Here is the  
>> code from
>> futures/thread.py:
>>
>>  while True:
>>    try:
>>        work_item = work_queue.get(block=True, timeout=0.1)
>>    except queue.Empty:
>>        executor = executor_reference()
>>        # Exit if:
>>        #   - The interpreter is shutting down OR
>>        #   - The executor that owns the worker has been collected OR
>>        #   - The executor that owns the worker has been shutdown.
>>        if _shutdown or executor is None or executor._shutdown:
>>            return
>
> Oh, yeah, that sounds like it might work. So why does shutdown exist?

It does work - there are tests and everything :-)

.shutdown exists for the same reason that .close exists on files:
- Python does not guarantee any particular GC strategy
- tracebacks and other objects may retain a reference in an unexpected  
way
- sometimes you want to free your resources before the function exits

>
>>> ** What happens when FutureList.wait(FIRST_COMPLETED) is called  
>>> twice?
>>> Does it return immediately the second time? Does it wait for the
>>> second task to finish? I'm inclined to think that FutureList  
>>> should go
>>> away and be replaced by functions that just take lists of Futures.
>>
>> It waits until a new future is completed.
>
> That seems confusing, since it's no longer the "FIRST" completed.

Maybe "NEXT_COMPLETED" would be better.

Cheers,
Brian

>
>>> In general, I think the has_done_futures(), exception_futures(),  
>>> etc.
>>> are fine even though their results may be out of date by the time  
>>> you
>>> inspect them. That's because any individual Future goes  
>>> monotonically
>>> from not-started->running->(exception|value), so users can take
>>> advantage of even an out-of-date done_futures() result. However,  
>>> it's
>>> dangerous to have several query functions, since users may think  
>>> that
>>> running_futures() `union` done_futures() `union` cancelled_futures()
>>> covers the whole FutureList, but instead a Future can move between  
>>> two
>>> of the sets between two of those calls. Instead, perhaps an atomic
>>> partition() function would be better, which returns a collection of
>>> sub-lists that cover the whole original set.
>>>
>>> I would rename result() to get() (or maybe Antoine's suggestion of
>>> __call__) to match Java. I'm not sure exception() needs to exist.
>>>
>>> --- More general points ---
>>>
>>> ** Java's Futures made a mistake in not supporting work stealing,  
>>> and
>>> this has caused deadlocks at Google. Specifically, in a bounded-size
>>> thread or process pool, when a task in the pool can wait for work
>>> running in the same pool, you can fill up the pool with tasks that  
>>> are
>>> waiting for tasks that haven't started running yet. To avoid this,
>>> Future.get() should be able to steal the task it's waiting on out of
>>> the pool's queue and run it immediately.
>>>
>>> ** I think both the Future-oriented blocking model and the
>>> callback-based model Deferreds support are important for different
>>> situations. Futures tend to be easier to program with, while  
>>> Deferreds
>>> use fewer threads and can have more predictable latencies. It should
>>> be possible to create a Future from a Deferred or a Deferred from a
>>> Future without taking up a thread.


From brian at sweetapp.com  Mon Nov  9 08:58:03 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Mon, 9 Nov 2009 18:58:03 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <F516D09C-3615-4B95-8B86-144B5E1DFD8C@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com>
	<5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com>
	<F516D09C-3615-4B95-8B86-144B5E1DFD8C@sweetapp.com>
Message-ID: <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com>

Hey everyone,

Thanks for all the great feedback!

I'm going to compile everyone's feedback and then send out a list of  
proposed changes. In the meantime, more discussion is welcome :-)

Cheers,
Brian


From ubershmekel at gmail.com  Mon Nov  9 09:02:59 2009
From: ubershmekel at gmail.com (Yuvgoog Greenle)
Date: Mon, 9 Nov 2009 10:02:59 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <200911090731.nA97Vdj4014042@theraft.openend.se>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<ben+python@benfinney.id.au> <87r5s82ojs.fsf@benfinney.id.au>
	<200911090731.nA97Vdj4014042@theraft.openend.se>
Message-ID: <9d153b7c0911090002u2f57b2efm99b5da7d579bced9@mail.gmail.com>

from http://eight.pairlist.net/pipermail/geeklog-devel/2008-December/003950.html
>?FYI: When pulling from / pushing to the Mercurial repository, you may
see warning messages like these:
[snip]
> You can ignore them for now - things should work just fine nonetheless.
>
> A Python upgrade on the webserver broke the repository viewer and
> after repairing this, you now get these messages. I guess we'll have
> to upgrade to Mercurial 1.1 eventually.
>
> bye, Dirk

What would happen if a single untested mercurial feature just broke? I
hope you realize this IS going to happen. Someone, somewhere is going
to one day see his code just breaking without warning.

If the warnings are annoying to you, lengthen the deprecation period
so the day of magic breakage will be the day the annoying warning
starts. People who are annoyed by warnings have no point in the
argument except for shortening the cycle of change which is something
I noticed python is against nowadays.


== a visualization ==
today:
    working, working*, warning, breaking!

Guido suggests:
    working, working*, working*, breaking!

MIchael Foord suggests a win-win proposal:
    working, working*, working*, warning, breaking!


* a special flag is needed for seeing silent warnings

--yuv

From ubershmekel at gmail.com  Mon Nov  9 09:12:38 2009
From: ubershmekel at gmail.com (Yuvgoog Greenle)
Date: Mon, 9 Nov 2009 10:12:38 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <200911090731.nA97Vdj4014042@theraft.openend.se>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<ben+python@benfinney.id.au> <87r5s82ojs.fsf@benfinney.id.au>
	<200911090731.nA97Vdj4014042@theraft.openend.se>
Message-ID: <9d153b7c0911090012r416f6926pa65a2b5e96de784f@mail.gmail.com>

On Mon, Nov 9, 2009 at 9:31 AM, Laura Creighton <lac at openend.se> wrote:
> Because casual programmers are not motivated to go after Deprecation
> Warnings
[ snip ]
> So lots of DeprecationWarnings
> will only train them to ignore DeprecationWarnings, or all Warnings.
>
> Laura
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>

This isn't just about casual programmers, everybody who uses python on
a daily basis for just interpreter stuff, are they supposed to change
their interpreter shortcut to have the warnings option just to stay up
to date ?

Not everyone that uses python is a package designer with unit tests
and a buildbot that runs with all the most advanced configurations
(that show hidden warnings).

This warning hiding feature's just making python a bit more of a "dark
art of the few who know it well". What ever happened to "unless
explicitly silenced"?

--yuv

From lac at openend.se  Mon Nov  9 09:17:01 2009
From: lac at openend.se (Laura Creighton)
Date: Mon, 09 Nov 2009 09:17:01 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: Message from Antoine Pitrou <solipsis@pitrou.net> of "Mon,
	09 Nov 2009 07:39:30 +0100." <1257748770.3507.12.camel@localhost> 
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost> 
Message-ID: <200911090817.nA98H1ol017072@theraft.openend.se>

In a message of Mon, 09 Nov 2009 07:39:30 +0100, Antoine Pitrou writes:
>
>> Experience has shown that when people get used to seeing 'a bunch of
>> warnings that don't really matter' they either a) turn them off or
>> b) ignore them, even when they are telling them valuable things that
>> they should be paying attention to.  So constantly spitting out
>> DeprecationWarnings as soon as something becomes deprecated is a
>> most excellent way to train people to ignore DeprecationWarnings.
>
>Well at least people get a chance to see them. If some people think the
>warnings are useless (even though the messages warn about removal of a
>construct), they won't run a code checker either.

This is not true.  Once people get into the habit of 'not seeing'
things, they don't see them even when they are important.  This
is not only true of program warnings -- somebody has done a
study of how cancerous tumours are overlooked.  In one particularly
frightening study, doctors at the Mayo clinic went back and checked
the previously 'normal' chest X-rays of patients who subsequently
developed cancer.  What they found was horrifying: 90% of the
tumours were visible on the previous Xrays.

(see Lorenz, G.B.A. et. al 1999 Miss Rate of Lung Cancer on the
Chest Radiograph in Clinical Practice in _Chest_ also
Berlin, Leonard 2000 Hindsight Bias in _American Journal of
Roentgenology_)

A test in 2002 indicated that the TSA missed 1 in every 4 guns
that were attempted to be smuggled in.  In 2004 a test at Newark's
airport indicated the same thing.  At 2005 test at O'Hare's airport
had 60% of bombs and explosive materials missed, while security
officials in L.A. missed 75%.

(I don't have the papers for this, I read this as a clipping from
the Wall Street Journal, which didn't list its sources.)

So this 'it's there, but we didn't see it' is a serious problem 
with wideranging implications outside of generating computer
warnings.  But we know some things about this.  The reason we
are lousy at detecting such things is in large part because they
are rare.  Most chest Xrays show no cancer.  Most bags do not
contain weapons and guns.  We're only good at detecting things
when a) we are looking for them and b) they actually occur
relatively commonly where we are looking.

As what we are looking for becomes more and more rare, then we are
more and more likely to overlook them when they do occur.  There
are lots of cognitive psychology experiments to test this, for
instance: Levin and Simons 1997 Failure to Detect Changes to
Attended Objects in Motion Pictures _Psychonomic Bulletin and Review_

The conclusion is that 'surprising' people with unexpected warnings
is less useful than one would think -- people tend to overlook them,
and thus not be surprised.  It's better when people get warnings
which they have asked to see.  Then they tend to notice them.  But
poorest of all is when people have trained themselves not to see
warnings at all.

>If Mercurial users and developers hadn't seen those warnings at all,
>perhaps Mercurial would have continued using deprecated constructs, and
>ended up broken when the N+1 Python version had been released. If even
>an established FLOSS project such as Mercurial is vulnerable to this
>kind of risk, then any in-house or one-man project will be even more
>vulnerable.

I agree, but I think that Mercurial developers ought to include a
set of people who are interested in looking at warnings precisely 
to prevent such things.  Thus their use case is different from that
of the casual programmer, or the user of Mercurial who has no
intention of doing anything with any warnings that ever show up.
The first group will notice things the most if they get warnings
when they explicitly ask for it.  If the warnings show up unasked for,
they too can be expected to get into the habit of ignoring them.
('oh they are just warnings').  But I don't know the precise point
where producing such warnings becomes more harmful than helpful.
As far as I know, nobody has run any experiements to determine 
this.

But I would suspect that blasting out all the DeprecationWarnings
for 3 whole releases before something goes away would err on the
'so frequent that it is ignored' side.

>Besides, do we have such a code checker that is able to find out
>deprecated constructs (not talking about 2to3 here) ?

I was thinking of something more primative, such as running your
code with all warnings on from time to time.

Laura


From solipsis at pitrou.net  Mon Nov  9 12:37:05 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 09 Nov 2009 12:37:05 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <200911090817.nA98H1ol017072@theraft.openend.se>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<200911090817.nA98H1ol017072@theraft.openend.se>
Message-ID: <1257766625.3888.15.camel@localhost>

Le lundi 09 novembre 2009 ? 09:17 +0100, Laura Creighton a ?crit :
> The conclusion is that 'surprising' people with unexpected warnings
> is less useful than one would think -- people tend to overlook them,
> and thus not be surprised.

Whether or not it's "less useful than one would think" doesn't make it
useless. There are many things which fall under the former predicate and
not under the latter. For example documentation (many people don't read
it), unit tests (they give a false sense of security)... do you suggest
we drop them too?

> It's better when people get warnings
> which they have asked to see.

Well, I don't see a contention here: they already can. If they want to
get warnings, they just have to run the program. It's not like Python
writes out lots of warnings, which is why you normally don't /need/ to
choose a specific kind of warning to display.

(-3 is an exception because it /can/ output many more warnings than is
reasonable, but that's part of why it is opt-in rather than opt-out)

> But I would suspect that blasting out all the DeprecationWarnings
> for 3 whole releases before something goes away would err on the
> 'so frequent that it is ignored' side.

I don't think anybody suggested that.

> I was thinking of something more primative, such as running your
> code with all warnings on from time to time.

Which gives far less code coverage than enabling them by default.



From lac at openend.se  Mon Nov  9 13:33:35 2009
From: lac at openend.se (Laura Creighton)
Date: Mon, 09 Nov 2009 13:33:35 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: Message from Antoine Pitrou <solipsis@pitrou.net> of "Mon,
	09 Nov 2009 12:37:05 +0100." <1257766625.3888.15.camel@localhost> 
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<200911090817.nA98H1ol017072@theraft.openend.se>
	<1257766625.3888.15.camel@localhost> 
Message-ID: <200911091233.nA9CXZqj002387@theraft.openend.se>

In a message of Mon, 09 Nov 2009 12:37:05 +0100, Antoine Pitrou writes:
>Le lundi 09 novembre 2009 ?? 09:17 +0100, Laura Creighton a ??crit :
>> The conclusion is that 'surprising' people with unexpected warnings
>> is less useful than one would think -- people tend to overlook them,
>> and thus not be surprised.
>
>Whether or not it's "less useful than one would think" doesn't make it
>useless. There are many things which fall under the former predicate and
>not under the latter. For example documentation (many people don't read
>it), unit tests (they give a false sense of security)... do you suggest
>we drop them too?

Ok, I was not strong enough.

Here is what I believe:

Take some sample set of 2000 programs, all of which use features
which are scheduled to be deprecated.  Divide them into two groups
of 1000.  In group A you issue DeprecationWarnings for one whole
release before the one where this code will break, unless the
programmer explicitly turns them off.  In group B you don't issue
any warnings unless the programmer asks for them.

Come the day when /env/python now points to the new release, which
group do you believe will have done a better job of modernising their
programs?  My belief is that this will be group B.  I think that the
number of programmers in group A who no longer saw the warnings 
because they were always there will greatly outnumber the ones who 
never run their programs with warnings turned on, and thus were 
completely unaware of the problem in the first place.  But I am
making some assumptions here -- one is that these programs will
be run fairly frequently.  The second assumption is that the
number of programs for whom the fix will be 'change the first line
to #! /usr/bin/env/some-older-version-of-python' is small.  I 
don't know if those are decent assumptions to make.  I don't
have anything approaching real data as to how Python programs
exist out there in the world.

The "does providing Deprecation Warnings when unasked for actually
help" experiement would be interesting to run.  This is the sort of
thing we need real numbers for, but I haven't seen any experiments
to test this so far.  All I have seen is indications from cognative
psychology that it is really hard for people to see things when
they are concentrating on something else.

For instance there is the test here
http://viscog.beckman.illinois.edu/media/dailytelegraph.html

'view the basketball video'

which is explained here:
http://www.scholarpedia.org/article/Inattentional_blindness

I think that we have two things to worry about here, inattentional
blindness (which is what the video tests) and problems caused by
insufficient attention (as opposed to inattention).  And, links
that I posted before which say that people really only see what
they expect, rather than what is there, and that our brains are
heavily optimised for 'ignoring stuff that doesn't matter' which
gets in the way of us noticing stuff that does matter.  Unless 
we expect this.

Laura



From solipsis at pitrou.net  Mon Nov  9 13:45:56 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 09 Nov 2009 13:45:56 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <200911091233.nA9CXZqj002387@theraft.openend.se>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<200911090817.nA98H1ol017072@theraft.openend.se>
	<1257766625.3888.15.camel@localhost>
	<200911091233.nA9CXZqj002387@theraft.openend.se>
Message-ID: <1257770756.3888.24.camel@localhost>


> I think that we have two things to worry about here, inattentional
> blindness (which is what the video tests) and problems caused by
> insufficient attention (as opposed to inattention).  And, links
> that I posted before which say that people really only see what
> they expect, rather than what is there, and that our brains are
> heavily optimised for 'ignoring stuff that doesn't matter' which
> gets in the way of us noticing stuff that does matter.

I guess we are all more or less acquainted with these phenomena. But it
still doesn't equate with "warnings are useless".

(and, by the way, the analogies you are trying to make with
aforementioned scientific research look very poor to me)



From betatim at gmail.com  Mon Nov  9 18:42:49 2009
From: betatim at gmail.com (Tim Head)
Date: Mon, 9 Nov 2009 11:42:49 -0600
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <200911090731.nA97Vdj4014042@theraft.openend.se>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<ben+python@benfinney.id.au> <87r5s82ojs.fsf@benfinney.id.au>
	<200911090731.nA97Vdj4014042@theraft.openend.se>
Message-ID: <85dba0990911090942x2c9a20d1mbbdb5e9eebbee454@mail.gmail.com>

2009/11/9 Laura Creighton <lac at openend.se>:
> In a message of Mon, 09 Nov 2009 17:35:35 +1100, Ben Finney writes:
>>Laura Creighton <lac at openend.se> writes:
>>
>>> So constantly spitting out DeprecationWarnings as soon as something
>>> becomes deprecated is a most excellent way to train people to ignore
>>> DeprecationWarnings.
>>[???]
>>
>>> Teaching people to run their code through some sort of code-checker
>>> every so often strikes me as more likely to be [helpful].
>>
>>Why is one of these helpful but the other one not so? Why can't the run
>>your code through a code-checker remain as easy as running the code
>>with the next version of Python?
>>
>
>>Ben Finney
>
> Because casual programmers are not motivated to go after Deprecation
> Warnings and modify their code to make such things go away. They're
> coding to a different standard, one where you don't go off and change
> things unless you absolutely have to. ?So lots of DeprecationWarnings
> will only train them to ignore DeprecationWarnings, or all Warnings.
>
> Laura

When I run some code that isn't mine and it produces
DeprecationWarnings "everytime" I start it or do my usual stuff with
it, it annoys me. I then consider doing one of three things:

* update to most recent version of package and hope they have gone away,
* hack/update code myself to make them go away,
* search for a way to silence warnings because they are annoying.

Which of the three I do depends on which one is the easiest/least
effort. If the deprecation warning tells me which line to go to, and
basically what change to make (think the error sum(s for s in
list_of_strings) gives you) then I will do that. So maybe what we need
is good deprecation warnings that tell you how to fix the code to make
them go away.

Tim

> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>



-- 
http://tim.jottit.com/

From guido at python.org  Mon Nov  9 19:02:35 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2009 10:02:35 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <1257748770.3507.12.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com> 
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com> 
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se> 
	<1257748770.3507.12.camel@localhost>
Message-ID: <ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>

[Laura]
>> Experience has shown that when people get used to seeing 'a bunch of
>> warnings that don't really matter' they either a) turn them off or
>> b) ignore them, even when they are telling them valuable things that
>> they should be paying attention to. ?So constantly spitting out
>> DeprecationWarnings as soon as something becomes deprecated is a
>> most excellent way to train people to ignore DeprecationWarnings.

On Sun, Nov 8, 2009 at 10:39 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Well at least people get a chance to see them. If some people think the
> warnings are useless (even though the messages warn about removal of a
> construct), they won't run a code checker either.

You're misreading Laura. She's not saying that some people choose to
ignore the warnings. She's pointing out that it is a scientifically
known fact that people will *learn to ignore warnings* if they occur
frequently enough, and that once this is learned the warnings have no
effect. Look up Jef Raskin's diatribes against modal dialogs. (Or read
the "cry wolf" fairy tale. :-)

> If Mercurial users and developers hadn't seen those warnings at all,
> perhaps Mercurial would have continued using deprecated constructs, and
> ended up broken when the N+1 Python version had been released. If even
> an established FLOSS project such as Mercurial is vulnerable to this
> kind of risk, then any in-house or one-man project will be even more
> vulnerable.

If the Mercurial developers claimed that Mercurial supported Python
2.6 without extensive testing they would be rank amateurs. I don't
believe they are.

> Besides, do we have such a code checker that is able to find out
> deprecated constructs (not talking about 2to3 here) ?

At Google we use a hacked-up version of pylint which seems infinitely
flexible in the checks you can specify. I assume that the public
version of pylint is just as capable -- someone just needs to write a
rule for it.

-- 
--Guido van Rossum (python.org/~guido)

From ssteinerx at gmail.com  Mon Nov  9 19:05:26 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 9 Nov 2009 13:05:26 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
Message-ID: <76E6A8F2-4D50-4232-887F-118523352414@gmail.com>


On Nov 9, 2009, at 1:02 PM, Guido van Rossum wrote:
> At Google we use a hacked-up version of pylint which seems infinitely
> flexible in the checks you can specify. I assume that the public
> version of pylint is just as capable -- someone just needs to write a
> rule for it.

Is that "hacked-up" version publicly available, or could it be?

Thanks,

S


From guido at python.org  Mon Nov  9 19:11:52 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2009 10:11:52 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <76E6A8F2-4D50-4232-887F-118523352414@gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com> 
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com> 
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se> 
	<1257748770.3507.12.camel@localhost>
	<ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com> 
	<76E6A8F2-4D50-4232-887F-118523352414@gmail.com>
Message-ID: <ca471dc20911091011y22ba38fcvcbc22f8e7ecbb791@mail.gmail.com>

On Mon, Nov 9, 2009 at 10:05 AM, ssteinerX at gmail.com
<ssteinerx at gmail.com> wrote:
>
> On Nov 9, 2009, at 1:02 PM, Guido van Rossum wrote:
>>
>> At Google we use a hacked-up version of pylint which seems infinitely
>> flexible in the checks you can specify. I assume that the public
>> version of pylint is just as capable -- someone just needs to write a
>> rule for it.
>
> Is that "hacked-up" version publicly available, or could it be?

I don't know and have no time to dive into this, but I suspect it
isn't of general interest, since AFAIK what we added is mostly rules
to enforce Google-specific coding standards.

-- 
--Guido van Rossum (python.org/~guido)

From ssteinerx at gmail.com  Mon Nov  9 19:13:12 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 9 Nov 2009 13:13:12 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911091011y22ba38fcvcbc22f8e7ecbb791@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
	<76E6A8F2-4D50-4232-887F-118523352414@gmail.com>
	<ca471dc20911091011y22ba38fcvcbc22f8e7ecbb791@mail.gmail.com>
Message-ID: <866AA07A-3BC5-46FE-AD53-DFBFDC8D2F18@gmail.com>


On Nov 9, 2009, at 1:11 PM, Guido van Rossum wrote:

> On Mon, Nov 9, 2009 at 10:05 AM, ssteinerX at gmail.com
> <ssteinerx at gmail.com> wrote:
>>
>> On Nov 9, 2009, at 1:02 PM, Guido van Rossum wrote:
>>>
>>> At Google we use a hacked-up version of pylint which seems  
>>> infinitely
>>> flexible in the checks you can specify. I assume that the public
>>> version of pylint is just as capable -- someone just needs to  
>>> write a
>>> rule for it.
>>
>> Is that "hacked-up" version publicly available, or could it be?
>
> I don't know and have no time to dive into this, but I suspect it
> isn't of general interest, since AFAIK what we added is mostly rules
> to enforce Google-specific coding standards.

Ok, thanks,

S


From ubershmekel at gmail.com  Mon Nov  9 19:25:19 2009
From: ubershmekel at gmail.com (Yuvgoog Greenle)
Date: Mon, 9 Nov 2009 20:25:19 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
Message-ID: <9d153b7c0911091025q68e36b5fjffd732bf3a522f03@mail.gmail.com>

I understand some people on this list ran code that kept giving them
warnings until they were "warning-blind" so I'd like to know why they
didn't run python with "-W ignore" and enjoyed a warning-less world of
wonder?

Either that or I'd rather these
ultimate-and-bug-less-that-run-with-special-options programmers to add
the ignore warnings stuff in their code, that way when their code
magically breaks upon a language change, there's proof that the real
problem was their ego.

If you feel it's hard to ignore warnings, then make it easier, don't
break the system because it's a good one that's saved me and many
others from doing silly things we would have never noticed until it
was too late.

--yuv

From g.brandl at gmx.net  Mon Nov  9 20:40:48 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 09 Nov 2009 19:40:48 +0000
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911081744l276f2563xe548bd701e3ce405@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<hd7lqc$59t$1@ger.gmane.org>
	<bbaeab100911081744l276f2563xe548bd701e3ce405@mail.gmail.com>
Message-ID: <hd9r4j$aon$1@ger.gmane.org>

Brett Cannon schrieb:
> On Sun, Nov 8, 2009 at 16:53, Georg Brandl <g.brandl at gmx.net> wrote:
>> Brett Cannon schrieb:
>>> During the moratorium PEP discussions Guido said he wanted to quiet
>>> down deprecation warnings. I see there being two options on this.
>>>
>>> One is to keep things as is, but to require two releases with
>>> PendingDeprecationWarning so there are three years of
>>> silent-by-default warnings to update your code. But that last release
>>> before removal came would still be noisy.
>>>
>>> The other option is to simply have all warnings filtered out by
>>> default. We could alter -W so that when it is used w/o an argument it
>>> turns to what is currently the default behaviour (or even turn all
>>> warnings which is more than what happens now).
>>
>> As a technical note, optional option arguments are a bad idea.
>>
>>   python -W ignore
>>
>> Am I calling the file "ignore" with all warnings enabled or running the
>> interactive interpreter ignoring all warnings?
> 
> You are calling the file called "ignore"; the argument to -W must
> contain colons to be valid.

It doesn't need to now; and IMO that is quite helpful, since it allows e.g. a
very short -Wi argument without me having to remember just how many colons
I have to put there.

And even if that change is deemed to be fine, why shouldn't I have colons in
my filename?  Special cases are not special enough... ;)

Georg


From brett at python.org  Mon Nov  9 21:45:34 2009
From: brett at python.org (Brett Cannon)
Date: Mon, 9 Nov 2009 12:45:34 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <hd9r4j$aon$1@ger.gmane.org>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<hd7lqc$59t$1@ger.gmane.org>
	<bbaeab100911081744l276f2563xe548bd701e3ce405@mail.gmail.com> 
	<hd9r4j$aon$1@ger.gmane.org>
Message-ID: <bbaeab100911091245x1f669e2bq6a896d7e4471e4b3@mail.gmail.com>

On Mon, Nov 9, 2009 at 11:40, Georg Brandl <g.brandl at gmx.net> wrote:
> Brett Cannon schrieb:
>> On Sun, Nov 8, 2009 at 16:53, Georg Brandl <g.brandl at gmx.net> wrote:
>>> Brett Cannon schrieb:
>>>> During the moratorium PEP discussions Guido said he wanted to quiet
>>>> down deprecation warnings. I see there being two options on this.
>>>>
>>>> One is to keep things as is, but to require two releases with
>>>> PendingDeprecationWarning so there are three years of
>>>> silent-by-default warnings to update your code. But that last release
>>>> before removal came would still be noisy.
>>>>
>>>> The other option is to simply have all warnings filtered out by
>>>> default. We could alter -W so that when it is used w/o an argument it
>>>> turns to what is currently the default behaviour (or even turn all
>>>> warnings which is more than what happens now).
>>>
>>> As a technical note, optional option arguments are a bad idea.
>>>
>>> ? python -W ignore
>>>
>>> Am I calling the file "ignore" with all warnings enabled or running the
>>> interactive interpreter ignoring all warnings?
>>
>> You are calling the file called "ignore"; the argument to -W must
>> contain colons to be valid.
>
> It doesn't need to now; and IMO that is quite helpful, since it allows e.g. a
> very short -Wi argument without me having to remember just how many colons
> I have to put there.
>
> And even if that change is deemed to be fine, why shouldn't I have colons in
> my filename? ?Special cases are not special enough... ;)

OK, fine, I didn't think it through very much when I proposed -W. We
can add -w for the same purpose.

-Brett

From brett at python.org  Mon Nov  9 22:18:52 2009
From: brett at python.org (Brett Cannon)
Date: Mon, 9 Nov 2009 13:18:52 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
Message-ID: <bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>

On Sun, Nov 8, 2009 at 13:40, Guido van Rossum <guido at python.org> wrote:
[SNIP]

> On Sun, Nov 8, 2009 at 1:35 PM, Michael Foord <michael at voidspace.org.uk> wrote:
>> Almost no-one is ever going to run Python with PendingDeprecation
>> warnings switched on, so there should be at least one 'noisy' release in
>> my opinion.
>
> I disagree. The -3 option is an example of a better approach: silent
> by default, warnings enabled by a command line flag. If we can trust
> developers to use -3 to check for Py3k incompatibilities, we should
> also be able to trust them to check for deprecation warnings
> explicitly.
>
> (Another argument to think about: if you download and install some 3rd
> party code, deprecation warnings about that code are *only noise*
> since they are not yours to fix. Warnings in a dynamic language work
> very different than warnings in a compiled language.)

Reflecting upon Guido's comment about compiled languages I have come
around to his view and want to make warnings silent by default. Here
are my reasons.

First, there is the app user perspective. If I use an application I do
not need to see any warnings, ever. Just because my interpreter got
upgraded (whether explicitly by me, my company's sysadmin, or my OS),
that should not suddenly start seeing warnings for some Python-based
application I use. We cannot expect all developers to release a new
version just before or the day of a new interpreter release. This is
what separates interpreted languages from compiled ones; just because
you upgrade your compiler does not mean new warnings will suddenly
show up for the compiled languages.

Now the idea has been floated to solve this by modifying distutils or
any other installers to include the line ``-W ignore`` in the shell
scripts they install. The problem with this is it assumes all users
use the shell script to launch the app. With the introduction of
__main__.py support for packages and zipfiles I can easily see people
starting to launch apps from the command-line with the newest
interpreter they have installed, e.g. ``python2.6 -m someapp``. And
just because I have enough knowledge to know how to specify what
Python interpreter to use does not mean I care about warnings in
Python (might not even know how to program in Python), nor know about
the ``-W ignore`` trick (let alone want to have to type that every
time I use the app).

Second is the perspective of someone not running with the warnings
turned on when they need to for tests. Some have said that there is
the possibility that they won't have the proper test coverage to pick
up the warnings. OK, fine, but that's not our problem. We cannot
dictate proper coding practices (nor should we for that leads down the
road of Java and checked exceptions). Python has always been about
consenting adults. If a development team does not dogfood their own
app with warnings turned on then that is their choice and they will be
bit in the ass at a later date. If a user gets bit by poor development
by a third-party that's unfortunate but once again not our fault since
we cannot dictate that people only use quality software.

In other words I think having warnings off by default is the best way
to improve the situation for users while still empowering developers
to know about things they need to change.

-Brett

From solipsis at pitrou.net  Mon Nov  9 22:47:45 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 09 Nov 2009 22:47:45 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
Message-ID: <1257803265.3888.114.camel@localhost>


> First, there is the app user perspective. If I use an application I do
> not need to see any warnings, ever.

But most end users don't launch their applications via the command-line,
so they won't get to see any stderr warnings anyway.
(have you ever launched a Gnome or KDE app from the command-line? they
can spout *tons* of warnings)

> We cannot expect all developers to release a new
> version just before or the day of a new interpreter release.

And that's precisely why deprecation warnings are necessary to warn them
beforehand. If you silence deprecation warnings by default,
developers /will/ end up in situations where they have to do an
emergency release to support the latest Python interpreter.

(OTOH, noisy deprecation warnings, however annoying they may be to you,
don't force developers to do an emergency release in order to fix them;
they allow developers to take their time)

>  With the introduction of
> __main__.py support for packages and zipfiles I can easily see people
> starting to launch apps from the command-line with the newest
> interpreter they have installed, e.g. ``python2.6 -m someapp``.

Uh? These would be pretty advanced users. The "-m" option is certainly
not known let alone used by the average non-Python savvy user.
And, of course, the Python savvy user is perfectly able to report a bug
to the application developer.

> And
> just because I have enough knowledge to know how to specify what
> Python interpreter to use does not mean I care about warnings in
> Python (might not even know how to program in Python), nor know about
> the ``-W ignore`` trick (let alone want to have to type that every
> time I use the app).

This sounds really sophistic. If you know about "-m", you know to type
"python --help" which immediately will give you the list of available
options, including a possible "-W ignore", "-w" or whatever is added to
that purpose.

> Some have said that there is
> the possibility that they won't have the proper test coverage to pick
> up the warnings. OK, fine, but that's not our problem.

How condescending is that?
Even /we/ are far from 100% test coverage.

> Python has always been about
> consenting adults.

Being consenting adults, however, does not mean giving people a gun to
shoot themselves in the foot, or removing any useful feature which might
let them know that their code should be fixed. We may not be Java, but
neither are we C++ or C.

> If a development team does not dogfood their own
> app with warnings turned on then that is their choice and they will be
> bit in the ass at a later date. If a user gets bit by poor development
> by a third-party that's unfortunate but once again not our fault since
> we cannot dictate that people only use quality software.

Brett, Python is a rich and diverse ecosystem. We as core developers may
live in an ideal world where perfect development practices are followed
(and even that is a lie, since we don't have 100% test coverage, some of
our tests are flaky and our buildbots are often red), but our software
is used in many situations, including by people (scientists, teachers,
system administrators, webmasters) who are not professional Python
programmers.

> In other words I think having warnings off by default is the best way
> to improve the situation for users while still empowering developers
> to know about things they need to change.

Actually, it doesn't make the situation significantly better for users,
or even for developers, it just satisfies our own little egoistical
comfort as power users who don't like the annoying sight of warnings in
a terminal.

I think this proposal is totally counter-productive, and counter to the
idea that Python caters for diverse categories of developers.



From guido at python.org  Mon Nov  9 22:51:03 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2009 13:51:03 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <1257803265.3888.114.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com> 
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com> 
	<1257803265.3888.114.camel@localhost>
Message-ID: <ca471dc20911091351u391cc6fcj6fb0bf03363719cd@mail.gmail.com>

On Mon, Nov 9, 2009 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> (have you ever launched a Gnome or KDE app from the command-line? they
> can spout *tons* of warnings)

Yes, this is a perfect example of what I *don't* want to happen. I
sometimes have to invoke these apps from the command line (seeing no
other choice) and I will *not* read the warnings no matter what they
say, since I cannot do anything about them.

-- 
--Guido van Rossum (python.org/~guido)

From ubershmekel at gmail.com  Mon Nov  9 22:52:50 2009
From: ubershmekel at gmail.com (Yuvgoog Greenle)
Date: Mon, 9 Nov 2009 23:52:50 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
Message-ID: <9d153b7c0911091352g5b708e78le3aabcc5fddc7ea2@mail.gmail.com>

On Mon, Nov 9, 2009 at 11:18 PM, Brett Cannon <brett at python.org> wrote:
> First, there is the app user perspective. If I use an application I do
> not need to see any warnings, ever.

So it comes down to either:

A. As things are - every ballsy app developer has to have this piece
of code laying around:
    import warnings
    warnings.filterwarnings("ignore")

or

B. we render warnings useless in our every day python lives except for
package buildbots that remember to run with -w (or whatever).

I would like a demographic on this, but I'm sure that either way
muting warnings will be a devastating blow to the spread of
information about what's new/old in python.

--yuv

From solipsis at pitrou.net  Mon Nov  9 22:55:22 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 09 Nov 2009 22:55:22 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <ca471dc20911091351u391cc6fcj6fb0bf03363719cd@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<ca471dc20911091351u391cc6fcj6fb0bf03363719cd@mail.gmail.com>
Message-ID: <1257803722.3888.125.camel@localhost>

Le lundi 09 novembre 2009 ? 13:51 -0800, Guido van Rossum a ?crit :
> On Mon, Nov 9, 2009 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > (have you ever launched a Gnome or KDE app from the command-line? they
> > can spout *tons* of warnings)
> 
> Yes, this is a perfect example of what I *don't* want to happen.

But it doesn't. Most Python applications display at worse one or two
deprecation warnings, which is perfectly bearable.
(the exception is -3, and that's why it's good to have it opt-in rather
than opt-out)



From jnoller at gmail.com  Mon Nov  9 23:11:15 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Mon, 9 Nov 2009 17:11:15 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <1257803722.3888.125.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<ca471dc20911091351u391cc6fcj6fb0bf03363719cd@mail.gmail.com>
	<1257803722.3888.125.camel@localhost>
Message-ID: <4222a8490911091411v77e339bby61e8c4f3991dd82e@mail.gmail.com>

On Mon, Nov 9, 2009 at 4:55 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le lundi 09 novembre 2009 ? 13:51 -0800, Guido van Rossum a ?crit :
>> On Mon, Nov 9, 2009 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> > (have you ever launched a Gnome or KDE app from the command-line? they
>> > can spout *tons* of warnings)
>>
>> Yes, this is a perfect example of what I *don't* want to happen.
>
> But it doesn't. Most Python applications display at worse one or two
> deprecation warnings, which is perfectly bearable.
> (the exception is -3, and that's why it's good to have it opt-in rather
> than opt-out)
>

No. It's not. Here's an example - brand new python hacker I know is
using third party libraries in his app. He runs that app hundreds of
times a day, and gets 3-4 deprecation warnings every time he runs that
application.

At first I told him to ignore it - then he showed me the logs from the
app which were rife with them. I gave him the snippet of code to
filter them out pretty quickly.

jesse

From ben+python at benfinney.id.au  Mon Nov  9 23:16:02 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 10 Nov 2009 09:16:02 +1100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
Message-ID: <87vdhj1h0d.fsf@benfinney.id.au>

Brett Cannon <brett at python.org> writes:

> We cannot expect all developers to release a new version just before
> or the day of a new interpreter release. This is what separates
> interpreted languages from compiled ones; just because you upgrade
> your compiler does not mean new warnings will suddenly show up for the
> compiled languages.

This is the most convincing point I see in favour of disabling
DeprecationWarning by default. Thanks for clearly stating why it applies
only to language interpreter implementations.

> In other words I think having warnings off by default is the best way
> to improve the situation for users while still empowering developers
> to know about things they need to change.

It doesn't sit well with the Python Zen admonitions about errors. But
I guess warnings aren't errors, so are excluded on that basis.

-- 
 \           ?Never express yourself more clearly than you are able to |
  `\                                               think.? ?Niels Bohr |
_o__)                                                                  |
Ben Finney


From brett at python.org  Mon Nov  9 23:19:05 2009
From: brett at python.org (Brett Cannon)
Date: Mon, 9 Nov 2009 14:19:05 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <1257803265.3888.114.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com> 
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com> 
	<1257803265.3888.114.camel@localhost>
Message-ID: <bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>

On Mon, Nov 9, 2009 at 13:47, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
>> First, there is the app user perspective. If I use an application I do
>> not need to see any warnings, ever.
>
> But most end users don't launch their applications via the command-line,
> so they won't get to see any stderr warnings anyway.
> (have you ever launched a Gnome or KDE app from the command-line? they
> can spout *tons* of warnings)

No, I haven't since I am a Mac user. But when I have launched Mac apps
from the command-line I have not had issues with warnings. Plus I do
launch plenty of command-line apps that could output warnings but
typically don't (but as I have said, this is not always the case when
I use a new interpreter on an app, e.g. Mercurial under 2.6 the day
Python 2.6 was released).

>
>> We cannot expect all developers to release a new
>> version just before or the day of a new interpreter release.
>
> And that's precisely why deprecation warnings are necessary to warn them
> beforehand. If you silence deprecation warnings by default,
> developers /will/ end up in situations where they have to do an
> emergency release to support the latest Python interpreter.
>

I don't quite follow that. If they hit deprecation warnings on a new
release and warnings are silent by default, how does that force an
emergency release? They just found out that the NEXT release of Python
will cause them trouble, not the new one that just came out.

> (OTOH, noisy deprecation warnings, however annoying they may be to you,
> don't force developers to do an emergency release in order to fix them;
> they allow developers to take their time)

Right, while I have to put up with them. As a user I don't want that.
If a Ruby app was spitting warnings out I would have to be searching
Google on how to turn them off and that's annoying.

>
>> ?With the introduction of
>> __main__.py support for packages and zipfiles I can easily see people
>> starting to launch apps from the command-line with the newest
>> interpreter they have installed, e.g. ``python2.6 -m someapp``.
>
> Uh? These would be pretty advanced users. The "-m" option is certainly
> not known let alone used by the average non-Python savvy user.

At the moment. I know Paul Moore has said he wished more apps were
released this way as it is much easier to launch command-line apps
that way on Windows. Plus I released a zip file of an app like this
and have not had issues.

> And, of course, the Python savvy user is perfectly able to report a bug
> to the application developer.
>

But reporting a bug is not going to turn the warnings off. They still
have to be savvy enough to know to switch them off.

>> And
>> just because I have enough knowledge to know how to specify what
>> Python interpreter to use does not mean I care about warnings in
>> Python (might not even know how to program in Python), nor know about
>> the ``-W ignore`` trick (let alone want to have to type that every
>> time I use the app).
>
> This sounds really sophistic. If you know about "-m", you know to type
> "python --help" which immediately will give you the list of available
> options, including a possible "-W ignore", "-w" or whatever is added to
> that purpose.
>

Not if you are simply told "run this code using -m". And would you
really want to have to type ``-W ignore`` every time you ran an app? I
sure don't.

>> Some have said that there is
>> the possibility that they won't have the proper test coverage to pick
>> up the warnings. OK, fine, but that's not our problem.
>
> How condescending is that?

IMO it's not not meant to be. Sorry if you read it that way.

> Even /we/ are far from 100% test coverage.

I never claimed we had 100% coverage. But I would expect us all to be
running off of trunk for some things with warnings turned on.

>
>> Python has always been about
>> consenting adults.
>
> Being consenting adults, however, does not mean giving people a gun to
> shoot themselves in the foot, or removing any useful feature which might
> let them know that their code should be fixed. We may not be Java, but
> neither are we C++ or C.
>

This is obviously coming down to difference of opinion. I don't view
switching warnings off by default as shooting anyone in the foot. I
put it on the same level as making sure you have unit tests; it's just
part of your testing and verification process that your users just
don't need to know about.

>> If a development team does not dogfood their own
>> app with warnings turned on then that is their choice and they will be
>> bit in the ass at a later date. If a user gets bit by poor development
>> by a third-party that's unfortunate but once again not our fault since
>> we cannot dictate that people only use quality software.
>
> Brett, Python is a rich and diverse ecosystem. We as core developers may
> live in an ideal world where perfect development practices are followed
> (and even that is a lie, since we don't have 100% test coverage, some of
> our tests are flaky and our buildbots are often red), but our software
> is used in many situations, including by people (scientists, teachers,
> system administrators, webmasters) who are not professional Python
> programmers.

I know that. One of my best friends is a neurobiology PhD and they use
Python in her lab. They are not expert programmers. But if something
broke one day because they got a traceback which could have been
avoided had they turned on warnings I would hope they can figure out
how to fix that. Otherwise, warnings or not, they have bigger issues
to deal with.

>
>> In other words I think having warnings off by default is the best way
>> to improve the situation for users while still empowering developers
>> to know about things they need to change.
>
> Actually, it doesn't make the situation significantly better for users,
> or even for developers, it just satisfies our own little egoistical
> comfort as power users who don't like the annoying sight of warnings in
> a terminal.

I obviously disagree.

>
> I think this proposal is totally counter-productive, and counter to the
> idea that Python caters for diverse categories of developers.

Obviously we both are not about to change our minds on this one as we
are coming from different perspectives. So I agree that we disagree
and I am happy to leave it at that.

-Brett

From mal at egenix.com  Mon Nov  9 23:22:51 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 09 Nov 2009 23:22:51 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and
 how	noisy they are)
In-Reply-To: <1257803722.3888.125.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>	<1257803265.3888.114.camel@localhost>	<ca471dc20911091351u391cc6fcj6fb0bf03363719cd@mail.gmail.com>
	<1257803722.3888.125.camel@localhost>
Message-ID: <4AF8963B.4060503@egenix.com>

Antoine Pitrou wrote:
> Le lundi 09 novembre 2009 ? 13:51 -0800, Guido van Rossum a ?crit :
>> On Mon, Nov 9, 2009 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> (have you ever launched a Gnome or KDE app from the command-line? they
>>> can spout *tons* of warnings)
>>
>> Yes, this is a perfect example of what I *don't* want to happen.
> 
> But it doesn't. Most Python applications display at worse one or two
> deprecation warnings, which is perfectly bearable.
> (the exception is -3, and that's why it's good to have it opt-in rather
> than opt-out)

I'm with Antoine here: I haven't seen a single warning from Python
in years... and I do use lots of 3rd party code in projects.
To me, this discussion appears a bit artifical.

Instead of worrying about too much noise on stderr we should reflect
more on whether certain deprecations are useful or not in the first
place, e.g. take the md5 module deprecation:

It would have been very easy to provide proxy implemenations for the
md5 and sha modules, redirecting to the hashlib module, instead of
going through the whole deprecation process, forcing developers to
clutter up their code with conditional imports and confusing newcomers
reading about the md5 module in their Python books.

After all, there's a reason why we still have the string module in
Python 2.7 without generating any warning whatsoever and properly
redirecting to the string methods.

IMHO, deprecations are only useful for things that are bound to
cause a change which doesn't allow for an option for providing
backwards compatible ways of letting older code continue to work
with more recent Python releases (even if it gets slower as result
of this indirection), e.g. Python syntax changes (such as changes
to the except clause) or changes to language semantics (such as
integer division).

In summary: Don't raise the bar for warning messages, raise
the bar for deprecations themsselves.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 09 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From solipsis at pitrou.net  Tue Nov 10 00:00:43 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Nov 2009 00:00:43 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
Message-ID: <1257807643.3888.222.camel@localhost>


> I don't quite follow that. If they hit deprecation warnings on a new
> release and warnings are silent by default, how does that force an
> emergency release?

Because nobody ran the code with warnings on (or they just ran the test
suite and it didn't test all code paths), so that when the Python
version removing the deprecated constructs goes out, the library or app
is suddenly broken.

> Right, while I have to put up with them. As a user I don't want that.
> If a Ruby app was spitting warnings out I would have to be searching
> Google on how to turn them off and that's annoying.

I don't know about Ruby but "ruby --help" or "man ruby" should be
sufficient, shouldn't it? Looking up command-line options in the bundled
documentation is pretty standard, and shouldn't pass as an annoyance.

> At the moment. I know Paul Moore has said he wished more apps were
> released this way as it is much easier to launch command-line apps
> that way on Windows.

End users are even less likely to use the command-line under Windows
than they are under Linux... Such an option is obviously for power users
only, and it is quite reasonable to tell them to use "-Wwhatever" if
they want to switch off warnings.

Please remember that warnings are /not/ the common situation, they are
the exception. Your argument makes it sound like all applications will
print out warnings and annoy their command-line users, which is /not/
true.

> But reporting a bug is not going to turn the warnings off. They still
> have to be savvy enough to know to switch them off.

It is still much better to let people report a bug than to hide the
problem until it becomes critical. That's how we help developers; not by
making them blind to the issues they may be facing ten months from now.

> I never claimed we had 100% coverage. But I would expect us all to be
> running off of trunk for some things with warnings turned on.

For me, I usually don't run any third-party app or library using trunk.
And if I do, I certainly accept the existence of deprecation warnings;
after all, it's better than having the app totally break because of
whatever issue ignored by the developers.

> > Actually, it doesn't make the situation significantly better for users,
> > or even for developers, it just satisfies our own little egoistical
> > comfort as power users who don't like the annoying sight of warnings in
> > a terminal.
> 
> I obviously disagree.

Certainly, but that proposal still does not seem to be solving a serious
problem. How many people have asked for this to be changed? Do we have
regular reports about it on the bug tracker? Or perhaps people are
complaining about it on the mailing-lists or on other public fora? Is
Python renowned for making the command-line a painful experience?

I don't reckon so, but perhaps I've missed something. Bringing such
evidence would be helpful, and not only for my own edification :)




From ubershmekel at gmail.com  Tue Nov 10 00:09:25 2009
From: ubershmekel at gmail.com (Yuvgoog Greenle)
Date: Tue, 10 Nov 2009 01:09:25 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <87vdhj1h0d.fsf@benfinney.id.au>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<87vdhj1h0d.fsf@benfinney.id.au>
Message-ID: <9d153b7c0911091509q57212e68nbca5f3ec386c1e2e@mail.gmail.com>

On Tue, Nov 10, 2009 at 12:16 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Brett Cannon <brett at python.org> writes:
>
>> We cannot expect all developers to release a new version just before
>> or the day of a new interpreter release. This is what separates
>> interpreted languages from compiled ones; just because you upgrade
>> your compiler does not mean new warnings will suddenly show up for the
>> compiled languages.
>
> This is the most convincing point I see in favour of disabling
> DeprecationWarning by default. Thanks for clearly stating why it applies
> only to language interpreter implementations.
>

Sorry Benny but you were convinced by an invalid argument.

This move won't save developers from releasing "a new version just
before or the day of a new interpreter release" because things are
bound to be deprecated. Developers will have to work extra hard when
they find out their app just breaks completely on a sunny interpreter
release. No pesky warnings - just plain old traceback and a dead
process.

Though I do agree with Brett on agreeing to disagree,

--yuv

From python at rcn.com  Tue Nov 10 00:09:56 2009
From: python at rcn.com (Raymond Hettinger)
Date: Mon, 9 Nov 2009 15:09:56 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
Message-ID: <E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>


[Brett Cannon]
> This is obviously coming down to difference of opinion. I don't view
> switching warnings off by default as shooting anyone in the foot. I
> put it on the same level as making sure you have unit tests; it's just
> part of your testing and verification process that your users just
> don't need to know about.

Nice summary.  FWIW, I concur with Brett.  No one needs to see
warnings/deprecations except for the developer who controls the code.
Even that developer may only need to see it at one point during the
development process.  Further, the developer controlling the code
just may not care -- the script can be for single use, for a class project,
or some other purpose that doesn't require being notified of what may
change in a later version of Python.


Raymond

From debatem1 at gmail.com  Tue Nov 10 00:14:07 2009
From: debatem1 at gmail.com (geremy condra)
Date: Mon, 9 Nov 2009 18:14:07 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
Message-ID: <f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>

On Mon, Nov 9, 2009 at 6:09 PM, Raymond Hettinger <python at rcn.com> wrote:
>
> [Brett Cannon]
>>
>> This is obviously coming down to difference of opinion. I don't view
>> switching warnings off by default as shooting anyone in the foot. I
>> put it on the same level as making sure you have unit tests; it's just
>> part of your testing and verification process that your users just
>> don't need to know about.
>
> Nice summary. ?FWIW, I concur with Brett. ?No one needs to see
> warnings/deprecations except for the developer who controls the code.
> Even that developer may only need to see it at one point during the
> development process. ?Further, the developer controlling the code
> just may not care -- the script can be for single use, for a class project,
> or some other purpose that doesn't require being notified of what may
> change in a later version of Python.
>
>
> Raymond

If they don't care, why does it matter whether they see it or not?
Seems like an argument for the status quo.

Geremy Condra

From solipsis at pitrou.net  Tue Nov 10 00:23:54 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Nov 2009 00:23:54 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
 they are)
In-Reply-To: <E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
Message-ID: <1257809034.3888.228.camel@localhost>

Le lundi 09 novembre 2009 ? 15:09 -0800, Raymond Hettinger a ?crit :
> Further, the developer controlling the code
> just may not care -- the script can be for single use, for a class project,
> or some other purpose that doesn't require being notified of what may
> change in a later version of Python.

If it's for single use or for a class project, you are certainly using
up-to-date constructs which won't emit any deprecation warning. The
issue of deprecation warnings should only appear when you write
something that gets used for more than a year or two.




From guido at python.org  Tue Nov 10 00:24:18 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2009 15:24:18 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <9d153b7c0911091509q57212e68nbca5f3ec386c1e2e@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com> 
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com> 
	<87vdhj1h0d.fsf@benfinney.id.au>
	<9d153b7c0911091509q57212e68nbca5f3ec386c1e2e@mail.gmail.com>
Message-ID: <ca471dc20911091524o42e0c7a5td4a5430d83ad1194@mail.gmail.com>

On Mon, Nov 9, 2009 at 3:09 PM, Yuvgoog Greenle <ubershmekel at gmail.com> wrote:
> On Tue, Nov 10, 2009 at 12:16 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
[...]

> Sorry Benny but you were convinced by an invalid argument.

Maybe you know Ben personally and maybe it's different in your
culture, but to me it sounds pretty derogatory to call someone "Benny"
who signs as "Ben".

(BTW are you using a pseudonym? Yuvgoog sounds like a made-up name.
While we are talking about how to address people, I just had to ask.)

>No pesky warnings - just plain old traceback and a dead process.

Have you actually ever tried to debug a warning? Typically the easiest
thing to do is to turn warnings into errors so you can see exactly
where it comes from...

-- 
--Guido van Rossum (python.org/~guido)

From python at rcn.com  Tue Nov 10 00:24:56 2009
From: python at rcn.com (Raymond Hettinger)
Date: Mon, 9 Nov 2009 15:24:56 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
Message-ID: <7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>


[Raymond]
>. Further, the developer controlling the code
> just may not care -- the script can be for single use, for a class project,
> or some other purpose that doesn't require being notified of what may
> change in a later version of Python.

[Geremy Condra debatem1 at gmail.com]
> If they don't care, why does it matter whether they see it or not?

There is a difference about not caring about the content of the warning
versus caring about whether the messages clutter the screen.  Noise is always bad.
There are legitimate reasons for not being interested in what is going to change
in a future version of Python -- for those people, the messages are a distraction and nuisance.


Raymond


From pjenvey at underboss.org  Tue Nov 10 00:21:13 2009
From: pjenvey at underboss.org (Philip Jenvey)
Date: Mon, 9 Nov 2009 15:21:13 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
Message-ID: <A8BF14BC-F95F-4C39-B826-8BB400F424C6@underboss.org>


On Nov 8, 2009, at 1:26 PM, Brett Cannon wrote:

> During the moratorium PEP discussions Guido said he wanted to quiet
> down deprecation warnings. I see there being two options on this.
>
> One is to keep things as is, but to require two releases with
> PendingDeprecationWarning so there are three years of
> silent-by-default warnings to update your code. But that last release
> before removal came would still be noisy.

DeprecationWarnings are annoying but for good enough reason. I'd  
rather encourage folks to deal with them sooner rather than later,  
which is what PendingDeprecationWarnings are for.

So I like requiring two (or one is enough for me) releases of  
PendingDeprecationWarnings, but combined with the requirement that any  
new warnings be explicitly documented in their own section of the  
release's What's New doc. That section should also remind you how to  
enable PendingDeprecationWarnings via the command line and the  
warnings module itself. I'd also consider adding a PYTHONWARNINGS  
environment variable to ease toggling of warnings when running Python  
code via an external script.

--
Philip Jenvey

From kevin at bud.ca  Tue Nov 10 00:43:58 2009
From: kevin at bud.ca (Kevin Teague)
Date: Mon, 9 Nov 2009 15:43:58 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
Message-ID: <3B477F08-E184-471D-B346-CC46659AF79A@bud.ca>


On Nov 9, 2009, at 10:02 AM, Guido van Rossum wrote:

> [Laura]
>>> Experience has shown that when people get used to seeing 'a bunch of
>>> warnings that don't really matter' they either a) turn them off or
>>> b) ignore them, even when they are telling them valuable things that
>>> they should be paying attention to.  So constantly spitting out
>>> DeprecationWarnings as soon as something becomes deprecated is a
>>> most excellent way to train people to ignore DeprecationWarnings.
>
> On Sun, Nov 8, 2009 at 10:39 PM, Antoine Pitrou  
> <solipsis at pitrou.net> wrote:
>> Well at least people get a chance to see them. If some people think  
>> the
>> warnings are useless (even though the messages warn about removal  
>> of a
>> construct), they won't run a code checker either.
>
> You're misreading Laura. She's not saying that some people choose to
> ignore the warnings. She's pointing out that it is a scientifically
> known fact that people will *learn to ignore warnings* if they occur
> frequently enough, and that once this is learned the warnings have no
> effect. Look up Jef Raskin's diatribes against modal dialogs. (Or read
> the "cry wolf" fairy tale. :-)


Case in point, Plone collected more and more deprecation warnings over  
the years. When Zope started down the DeprecationWarning road, there  
were just a few warnings, but people got used to them. After a few  
years went by, starting the application server used to emit a truly  
hideous amount of warnings. End-users saw these warnings and it was  
embarrassing. 3rd party developers using it as a framework saw these  
warnings and it just frightened them off. Despite being such a bad  
situation, the Plone core developers saw these warnings and did  
nothing to fix them, they just learned to ignore the situation when  
starting Plone - despite it being a major source of embarassment for  
them, everyone just got used to ignoring them (until finally Hanno  
took mercy and the led the charge to clean-up the morass).

Andy McKay complained, "Do I really need to see all this? Can we put  
this into a unit test that will do something like test --show- 
deprecations? Deprecation warnings in products I develop are useful,  
but I fear, I like most or just ignoring these logs. If that's the  
case we are losing the battle."

http://www.agmweb.ca/blog/andy/1849/


From debatem1 at gmail.com  Tue Nov 10 00:44:59 2009
From: debatem1 at gmail.com (geremy condra)
Date: Mon, 9 Nov 2009 18:44:59 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>
Message-ID: <f3cc57c60911091544w774f1a44s49600063c22f5490@mail.gmail.com>

On Mon, Nov 9, 2009 at 6:24 PM, Raymond Hettinger <python at rcn.com> wrote:
>
> [Raymond]
>>
>> . Further, the developer controlling the code
>> just may not care -- the script can be for single use, for a class
>> project,
>> or some other purpose that doesn't require being notified of what may
>> change in a later version of Python.
>
> [Geremy Condra debatem1 at gmail.com]
>>
>> If they don't care, why does it matter whether they see it or not?
>
> There is a difference about not caring about the content of the warning
> versus caring about whether the messages clutter the screen. ?Noise is
> always bad.

I don't really have a horse in this race one way or the other, but the idea
that we shouldn't put blinking lights and sirens on the crushomatic 9000
because people might eventually ignore them doesn't strike me as being
all that smart. If the warnings exist for a good reason, then they
constitute information and should be seen. If they don't, they constitute
noise and should be removed.

> There are legitimate reasons for not being interested in what is going to
> change
> in a future version of Python -- for those people, the messages are a
> distraction and nuisance.

And thats why you should be able to silence the warnings- which you
can currently do, if they bother you so much.

To me this seems like a lot of drama over a non-feature.

Geremy Condra

From solipsis at pitrou.net  Tue Nov 10 00:52:06 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Nov 2009 00:52:06 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
 noisy they are)
In-Reply-To: <3B477F08-E184-471D-B346-CC46659AF79A@bud.ca>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
	<3B477F08-E184-471D-B346-CC46659AF79A@bud.ca>
Message-ID: <1257810726.3888.234.camel@localhost>

Le lundi 09 novembre 2009 ? 15:43 -0800, Kevin Teague a ?crit :
> Despite being such a bad  
> situation, the Plone core developers saw these warnings and did  
> nothing to fix them, they just learned to ignore the situation when  
> starting Plone - despite it being a major source of embarassment for  
> them, everyone just got used to ignoring them (until finally Hanno  
> took mercy and the led the charge to clean-up the morass).

So, the question is: would anyone have led the charge if there had been
no visible warnings, and no morass to clean up? As an insider, what is
your thought about it?



From guido at python.org  Tue Nov 10 01:01:43 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2009 16:01:43 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <f3cc57c60911091544w774f1a44s49600063c22f5490@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com> 
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com> 
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com> 
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com> 
	<7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>
	<f3cc57c60911091544w774f1a44s49600063c22f5490@mail.gmail.com>
Message-ID: <ca471dc20911091601w1b6e0636md7486949e1fb024a@mail.gmail.com>

On Mon, Nov 9, 2009 at 3:44 PM, geremy condra <debatem1 at gmail.com> wrote:
> the idea
> that we shouldn't put blinking lights and sirens on the crushomatic 9000
> because people might eventually ignore them doesn't strike me as being
> all that smart.

I'm sorry, but you're really not getting the point. The crushomatic
already *has* a blinking light and a siren and everyone is *already*
ignoring them, because they go off all the time, whenever a person
gets closer than 6 feet. So we have no way to warn them when there
*really* is a danger, like when a person is about to put their hand
into the blender.

The solution is to make the machine less dangerous (e.g. if you open
the lid the motor is cut off), not to argue that the siren is really
important.

-- 
--Guido van Rossum (python.org/~guido)

From ubershmekel at gmail.com  Tue Nov 10 01:13:00 2009
From: ubershmekel at gmail.com (Yuvgoog Greenle)
Date: Tue, 10 Nov 2009 02:13:00 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911091524o42e0c7a5td4a5430d83ad1194@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<87vdhj1h0d.fsf@benfinney.id.au>
	<9d153b7c0911091509q57212e68nbca5f3ec386c1e2e@mail.gmail.com>
	<ca471dc20911091524o42e0c7a5td4a5430d83ad1194@mail.gmail.com>
Message-ID: <9d153b7c0911091613o283cb098m2648c7fd81d9aa7e@mail.gmail.com>

On Tue, Nov 10, 2009 at 1:24 AM, Guido van Rossum <guido at python.org> wrote:
> On Mon, Nov 9, 2009 at 3:09 PM, Yuvgoog Greenle <ubershmekel at gmail.com> wrote:
>> On Tue, Nov 10, 2009 at 12:16 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> [...]
>
>> Sorry Benny but you were convinced by an invalid argument.
>
> Maybe you know Ben personally and maybe it's different in your
> culture, but to me it sounds pretty derogatory to call someone "Benny"
> who signs as "Ben".
>
> (BTW are you using a pseudonym? Yuvgoog sounds like a made-up name.
> While we are talking about how to address people, I just had to ask.)
>
>>No pesky warnings - just plain old traceback and a dead process.
>
> Have you actually ever tried to debug a warning? Typically the easiest
> thing to do is to turn warnings into errors so you can see exactly
> where it comes from...
>
> --
> --Guido van Rossum (python.org/~guido)
>

First of all I'd like to apologize to Ben, this may sound like a lame
excuse but I guess Ben Finney somehow turned into Benny because of my
lack of sleep. So again sorry for that, I really didn't mean it that
way.

Yuvgoog is a pseudonym indeed, my real name is Yuval Greenfield. I'd
grown a habit of trying to see where I get spam mail from so I use
names mangled with the service name. You might see me as Yuvpic
Greenasa, Yuvfaceal Greenbookfield, etc. I guess this habit's one I
should get rid of, especially seeing as it really hasn't paid off (I
haven't caught a single spammer yet after probably 6 years of it).

Concerning debugging warnings, I have to say I currently remember my
experience with 2: the deprecation of sets and the deprecation of
string exceptions. On both occasions the warnings helped me figure
things out before my code broke. Specifically with raising string
exceptions, warnings saved me and my colleagues from alot of bad code,
string exceptions were the convention at my office and I only found
out early because I installed a new python that warned me. Finding
this out as an error might not have been too bad either but it would
have probably taken more time until I got a python that doesn't allow
raising strings (2.6!).


--Yuval Greenfield (yuv)

From lac at openend.se  Tue Nov 10 01:18:48 2009
From: lac at openend.se (Laura Creighton)
Date: Tue, 10 Nov 2009 01:18:48 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: Message from geremy condra <debatem1@gmail.com> of "Mon,
	09 Nov 2009 18:14:07 EST."
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com> 
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com> 
Message-ID: <200911100018.nAA0IntE017750@theraft.openend.se>

In a message of Mon, 09 Nov 2009 18:14:07 EST, geremy condra writes:
>On Mon, Nov 9, 2009 at 6:09 PM, Raymond Hettinger <python at rcn.com> wrote:
>>
>> [Brett Cannon]
>>>
>>> This is obviously coming down to difference of opinion. I don't view
>>> switching warnings off by default as shooting anyone in the foot. I
>>> put it on the same level as making sure you have unit tests; it's just
>>> part of your testing and verification process that your users just
>>> don't need to know about.
>>
>> Nice summary. ?FWIW, I concur with Brett. ?No one needs to see
>> warnings/deprecations except for the developer who controls the code.
>> Even that developer may only need to see it at one point during the
>> development process. ?Further, the developer controlling the code
>> just may not care -- the script can be for single use, for a class proj
>ect,
>> or some other purpose that doesn't require being notified of what may
>> change in a later version of Python.
>>
>>
>> Raymond
>
>If they don't care, why does it matter whether they see it or not?
>Seems like an argument for the status quo.
>
>Geremy Condra

The reason that it matters is that having seen it a few times may make
it impossible for them to see it in the future.

We studied precisely this in determining 'how often should C
programmers run lint' back in the 1980s for DCIEM (The Canadian
department of War).  I started out as a firm proponent of 'you should
run lint every time' and 'you should fix your lint errors as soon as
you see them'.  It turned out that this was the wrong idea.  Most
people were completely unable to fix their lint errors as they showed
up.  They were unable to _see_ their lint errors as they showed up.
Because they were concentrating on the code that they were writing,
all errors and warnings about code that wasn't the focus of their
concentration didn't register with them consciously.  They were blind
to these errors and warnings because their concentration was somewhere
else.

Which wouldn't be so bad if, once they stopped concentrating on the
code they were writing, they were then free to see the lint errors
again, and fix them then.  But a very strange thing had happened by
that point.  They had become used to the lint errors that they weren't
paying attention to, and so they _still_ couldn't see them, even when
they wanted to, because they were habituated to them.  It was
striking.  You could ask the people to read their lint output, looking
for 'Warning: long assignment may lose accuracy' messages, and they
would read right by that message, and say, no, they didn't have one.
You could get them to read all of their lint messages, out loud, to
the whole room, and they would read them one by one, including the one
that said 'Warning: long assignment may lose accuracy' and go on to
the next message and _still_ not notice that they had found what you
had asked them to look for.

These are people who are most sincere in looking for these errors,
those who dearly wanted to fix them at this point in time.  But 
the development process that they had at this time destroyed their
ability to make use of the warnings that lint was giving them.  To
use lint well, they needed to use it _only_ when they were at a
point in their thinking when they were receptive to seeing such
messages.

(And, given that we were giving the poor bastards problems in 
string manipulation, which they all had to do with C pointer 
arithmetic, and that many of the soldiers were sticking with
ed, the editor they knew, rather than newfangled things such
as emacs, and the HCR screen editor -- this was before vi,
they had one heck of a lot to keep in their heads when they
were programming.  Lots of the time they were not receptive to
seeing any warnings at all.)

What we concluded was that people should only run lint on their
C programs when they wanted to fix those errors lint found.  While they
were busy concentrating on other things, they shouldn't run it.  And
it is bad to warn people unless you already have their attention, and
they are looking for warnings.  Otherwise, they may habituate themselves
to seeing the warning, and you will lose your chance to ever warn them.

Does this generalise to 'Python should never raise warnings unless
they are asked for'?  I don't know.  The v7 lint of 1983 or so was
one of the chattiest programs on the planet.  And most of the warnings
that our soldiers got were about bad code in somebody else's library
which they couldn't modify anyway.  So the pressure to become blind
to warnings -- all warnings -- was pretty high.  This isn't the case
with Python.  And the cognitive load to program at all, doing
string manipulation in C using ed as your programming language of
choice ... I know that at the time I did not think I was doing anything
particularly difficult, but I was 20 then.  I cannot imagine doing it
now.

What I object to is the blind acceptance of the notion that 'it
will do good for programmers to see deprecation warnings that they
were not expecting or looking for'.  All the evidence I know runs
against this one - first they won't see them because they weren't
looking for them ... their minds are on other things, and then
they won't see them because they _CANNOT_ any more, because they
have been overly exposed to such things.

It would be really nice to run a cognitive experiment to test how
well warning messages are seen.  We ran one at DCIEM to test this.
We had our soldiers' lint runs changed so that for each of them,
8 times in a day, they would get a lint warning that would
instruct them to say a certain word to a certain person (in the
room where the printer was) for a small reward ... a chocolate bar,
a piece of pie, and the important note on their service record
that they were an exceptionally observant person.  In a group of
35 soldiers, we generally got 4-5 who saw every message and 18-22
who saw none.  Of course, this was _after_ they had been habituated
to seeing the messages.  I don't know how it would work for fresh
students.

One thing we were able to prove -- which has been amply demonstrated
by others -- is that newbies who lack the experience to trust their
ability to 'overlook those things that do not matter' are those best
served by warning messages.  They pay attention better than the
experienced.  But I don't see any way to identify new python
users and give them more warnings than others showing up any time soon.

Laura


From kevin at bud.ca  Tue Nov 10 01:48:17 2009
From: kevin at bud.ca (Kevin Teague)
Date: Mon, 9 Nov 2009 16:48:17 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <1257810726.3888.234.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
	<ca471dc20911091002yabec1c6g9a9c0e5d9ede8d86@mail.gmail.com>
	<3B477F08-E184-471D-B346-CC46659AF79A@bud.ca>
	<1257810726.3888.234.camel@localhost>
Message-ID: <7ECCF3C8-36BD-49C6-A0AF-C5C998F4720D@bud.ca>


On Nov 9, 2009, at 3:52 PM, Antoine Pitrou wrote:

> Le lundi 09 novembre 2009 ? 15:43 -0800, Kevin Teague a ?crit :
>> Despite being such a bad
>> situation, the Plone core developers saw these warnings and did
>> nothing to fix them, they just learned to ignore the situation when
>> starting Plone - despite it being a major source of embarassment for
>> them, everyone just got used to ignoring them (until finally Hanno
>> took mercy and the led the charge to clean-up the morass).
>
> So, the question is: would anyone have led the charge if there had  
> been
> no visible warnings, and no morass to clean up? As an insider, what is
> your thought about it?
>

The morass would have been cleaned up anyways, since the excessively  
large amount of deprecations was acknowledged that Plone needed major  
refactoring towards a cleaner code-base. I'm not exactly sure how much  
the annoyance of the deprecations warnings played a role in  
accelerating this - probably Hanno or Wichert would be better people  
to ask. I only observed these warnings with annoyance since I only do  
Plone work as an integrator (e.g. used Plone-as-framework).

Another thing that happened with all the deprecations in the Zope  
world was DeprecationWarnings that had a long shelf life, generally a  
couple of major releases, so around is three years or more. And at the  
end of that point whomever initiated the deprecation is no longer  
leading the charge for it, since they've moved on to other things. The  
project itself shifted direction, and despite the deprecation  
warnings, people would still use the old APIs (newbie developers) or  
people would not want to go in and maintain their 3-year old code that  
relyed on it and not want it to break needlessly. So there would be  
opposition to removing the deprecation at that point. For example,  
Zope 3.4 is the last proper Zope 3 release, but there are still  
packages with deprecation warnings such as, "This function has been  
deprecated and will go away in Zope 3.6." or "The ``localUtility``  
directive has been deprecated and will be removed after 09/2007."

M.-A. Lemburg made a good point, "Don't raise the bar for warning  
messages, raise the bar for deprecations themselves.". In Zope this  
was generally adopted for many of those deprecations. Warnings were  
spewed, people got annoyed, people lost interesting refactoring/ 
cleaning that particular code, and in the end after building up  
"warnings immunity" it was agreed to just leave the BBB shims in  
there ... they weren't hurting anything beyond adding a few kilobytes  
to the size of the tarballs. (although there has been quite a bit of  
movement just recently towards cleaning up the better packages from  
Zope 3 and releasing them in the form of the Zope Toolkit).

As for me, I've become so blind to warnings that I couldn't tell you  
of the top of my head if Python deprecated md5 -> hashlib or vice- 
versa. Nor do I really care, I've never needed to interact with those  
modules directly, but I've seen them spewed on the console a lot.  
There are lots of cases where Python code is run on the command-line  
and you aren't directly invoking it with the interpreter where you can  
easily silence the warnings (although again, I've never bothered with  
silencing them, I, like many, just learned to ignore them ... they're  
annoying but not so annoying to want to have to rememeber to type  
extra syntax every time you run the interpreter). Some notable command- 
line Python apps where you can't easily put in a -W:

  $ hg
  $ buildout
  $ pip
  $ zopectl
  $ paster

As for the code I write, I tend to favor packages outside the standard  
library these days since they can move forward with refactoring in a  
more timely manner, and I can make explicit those dependencies in the  
setup.py's install_requires field. If a package requires 'argparse',  
then it specifies that. If a package requires 'opster', then it  
specifies that. No need for deprecation warnings saying argparse ->  
opster or vice-versa. When the package is installed, it pulls in the  
packages that it says that it needs. Simple as that.

But I don't think the DeprecationWarning discussion is entirely  
fruitless. I'd imagine that Distutils in particular is going to need  
to do some deprecations in order to make things "right" in packaging  
land.


From debatem1 at gmail.com  Tue Nov 10 02:03:33 2009
From: debatem1 at gmail.com (geremy condra)
Date: Mon, 9 Nov 2009 20:03:33 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <ca471dc20911091601w1b6e0636md7486949e1fb024a@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>
	<f3cc57c60911091544w774f1a44s49600063c22f5490@mail.gmail.com>
	<ca471dc20911091601w1b6e0636md7486949e1fb024a@mail.gmail.com>
Message-ID: <f3cc57c60911091703l6d14e541jef10ac68d8efb7e6@mail.gmail.com>

On Mon, Nov 9, 2009 at 7:01 PM, Guido van Rossum <guido at python.org> wrote:
> On Mon, Nov 9, 2009 at 3:44 PM, geremy condra <debatem1 at gmail.com> wrote:
>> the idea
>> that we shouldn't put blinking lights and sirens on the crushomatic 9000
>> because people might eventually ignore them doesn't strike me as being
>> all that smart.
>
> I'm sorry, but you're really not getting the point. The crushomatic
> already *has* a blinking light and a siren and everyone is *already*
> ignoring them, because they go off all the time, whenever a person
> gets closer than 6 feet. So we have no way to warn them when there
> *really* is a danger, like when a person is about to put their hand
> into the blender.
>
> The solution is to make the machine less dangerous (e.g. if you open
> the lid the motor is cut off), not to argue that the siren is really
> important.

Rereading my post I don't see any place where I seem to have
argued that deprecation warnings were "really important", but
in case that's how it sounded, let me clarify: I don't think this
argument is anything other than lots and lots of bikeshedding,
since these warnings can already be explicitly silenced by the
very people this is designed to protect.

Having said that, if you think that deprecation warnings are
mostly noise, why not ask the pylint and pychecker devs to
handle that and drop the warnings altogether?

Geremy Condra

From lac at openend.se  Tue Nov 10 02:22:36 2009
From: lac at openend.se (Laura Creighton)
Date: Tue, 10 Nov 2009 02:22:36 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: Message from geremy condra <debatem1@gmail.com> of "Mon,
	09 Nov 2009 20:03:33 EST."
	<f3cc57c60911091703l6d14e541jef10ac68d8efb7e6@mail.gmail.com> 
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>
	<f3cc57c60911091544w774f1a44s49600063c22f5490@mail.gmail.com>
	<ca471dc20911091601w1b6e0636md7486949e1fb024a@mail.gmail.com>
	<f3cc57c60911091703l6d14e541jef10ac68d8efb7e6@mail.gmail.com> 
Message-ID: <200911100122.nAA1Ma1i022342@theraft.openend.se>

In a message of Mon, 09 Nov 2009 20:03:33 EST, geremy condra writes:
>On Mon, Nov 9, 2009 at 7:01 PM, Guido van Rossum <guido at python.org> wrote
>:
>> On Mon, Nov 9, 2009 at 3:44 PM, geremy condra <debatem1 at gmail.com> wrot
>e:
>>> the idea
>>> that we shouldn't put blinking lights and sirens on the crushomatic 90
>00
>>> because people might eventually ignore them doesn't strike me as being
>>> all that smart.
>>
>> I'm sorry, but you're really not getting the point. The crushomatic
>> already *has* a blinking light and a siren and everyone is *already*
>> ignoring them, because they go off all the time, whenever a person
>> gets closer than 6 feet. So we have no way to warn them when there
>> *really* is a danger, like when a person is about to put their hand
>> into the blender.
>>
>> The solution is to make the machine less dangerous (e.g. if you open
>> the lid the motor is cut off), not to argue that the siren is really
>> important.
>
>Rereading my post I don't see any place where I seem to have
>argued that deprecation warnings were "really important", but
>in case that's how it sounded, let me clarify: I don't think this
>argument is anything other than lots and lots of bikeshedding,
>since these warnings can already be explicitly silenced by the
>very people this is designed to protect.

My point is that 'designing to protect people' is not enough, you need
to actually protect them.  And while in general 'logically this should
protect them' is enough -- (and what intelligent people often mean by
'designing to protect people') -- when it comes to issues of cognative
psychology, the important lessons to learn are the ones that teach you
is where this one fails ---- by trying to protect X, or at least
make it easier for X, you end up making it harder.

>Having said that, if you think that deprecation warnings are
>mostly noise, why not ask the pylint and pychecker devs to
>handle that and drop the warnings altogether?

I am all for this.  And I am all for Marc Andr?'s suggestion to
raise the barrier for deprecations themselves as well.

Laura

>
>Geremy Condra
>_______________________________________________
>stdlib-sig mailing list
>stdlib-sig at python.org
>http://mail.python.org/mailman/listinfo/stdlib-sig

From debatem1 at gmail.com  Tue Nov 10 02:41:40 2009
From: debatem1 at gmail.com (geremy condra)
Date: Mon, 9 Nov 2009 20:41:40 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <200911100122.nAA1Ma1i022342@theraft.openend.se>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>
	<f3cc57c60911091544w774f1a44s49600063c22f5490@mail.gmail.com>
	<ca471dc20911091601w1b6e0636md7486949e1fb024a@mail.gmail.com>
	<debatem1@gmail.com>
	<f3cc57c60911091703l6d14e541jef10ac68d8efb7e6@mail.gmail.com>
	<200911100122.nAA1Ma1i022342@theraft.openend.se>
Message-ID: <f3cc57c60911091741m32fcd2f7jffad3bbc2e466ff0@mail.gmail.com>

On Mon, Nov 9, 2009 at 8:22 PM, Laura Creighton <lac at openend.se> wrote:
> In a message of Mon, 09 Nov 2009 20:03:33 EST, geremy condra writes:
>>On Mon, Nov 9, 2009 at 7:01 PM, Guido van Rossum <guido at python.org> wrote
>>:
>>> On Mon, Nov 9, 2009 at 3:44 PM, geremy condra <debatem1 at gmail.com> wrot
>>e:
>>>> the idea
>>>> that we shouldn't put blinking lights and sirens on the crushomatic 90
>>00
>>>> because people might eventually ignore them doesn't strike me as being
>>>> all that smart.
>>>
>>> I'm sorry, but you're really not getting the point. The crushomatic
>>> already *has* a blinking light and a siren and everyone is *already*
>>> ignoring them, because they go off all the time, whenever a person
>>> gets closer than 6 feet. So we have no way to warn them when there
>>> *really* is a danger, like when a person is about to put their hand
>>> into the blender.
>>>
>>> The solution is to make the machine less dangerous (e.g. if you open
>>> the lid the motor is cut off), not to argue that the siren is really
>>> important.
>>
>>Rereading my post I don't see any place where I seem to have
>>argued that deprecation warnings were "really important", but
>>in case that's how it sounded, let me clarify: I don't think this
>>argument is anything other than lots and lots of bikeshedding,
>>since these warnings can already be explicitly silenced by the
>>very people this is designed to protect.
>
> My point is that 'designing to protect people' is not enough, you need
> to actually protect them. ?And while in general 'logically this should
> protect them' is enough -- (and what intelligent people often mean by
> 'designing to protect people') -- when it comes to issues of cognative
> psychology, the important lessons to learn are the ones that teach you
> is where this one fails ---- by trying to protect X, or at least
> make it easier for X, you end up making it harder.

I'm having some difficulty reconciling the idea of "protecting"
developers from warnings with "we're all adults here" and
"errors should never pass silently, unless explicitly silenced".
Should I be?

Geremy Condra

From guido at python.org  Tue Nov 10 03:00:41 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2009 18:00:41 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <f3cc57c60911091741m32fcd2f7jffad3bbc2e466ff0@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com> 
	<7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>
	<f3cc57c60911091544w774f1a44s49600063c22f5490@mail.gmail.com> 
	<ca471dc20911091601w1b6e0636md7486949e1fb024a@mail.gmail.com> 
	<debatem1@gmail.com>
	<f3cc57c60911091703l6d14e541jef10ac68d8efb7e6@mail.gmail.com> 
	<200911100122.nAA1Ma1i022342@theraft.openend.se>
	<f3cc57c60911091741m32fcd2f7jffad3bbc2e466ff0@mail.gmail.com>
Message-ID: <ca471dc20911091800x5dae98bfma4a4c8812683a674@mail.gmail.com>

On Mon, Nov 9, 2009 at 5:41 PM, geremy condra <debatem1 at gmail.com> wrote:
> I'm having some difficulty reconciling the idea of "protecting"
> developers from warnings with "we're all adults here" and
> "errors should never pass silently, unless explicitly silenced".
> Should I be?

Yes. Warnings are effectively errors that are allowed to pass -- in
effect, silently, since users end up silencing them.

The "manly" thing to do (to pick an expression popularized by Linus
Torvalds) would be to have only errors, and test your code with new
Python versions before allowing it to run there.

-- 
--Guido van Rossum (python.org/~guido)

From debatem1 at gmail.com  Tue Nov 10 03:41:35 2009
From: debatem1 at gmail.com (geremy condra)
Date: Mon, 9 Nov 2009 21:41:35 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <ca471dc20911091800x5dae98bfma4a4c8812683a674@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<7A0362F9252D4FE9B083F743150C1F7B@RaymondLaptop1>
	<f3cc57c60911091544w774f1a44s49600063c22f5490@mail.gmail.com>
	<ca471dc20911091601w1b6e0636md7486949e1fb024a@mail.gmail.com>
	<debatem1@gmail.com>
	<f3cc57c60911091703l6d14e541jef10ac68d8efb7e6@mail.gmail.com>
	<200911100122.nAA1Ma1i022342@theraft.openend.se>
	<f3cc57c60911091741m32fcd2f7jffad3bbc2e466ff0@mail.gmail.com>
	<ca471dc20911091800x5dae98bfma4a4c8812683a674@mail.gmail.com>
Message-ID: <f3cc57c60911091841l3ed580cdk7524ab7fc8a78414@mail.gmail.com>

On Mon, Nov 9, 2009 at 9:00 PM, Guido van Rossum <guido at python.org> wrote:
> On Mon, Nov 9, 2009 at 5:41 PM, geremy condra <debatem1 at gmail.com> wrote:
>> I'm having some difficulty reconciling the idea of "protecting"
>> developers from warnings with "we're all adults here" and
>> "errors should never pass silently, unless explicitly silenced".
>> Should I be?
>
> Yes. Warnings are effectively errors that are allowed to pass -- in
> effect, silently, since users end up silencing them.

Ok, so whats the problem with letting them keep silencing them?
I don't see how you can get error ennui from warnings you never
see.

> The "manly" thing to do (to pick an expression popularized by Linus
> Torvalds) would be to have only errors, and test your code with new
> Python versions before allowing it to run there.

Lets say I wrestle wolverines in my spare time- are you saying that
the warnings should be silent or that there just shouldn't be any?

As an aside, given how often the discussion of running on a version
you haven't tested comes up, is there a convenience function that
raises an error in the event that you're outside of the tested range?
Something akin to:

def check_version(minimum, maximum):
	from sys import version_info
	maximum = tuple(int(x) for x in maximum.split("."))
	minimum = tuple(int(x) for x in minimum.split("."))
	if not minimum < version_info < maximum:
		raise RuntimeError("Improper version of Python")

Geremy Condra

From guido at python.org  Tue Nov 10 04:31:12 2009
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2009 19:31:12 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <9d153b7c0911091613o283cb098m2648c7fd81d9aa7e@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com> 
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com> 
	<87vdhj1h0d.fsf@benfinney.id.au>
	<9d153b7c0911091509q57212e68nbca5f3ec386c1e2e@mail.gmail.com> 
	<ca471dc20911091524o42e0c7a5td4a5430d83ad1194@mail.gmail.com> 
	<9d153b7c0911091613o283cb098m2648c7fd81d9aa7e@mail.gmail.com>
Message-ID: <ca471dc20911091931u9ebdb0cgae86699d79a10a9f@mail.gmail.com>

On Mon, Nov 9, 2009 at 4:13 PM, Yuvgoog Greenle <ubershmekel at gmail.com> wrote:
> First of all I'd like to apologize to Ben, this may sound like a lame
> excuse but I guess Ben Finney somehow turned into Benny because of my
> lack of sleep. So again sorry for that, I really didn't mean it that
> way.
>
> Yuvgoog is a pseudonym indeed, my real name is Yuval Greenfield. I'd
> grown a habit of trying to see where I get spam mail from so I use
> names mangled with the service name. You might see me as Yuvpic
> Greenasa, Yuvfaceal Greenbookfield, etc. I guess this habit's one I
> should get rid of, especially seeing as it really hasn't paid off (I
> haven't caught a single spammer yet after probably 6 years of it).

Thanks Yuval. This puts a much more human face on your posts, which
matters to me. A lot, actually.

> Concerning debugging warnings, I have to say I currently remember my
> experience with 2: the deprecation of sets and the deprecation of
> string exceptions. On both occasions the warnings helped me figure
> things out before my code broke. Specifically with raising string
> exceptions, warnings saved me and my colleagues from alot of bad code,
> string exceptions were the convention at my office and I only found
> out early because I installed a new python that warned me. Finding
> this out as an error might not have been too bad either but it would
> have probably taken more time until I got a python that doesn't allow
> raising strings (2.6!).

In the balance, I think it's easy enough to find these things by using
the version that actually breaks things. If you are your only
customer, until then, there's little benefit in fixing the warnings
ahead of time (except perhaps for learning -- but, strangely, there
are ways of learning about programming besides trying things out :-).
If you have other customers, you should get used to testing with the
latest version of Python anyway -- running without warnings in a
previous version really isn't a good enough test for future
compatibility.

-- 
--Guido van Rossum (python.org/~guido)

From ubershmekel at gmail.com  Tue Nov 10 08:32:41 2009
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Tue, 10 Nov 2009 09:32:41 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911091931u9ebdb0cgae86699d79a10a9f@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<87vdhj1h0d.fsf@benfinney.id.au>
	<9d153b7c0911091509q57212e68nbca5f3ec386c1e2e@mail.gmail.com>
	<ca471dc20911091524o42e0c7a5td4a5430d83ad1194@mail.gmail.com>
	<9d153b7c0911091613o283cb098m2648c7fd81d9aa7e@mail.gmail.com>
	<ca471dc20911091931u9ebdb0cgae86699d79a10a9f@mail.gmail.com>
Message-ID: <9d153b7c0911092332u6a93f977wd9bc776521e4249a@mail.gmail.com>

On Tue, Nov 10, 2009 at 5:31 AM, Guido van Rossum <guido at python.org> wrote:
[snip]
> In the balance, I think it's easy enough to find these things by using
> the version that actually breaks things. If you are your only
> customer, until then, there's little benefit in fixing the warnings
> ahead of time (except perhaps for learning -- but, strangely, there
> are ways of learning about programming besides trying things out :-).
> If you have other customers, you should get used to testing with the
> latest version of Python anyway -- running without warnings in a
> previous version really isn't a good enough test for future
> compatibility.
>
> --
> --Guido van Rossum (python.org/~guido)
>

Yeah, it's easy to find and fix the bugs using the version that incurs
them. I'm not worried about that, I'm more worried about how
information will spread, a lot of pythonistas I know won't learn about
deprecations until their program is broken. Deprecation warnings
spread the word more elegantly and faster than hard deprecations IMO.

I guess an upside to silent warnings would be that we could have a lot
more of them. It could be nice if pylint was integrated as a warning
level, lets say with 2 popular conventions in the std-lib. I
personally always maximize the warning level on my C compiler and I
like the insight.

--yuv

From nicolas.chauvat at logilab.fr  Tue Nov 10 10:26:47 2009
From: nicolas.chauvat at logilab.fr (Nicolas Chauvat)
Date: Tue, 10 Nov 2009 10:26:47 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy	they are)
Message-ID: <20091110092647.GD5378@volans.logilab.fr>

Hi List,

Guido said:
> At Google we use a hacked-up version of pylint which seems infinitely
> flexible in the checks you can specify. I assume that the public
> version of pylint is just as capable -- someone just needs to write a
> rule for it.

Now that you mention it... I have tried in the past to get people from
Google to contribute these changes back to the trunk_, but without
success.

There will be a `pylint bug day`_ via internet on nov 25th, 2009.

Any chance you could pass the info to the Google employees in charge
of the hacked-up version of pylint ?

.. _trunk: http://www.logilab.org/hg/pylint
.. _`pylint bug day`: http://www.logilab.org/blogentry/18781

Thanks in advance,

PS: If someone else from the list is reading this and wants to join
and maybe implement the checking rule mentionned above in the thread,
please do!

-- 
Nicolas Chauvat

logilab.fr - services en informatique scientifique et gestion de connaissances  

From nad at acm.org  Tue Nov 10 00:34:06 2009
From: nad at acm.org (Ned Deily)
Date: Mon, 09 Nov 2009 15:34:06 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
Message-ID: <nad-966ADB.15340609112009@news.gmane.org>

In article 
<f3cc57c60911091514i43f67896q489adca0585bc460 at mail.gmail.com>,
 geremy condra <debatem1 at gmail.com> 
 wrote:
> On Mon, Nov 9, 2009 at 6:09 PM, Raymond Hettinger 
> <python at rcn.com> wrote:
> > Nice summary. ?FWIW, I concur with Brett. ?No one needs to see
> > warnings/deprecations except for the developer who controls the code.
> > Even that developer may only need to see it at one point during the
> > development process. ?Further, the developer controlling the code
> > just may not care -- the script can be for single use, for a class project,
> > or some other purpose that doesn't require being notified of what may
> > change in a later version of Python.

+1

> If they don't care, why does it matter whether they see it or not?
> Seems like an argument for the status quo.

Because, as it stands in the case of third-party packages, the 
deprecation warnings target the wrong audience, the end-users.  At best 
the status quo behavior is a constant annoyance and, at worst, 
encourages end users to learn to ignore them, even in cases where it 
might eventually matter in their own code.

-- 
 Ned Deily,
 nad at acm.org


From barry at python.org  Tue Nov 10 15:37:48 2009
From: barry at python.org (Barry Warsaw)
Date: Tue, 10 Nov 2009 08:37:48 -0600
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
Message-ID: <8B3BE52C-3EF0-40E0-8362-1F5F28C1A6C3@python.org>

On Nov 8, 2009, at 3:26 PM, Brett Cannon wrote:

> During the moratorium PEP discussions Guido said he wanted to quiet
> down deprecation warnings. I see there being two options on this.

I'm probably unable to catch up on this whole thread right now, but  
having been recently defeated by DeprecationWarnings when attempting  
to update Launchpad to Python 2.6, I'd say the one thing that would  
have really helped us is an environment variable tied to -W, e.g  
$PYTHONWARNINGS.  If I could have shut up DeprecationWarnings with one  
wave of my magic setenv, I think things would have gone much quicker  
for us.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091110/7ecfd8e6/attachment.pgp>

From barry at python.org  Tue Nov 10 15:41:32 2009
From: barry at python.org (Barry Warsaw)
Date: Tue, 10 Nov 2009 08:41:32 -0600
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
Message-ID: <9306AD19-7365-4912-B5AA-860A4C68628D@python.org>

On Nov 8, 2009, at 3:40 PM, Guido van Rossum wrote:

> I was specifically after all kinds of deprecation warnings, which
> generally are telling you that you need to make a change in order to
> remain compatible in the future. That's sometimes interesting but
> often irrelevant noise. So I would agree with Gregory P Smith's
> proposal to just treat all deprecation warnings as silent.

+1

> There are other kinds warnings which might be useful for other reasons
> -- they typically point out code that does not do what you might think
> it does. A good example is the warning added in 2.6 about "assert (x,
> y)". This is something you ignore at your peril.

Yes, I was quite surprised at the few small example of this in  
Launchpad.  This was actually something I was glad to see warned  
about, and of course we landed fixes for these independent of our  
Python 2.6 migration work.

> I disagree. The -3 option is an example of a better approach: silent
> by default, warnings enabled by a command line flag. If we can trust
> developers to use -3 to check for Py3k incompatibilities, we should
> also be able to trust them to check for deprecation warnings
> explicitly.

+1

> (Another argument to think about: if you download and install some 3rd
> party code, deprecation warnings about that code are *only noise*
> since they are not yours to fix. Warnings in a dynamic language work
> very different than warnings in a compiled language.)

Very +1

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091110/97077d85/attachment.pgp>

From solipsis at pitrou.net  Tue Nov 10 15:45:33 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Nov 2009 15:45:33 +0100
Subject: [stdlib-sig] -Wi
In-Reply-To: <8B3BE52C-3EF0-40E0-8362-1F5F28C1A6C3@python.org>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<8B3BE52C-3EF0-40E0-8362-1F5F28C1A6C3@python.org>
Message-ID: <1257864333.3579.10.camel@localhost>


> I'm probably unable to catch up on this whole thread right now, but  
> having been recently defeated by DeprecationWarnings when attempting  
> to update Launchpad to Python 2.6, I'd say the one thing that would  
> have really helped us is an environment variable tied to -W, e.g  
> $PYTHONWARNINGS.  If I could have shut up DeprecationWarnings with one  
> wave of my magic setenv, I think things would have gone much quicker  
> for us.

That's a good point, I think. How about opening a ticket?



From barry at python.org  Tue Nov 10 15:49:49 2009
From: barry at python.org (Barry Warsaw)
Date: Tue, 10 Nov 2009 08:49:49 -0600
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <1257748770.3507.12.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911081805x2977064bw698550aa4d8b9fff@mail.gmail.com>
	<1257747008.3507.5.camel@localhost>
	<200911090625.nA96PYfX009999@theraft.openend.se>
	<1257748770.3507.12.camel@localhost>
Message-ID: <F969A113-86B7-49E8-A675-59B72E0B4AB8@python.org>

On Nov 9, 2009, at 12:39 AM, Antoine Pitrou wrote:

> If Mercurial users and developers hadn't seen those warnings at all,
> perhaps Mercurial would have continued using deprecated constructs,  
> and
> ended up broken when the N+1 Python version had been released. If even
> an established FLOSS project such as Mercurial is vulnerable to this
> kind of risk, then any in-house or one-man project will be even more
> vulnerable.

There are two different use cases here.  I'll s/Mercurial/Bazaar/  
since I'm more familiar with the latter.

We have bzr the command line script and bzrlib the library.  When I  
use or develop bzr, I would be fine with seeing the  
DeprecationWarnings, and using those to pressure upstream to make bzr  
compatible with newer versions of Python.

As a client of bzrlib from a different application, I'm much less  
happy about those DeprecationWarnings because there's probably much  
less I can do about it.  Maybe I didn't even realize that bzrlib got  
pulled into my application as a dependency, well until it started  
screaming at me. ;)  I may not be able to fix those problems, and even  
if I could pressure upstream to fix them, it may not help me much  
because I'm using an egg, or even more glacially a package from my  
distro.  So in this case, those warnings /are/ just noise and not  
always easy to shut off.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091110/142eb42d/attachment.pgp>

From debatem1 at gmail.com  Tue Nov 10 15:50:30 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 09:50:30 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <nad-966ADB.15340609112009@news.gmane.org>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
Message-ID: <f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>

On Mon, Nov 9, 2009 at 6:34 PM, Ned Deily <nad at acm.org> wrote:
> In article
> <f3cc57c60911091514i43f67896q489adca0585bc460 at mail.gmail.com>,
> ?geremy condra <debatem1 at gmail.com>
> ?wrote:
>> On Mon, Nov 9, 2009 at 6:09 PM, Raymond Hettinger
>> <python at rcn.com> wrote:
>> > Nice summary. ?FWIW, I concur with Brett. ?No one needs to see
>> > warnings/deprecations except for the developer who controls the code.
>> > Even that developer may only need to see it at one point during the
>> > development process. ?Further, the developer controlling the code
>> > just may not care -- the script can be for single use, for a class project,
>> > or some other purpose that doesn't require being notified of what may
>> > change in a later version of Python.
>
> +1
>
>> If they don't care, why does it matter whether they see it or not?
>> Seems like an argument for the status quo.
>
> Because, as it stands in the case of third-party packages, the
> deprecation warnings target the wrong audience, the end-users. ?At best
> the status quo behavior is a constant annoyance and, at worst,
> encourages end users to learn to ignore them, even in cases where it
> might eventually matter in their own code.

Ok, so whats wrong with just saying

import warnings
warnings.simplefilter("ignore")

and walking away?

Geremy Condra

From barry at python.org  Tue Nov 10 16:06:49 2009
From: barry at python.org (Barry Warsaw)
Date: Tue, 10 Nov 2009 09:06:49 -0600
Subject: [stdlib-sig] -Wi
In-Reply-To: <1257864333.3579.10.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<8B3BE52C-3EF0-40E0-8362-1F5F28C1A6C3@python.org>
	<1257864333.3579.10.camel@localhost>
Message-ID: <6E417155-062E-48C3-B595-44F7328F0680@python.org>

On Nov 10, 2009, at 8:45 AM, Antoine Pitrou wrote:

>> I'm probably unable to catch up on this whole thread right now, but
>> having been recently defeated by DeprecationWarnings when attempting
>> to update Launchpad to Python 2.6, I'd say the one thing that would
>> have really helped us is an environment variable tied to -W, e.g
>> $PYTHONWARNINGS.  If I could have shut up DeprecationWarnings with  
>> one
>> wave of my magic setenv, I think things would have gone much quicker
>> for us.
>
> That's a good point, I think. How about opening a ticket?

http://bugs.python.org/issue7301

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091110/4ccd595f/attachment.pgp>

From nad at acm.org  Tue Nov 10 16:19:40 2009
From: nad at acm.org (Ned Deily)
Date: Tue, 10 Nov 2009 07:19:40 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
Message-ID: <nad-722C39.07193910112009@news.gmane.org>

In article 
<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6 at mail.gmail.com>,
 geremy condra <debatem1 at gmail.com> 
 wrote:
> Ok, so whats wrong with just saying
> 
> import warnings
> warnings.simplefilter("ignore")
> 
> and walking away?

If the package is a stand-alone application (c.f. Barry's bzr example), 
it's not reasonable to ask end users to modify its code; they may not 
even be able to easily (i.e. root privileges required).  More generally, 
it seems unfair and unwise to ask the 10 000 users of a package to take 
action when ultimately the 1 maintainer of the package is the one who 
needs to do so.

-- 
 Ned Deily,
 nad at acm.org


From solipsis at pitrou.net  Tue Nov 10 16:28:47 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Nov 2009 16:28:47 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
 they are)
In-Reply-To: <nad-722C39.07193910112009@news.gmane.org>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
Message-ID: <1257866927.3579.20.camel@localhost>


> If the package is a stand-alone application (c.f. Barry's bzr example), 
> it's not reasonable to ask end users to modify its code; they may not 
> even be able to easily (i.e. root privileges required).  More generally, 
> it seems unfair and unwise to ask the 10 000 users of a package to take 
> action when ultimately the 1 maintainer of the package is the one who 
> needs to do so.

That's why it is advised to report the problem to the maintainer (or
perhaps the packager, in case e.g. of a linux distro), like you do for
any other bug.

It should be stressed again that it is not a very common problem (how
many Python apps do you routinely launch from the command-line, apart
from hg, bzr, buildout & friends? (*)), and it's not a critical one
either (you can perfectly live with it, like you can live with the
occasional warning about a self-signed HTTPS certificate).
On the other had, having the application break when upgrading to Python
N+1 would be critical.

(*) All these are developer tools by the way, not end-user apps.



From debatem1 at gmail.com  Tue Nov 10 16:31:47 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 10:31:47 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <nad-722C39.07193910112009@news.gmane.org>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
Message-ID: <f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>

On Tue, Nov 10, 2009 at 10:19 AM, Ned Deily <nad at acm.org> wrote:
> In article
> <f3cc57c60911100650q551ca79em59d7c47ff61aa5a6 at mail.gmail.com>,
> ?geremy condra <debatem1 at gmail.com>
> ?wrote:
>> Ok, so whats wrong with just saying
>>
>> import warnings
>> warnings.simplefilter("ignore")
>>
>> and walking away?
>
> If the package is a stand-alone application (c.f. Barry's bzr example),
> it's not reasonable to ask end users to modify its code; they may not
> even be able to easily (i.e. root privileges required). ?More generally,
> it seems unfair and unwise to ask the 10 000 users of a package to take
> action when ultimately the 1 maintainer of the package is the one who
> needs to do so.
>
> --
> ?Ned Deily,
> ?nad at acm.org

Let me rephrase- I'm not asking end users to silence them, I'm
saying that if it annoys the end users so much, the devs should
do it themselves.

Geremy Condra

From ronaldoussoren at mac.com  Tue Nov 10 16:48:25 2009
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 10 Nov 2009 16:48:25 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
 they are)
In-Reply-To: <f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
Message-ID: <F91E4361-58F2-4BA5-9D00-F82B75544FFC@mac.com>


On 10 Nov, 2009, at 16:31, geremy condra wrote:

> On Tue, Nov 10, 2009 at 10:19 AM, Ned Deily <nad at acm.org> wrote:
>> In article
>> <f3cc57c60911100650q551ca79em59d7c47ff61aa5a6 at mail.gmail.com>,
>>  geremy condra <debatem1 at gmail.com>
>>  wrote:
>>> Ok, so whats wrong with just saying
>>> 
>>> import warnings
>>> warnings.simplefilter("ignore")
>>> 
>>> and walking away?
>> 
>> If the package is a stand-alone application (c.f. Barry's bzr example),
>> it's not reasonable to ask end users to modify its code; they may not
>> even be able to easily (i.e. root privileges required).  More generally,
>> it seems unfair and unwise to ask the 10 000 users of a package to take
>> action when ultimately the 1 maintainer of the package is the one who
>> needs to do so.
>> 
>> --
>>  Ned Deily,
>>  nad at acm.org
> 
> Let me rephrase- I'm not asking end users to silence them, I'm
> saying that if it annoys the end users so much, the devs should
> do it themselves.

How do I do that for the libraries I distribute? Users of my libraries should not get DeprecationWarnings about my code, but I should be able to generate DeprecationWarnings of my own when I deprecate some of my APIs. Oh, and it should still be possible for me to check for DeprecationWarnings in my code.

Ronald

> 
> Geremy Condra
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3567 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091110/625ab768/attachment.bin>

From santagada at gmail.com  Tue Nov 10 16:54:28 2009
From: santagada at gmail.com (Leonardo Santagada)
Date: Tue, 10 Nov 2009 13:54:28 -0200
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <1257866927.3579.20.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
Message-ID: <1DFA3EFA-2663-42B3-B2C3-3067FED47F20@gmail.com>


On Nov 10, 2009, at 1:28 PM, Antoine Pitrou wrote:

> That's why it is advised to report the problem to the maintainer (or
> perhaps the packager, in case e.g. of a linux distro), like you do for
> any other bug.

So what you are saying is that all python app developer should receive  
hundreds of bug reports (or at least "me too" comments) for months if  
he decides that he has other priorities than to fix deprecation  
warnings?

There is a reason it takes more than a year to remove something from  
the language and that is so that people can adapt on their own time,  
so you are either saying that users should ignore errors (giving way  
to Laura's argument) or that they should be bullied by its users. Or  
are you suggesting that the default should be warnings on and everyone  
should release their software with code to turn it off?

If I could vote I would be +1 on the idea of warnings off and moving  
that to pylint or some other tool, but with a good paragraph or two on  
the docs about it. Or a switch to turn warnings on with an api so that  
unittest and py.test could turn them on.

--
Leonardo Santagada
santagada at gmail.com




From solipsis at pitrou.net  Tue Nov 10 16:55:59 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 10 Nov 2009 16:55:59 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
 they are)
In-Reply-To: <4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
	<4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
Message-ID: <1257868559.3579.36.camel@localhost>


> Why is that relevant? Bzr and hg are not only for Python developers,
> they can just as well be used for other code or even configuration
> files.

It is relevant because developers (of Python or not) should understand
that warning messages are a necessary evil in order to warn of a
potential pitfall.
In other words, there's no reason for a sensible developer to be angered
by the sight of warning messages when using a developer-oriented tool.
And, similarly, a system administrator editing configuration files knows
that warning messages exist for a reason.

> I know that I get annoyed by "random" messages from Java tools that I
> use, which doesn't help improve my opinion of that language.

Why does it impact your opinion of Java, rather than your opinion of the
developers who did nothing to fix the problem in their package?
(of course, the deprecation themselves are perhaps mistaken, see
Marc-Andr?'s message about that)

As the user of a Python package, don't you want to know that your
current version of a package may break when you switch to Python N+1?
Do you prefer the pleasant surprise of discovering it after the fact?

Regards

Antoine.



From guido at python.org  Tue Nov 10 17:00:27 2009
From: guido at python.org (Guido van Rossum)
Date: Tue, 10 Nov 2009 08:00:27 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com> 
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com> 
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com> 
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com> 
	<nad-722C39.07193910112009@news.gmane.org>
	<f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
Message-ID: <ca471dc20911100800o2f4e2f05kd30f6591c2e24455@mail.gmail.com>

On Tue, Nov 10, 2009 at 7:31 AM, geremy condra <debatem1 at gmail.com> wrote:
> Let me rephrase- I'm not asking end users to silence them, I'm
> saying that if it annoys the end users so much, the devs should
> do it themselves.

So either way the developers must be aware of the warnings. In your
proposal, every developer must add a special hack to their code as
distribute, which they must disable when they test. In my proposal,
developers must explicitly run with some flag (e.g. -w) when they
test. The burden for the developer is about the same in the case of a
conscientious developer, while in your proposal lazy developers place
the burden on their users. I really don't see how your proposal is
better.

-- 
--Guido van Rossum (python.org/~guido)

From ronaldoussoren at mac.com  Tue Nov 10 17:17:59 2009
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 10 Nov 2009 17:17:59 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
 they are)
In-Reply-To: <1257868559.3579.36.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
	<4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
	<1257868559.3579.36.camel@localhost>
Message-ID: <B59B6401-2207-4A36-AA4F-2AA7E477B429@mac.com>


On 10 Nov, 2009, at 16:55, Antoine Pitrou wrote:

> 
>> Why is that relevant? Bzr and hg are not only for Python developers,
>> they can just as well be used for other code or even configuration
>> files.
> 
> It is relevant because developers (of Python or not) should understand
> that warning messages are a necessary evil in order to warn of a
> potential pitfall.
> In other words, there's no reason for a sensible developer to be angered
> by the sight of warning messages when using a developer-oriented tool.
> And, similarly, a system administrator editing configuration files knows
> that warning messages exist for a reason.

That doesn't make warnings for code that I have no control over any less annoying.  And worse, DeprecationWarnings aren't even something that require immediate followup because nothing is broken.  It's not, "I installed Python 2.6 and now my tools no longer work", but "I installed Python 2.6 and am now getting useless crap when I run tools". 


> 
>> I know that I get annoyed by "random" messages from Java tools that I
>> use, which doesn't help improve my opinion of that language.
> 
> Why does it impact your opinion of Java, rather than your opinion of the
> developers who did nothing to fix the problem in their package?
> (of course, the deprecation themselves are perhaps mistaken, see
> Marc-Andr?'s message about that)


> 
> As the user of a Python package, don't you want to know that your
> current version of a package may break when you switch to Python N+1?
> Do you prefer the pleasant surprise of discovering it after the fact?

That's why I test before upgrading a production machine.  Running without DeprecationWarnings is not a perfect predicator for running on a future version of Python.

Furthermore, I get DeprecationWarnings the day I install Python N, even though Python N+1 is still a long time away (18 months on average).

Ronald
> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3567 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091110/c6cdd7ef/attachment.bin>

From ronaldoussoren at mac.com  Tue Nov 10 16:38:55 2009
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 10 Nov 2009 16:38:55 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
 they are)
In-Reply-To: <1257866927.3579.20.camel@localhost>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
Message-ID: <4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>


On 10 Nov, 2009, at 16:28, Antoine Pitrou wrote:

> 
>> If the package is a stand-alone application (c.f. Barry's bzr example), 
>> it's not reasonable to ask end users to modify its code; they may not 
>> even be able to easily (i.e. root privileges required).  More generally, 
>> it seems unfair and unwise to ask the 10 000 users of a package to take 
>> action when ultimately the 1 maintainer of the package is the one who 
>> needs to do so.
> 
> That's why it is advised to report the problem to the maintainer (or
> perhaps the packager, in case e.g. of a linux distro), like you do for
> any other bug.
> 
> It should be stressed again that it is not a very common problem (how
> many Python apps do you routinely launch from the command-line, apart
> from hg, bzr, buildout & friends? (*)), and it's not a critical one
> either (you can perfectly live with it, like you can live with the
> occasional warning about a self-signed HTTPS certificate).
> On the other had, having the application break when upgrading to Python
> N+1 would be critical.

A significant part of the code I write are command-line tools for system administrators. When they see the deprecation warnings they ignore them at best, and at worst get the impression that Python sucks because of these (to them) annoying messages.

As Brett and others noted DeprecationWarnings are useful for the developer of a module, not for their end-users. Unlike to compiled languages end-users get to see Python's Deprecation warnings.

> 
> (*) All these are developer tools by the way, not end-user apps.

Why is that relevant? Bzr and hg are not only for Python developers, they can just as well be used for other code or even configuration files.    I know that I get annoyed by "random" messages from Java tools that I use, which doesn't help improve my opinion of that language.

Ronald
> 
> 
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3567 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/stdlib-sig/attachments/20091110/91c618cb/attachment-0001.bin>

From nad at acm.org  Tue Nov 10 17:45:22 2009
From: nad at acm.org (Ned Deily)
Date: Tue, 10 Nov 2009 08:45:22 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
Message-ID: <nad-F55A88.08452210112009@news.gmane.org>

In article <1257866927.3579.20.camel at localhost>,
 Antoine Pitrou <solipsis at pitrou.net> 
 wrote:
> It should be stressed again that it is not a very common problem (how
> many Python apps do you routinely launch from the command-line, apart
> from hg, bzr, buildout & friends? (*)), and it's not a critical one
> either (you can perfectly live with it, like you can live with the
> occasional warning about a self-signed HTTPS certificate).

Not to belabor the issue, but I don't think that's a good analogy.  In 
the case of a self-signed certificate, I as an end-user do have a 
decision to make: whether to trust the certificate or not as it may have 
dire consequences in real life.  And I am the only one who can make that 
policy decision for me.  (Plus, the mail application and web browser I 
use anticipate this use case and provide user interfaces for me to 
easily evaluate the certificate and to capture my policy decision 
preference so I'm not pestered when the situation arises again.)

> On the other had, having the application break when upgrading to Python
> N+1 would be critical.
> 
> (*) All these are developer tools by the way, not end-user apps.

Roles are relative: I would consider a Mozilla developer to be an 
end-user of hg.

-- 
 Ned Deily,
 nad at acm.org


From debatem1 at gmail.com  Tue Nov 10 17:49:34 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 11:49:34 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <ca471dc20911100800o2f4e2f05kd30f6591c2e24455@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
	<ca471dc20911100800o2f4e2f05kd30f6591c2e24455@mail.gmail.com>
Message-ID: <f3cc57c60911100849m300ddf67t8c13d928fdf311d9@mail.gmail.com>

On Tue, Nov 10, 2009 at 11:00 AM, Guido van Rossum <guido at python.org> wrote:
> On Tue, Nov 10, 2009 at 7:31 AM, geremy condra <debatem1 at gmail.com> wrote:
>> Let me rephrase- I'm not asking end users to silence them, I'm
>> saying that if it annoys the end users so much, the devs should
>> do it themselves.
>
> So either way the developers must be aware of the warnings.

I don't understand how default suppressing all warnings implies
that a developer "must" be aware of them.

> In your proposal, every developer must add a special hack to
> their code as distribute, which they must disable when they test.

You only need the "special hack" (I'd point out that its in the
standard library docs) if you're running deprecated code. Leave
it off until a warning comes up, then decide what you're going
to do about the warning, then suppress it. The important thing
was that you were made aware of the problem and then made
a conscious decision to do something about it, rather than
being totally unaware of the problem until it became an error.
That's no better than no warnings at all.

> In my proposal, developers must explicitly run with some flag
> (e.g. -w) when they test. The burden for the developer is about
> the same in the case of a conscientious developer, while in your
> proposal lazy developers place the burden on their users.
> I really don't see how your proposal is better.

1) I'd see the burden of typing python -w every time I tested,
or changing the #! on my test code and breaking it on 3.1
and back, as being much larger than the burden of typing
fifty characters once.

2) a lazy developer probably isn't paying attention to warnings
because he has no intention of fixing them before they blow
up. I'm pretty sure that's not a development habit that should
be well supported by Python.

3) As I've said before, if you don't think the warnings are
important info, then ask pylint and pychecker to take them
up and take them out of Python. That's the case where
there isn't any burden on either the conscientious dev or
the end user.

Geremy Condra

From debatem1 at gmail.com  Tue Nov 10 17:50:09 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 11:50:09 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
	<4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
Message-ID: <f3cc57c60911100850w3ba37a56rd6ac7089741276bb@mail.gmail.com>

On Tue, Nov 10, 2009 at 10:38 AM, Ronald Oussoren
<ronaldoussoren at mac.com> wrote:
>
> On 10 Nov, 2009, at 16:28, Antoine Pitrou wrote:
>
>>
>>> If the package is a stand-alone application (c.f. Barry's bzr example),
>>> it's not reasonable to ask end users to modify its code; they may not
>>> even be able to easily (i.e. root privileges required). ?More generally,
>>> it seems unfair and unwise to ask the 10 000 users of a package to take
>>> action when ultimately the 1 maintainer of the package is the one who
>>> needs to do so.
>>
>> That's why it is advised to report the problem to the maintainer (or
>> perhaps the packager, in case e.g. of a linux distro), like you do for
>> any other bug.
>>
>> It should be stressed again that it is not a very common problem (how
>> many Python apps do you routinely launch from the command-line, apart
>> from hg, bzr, buildout & friends? (*)), and it's not a critical one
>> either (you can perfectly live with it, like you can live with the
>> occasional warning about a self-signed HTTPS certificate).
>> On the other had, having the application break when upgrading to Python
>> N+1 would be critical.
>
> A significant part of the code I write are command-line tools for system administrators. When they see the deprecation warnings they ignore them at best, and at worst get the impression that Python sucks because of these (to them) annoying messages.

<snip>

If you're so worried about the warnings, suppress them. You control the code.

Geremy Condra

From debatem1 at gmail.com  Tue Nov 10 17:58:49 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 11:58:49 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <F91E4361-58F2-4BA5-9D00-F82B75544FFC@mac.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
	<F91E4361-58F2-4BA5-9D00-F82B75544FFC@mac.com>
Message-ID: <f3cc57c60911100858s5f6ca253k8b38551ee3c48164@mail.gmail.com>

On Tue, Nov 10, 2009 at 10:48 AM, Ronald Oussoren
<ronaldoussoren at mac.com> wrote:
>
> On 10 Nov, 2009, at 16:31, geremy condra wrote:
>
>> On Tue, Nov 10, 2009 at 10:19 AM, Ned Deily <nad at acm.org> wrote:
>>> In article
>>> <f3cc57c60911100650q551ca79em59d7c47ff61aa5a6 at mail.gmail.com>,
>>> ?geremy condra <debatem1 at gmail.com>
>>> ?wrote:
>>>> Ok, so whats wrong with just saying
>>>>
>>>> import warnings
>>>> warnings.simplefilter("ignore")
>>>>
>>>> and walking away?
>>>
>>> If the package is a stand-alone application (c.f. Barry's bzr example),
>>> it's not reasonable to ask end users to modify its code; they may not
>>> even be able to easily (i.e. root privileges required). ?More generally,
>>> it seems unfair and unwise to ask the 10 000 users of a package to take
>>> action when ultimately the 1 maintainer of the package is the one who
>>> needs to do so.
>>>
>>> --
>>> ?Ned Deily,
>>> ?nad at acm.org
>>
>> Let me rephrase- I'm not asking end users to silence them, I'm
>> saying that if it annoys the end users so much, the devs should
>> do it themselves.
>
> How do I do that for the libraries I distribute? Users of my libraries should not get DeprecationWarnings about my code, but I should be able to generate DeprecationWarnings of my own when I deprecate some of my APIs. Oh, and it should still be possible for me to check for DeprecationWarnings in my code.
>
> Ronald

http://docs.python.org/3.1/library/warnings.html#module-warnings

Its a pretty well thought out interface, actually. In the cases you
mention (obviously without looking at your code) I'd suggest that
you subclass warning and filter based on that.

Geremy Condra

From guido at python.org  Tue Nov 10 19:19:33 2009
From: guido at python.org (Guido van Rossum)
Date: Tue, 10 Nov 2009 10:19:33 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <f3cc57c60911100850w3ba37a56rd6ac7089741276bb@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com> 
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com> 
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com> 
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost> 
	<4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
	<f3cc57c60911100850w3ba37a56rd6ac7089741276bb@mail.gmail.com>
Message-ID: <ca471dc20911101019i6c2931e6k6deb26386274e9aa@mail.gmail.com>

On Tue, Nov 10, 2009 at 8:50 AM, geremy condra <debatem1 at gmail.com> wrote:
> If you're so worried about the warnings, suppress them. You control the code.

No. The nuisance of warnings is that they are presented to the *user*,
not the developer, and the user (in many cases) doesn't control the
code.

-- 
--Guido van Rossum (python.org/~guido)

From debatem1 at gmail.com  Tue Nov 10 19:43:29 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 13:43:29 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <ca471dc20911101019i6c2931e6k6deb26386274e9aa@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
	<4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
	<f3cc57c60911100850w3ba37a56rd6ac7089741276bb@mail.gmail.com>
	<ca471dc20911101019i6c2931e6k6deb26386274e9aa@mail.gmail.com>
Message-ID: <f3cc57c60911101043x7e2bbb63o85fc078a4d1b6d18@mail.gmail.com>

On Tue, Nov 10, 2009 at 1:19 PM, Guido van Rossum <guido at python.org> wrote:
> On Tue, Nov 10, 2009 at 8:50 AM, geremy condra <debatem1 at gmail.com> wrote:
>> If you're so worried about the warnings, suppress them. You control the code.
>
> No. The nuisance of warnings is that they are presented to the *user*,
> not the developer, and the user (in many cases) doesn't control the
> code.
>
> --
> --Guido van Rossum (python.org/~guido)
>

In this case, however, he does, as per his comment:

"A significant part of the code I write are command-line tools for
system administrators. When they see the deprecation warnings they
ignore them at best, and at worst get the impression that Python sucks
because of these (to them) annoying messages.".

Geremy Condra

From santagada at gmail.com  Tue Nov 10 22:02:18 2009
From: santagada at gmail.com (Leonardo Santagada)
Date: Tue, 10 Nov 2009 19:02:18 -0200
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <f3cc57c60911100849m300ddf67t8c13d928fdf311d9@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<1257803265.3888.114.camel@localhost>
	<bbaeab100911091419p5b4abef2ia1542be6882160a4@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
	<ca471dc20911100800o2f4e2f05kd30f6591c2e24455@mail.gmail.com>
	<f3cc57c60911100849m300ddf67t8c13d928fdf311d9@mail.gmail.com>
Message-ID: <9B120D89-CC9B-4A3E-AC46-68D1F8C3F4FA@gmail.com>


On Nov 10, 2009, at 2:49 PM, geremy condra wrote:

> On Tue, Nov 10, 2009 at 11:00 AM, Guido van Rossum  
> <guido at python.org> wrote:
>> On Tue, Nov 10, 2009 at 7:31 AM, geremy condra <debatem1 at gmail.com>  
>> wrote:
>>> Let me rephrase- I'm not asking end users to silence them, I'm
>>> saying that if it annoys the end users so much, the devs should
>>> do it themselves.
>>
>> So either way the developers must be aware of the warnings.
>
> I don't understand how default suppressing all warnings implies
> that a developer "must" be aware of them.
>
>> In your proposal, every developer must add a special hack to
>> their code as distribute, which they must disable when they test.
>
> You only need the "special hack" (I'd point out that its in the
> standard library docs) if you're running deprecated code. Leave
> it off until a warning comes up, then decide what you're going
> to do about the warning, then suppress it. The important thing
> was that you were made aware of the problem and then made
> a conscious decision to do something about it, rather than
> being totally unaware of the problem until it became an error.
> That's no better than no warnings at all.

So you are saying that developers should re-release their software  
when a new python release happens just to put some code to disable  
warnings? It does take effort to make a software release, and every  
time you do one is a chance to screw things up (even python had  
releases issues). For me it is unacceptable.

> 3) As I've said before, if you don't think the warnings are
> important info, then ask pylint and pychecker to take them
> up and take them out of Python. That's the case where
> there isn't any burden on either the conscientious dev or
> the end user.


Good that you agree on taking the warnings out of python :)

--
Leonardo Santagada
santagada at gmail.com




From ben+python at benfinney.id.au  Wed Nov 11 00:00:14 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 11 Nov 2009 10:00:14 +1100
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
	<4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
	<f3cc57c60911100850w3ba37a56rd6ac7089741276bb@mail.gmail.com>
	<ca471dc20911101019i6c2931e6k6deb26386274e9aa@mail.gmail.com>
	<f3cc57c60911101043x7e2bbb63o85fc078a4d1b6d18@mail.gmail.com>
Message-ID: <87iqdi0yv5.fsf@benfinney.id.au>

geremy condra <debatem1 at gmail.com> writes:

> On Tue, Nov 10, 2009 at 1:19 PM, Guido van Rossum <guido at python.org> wrote:
> > On Tue, Nov 10, 2009 at 8:50 AM, geremy condra <debatem1 at gmail.com> wrote:
> >> If you're so worried about the warnings, suppress them. You control
> >> the code.
> >
> > No. The nuisance of warnings is that they are presented to the
> > *user*, not the developer, and the user (in many cases) doesn't
> > control the code.
> >
> > --
> > --Guido van Rossum (python.org/~guido)
> >
>
> In this case, however, he does, as per his comment:

Antoine (?he?, in your sentence) does. *The user does not*, and it is
the user who is seeing the warnings:

> "A significant part of the code I write are command-line tools for
> system administrators. When they see the deprecation warnings they
> ignore them at best, and at worst get the impression that Python sucks
> because of these (to them) annoying messages.".

-- 
 \        ?You can be a victor without having victims.? ?Harriet Woods |
  `\                                                                   |
_o__)                                                                  |
Ben Finney


From ben+python at benfinney.id.au  Wed Nov 11 00:08:27 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 11 Nov 2009 10:08:27 +1100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<87vdhj1h0d.fsf@benfinney.id.au>
	<9d153b7c0911091509q57212e68nbca5f3ec386c1e2e@mail.gmail.com>
Message-ID: <87eio60yhg.fsf@benfinney.id.au>

Guido van Rossum <guido at python.org> writes:

> On Mon, Nov 9, 2009 at 3:09 PM, Yuvgoog Greenle <ubershmekel at gmail.com> wrote:
> > On Tue, Nov 10, 2009 at 12:16 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> [...]
>
> > Sorry Benny but you were convinced by an invalid argument.
>
> Maybe you know Ben personally and maybe it's different in your
> culture, but to me it sounds pretty derogatory to call someone "Benny"
> who signs as "Ben".

This is all true, and I agree with Guido that care needs to be taken;
?Benny? is a diminutive of ?Ben?. Overly familiar, unless one knows the
person intimately.

Yuvgoog Greenle <ubershmekel at gmail.com>
writes:

> First of all I'd like to apologize to Ben, this may sound like a lame
> excuse but I guess Ben Finney somehow turned into Benny because of my
> lack of sleep. So again sorry for that, I really didn't mean it that
> way.

No harm done (in this case I assumed it was a cultural slip).

Apology accepted.

-- 
 \          ?And if I laugh at any mortal thing, / 'Tis that I may not |
  `\               weep.? ??Lord? George Gordon Noel Byron, _Don Juan_ |
_o__)                                                                  |
Ben Finney


From debatem1 at gmail.com  Wed Nov 11 00:37:22 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 18:37:22 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <9B120D89-CC9B-4A3E-AC46-68D1F8C3F4FA@gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<E1BE1CBC1B934B7D94FDFEF54F34CC37@RaymondLaptop1>
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
	<ca471dc20911100800o2f4e2f05kd30f6591c2e24455@mail.gmail.com>
	<f3cc57c60911100849m300ddf67t8c13d928fdf311d9@mail.gmail.com>
	<9B120D89-CC9B-4A3E-AC46-68D1F8C3F4FA@gmail.com>
Message-ID: <f3cc57c60911101537o77cdd797n46d06e04dde6de26@mail.gmail.com>

On Tue, Nov 10, 2009 at 4:02 PM, Leonardo Santagada <santagada at gmail.com> wrote:
>
> On Nov 10, 2009, at 2:49 PM, geremy condra wrote:
>
>> On Tue, Nov 10, 2009 at 11:00 AM, Guido van Rossum <guido at python.org>
>> wrote:
>>>
>>> On Tue, Nov 10, 2009 at 7:31 AM, geremy condra <debatem1 at gmail.com>
>>> wrote:
>>>>
>>>> Let me rephrase- I'm not asking end users to silence them, I'm
>>>> saying that if it annoys the end users so much, the devs should
>>>> do it themselves.
>>>
>>> So either way the developers must be aware of the warnings.
>>
>> I don't understand how default suppressing all warnings implies
>> that a developer "must" be aware of them.
>>
>>> In your proposal, every developer must add a special hack to
>>> their code as distribute, which they must disable when they test.
>>
>> You only need the "special hack" (I'd point out that its in the
>> standard library docs) if you're running deprecated code. Leave
>> it off until a warning comes up, then decide what you're going
>> to do about the warning, then suppress it. The important thing
>> was that you were made aware of the problem and then made
>> a conscious decision to do something about it, rather than
>> being totally unaware of the problem until it became an error.
>> That's no better than no warnings at all.
>
> So you are saying that developers should re-release their software when a
> new python release happens just to put some code to disable warnings? It
> does take effort to make a software release, and every time you do one is a
> chance to screw things up (even python had releases issues). For me it is
> unacceptable.

If its an absolute no-no that warnings be raised then blanket
suppress them. The snippet of code from the warnings module
should do the job.

>> 3) As I've said before, if you don't think the warnings are
>> important info, then ask pylint and pychecker to take them
>> up and take them out of Python. That's the case where
>> there isn't any burden on either the conscientious dev or
>> the end user.
>
>
> Good that you agree on taking the warnings out of python :)

I think this whole discussion is pointless bikeshedding. My
point here is that changing the way python is invoked doesn't
strike me as really solving the problem. If you believe that the
warnings are the problem, then lobby to have them taken out.
If you think the timeline for them is the problem, then change
that. If you just want to stop your users from seeing those
warnings, but think they're appropriate, then the tools to do
that already exist, and I would much prefer that you used
them.


Geremy Condra

From debatem1 at gmail.com  Wed Nov 11 00:40:36 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 18:40:36 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <87iqdi0yv5.fsf@benfinney.id.au>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<1257866927.3579.20.camel@localhost>
	<4BA8CB27-7B1C-41C9-964D-EAB9A6444322@mac.com>
	<f3cc57c60911100850w3ba37a56rd6ac7089741276bb@mail.gmail.com>
	<ca471dc20911101019i6c2931e6k6deb26386274e9aa@mail.gmail.com>
	<f3cc57c60911101043x7e2bbb63o85fc078a4d1b6d18@mail.gmail.com>
	<87iqdi0yv5.fsf@benfinney.id.au>
Message-ID: <f3cc57c60911101540y11a0d50bsfca065d5c16abbed@mail.gmail.com>

On Tue, Nov 10, 2009 at 6:00 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> geremy condra <debatem1 at gmail.com> writes:
>
>> On Tue, Nov 10, 2009 at 1:19 PM, Guido van Rossum <guido at python.org> wrote:
>> > On Tue, Nov 10, 2009 at 8:50 AM, geremy condra <debatem1 at gmail.com> wrote:
>> >> If you're so worried about the warnings, suppress them. You control
>> >> the code.
>> >
>> > No. The nuisance of warnings is that they are presented to the
>> > *user*, not the developer, and the user (in many cases) doesn't
>> > control the code.
>> >
>> > --
>> > --Guido van Rossum (python.org/~guido)
>> >
>>
>> In this case, however, he does, as per his comment:
>
> Antoine (?he?, in your sentence) does. *The user does not*, and it is
> the user who is seeing the warnings:

<snip>

...and so he (Antoine) should suppress the warnings if he is
worried about the user seeing them. Unless he's operating
on an incredibly slow release schedule his users will see his
patch a *long* time before they'll see any change in their
default version of python's behavior.

Geremy Condra

From guido at python.org  Wed Nov 11 03:58:42 2009
From: guido at python.org (Guido van Rossum)
Date: Tue, 10 Nov 2009 18:58:42 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <f3cc57c60911101537o77cdd797n46d06e04dde6de26@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<f3cc57c60911091514i43f67896q489adca0585bc460@mail.gmail.com> 
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com> 
	<nad-722C39.07193910112009@news.gmane.org>
	<f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com> 
	<ca471dc20911100800o2f4e2f05kd30f6591c2e24455@mail.gmail.com> 
	<f3cc57c60911100849m300ddf67t8c13d928fdf311d9@mail.gmail.com> 
	<9B120D89-CC9B-4A3E-AC46-68D1F8C3F4FA@gmail.com>
	<f3cc57c60911101537o77cdd797n46d06e04dde6de26@mail.gmail.com>
Message-ID: <ca471dc20911101858x6c545700x52f64b83833ca88f@mail.gmail.com>

On Tue, Nov 10, 2009 at 3:37 PM, geremy condra <debatem1 at gmail.com> wrote:
> If its an absolute no-no that warnings be raised then blanket
> suppress them.

Read the subject line again. It's about deprecation warnings
specifically -- not all warnings.

This is my last post on the subject, I don't think I can be any
clearer than I already have been.

-- 
--Guido van Rossum (python.org/~guido)

From debatem1 at gmail.com  Wed Nov 11 04:10:00 2009
From: debatem1 at gmail.com (geremy condra)
Date: Tue, 10 Nov 2009 22:10:00 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and hownoisy
	they are)
In-Reply-To: <ca471dc20911101858x6c545700x52f64b83833ca88f@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<nad-966ADB.15340609112009@news.gmane.org>
	<f3cc57c60911100650q551ca79em59d7c47ff61aa5a6@mail.gmail.com>
	<nad-722C39.07193910112009@news.gmane.org>
	<f3cc57c60911100731n6f5eceffk5b2e92e0848a9328@mail.gmail.com>
	<ca471dc20911100800o2f4e2f05kd30f6591c2e24455@mail.gmail.com>
	<f3cc57c60911100849m300ddf67t8c13d928fdf311d9@mail.gmail.com>
	<9B120D89-CC9B-4A3E-AC46-68D1F8C3F4FA@gmail.com>
	<f3cc57c60911101537o77cdd797n46d06e04dde6de26@mail.gmail.com>
	<ca471dc20911101858x6c545700x52f64b83833ca88f@mail.gmail.com>
Message-ID: <f3cc57c60911101910x65717433j9f13fa73923b608d@mail.gmail.com>

On Tue, Nov 10, 2009 at 9:58 PM, Guido van Rossum <guido at python.org> wrote:
> On Tue, Nov 10, 2009 at 3:37 PM, geremy condra <debatem1 at gmail.com> wrote:
>> If its an absolute no-no that warnings be raised then blanket
>> suppress them.
>
> Read the subject line again. It's about deprecation warnings
> specifically -- not all warnings.
>
> This is my last post on the subject, I don't think I can be any
> clearer than I already have been.
>
> --
> --Guido van Rossum (python.org/~guido)
>

Then blanket suppress the deprecation warnings:

import warnings
warnings.simplefilter('ignore', DeprecationWarning)

Geremy Condra

From g.brandl at gmx.net  Wed Nov 11 10:01:29 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 11 Nov 2009 10:01:29 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <9d153b7c0911091352g5b708e78le3aabcc5fddc7ea2@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<9d153b7c0911091352g5b708e78le3aabcc5fddc7ea2@mail.gmail.com>
Message-ID: <hdduh9$617$1@ger.gmane.org>

Yuvgoog Greenle schrieb:
> On Mon, Nov 9, 2009 at 11:18 PM, Brett Cannon <brett at python.org> wrote:
>> First, there is the app user perspective. If I use an application I do
>> not need to see any warnings, ever.
> 
> So it comes down to either:
> 
> A. As things are - every ballsy app developer has to have this piece
> of code laying around:
>     import warnings
>     warnings.filterwarnings("ignore")
> 
> or
> 
> B. we render warnings useless in our every day python lives except for
> package buildbots that remember to run with -w (or whatever).

On the contrary, I think that DeprecationWarnings will become much more
useful when silenced by default, because people will use them *more*.
>From experience, when thinking about emitting DeprecationWarnings in Sphinx,
which is a command-line tool, I often don't bother, because *it still works
anyway* and I don't want Sphinx users to see a screenful of warnings when
just building their docs.

If DeprecationWarnings were silenced by default, I (and probably others too)
would put them into the code more liberally, because I know that *when* they
are displayed, they are displayed to someone who actually *wants* to fix the
code that causes them.

Of course I could use the "filterwarnings" incantation given above, but that
defeats the purpose of the -W command line option, and *globally* silences
warnings I don't even want to control.

> I would like a demographic on this, but I'm sure that either way
> muting warnings will be a devastating blow to the spread of
> information about what's new/old in python.

Please stop trying to seed FUD.

Most Python developers are flexible enough to get to know and use a new and
useful tool when they are made aware of it, and using -w is a very simple
tool.

Georg


From debatem1 at gmail.com  Wed Nov 11 16:59:42 2009
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 11 Nov 2009 10:59:42 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <hdduh9$617$1@ger.gmane.org>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<9d153b7c0911091352g5b708e78le3aabcc5fddc7ea2@mail.gmail.com>
	<hdduh9$617$1@ger.gmane.org>
Message-ID: <f3cc57c60911110759o66b5dd77t3a6ac5e1e58997a6@mail.gmail.com>

On Wed, Nov 11, 2009 at 4:01 AM, Georg Brandl <g.brandl at gmx.net> wrote:
> Yuvgoog Greenle schrieb:
>> On Mon, Nov 9, 2009 at 11:18 PM, Brett Cannon <brett at python.org> wrote:
>>> First, there is the app user perspective. If I use an application I do
>>> not need to see any warnings, ever.
>>
>> So it comes down to either:
>>
>> A. As things are - every ballsy app developer has to have this piece
>> of code laying around:
>> ? ? import warnings
>> ? ? warnings.filterwarnings("ignore")
>>
>> or
>>
>> B. we render warnings useless in our every day python lives except for
>> package buildbots that remember to run with -w (or whatever).
>
> On the contrary, I think that DeprecationWarnings will become much more
> useful when silenced by default, because people will use them *more*.
> >From experience, when thinking about emitting DeprecationWarnings in Sphinx,
> which is a command-line tool, I often don't bother, because *it still works
> anyway* and I don't want Sphinx users to see a screenful of warnings when
> just building their docs.

If you don't feel like your users need that information, don't give it to them.
Same thing goes for both sphinx and python in general.

> If DeprecationWarnings were silenced by default, I (and probably others too)
> would put them into the code more liberally, because I know that *when* they
> are displayed, they are displayed to someone who actually *wants* to fix the
> code that causes them.
>
> Of course I could use the "filterwarnings" incantation given above, but that
> defeats the purpose of the -W command line option, and *globally* silences
> warnings I don't even want to control.

Please read the documentation, you can filter what and how you
want.

>> I would like a demographic on this, but I'm sure that either way
>> muting warnings will be a devastating blow to the spread of
>> information about what's new/old in python.
>
> Please stop trying to seed FUD.
>
> Most Python developers are flexible enough to get to know and use a new and
> useful tool when they are made aware of it, and using -w is a very simple
> tool.

But also an unnecessary one. Everything you've described can
be done simply and without changing python's behavior. There's
just no need to make the change.

Geremy Condra

From g.brandl at gmx.net  Wed Nov 11 23:51:12 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 11 Nov 2009 23:51:12 +0100
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <f3cc57c60911110759o66b5dd77t3a6ac5e1e58997a6@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>	<9d153b7c0911091352g5b708e78le3aabcc5fddc7ea2@mail.gmail.com>	<hdduh9$617$1@ger.gmane.org>
	<f3cc57c60911110759o66b5dd77t3a6ac5e1e58997a6@mail.gmail.com>
Message-ID: <hdff54$qsk$1@ger.gmane.org>

geremy condra schrieb:

>> If DeprecationWarnings were silenced by default, I (and probably others too)
>> would put them into the code more liberally, because I know that *when* they
>> are displayed, they are displayed to someone who actually *wants* to fix the
>> code that causes them.
>>
>> Of course I could use the "filterwarnings" incantation given above, but that
>> defeats the purpose of the -W command line option, and *globally* silences
>> warnings I don't even want to control.
> 
> Please read the documentation, you can filter what and how you
> want.

I don't read the documentation.  I write it.

Of course I can filter out only deprecation warnings occurring in a sphinx.*
module.  But then the extension author who wants to see them has to change my
code so that he can see the warnings again.  That's not easier than running
"python -w", is it?

And that's the problem with the warnings being displayed by default: it gives
me extra work to figure out what to filter and to add that filter to all my
programs, and it gives the user of a program or library extra work to figure
out where the warnings are filtered, and to remove that code.

Even worse, if no warnings are emitted, but would be emitted by default, people
probably wouldn't even look for a warnings filter but assume the code is OK.
With warnings silenced but activated with a simple -w flag, they *know* they
are silent and they can use -w to show them.  Much simpler, much more
consistent.

>>> I would like a demographic on this, but I'm sure that either way
>>> muting warnings will be a devastating blow to the spread of
>>> information about what's new/old in python.
>>
>> Please stop trying to seed FUD.
>>
>> Most Python developers are flexible enough to get to know and use a new and
>> useful tool when they are made aware of it, and using -w is a very simple
>> tool.
> 
> But also an unnecessary one. Everything you've described can
> be done simply and without changing python's behavior. There's
> just no need to make the change.

There's never a "need" to make any change.  Why introduce "with"?  A simple
try-finally is all you need.  Etc.  We make these changes because we think
that they benefit our users, make life a bit easier and coding a bit more
fun.  Seeing deprecation warnings every time I run hg was no fun at all.

Georg


From debatem1 at gmail.com  Thu Nov 12 04:44:33 2009
From: debatem1 at gmail.com (geremy condra)
Date: Wed, 11 Nov 2009 22:44:33 -0500
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <hdff54$qsk$1@ger.gmane.org>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<ca471dc20911081340t701c0202tf43d9bcad76bad0c@mail.gmail.com>
	<bbaeab100911091318i10d7b9efp2ea2bbc158382b72@mail.gmail.com>
	<9d153b7c0911091352g5b708e78le3aabcc5fddc7ea2@mail.gmail.com>
	<hdduh9$617$1@ger.gmane.org>
	<f3cc57c60911110759o66b5dd77t3a6ac5e1e58997a6@mail.gmail.com>
	<hdff54$qsk$1@ger.gmane.org>
Message-ID: <f3cc57c60911111944u5dc30debl5850fd9bfa89f08f@mail.gmail.com>

On Wed, Nov 11, 2009 at 5:51 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> geremy condra schrieb:
>
>>> If DeprecationWarnings were silenced by default, I (and probably others too)
>>> would put them into the code more liberally, because I know that *when* they
>>> are displayed, they are displayed to someone who actually *wants* to fix the
>>> code that causes them.
>>>
>>> Of course I could use the "filterwarnings" incantation given above, but that
>>> defeats the purpose of the -W command line option, and *globally* silences
>>> warnings I don't even want to control.
>>
>> Please read the documentation, you can filter what and how you
>> want.
>
> I don't read the documentation. ?I write it.

I'm well aware of your contributions to Python- I simply assumed that since
you were insinuating something wasn't possible when the docs made a
special point of mentioning that it was, you weren't aware that the
capability existed. No offense was intended.

> Of course I can filter out only deprecation warnings occurring in a sphinx.*
> module. ?But then the extension author who wants to see them has to change my
> code so that he can see the warnings again. ?That's not easier than running
> "python -w", is it?

I'd just extend sphinx's -w flag so that if it isn't given a filename it would
write to stdout. It also seems likely to me that that change would land
on your users machines much sooner than any change to python would,
although you would obviously know more about that than I would.

> And that's the problem with the warnings being displayed by default: it gives
> me extra work to figure out what to filter and to add that filter to all my
> programs, and it gives the user of a program or library extra work to figure
> out where the warnings are filtered, and to remove that code.

IMHO, thats the idea behind Python emitting warnings at all- that one of
the unspoken contracts between the program and Python has been broken
in a way that, while not immediately disastrous, will nevertheless
eventually end in tears. It is then the developer's responsibility to make
sure that the problem has been appropriately resolved, and to write
the appropriate (explicit, hopefully narrowly targeted) code to suppress
the warning in the event that it *is* superfluous.

I'd also like to point out that if you get the same passel of superfluous
warnings every time that you run under -w that you do normally, the
arguments raised earlier about error ennui become doubly relevant,
since now you now have the option of making them 'go play over
there' rather than cluttering up your screen.

> Even worse, if no warnings are emitted, but would be emitted by default, people
> probably wouldn't even look for a warnings filter but assume the code is OK.
> With warnings silenced but activated with a simple -w flag, they *know* they
> are silent and they can use -w to show them. ?Much simpler, much more
> consistent.

This is the most convincing thing I've heard on this discussion so far.
Its a little bit of a consolation prize (sure, we broke your expectations
in a way that left you unaware that your code would blow up until it did,
but at least we did it consistently!) but assuming that every other option
were taken off the table I could understand this being considered a
virtue.

>>>> I would like a demographic on this, but I'm sure that either way
>>>> muting warnings will be a devastating blow to the spread of
>>>> information about what's new/old in python.
>>>
>>> Please stop trying to seed FUD.
>>>
>>> Most Python developers are flexible enough to get to know and use a new and
>>> useful tool when they are made aware of it, and using -w is a very simple
>>> tool.
>>
>> But also an unnecessary one. Everything you've described can
>> be done simply and without changing python's behavior. There's
>> just no need to make the change.
>
> There's never a "need" to make any change. ?Why introduce "with"? ?A simple
> try-finally is all you need. ?Etc. ?We make these changes because we think
> that they benefit our users, make life a bit easier and coding a bit more
> fun. ?Seeing deprecation warnings every time I run hg was no fun at all.

You're right, of course- 'need' is an impossibly high standard to meet
in a project as mature as python. The point I was trying to make was
that this proposal alters two of the (IMO) most important contracts
between developers and python (invocation and error notification),
that the bar for such a change should be set pretty high, and that I
don't think that things that can be easily accomplished inside of
the standard library should be seen as meeting that standard. Again,
YMMV.

Geremy Condra

From brian at sweetapp.com  Fri Nov 13 03:38:33 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 13 Nov 2009 13:38:33 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com>
	<5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com>
	<F516D09C-3615-4B95-8B86-144B5E1DFD8C@sweetapp.com>
	<216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com>
Message-ID: <C4525BFA-E6E5-4129-8B93-BCE57D59ED69@sweetapp.com>

Hey all,

I compiled a summary of people's feedback (about technical issues - I  
agree that the docs could be better but agreeing on the API seems like  
the first step) and have some API change proposals.

Here is a summary of the feedback:
- Use Twisted Deferreds rather than Futures
- The API too complex
- Make Future a callable and drop the .result()/.exception() methods
- Remove .wait() from Executor
- Make it easy to process results in the order of completion rather  
than in the order that the futures were generated
- Executor context managers should wait until their workers complete  
before exiting
- Extract Executor.map, etc. into separate functions/modules
- FutureList has too many methods or is not necessary
- Executor should have an easy way to produce a single future
- Should be able to wait on an arbitrary list of futures
- Should have a way of avoiding deadlock (will follow-up on this  
separately)

Here is what I suggest as far as API changes (the docs suck, I'll  
polish them when we reach consensus):

FutureList is eliminated completely.

Future remains unchanged - I disagree that Deferreds would be better,  
that .exception() is not useful, and that .result() should be  
renamed .get() or .__call__(). But I am easily persuadable :-)

The Executor ABC is simplified to only contain a single method:

def Executor.submit(self, fn, *args, **kwargs) :

Submits a call for execution and returns a Future representing the  
pending results of fn(*args, **kwargs)

map becomes a utility function:

def map(executor, *iterables, timeout=None)

Equivalent to map(func, *iterables) but executed asynchronously and  
possibly out-of-order. The returned iterator raises a TimeoutError if  
__next__() is called and the result isn?t available after timeout  
seconds from the original call to run_to_results(). If timeout is not  
specified or None then there is no limit to the wait time. If a call  
raises an exception then that exception will be raised when its value  
is retrieved from the iterator.

wait becomes a utility function that can wait on any iterable of  
Futures:

def wait(futures, return_when=ALL_COMPLETED)

Wait until the given condition is met for the given futures. This  
method should always be called using keyword arguments, which are:

timeout can be used to control the maximum number of seconds to wait  
before returning. If timeout is not specified or None then there is no  
limit to the wait time.

return_when indicates when the method should return. It must be one of  
the following constants:

     NEXT_COMPLETED
     NEXT_EXCEPTION
     ALL_COMPLETED

a new utility function is added that iterates over the given Futures  
and returns the as they are completed:

def itercompleted(futures, timeout=None):

Returns an iterator that returns a completed Future from the given  
list when __next__() is called. If no Futures are completed then  
__next__() is called then __next__() waits until one does complete.  
Raises a TimeoutError if __next__() is called and no completed future  
is available after timeout seconds from the original call.

The URL loading example becomes:

import functools
import urllib.request
import futures

URLS = ['http://www.foxnews.com/',
         'http://www.cnn.com/',
         'http://europe.wsj.com/',
         'http://www.bbc.co.uk/',
         'http://some-made-up-domain.com/']

def load_url(url, timeout):
     return urllib.request.urlopen(url, timeout=timeout).read()

with futures.ThreadPoolExecutor(50) as executor:
   fs = [executor.submit(load_url, url, timeout=30) for url in URLS]

for future in futures.itercompleted(fs):
     if future.exception() is not None:
         print('%r generated an exception: %s' % (url,  
future.exception()))
     else:
         print('%r page is %d bytes' % (url, len(future.result())))

What do you think? Are we moving in the right direction?

Cheers,
Brian


From jyasskin at gmail.com  Fri Nov 13 06:19:05 2009
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Thu, 12 Nov 2009 21:19:05 -0800
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <C4525BFA-E6E5-4129-8B93-BCE57D59ED69@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com>
	<5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com>
	<F516D09C-3615-4B95-8B86-144B5E1DFD8C@sweetapp.com>
	<216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com>
	<C4525BFA-E6E5-4129-8B93-BCE57D59ED69@sweetapp.com>
Message-ID: <5d44f72f0911122119l1f2fd875va7dcbb22a46f9c7f@mail.gmail.com>

I am very happy with those changes.

I think deadlock should be addressed before the first release as it
changes the detailed semantics of some of the operations, but you've
promised to do that, so cool. :)

I think it's fine to leave the embedding of Deferred-like things into
futures and the embedding of futures into Deferred-like things until a
later release. I expect it to be requested, but I don't think it'll be
hard to add later.

On Thu, Nov 12, 2009 at 6:38 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hey all,
>
> I compiled a summary of people's feedback (about technical issues - I agree
> that the docs could be better but agreeing on the API seems like the first
> step) and have some API change proposals.
>
> Here is a summary of the feedback:
> - Use Twisted Deferreds rather than Futures
> - The API too complex
> - Make Future a callable and drop the .result()/.exception() methods
> - Remove .wait() from Executor
> - Make it easy to process results in the order of completion rather than in
> the order that the futures were generated
> - Executor context managers should wait until their workers complete before
> exiting
> - Extract Executor.map, etc. into separate functions/modules
> - FutureList has too many methods or is not necessary
> - Executor should have an easy way to produce a single future
> - Should be able to wait on an arbitrary list of futures
> - Should have a way of avoiding deadlock (will follow-up on this separately)
>
> Here is what I suggest as far as API changes (the docs suck, I'll polish
> them when we reach consensus):
>
> FutureList is eliminated completely.
>
> Future remains unchanged - I disagree that Deferreds would be better, that
> .exception() is not useful, and that .result() should be renamed .get() or
> .__call__(). But I am easily persuadable :-)
>
> The Executor ABC is simplified to only contain a single method:
>
> def Executor.submit(self, fn, *args, **kwargs) :
>
> Submits a call for execution and returns a Future representing the pending
> results of fn(*args, **kwargs)
>
> map becomes a utility function:
>
> def map(executor, *iterables, timeout=None)
>
> Equivalent to map(func, *iterables) but executed asynchronously and possibly
> out-of-order. The returned iterator raises a TimeoutError if __next__() is
> called and the result isn?t available after timeout seconds from the
> original call to run_to_results(). If timeout is not specified or None then
> there is no limit to the wait time. If a call raises an exception then that
> exception will be raised when its value is retrieved from the iterator.
>
> wait becomes a utility function that can wait on any iterable of Futures:
>
> def wait(futures, return_when=ALL_COMPLETED)
>
> Wait until the given condition is met for the given futures. This method
> should always be called using keyword arguments, which are:
>
> timeout can be used to control the maximum number of seconds to wait before
> returning. If timeout is not specified or None then there is no limit to the
> wait time.
>
> return_when indicates when the method should return. It must be one of the
> following constants:
>
> ? ?NEXT_COMPLETED
> ? ?NEXT_EXCEPTION
> ? ?ALL_COMPLETED
>
> a new utility function is added that iterates over the given Futures and
> returns the as they are completed:
>
> def itercompleted(futures, timeout=None):
>
> Returns an iterator that returns a completed Future from the given list when
> __next__() is called. If no Futures are completed then __next__() is called
> then __next__() waits until one does complete. Raises a TimeoutError if
> __next__() is called and no completed future is available after timeout
> seconds from the original call.
>
> The URL loading example becomes:
>
> import functools
> import urllib.request
> import futures
>
> URLS = ['http://www.foxnews.com/',
> ? ? ? ?'http://www.cnn.com/',
> ? ? ? ?'http://europe.wsj.com/',
> ? ? ? ?'http://www.bbc.co.uk/',
> ? ? ? ?'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
> ? ?return urllib.request.urlopen(url, timeout=timeout).read()
>
> with futures.ThreadPoolExecutor(50) as executor:
> ?fs = [executor.submit(load_url, url, timeout=30) for url in URLS]
>
> for future in futures.itercompleted(fs):
> ? ?if future.exception() is not None:
> ? ? ? ?print('%r generated an exception: %s' % (url, future.exception()))
> ? ?else:
> ? ? ? ?print('%r page is %d bytes' % (url, len(future.result())))
>
> What do you think? Are we moving in the right direction?
>
> Cheers,
> Brian
>
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>



-- 
Namast?,
Jeffrey Yasskin
http://jeffrey.yasskin.info/

From brian at sweetapp.com  Fri Nov 13 06:19:40 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 13 Nov 2009 16:19:40 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
Message-ID: <21CB1702-5680-4C43-B1FB-BA438C4D7E98@sweetapp.com>

On Nov 8, 2009, at 6:37 AM, Jeffrey Yasskin wrote:
> --- More general points ---
>
> ** Java's Futures made a mistake in not supporting work stealing, and
> this has caused deadlocks at Google. Specifically, in a bounded-size
> thread or process pool, when a task in the pool can wait for work
> running in the same pool, you can fill up the pool with tasks that are
> waiting for tasks that haven't started running yet. To avoid this,
> Future.get() should be able to steal the task it's waiting on out of
> the pool's queue and run it immediately.

Hey Jeff,

I understand the deadlock possibilities of the executor model, could  
you explain your proposal would work?

Would it be some sort of flag on the Future.get method e.g.  
Future.get(timeout=None, immediate_execution=False)?

Cheers,
Brian

From jyasskin at gmail.com  Fri Nov 13 06:27:31 2009
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Thu, 12 Nov 2009 21:27:31 -0800
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <21CB1702-5680-4C43-B1FB-BA438C4D7E98@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<21CB1702-5680-4C43-B1FB-BA438C4D7E98@sweetapp.com>
Message-ID: <5d44f72f0911122127v5876cad8i73e5bf25d0ddc5e1@mail.gmail.com>

On Thu, Nov 12, 2009 at 9:19 PM, Brian Quinlan <brian at sweetapp.com> wrote:
> On Nov 8, 2009, at 6:37 AM, Jeffrey Yasskin wrote:
>>
>> --- More general points ---
>>
>> ** Java's Futures made a mistake in not supporting work stealing, and
>> this has caused deadlocks at Google. Specifically, in a bounded-size
>> thread or process pool, when a task in the pool can wait for work
>> running in the same pool, you can fill up the pool with tasks that are
>> waiting for tasks that haven't started running yet. To avoid this,
>> Future.get() should be able to steal the task it's waiting on out of
>> the pool's queue and run it immediately.
>
> Hey Jeff,
>
> I understand the deadlock possibilities of the executor model, could you
> explain your proposal would work?
>
> Would it be some sort of flag on the Future.get method e.g.
> Future.get(timeout=None, immediate_execution=False)?

I don't think a flag is the way to go at first glance, although there
could be upsides I haven't thought of. Here's what I had in mind:

After I call "fut = executor.submit(task)", the task can be in 3
states: queued, running, and finished. The simplest deadlock happens
in a 1-thread pool when the running thread calls fut.result(), and the
task is queued on the same pool. So instead of just waiting for the
task to finish running, the current thread atomically(checks what
state it's in, and if it's queued, marks it as stolen instead) and
calls it in the current thread. When a stolen task gets to the front
of its queue and starts running, it just acts like a no-op.

This can't introduce any new lock-order deadlocks, but it can be
observable if the task looks at thread-local variables.

From brian at sweetapp.com  Fri Nov 13 07:13:18 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 13 Nov 2009 17:13:18 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <5d44f72f0911122127v5876cad8i73e5bf25d0ddc5e1@mail.gmail.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<21CB1702-5680-4C43-B1FB-BA438C4D7E98@sweetapp.com>
	<5d44f72f0911122127v5876cad8i73e5bf25d0ddc5e1@mail.gmail.com>
Message-ID: <5226789A-EDEF-4A9A-84AD-D2E83438BF61@sweetapp.com>


On Nov 13, 2009, at 4:27 PM, Jeffrey Yasskin wrote:

> On Thu, Nov 12, 2009 at 9:19 PM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
>> On Nov 8, 2009, at 6:37 AM, Jeffrey Yasskin wrote:
>>>
>>> --- More general points ---
>>>
>>> ** Java's Futures made a mistake in not supporting work stealing,  
>>> and
>>> this has caused deadlocks at Google. Specifically, in a bounded-size
>>> thread or process pool, when a task in the pool can wait for work
>>> running in the same pool, you can fill up the pool with tasks that  
>>> are
>>> waiting for tasks that haven't started running yet. To avoid this,
>>> Future.get() should be able to steal the task it's waiting on out of
>>> the pool's queue and run it immediately.
>>
>> Hey Jeff,
>>
>> I understand the deadlock possibilities of the executor model,  
>> could you
>> explain your proposal would work?
>>
>> Would it be some sort of flag on the Future.get method e.g.
>> Future.get(timeout=None, immediate_execution=False)?
>
> I don't think a flag is the way to go at first glance, although there
> could be upsides I haven't thought of. Here's what I had in mind:
>
> After I call "fut = executor.submit(task)", the task can be in 3
> states: queued, running, and finished. The simplest deadlock happens
> in a 1-thread pool when the running thread calls fut.result(), and the
> task is queued on the same pool. So instead of just waiting for the
> task to finish running, the current thread atomically(checks what
> state it's in, and if it's queued, marks it as stolen instead) and
> calls it in the current thread. When a stolen task gets to the front
> of its queue and starts running, it just acts like a no-op.
>
> This can't introduce any new lock-order deadlocks, but it can be
> observable if the task looks at thread-local variables.

So you have something like this:

def Future.result(self, timeout=None):
   with some_lock:  # would have to think about locking here
     do_work_locally =  (threading.current_thread in  
self._my_executor.threads and
         self._my_executor.free_threads == 0 and
         timeout is None):

That's pretty clever. Some things that I don't like:
1. it might only be applicable to executors using a thread pool so  
people
     shouldn't count on it (but maybe only thread pool executors have  
this
     deadlock problem so it doesn't matter?)
2. it makes the implementation of Future dependent on the executor that
      created it - but maybe that's OK too, Future can be an ABC and
      executor implementations that need customer Futures can subclass  
it

Cheers,
Brian

From jyasskin at gmail.com  Fri Nov 13 08:09:59 2009
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Thu, 12 Nov 2009 23:09:59 -0800
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <5226789A-EDEF-4A9A-84AD-D2E83438BF61@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<21CB1702-5680-4C43-B1FB-BA438C4D7E98@sweetapp.com>
	<5d44f72f0911122127v5876cad8i73e5bf25d0ddc5e1@mail.gmail.com>
	<5226789A-EDEF-4A9A-84AD-D2E83438BF61@sweetapp.com>
Message-ID: <5d44f72f0911122309q17fb31b0k7770cae24ab3433f@mail.gmail.com>

On Thu, Nov 12, 2009 at 10:13 PM, Brian Quinlan <brian at sweetapp.com> wrote:
>
> On Nov 13, 2009, at 4:27 PM, Jeffrey Yasskin wrote:
>
>> On Thu, Nov 12, 2009 at 9:19 PM, Brian Quinlan <brian at sweetapp.com> wrote:
>>>
>>> On Nov 8, 2009, at 6:37 AM, Jeffrey Yasskin wrote:
>>>>
>>>> --- More general points ---
>>>>
>>>> ** Java's Futures made a mistake in not supporting work stealing, and
>>>> this has caused deadlocks at Google. Specifically, in a bounded-size
>>>> thread or process pool, when a task in the pool can wait for work
>>>> running in the same pool, you can fill up the pool with tasks that are
>>>> waiting for tasks that haven't started running yet. To avoid this,
>>>> Future.get() should be able to steal the task it's waiting on out of
>>>> the pool's queue and run it immediately.
>>>
>>> Hey Jeff,
>>>
>>> I understand the deadlock possibilities of the executor model, could you
>>> explain your proposal would work?
>>>
>>> Would it be some sort of flag on the Future.get method e.g.
>>> Future.get(timeout=None, immediate_execution=False)?
>>
>> I don't think a flag is the way to go at first glance, although there
>> could be upsides I haven't thought of. Here's what I had in mind:
>>
>> After I call "fut = executor.submit(task)", the task can be in 3
>> states: queued, running, and finished. The simplest deadlock happens
>> in a 1-thread pool when the running thread calls fut.result(), and the
>> task is queued on the same pool. So instead of just waiting for the
>> task to finish running, the current thread atomically(checks what
>> state it's in, and if it's queued, marks it as stolen instead) and
>> calls it in the current thread. When a stolen task gets to the front
>> of its queue and starts running, it just acts like a no-op.
>>
>> This can't introduce any new lock-order deadlocks, but it can be
>> observable if the task looks at thread-local variables.
>
> So you have something like this:
>
> def Future.result(self, timeout=None):
> ?with some_lock: ?# would have to think about locking here
> ? ?do_work_locally = ?(threading.current_thread in self._my_executor.threads
> and
> ? ? ? ?self._my_executor.free_threads == 0 and
> ? ? ? ?timeout is None):

You can deadlock from a cycle between multiple pools, too, so it's
probably a bad idea to limit it to only steal if self is one of the
pool's threads, and there's no real reason to limit the stealing to
when there are exactly 0 waiting threads. Depending on the internal
implementation, Future.result() might look something like (untested,
sorry if there are obvious bugs):

class Future:
  def __init__(self, f, args, kwargs):
    self.f, self.args, self.kwargs = f, args, kwargs
    self.state, self.lock = QUEUED, Executor.Lock()

  def run(self):
    with self.lock:
      if self.state != QUEUED: return
      self.state = RUNNING
    self._result = self.f(*self.args, **self.kwargs)
    with self.lock:
      self.state = DONE
      self.notify()

  def result(self, timeout=None):
    if timeout is None:  # Good catch.
      self.run()
    self.wait(timeout)
    return self._result

> That's pretty clever. Some things that I don't like:
> 1. it might only be applicable to executors using a thread pool so people
> ? ?shouldn't count on it (but maybe only thread pool executors have this
> ? ?deadlock problem so it doesn't matter?)

Process pools have the same deadlock problem, unless it's impossible
for a task in a process pool to hold a reference to the same pool, or
another pool whose tasks have a reference to the first one?

> 2. it makes the implementation of Future dependent on the executor that
> ? ? created it - but maybe that's OK too, Future can be an ABC and
> ? ? executor implementations that need customer Futures can subclass it

It makes some piece dependent on the executor, although not
necessarily the whole Future. For example, the run() method above
could be wrapped into a Task class that only knows how to mark work
stolen and run stuff locally.

From solipsis at pitrou.net  Fri Nov 13 15:22:18 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 13 Nov 2009 15:22:18 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <C4525BFA-E6E5-4129-8B93-BCE57D59ED69@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com>
	<5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com>
	<F516D09C-3615-4B95-8B86-144B5E1DFD8C@sweetapp.com>
	<216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com>
	<C4525BFA-E6E5-4129-8B93-BCE57D59ED69@sweetapp.com>
Message-ID: <1258122138.3574.23.camel@localhost>


Hey,

> Future remains unchanged - I disagree that Deferreds would be better,  
> that .exception() is not useful, and that .result() should be  
> renamed .get() or .__call__().

On what grounds do you disagree with the latter?

Another question: is the caught exception an attribute of the future? If
so, is there any mechanism to clean it up (and its traceback) once the
future has been "consumed"?

> map becomes a utility function:
> 
> def map(executor, *iterables, timeout=None)

Why? map() can be defined on the ABC, so that subclasses don't have to
provide their own implementation.

An utility function which looks like a method and shadows the name of a
built-in looks like a bad choice to me.

> wait becomes a utility function that can wait on any iterable of  
> Futures:
> 
> def wait(futures, return_when=ALL_COMPLETED)

Does it work if the futures are executed by different executors?
If not, it should be an Executor method.

> return_when indicates when the method should return. It must be one of  
> the following constants:
> 
>      NEXT_COMPLETED
>      NEXT_EXCEPTION
>      ALL_COMPLETED

Can you outline the difference between NEXT_COMPLETED and
NEXT_EXCEPTION? What happens if I ask for NEXT_COMPLETED but the next
future to complete raises an exception? etc.

> def itercompleted(futures, timeout=None):
> 
> Returns an iterator that returns a completed Future from the given  
> list when __next__() is called. If no Futures are completed then  
> __next__() is called then __next__() waits until one does complete.

What about futures which complete with an exception?

> with futures.ThreadPoolExecutor(50) as executor:
>    fs = [executor.submit(load_url, url, timeout=30) for url in URLS]

The use of "with" here still is counter-intuitive, because it does not
clean up resources immediately as it would seem to do. "with" is always
synchronous in other situations.

> What do you think? Are we moving in the right direction?

Perhaps, yes, but there are still lots of dark areas.

Besides, it's obvious that the package has to mature, and should be
tested by other people.



From brett at python.org  Sat Nov 14 01:23:23 2009
From: brett at python.org (Brett Cannon)
Date: Fri, 13 Nov 2009 16:23:23 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
Message-ID: <bbaeab100911131623j3599833sb36fff28d7c2298d@mail.gmail.com>

I asked Guido and he has made his call: DeprecationWarning will be
silent by default. I have started http://bugs.python.org/issue7319 to
track the work. As of right now it has a patch against trunk to add
the silencing to the warnings filter by default. Still need to come up
with the proper doc changes before it can be committed on trunk and
py3k (probably the tutorial and warnings module; anywhere else?).

On Sun, Nov 8, 2009 at 13:26, Brett Cannon <brett at python.org> wrote:
> During the moratorium PEP discussions Guido said he wanted to quiet
> down deprecation warnings. I see there being two options on this.
>
> One is to keep things as is, but to require two releases with
> PendingDeprecationWarning so there are three years of
> silent-by-default warnings to update your code. But that last release
> before removal came would still be noisy.
>
> The other option is to simply have all warnings filtered out by
> default. We could alter -W so that when it is used w/o an argument it
> turns to what is currently the default behaviour (or even turn all
> warnings which is more than what happens now). This will require that
> people proactively check for warnings when updating for compatibility,
> else they will eventually use a Python release where there code will
> simply break because something changed. This route means we do not
> have to specify any deprecation policy right now (that would be a
> separate discussion).
>
> Channeling Guido he is after the latter, but a general discussion
> would still be good since he didn't explicitly say what he was after
> other than to quiet down warnings.
>
> -Brett
>

From ubershmekel at gmail.com  Sat Nov 14 12:38:42 2009
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Sat, 14 Nov 2009 13:38:42 +0200
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <bbaeab100911131623j3599833sb36fff28d7c2298d@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com>
	<bbaeab100911131623j3599833sb36fff28d7c2298d@mail.gmail.com>
Message-ID: <9d153b7c0911140338o1a9a3ecfy3dc55f034fa3ebb7@mail.gmail.com>

On Sat, Nov 14, 2009 at 2:23 AM, Brett Cannon <brett at python.org> wrote:
> py3k (probably the tutorial and warnings module; anywhere else?).

Maybe all the "what's new" that talk about porting to py3k.

--yuv

From brett at python.org  Sat Nov 14 20:15:36 2009
From: brett at python.org (Brett Cannon)
Date: Sat, 14 Nov 2009 11:15:36 -0800
Subject: [stdlib-sig] standardizing the deprecation policy (and how
	noisy they are)
In-Reply-To: <9d153b7c0911140338o1a9a3ecfy3dc55f034fa3ebb7@mail.gmail.com>
References: <bbaeab100911081326v56df86abl3a3b1ee5e11df24d@mail.gmail.com> 
	<bbaeab100911131623j3599833sb36fff28d7c2298d@mail.gmail.com> 
	<9d153b7c0911140338o1a9a3ecfy3dc55f034fa3ebb7@mail.gmail.com>
Message-ID: <bbaeab100911141115s3ad65743p362273566ace1ec1@mail.gmail.com>

On Sat, Nov 14, 2009 at 03:38, Yuval Greenfield <ubershmekel at gmail.com> wrote:
> On Sat, Nov 14, 2009 at 2:23 AM, Brett Cannon <brett at python.org> wrote:
>> py3k (probably the tutorial and warnings module; anywhere else?).
>
> Maybe all the "what's new" that talk about porting to py3k.

Well, this will be in 2.7 so it won't require specific mention in
porting. But it will be covered in "What's New" for 2.7 and 3.2.

-Brett

From brian at sweetapp.com  Sat Nov 28 03:18:37 2009
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 28 Nov 2009 13:18:37 +1100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <1258122138.3574.23.camel@localhost>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com>
	<5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com>
	<F516D09C-3615-4B95-8B86-144B5E1DFD8C@sweetapp.com>
	<216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com>
	<C4525BFA-E6E5-4129-8B93-BCE57D59ED69@sweetapp.com>
	<1258122138.3574.23.camel@localhost>
Message-ID: <4BEFEDD0-6B42-4B81-876C-BB05839CB601@sweetapp.com>

Hey Antoine,

Sorry for not getting back to you sooner - I actually thought that I  
did reply but I see now that I didn't.

On 14 Nov 2009, at 01:22, Antoine Pitrou wrote:

>
> Hey,
>
>> Future remains unchanged - I disagree that Deferreds would be better,
>> that .exception() is not useful, and that .result() should be
>> renamed .get() or .__call__().
>
> On what grounds do you disagree with the latter?

It feels hacky. Getting the result doesn't feel so special that it  
deserves to be a call rather than a simple getter.

> Another question: is the caught exception an attribute of the future?

Yes

> If
> so, is there any mechanism to clean it up (and its traceback) once the
> future has been "consumed"?

No there isn't. That's a good point though. I wonder if futures will  
tend to long-lived after there results are available?

>
>> map becomes a utility function:
>>
>> def map(executor, *iterables, timeout=None)
>
> Why? map() can be defined on the ABC, so that subclasses don't have to
> provide their own implementation.
>
> An utility function which looks like a method and shadows the name  
> of a
> built-in looks like a bad choice to me.

Good point.

>> wait becomes a utility function that can wait on any iterable of
>> Futures:
>>
>> def wait(futures, return_when=ALL_COMPLETED)
>
> Does it work if the futures are executed by different executors?
> If not, it should be an Executor method.

Yes, it goes.

>
>> return_when indicates when the method should return. It must be one  
>> of
>> the following constants:
>>
>>     NEXT_COMPLETED
>>     NEXT_EXCEPTION
>>     ALL_COMPLETED
>
> Can you outline the difference between NEXT_COMPLETED and
> NEXT_EXCEPTION? What happens if I ask for NEXT_COMPLETED but the next
> future to complete raises an exception? etc.

NEXT_COMPLETED includes futures that raise. Completed in this sense  
means "done running".

>
>> def itercompleted(futures, timeout=None):
>>
>> Returns an iterator that returns a completed Future from the given
>> list when __next__() is called. If no Futures are completed then
>> __next__() is called then __next__() waits until one does complete.
>
> What about futures which complete with an exception?

They are included.


>> with futures.ThreadPoolExecutor(50) as executor:
>>   fs = [executor.submit(load_url, url, timeout=30) for url in URLS]
>
> The use of "with" here still is counter-intuitive, because it does not
> clean up resources immediately as it would seem to do. "with" is  
> always
> synchronous in other situations.

Maybe waiting until all pending futures are done executing would be  
better.

>
>> What do you think? Are we moving in the right direction?
>
> Perhaps, yes, but there are still lots of dark areas.
>
> Besides, it's obvious that the package has to mature, and should be
> tested by other people.


It would be great if other people tested the API. I'm not sure what  
you mean by "mature" though.

Cheers,
Brian


From solipsis at pitrou.net  Sun Nov 29 11:59:05 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 29 Nov 2009 11:59:05 +0100
Subject: [stdlib-sig] futures - a new package for asynchronous execution
In-Reply-To: <4BEFEDD0-6B42-4B81-876C-BB05839CB601@sweetapp.com>
References: <E145302B-9F8D-4FF9-A4FD-FD2710E03F65@sweetapp.com>
	<bbaeab100911061712id912b7el986721bd65f2705c@mail.gmail.com>
	<4222a8490911061820j772ac247o536972153762a3e4@mail.gmail.com>
	<1257605583.3437.3.camel@localhost>
	<4222a8490911070707n6e71b3c9xdeb2218c879b128c@mail.gmail.com>
	<1257607273.3437.13.camel@localhost>
	<4222a8490911070732p5b5cdc6cj2a5d44416658e119@mail.gmail.com>
	<5d44f72f0911071137m3a499f99j9edc604bc8b9b127@mail.gmail.com>
	<37E25344-20F9-4D42-B982-CA43D24FA806@sweetapp.com>
	<5d44f72f0911080001off0d158n4c0c4d903a844516@mail.gmail.com>
	<F516D09C-3615-4B95-8B86-144B5E1DFD8C@sweetapp.com>
	<216378C8-6B77-4DAC-9292-841A8E5849B5@sweetapp.com>
	<C4525BFA-E6E5-4129-8B93-BCE57D59ED69@sweetapp.com>
	<1258122138.3574.23.camel@localhost>
	<4BEFEDD0-6B42-4B81-876C-BB05839CB601@sweetapp.com>
Message-ID: <1259492345.3376.14.camel@localhost>


> It feels hacky. Getting the result doesn't feel so special that it  
> deserves to be a call rather than a simple getter.

Well, it is special since the whole point of a future is to get that
result. Like the whole point of a weakref is to get the underlying
object.
Of course this is pretty much bikeshedding...

> It would be great if other people tested the API. I'm not sure what  
> you mean by "mature" though.

What I mean is that it would be nice if it got reviewed, tested and
criticized by actual users. I have not looked at the implementation
though.

> No there isn't. That's a good point though. I wonder if futures will  
> tend to long-lived after there results are available?

It's hard to tell without anyone actually using them, but for advanced
uses I suspect that futures may become more or less long-lived objects
(like Deferreds are :-)).
In a Web spider example, you could have a bunch of futures representing
pending or completed HTTP fetches, and a worker thread processing the
results on the fly when each of those futures gets ready. If the
processing is non-trivial (or if it involves say a busy database) the
worker thread could get quite a bit behind the completion of HTTP
requests.

Twisted has a whole machinery for that in its specialized "Failure"
class, so as to keep the traceback information in string representation
and at the same time relinquish all references to the frames involved in
the traceback. I'm not sure we need the same degree of sophistication
but we should keep in mind that it's a potential problem.

(actually, perhaps this would deserve built-in support in the
interpreter)

> > The use of "with" here still is counter-intuitive, because it does
> not
> > clean up resources immediately as it would seem to do. "with" is  
> > always
> > synchronous in other situations.
> 
> Maybe waiting until all pending futures are done executing would be  
> better.

I think it would be better indeed. At least it would be more in line
with the other uses of context managers.

Regards

Antoine.