From greg.ewing at canterbury.ac.nz  Mon Mar  1 00:22:19 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 01 Mar 2010 12:22:19 +1300
Subject: [Python-Dev] __file__
In-Reply-To: <4B89F5FA.2010604@g.nevcal.com>
References: <20100130190005.058c8187@freewill.wooz.org>
	<bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<ca471dc21002261429x5674ea95vc52e0255185a254a@mail.gmail.com>
	<bbaeab101002261455wf89eenf2796bd3d6cc8ddb@mail.gmail.com>
	<4B885AB4.7040004@g.nevcal.com>
	<bbaeab101002261713h5fd3e9f4hab56807e0831c266@mail.gmail.com>
	<4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz>
	<4B89F5FA.2010604@g.nevcal.com>
Message-ID: <4B8AFAAB.5020106@canterbury.ac.nz>

Glenn Linderman wrote:
> if the 
> command line/runpy can do it, the importer could do it.  Just a matter 
> of desire and coding.  Whether it is worth pursuing further depends on 
> people's perceptions of "kookiness" vs. functional and performance 
> considerations.

Having .py files around that aren't source text could
lead to a lot of confusion, given that most platforms
these days decide which application to open for a given
file based solely on the filename extension. I wouldn't
enjoy trying to open a .py file only to have my text
editor blow up because it was actually a binary file.

So on balance I think it's a bit too kooky for my
taste.

-- 
Greg

From floris.bruynooghe at gmail.com  Mon Mar  1 00:14:41 2010
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sun, 28 Feb 2010 23:14:41 +0000
Subject: [Python-Dev] __file__
In-Reply-To: <hmekmc$1p4$1@dough.gmane.org>
References: <4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com> <hmekmc$1p4$1@dough.gmane.org>
Message-ID: <20100228231441.GA18979@laurie.devork>

On Sun, Feb 28, 2010 at 09:45:56PM +0100, Baptiste Carvello wrote:
> However, making a difference between zipimport and the filesystem
> importer means the application will stop working if I unzip the
> library zip file, which is surprising. Unzipping the zip file can be
> handy when debugging a bug caused by a forgotten module.

That difference exists already, the zipimporter will happily run .pyo
files inside the zipfile even when you're not running with -O or
PYTHONOPTIMIZE.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From greg.ewing at canterbury.ac.nz  Mon Mar  1 00:35:17 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 01 Mar 2010 12:35:17 +1300
Subject: [Python-Dev] __file__
In-Reply-To: <20100228121907.GA13564@laurie.devork>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
Message-ID: <4B8AFDB5.8040302@canterbury.ac.nz>

Floris Bruynooghe wrote:

> Can't it look for a .py file in the source directory first (1st stat)?
> When it's there check for the .pyc in the cache directory (2nd stat,
> magic number encoded in filename), if it's not check for .pyc in the
> source directory (2nd stat + read for magic number check).

Yes, although that would then incur higher stat overheads for
people distributing .pyc files. There doesn't seem to be a
way of pleasing everyone.

This is all assuming that the extra stat calls are actually
a problem. Does anyone have any evidence that they would
really take significant time compared to loading the module?
Once you've looked for one file in a given directory, looking
for another one in the same directory ought to be quite fast,
since all the relevant directory blocks will be in the
filesystem cache.

-- 
Greg

From florent.xicluna at gmail.com  Mon Mar  1 00:36:27 2010
From: florent.xicluna at gmail.com (Florent XICLUNA)
Date: Mon, 1 Mar 2010 00:36:27 +0100
Subject: [Python-Dev] Update xml.etree.ElementTree for Python 2.7 and 3.2
In-Reply-To: <hmdid5$eqc$1@dough.gmane.org>
References: <loom.20100218T101639-70@post.gmane.org>
	<4B7DB541.4000604@v.loewis.de> <hll0l9$tg3$1@ger.gmane.org>
	<4B7E2430.2070804@v.loewis.de>
	<fb73205e1002182336p621f5dft96fdae095dd68c23@mail.gmail.com>
	<4B7F95FE.20703@v.loewis.de> <loom.20100220T120554-850@post.gmane.org>
	<4B7FD0C7.4050207@v.loewis.de> <hmdid5$eqc$1@dough.gmane.org>
Message-ID: <e43810331002281536i19089b54ka789339cf07393d2@mail.gmail.com>

2010/2/28 Stefan Behnel <stefan_ml at behnel.de>

> I would actually encourage Florent to do the opposite: act now and prepare
> a patch against the latest official ET 1.2 and cET releases (or their SVN
> version respectively) that integrates everything that is considered safe,
> i.e. everything that makes cET compatible with ET and everything that seems
> clearly stable in ET 1.3 and does not break compatibility for existing code
> that uses ET 1.2. If you send that to Fredrik, I expect little opposition
> to making that the base for a 1.2.8 release, which can then be folded back
> into the stdlib.
>
>
I exchanged some e-mails with Fredrik last week. Not sure if it will be
1.2.8 or 1.3, but now he is positive on the goals of the patch. I've
commited all the changes and external fixes to a branch of the Mercurial
repo owned by Fredrik. I'm expecting an answer soon.

Branch based on the official etree repository (Mercurial):
http://bitbucket.org/flox/et-2009-provolone/

Patch based on this branch:
http://codereview.appspot.com/207048
(patch set 7 almost identical to the tip of the Mercurial repo)

-- 
Florent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/1bcb1c49/attachment.html>

From robertc at robertcollins.net  Mon Mar  1 00:56:20 2010
From: robertc at robertcollins.net (Robert Collins)
Date: Mon, 01 Mar 2010 10:56:20 +1100
Subject: [Python-Dev] __file__
In-Reply-To: <4B8AFDB5.8040302@canterbury.ac.nz>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<4B8AFDB5.8040302@canterbury.ac.nz>
Message-ID: <1267401380.25053.116.camel@lifeless-64>

On Mon, 2010-03-01 at 12:35 +1300, Greg Ewing wrote:
> 
> Yes, although that would then incur higher stat overheads for
> people distributing .pyc files. There doesn't seem to be a
> way of pleasing everyone.
> 
> This is all assuming that the extra stat calls are actually
> a problem. Does anyone have any evidence that they would
> really take significant time compared to loading the module?
> Once you've looked for one file in a given directory, looking
> for another one in the same directory ought to be quite fast,
> since all the relevant directory blocks will be in the
> filesystem cache. 

We've done a bunch of testing in bzrlib. Basic things are:
 - statting /is/ expensive *if* you don't use the result.
 - loading code is the main cost *once* you have a hot disk cache

Specifically, stats for files that are *not present* incur page-in costs
for the dentries needed to determine the file is absent. In the special
case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit
the same pages and don't incur additional page in costs. (you'll hit the
same page in most file systems when you look for the second and third
entries).

In most file systems stats for files that *are present* also incur a
page-in for the inode of the file. If you then do not read the file,
this is I/O that doesn't really gain anything. 

Being able to disable .py file usage completely - so that only foo.pyc
and foo/__init__.pyc are probed for, could have a very noticable change
in the cold cache startup time.

# Startup time for bzr (cold cache):
$ drop-caches
$ time bzr --no-plugins revno
5061

real    0m8.875s
user    0m0.210s
sys     0m0.140s

# Hot cache
$ time bzr --no-plugins revno
5061

real    0m0.307s
user    0m0.250s
sys     0m0.040s


(revno is a small command that reads a small amount of data - just
enough to trigger demand loading of the core repository layers and so
on).

strace timings for those two operations:
cold cache:
$ strace -c bzr --no-plugins revno
5061
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 56.34    0.040000          76       527           read
 28.98    0.020573           9      2273      1905 open
 14.43    0.010248          14       734       625 stat
  0.15    0.000107           0       533           fstat
...

hot cache:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 45.10    0.000368          92         4           getdents
 19.49    0.000159           0       527           read
 16.91    0.000138           1       163           munmap
 10.05    0.000082           2        54           mprotect
  8.46    0.000069           0      2273      1905 open
  0.00    0.000000           0         8           write
  0.00    0.000000           0       367           close
  0.00    0.000000           0       734       625 stat
...

Cheers,
Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/22f9c569/attachment.pgp>

From greg.ewing at canterbury.ac.nz  Mon Mar  1 01:31:17 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 01 Mar 2010 13:31:17 +1300
Subject: [Python-Dev] __file__
In-Reply-To: <1267401380.25053.116.camel@lifeless-64>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<4B8AFDB5.8040302@canterbury.ac.nz>
	<1267401380.25053.116.camel@lifeless-64>
Message-ID: <4B8B0AD5.4090606@canterbury.ac.nz>

Robert Collins wrote:
> In the special
> case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit
> the same pages and don't incur additional page in costs.

So then looking for a .pyc alongside a .py or vice versa
should be almost free, and we shouldn't be worrying about
it.

> hot cache:
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  45.10    0.000368          92         4           getdents
>   0.00    0.000000           0       734       625 stat

Further supporting the idea that stat calls are negligible
once the cache is warmed up.

-- 
Greg

From pmaupin at gmail.com  Mon Mar  1 01:25:39 2010
From: pmaupin at gmail.com (Patrick Maupin)
Date: Sun, 28 Feb 2010 18:25:39 -0600
Subject: [Python-Dev] Draft PEP on RSON configuration file format
Message-ID: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>

All:

Finding .ini configuration files too limiting, JSON and XML to hard to
manually edit, and YAML too complex to parse quickly, I have started
work on a new configuration file parser.

I call the new format RSON (for "Readable Serial Object Notation"),
and it is designed to be a superset of JSON.

I would love for it to be considered valuable enough to be a part of
the standard library, but even if that does not come to pass, I would
be very interested in feedback to help me polish the specification,
and then possibly help for implementation and testing.

The documentation is in rst PEP form, at:

http://rson.googlecode.com/svn/trunk/doc/draftpep.txt

Thanks and best regards,
Pat

From benjamin at python.org  Mon Mar  1 01:29:48 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 28 Feb 2010 18:29:48 -0600
Subject: [Python-Dev] Draft PEP on RSON configuration file format
In-Reply-To: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
References: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
Message-ID: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com>

2010/2/28 Patrick Maupin <pmaupin at gmail.com>:
> All:
>
> Finding .ini configuration files too limiting, JSON and XML to hard to
> manually edit, and YAML too complex to parse quickly, I have started
> work on a new configuration file parser.

In that case, it should live in the user space for several years. If
the community decides that it is an excellent format, then it should
be considered for inclusion in the stand library.



-- 
Regards,
Benjamin

From pmaupin at gmail.com  Mon Mar  1 01:59:16 2010
From: pmaupin at gmail.com (Patrick Maupin)
Date: Sun, 28 Feb 2010 18:59:16 -0600
Subject: [Python-Dev] Draft PEP on RSON configuration file format
In-Reply-To: <1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com>
References: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
	<1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com>
Message-ID: <d09829f51002281659r241206cckf5bab84afea64b8@mail.gmail.com>

On Sun, Feb 28, 2010 at 6:29 PM, Benjamin Peterson <benjamin at python.org> wrote:
> In that case, it should live in the user space for several years. If
> the community decides that it is an excellent format, then it should
> be considered for inclusion in the stand library.

Agreed.

However, there are too many things which became de facto standards
without community input this way.  PEP 1 itself says:

Reference Implementation -- The reference implementation must be
completed before any PEP is given status "Final", but it need not be
completed before the PEP is accepted. It is better to finish the
specification and rationale first and reach consensus on it before
writing code.

So, I do not mind the code sitting outside the standard library, and
the PEP not reaching "Final" for several years, but I do believe that
the PEP process is itself a really good way to build a better
mousetrap by consensus.

If you do not care to participate in the building of this particular
mousetrap, that is OK, too.

Regards,
Pat

From solipsis at pitrou.net  Mon Mar  1 02:39:45 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 28 Feb 2010 20:39:45 -0500
Subject: [Python-Dev] Draft PEP on RSON configuration file format
References: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
	<1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com>
	<d09829f51002281659r241206cckf5bab84afea64b8@mail.gmail.com>
Message-ID: <20100228203945.62e1ea4b@msiwind>

Le Sun, 28 Feb 2010 18:59:16 -0600,
Patrick Maupin <pmaupin at gmail.com> a ?crit :
> 
> So, I do not mind the code sitting outside the standard library, and
> the PEP not reaching "Final" for several years, but I do believe that
> the PEP process is itself a really good way to build a better
> mousetrap by consensus.

In this case it is *at best* python-ideas material, or even
preferably comp.lang.python.

Just for the record, my only reaction when giving the PEP a glance was
"yet another configuration file format - yawn".

Good luck though,

Antoine.



From pmaupin at gmail.com  Mon Mar  1 02:46:30 2010
From: pmaupin at gmail.com (Patrick Maupin)
Date: Sun, 28 Feb 2010 19:46:30 -0600
Subject: [Python-Dev] Draft PEP on RSON configuration file format
In-Reply-To: <20100228203945.62e1ea4b@msiwind>
References: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
	<1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com>
	<d09829f51002281659r241206cckf5bab84afea64b8@mail.gmail.com>
	<20100228203945.62e1ea4b@msiwind>
Message-ID: <d09829f51002281746i7e35dffexe9a1db72b754bc79@mail.gmail.com>

On Sun, Feb 28, 2010 at 7:39 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> In this case it is *at best* python-ideas material, or even
> preferably comp.lang.python.

I was thinking about comp.lang.python at some point, but thought I
would try here first.

> Just for the record, my only reaction when giving the PEP a glance was
> "yet another configuration file format - yawn".

I suppose I have that sort of reaction about areas I am not interested
in, as well, but currently I am deeply interested in configuration
files due to my circumstances.  In any case, the observation that
there are already several preexisting file formats used for
configuration is certainly covered in the PEP draft, but if you have
anything constructive to add *about* configuration file formats, I
would certainly welcome the input.

Best regards,
Pat

From solipsis at pitrou.net  Mon Mar  1 02:51:20 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 28 Feb 2010 20:51:20 -0500
Subject: [Python-Dev] Draft PEP on RSON configuration file format
References: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
	<1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com>
	<d09829f51002281659r241206cckf5bab84afea64b8@mail.gmail.com>
	<20100228203945.62e1ea4b@msiwind>
	<d09829f51002281746i7e35dffexe9a1db72b754bc79@mail.gmail.com>
Message-ID: <20100228205120.6ac82f01@msiwind>

Le Sun, 28 Feb 2010 19:46:30 -0600,
Patrick Maupin <pmaupin at gmail.com> a ?crit :
> 
> I suppose I have that sort of reaction about areas I am not interested
> in, as well, but currently I am deeply interested in configuration
> files due to my circumstances.  In any case, the observation that
> there are already several preexisting file formats used for
> configuration is certainly covered in the PEP draft, but if you have
> anything constructive to add *about* configuration file formats, I
> would certainly welcome the input.

Well, a constructive approach would involve approaching projects
which have devised their own formats, so as to know what kind of
unified format they would be likely to accept (or not).

python-dev is probably not the place for such an approach, however.



From brett at python.org  Mon Mar  1 03:11:39 2010
From: brett at python.org (Brett Cannon)
Date: Sun, 28 Feb 2010 18:11:39 -0800
Subject: [Python-Dev] __file__
In-Reply-To: <4B8B0AD5.4090606@canterbury.ac.nz>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com> 
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com> 
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org> 
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz> 
	<20100228121907.GA13564@laurie.devork>
	<4B8AFDB5.8040302@canterbury.ac.nz> 
	<1267401380.25053.116.camel@lifeless-64>
	<4B8B0AD5.4090606@canterbury.ac.nz>
Message-ID: <bbaeab101002281811m6311b5far8f386b0155c921e6@mail.gmail.com>

On Sun, Feb 28, 2010 at 16:31, Greg Ewing <greg.ewing at canterbury.ac.nz>wrote:

> Robert Collins wrote:
>
>> In the special
>> case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit
>> the same pages and don't incur additional page in costs.
>>
>
> So then looking for a .pyc alongside a .py or vice versa
> should be almost free, and we shouldn't be worrying about
> it.
>

But that is making the assumption that all filesystems operate this way
(.e.g does NFS have the same performance characteristics?).


>
>  hot cache:
>> % time     seconds  usecs/call     calls    errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>>  45.10    0.000368          92         4           getdents
>>  0.00    0.000000           0       734       625 stat
>>
>
> Further supporting the idea that stat calls are negligible
> once the cache is warmed up.


But that's the point: once it's warmed up. This is not the case when
executing a script once every once in a while compared to something bzr
where you are most likely going to execute the command multiple times within
a small timeframe.

-Brett




>
>
> --
> Greg
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100228/f658ea33/attachment-0001.html>

From brett at python.org  Mon Mar  1 03:27:04 2010
From: brett at python.org (Brett Cannon)
Date: Sun, 28 Feb 2010 18:27:04 -0800
Subject: [Python-Dev] __file__
In-Reply-To: <4B8AD627.60103@gmail.com>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com> 
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org> 
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz> 
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> 
	<4B8A6A8F.7050008@gmail.com>
	<bbaeab101002281221g608db2bem798361786ba39f24@mail.gmail.com> 
	<4B8AD627.60103@gmail.com>
Message-ID: <bbaeab101002281827pbf73ecg8a7e5d30726aa093@mail.gmail.com>

On Sun, Feb 28, 2010 at 12:46, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Brett Cannon wrote:
> > Actually it's four: name/__init__.py, name/__init__.pyc, name.py, and
> > then name.pyc. And just so people have terminology to go with all of
> > this, this search is what the finder does to say whether it can or
> > cannot handle the requested module.
>
> Huh, I thought we checked for the directory first and only then checked
> for the __init__ module within it (hence the generation of ImportWarning
> when we don't find __init__ after finding a correctly named directory).
> So a normal miss (i.e. no directory) only needs one stat call.
>
> (However, I'll grant that I haven't looked at this particular chunk of
> code in a fairly long time, so I could easily be wrong).
>
> Robert raises a good point about the checks for extension modules as
> well - we should get an accurate count here so Barry's PEP can pitch the
> proportional reduction in stat calls accurately.
>

Here are the details (from Python/import.c:find_module) assuming that
everything has failed to the point of trying for the implicit sys.path
importers:

stat_info = stat(name)
if stat_info.exists and stat_info.is_dir:
  if stat(name/__init__.py) || stat(name/__init__.pyc):
     load(name)
else:
  for ext in ('.so', 'module.so', '.py', 'pyc'):  # Windows has an extra
check for .pyw files.
     if open(name + ext):
        load(name)

So there are a total of five to six depending on the OS (actually, VMS goes
up to eight!) before a search path is considered not to contain a module.

And thanks to doing this I realized importlib is not stat'ing the directory
first which should fail faster than checking for the __init__ files every
time.

-Brett





>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100228/7c538a8f/attachment.html>

From brett at python.org  Mon Mar  1 03:29:42 2010
From: brett at python.org (Brett Cannon)
Date: Sun, 28 Feb 2010 18:29:42 -0800
Subject: [Python-Dev] __file__
In-Reply-To: <hmekmc$1p4$1@dough.gmane.org>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com> 
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com> 
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org> 
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz> 
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> 
	<4B8A6A8F.7050008@gmail.com> <hmekmc$1p4$1@dough.gmane.org>
Message-ID: <bbaeab101002281829v5d25f0f5ra722dbd98128ed25@mail.gmail.com>

On Sun, Feb 28, 2010 at 12:45, Baptiste Carvello <baptiste13z at free.fr>wrote:

> Nick Coghlan a ?crit :
>
>
>> Another option is to remove bytecode-only support from the default
>> filesystem importer, but keep it for zipimport (since the stat call
>> savings don't apply in the latter case).
>>
>>
> bytecode-only in a zip is used by py2exe, cx_freeze and the like, for space
> reasons. Disabling it would probably hurt them.
>
> However, making a difference between zipimport and the filesystem importer
> means the application will stop working if I unzip the library zip file,
> which is surprising. Unzipping the zip file can be handy when debugging a
> bug caused by a forgotten module.
>
>
Is it really that hard to unzip a bunch of .pyc files, modify what you need
to, and then zip it back up? And if you are given a zip file of only .pyc
files you can't really debug anything anyway.

-Brett




> Cheers,
> Baptiste
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100228/369232a3/attachment.html>

From brett at python.org  Mon Mar  1 03:31:54 2010
From: brett at python.org (Brett Cannon)
Date: Sun, 28 Feb 2010 18:31:54 -0800
Subject: [Python-Dev] The fate of Distutils in Python 2.7
In-Reply-To: <94bdd2611002261415p5342b6dbv6a4a87ba0f4c7700@mail.gmail.com>
References: <94bdd2611002261344q361e702av10fdf99ea5eb9f71@mail.gmail.com> 
	<bbaeab101002261413g7d72ff7br84bc6b59d8f22245@mail.gmail.com> 
	<94bdd2611002261415p5342b6dbv6a4a87ba0f4c7700@mail.gmail.com>
Message-ID: <bbaeab101002281831o43b79989j7710441f8a3919d2@mail.gmail.com>

On Fri, Feb 26, 2010 at 14:15, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> On Fri, Feb 26, 2010 at 11:13 PM, Brett Cannon <brett at python.org> wrote:
> [..]
> > I assume you want the Distutils2 component to auto-assign to you like
> > Distutils currently does? If so I can add the component for you if people
> > don't object to the new component.
>
> Sounds good -- Thanks
>

Done.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100228/07e761ed/attachment.html>

From pmaupin at gmail.com  Mon Mar  1 03:33:05 2010
From: pmaupin at gmail.com (Patrick Maupin)
Date: Sun, 28 Feb 2010 20:33:05 -0600
Subject: [Python-Dev] Draft PEP on RSON configuration file format
In-Reply-To: <20100228205120.6ac82f01@msiwind>
References: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
	<1afaf6161002281629m5b270b40j51ae966cecb09f4e@mail.gmail.com>
	<d09829f51002281659r241206cckf5bab84afea64b8@mail.gmail.com>
	<20100228203945.62e1ea4b@msiwind>
	<d09829f51002281746i7e35dffexe9a1db72b754bc79@mail.gmail.com>
	<20100228205120.6ac82f01@msiwind>
Message-ID: <d09829f51002281833t433c759aqb18acd517176b1ac@mail.gmail.com>

On Sun, Feb 28, 2010 at 7:51 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Well, a constructive approach would involve approaching projects
> which have devised their own formats, so as to know what kind of
> unified format they would be likely to accept (or not).

Trying to poll "selected projects which have configuration files" may
or may not be a constructive approach.  Most projects which have
predefined formats are unlikely to change, unless there is
standardization on a new format.  It is very much a chicken and egg
problem, although I agree with (and have implemented) the suggestion
that I discuss this on python-list.

Having said that, one of the reasons I wrote the PEP and am working on
a parser is because of a few projects I use and/or am personally
involved in.  For example, rst2pdf stylesheets are in JSON, e.g.

http://rst2pdf.googlecode.com/svn/trunk/rst2pdf/styles/styles.json

Now, we're all programmers here, and we can read this, and can even
modify it, but it is easy to get wrong, and very verbose with lots of
syntax gotchas.  For example, unlike Python, JSON won't even let you
have a trailing comma.

But JSON *is* a great format, and RSON (like YAML) is designed to
parse properly formatted JSON, so the goal is that any project which
uses JSON could use RSON as a drop-in replacement, and then update its
configuration data.

Of course, it is extremely easy (hence your yawn) to create a new
configuration format, even if it is specified that it is upwards
compatible with JSON.  The trick is to create the *correct* new
format, that at least some people can agree on.

In order to do this, I have chosen to poll, not preexisting projects,
which have entrenched configuration data and a reluctance to change,
but brand new projects which haven't been invented yet.  Many of the
inventors of those projects hang out on python-dev, so this seemed
like a reasonable place to do polling.

As I tried to make clear, I will not be too disappointed if I do not
come up with something worthy of the standard library for a long time
(if ever), but the PEP process is very valuable, and I would like to
start off on the right foot by soliciting feedback before I do too
much coding.

Sorry if it feels like spam; this is my last message on the matter
until and unless somebody wants to constructively discuss the actual
contents of the PEP.  Please feel free to email me privately if you
don't want to clutter up this list.

Thanks and best regards,
Pat

From v+python at g.nevcal.com  Mon Mar  1 04:32:09 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Sun, 28 Feb 2010 19:32:09 -0800
Subject: [Python-Dev] __file__
In-Reply-To: <4B8AFAAB.5020106@canterbury.ac.nz>
References: <20100130190005.058c8187@freewill.wooz.org>	<bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>	<20100225160832.6e7a3063@freewill.wooz.org>	<4B870E2A.8090406@canterbury.ac.nz>
	<4B870CD5.8020105@voidspace.org.uk>	<4B871227.2030707@canterbury.ac.nz>	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>	<ca471dc21002261429x5674ea95vc52e0255185a254a@mail.gmail.com>	<bbaeab101002261455wf89eenf2796bd3d6cc8ddb@mail.gmail.com>	<4B885AB4.7040004@g.nevcal.com>	<bbaeab101002261713h5fd3e9f4hab56807e0831c266@mail.gmail.com>	<4B889AB1.4050402@g.nevcal.com>
	<4B89C5FB.40104@canterbury.ac.nz>	<4B89F5FA.2010604@g.nevcal.com>
	<4B8AFAAB.5020106@canterbury.ac.nz>
Message-ID: <4B8B3539.5060709@g.nevcal.com>

On approximately 2/28/2010 3:22 PM, came the following characters from 
the keyboard of Greg Ewing:
> Glenn Linderman wrote:
>> if the command line/runpy can do it, the importer could do it.  Just 
>> a matter of desire and coding.  Whether it is worth pursuing further 
>> depends on people's perceptions of "kookiness" vs. functional and 
>> performance considerations.
>
> Having .py files around that aren't source text could
> lead to a lot of confusion, given that most platforms
> these days decide which application to open for a given
> file based solely on the filename extension. I wouldn't
> enjoy trying to open a .py file only to have my text
> editor blow up because it was actually a binary file.
>
> So on balance I think it's a bit too kooky for my
> taste.

I understand your thoughts, but have some rebuttal comments.  Mind you, 
if there is a better solution that can improve performance for both the 
source+binary and the binary-only distributions, I'm all for it.  But in 
general, I'm all for performance improvements, even if there is some 
kookiness :)  Thankful for Brett's posting of the actual search code 
fragment.

If your text editor blows up because it is binary, it is a sad text editor.

If you have .py mapped to a text editor, that's sort of kooky too; I 
have it mapped to Python.

The .py files that are binary would generally be part of an application 
distribution in binary form, and therefore would be installed in some 
place like /bin or C:\Program Files ... not the place you'd look for 
source code, to confuse your text editor.

-- 
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking


From solipsis at pitrou.net  Mon Mar  1 04:40:20 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 28 Feb 2010 22:40:20 -0500
Subject: [Python-Dev] __file__
References: <20100130190005.058c8187@freewill.wooz.org>
	<bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz>
	<4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<ca471dc21002261429x5674ea95vc52e0255185a254a@mail.gmail.com>
	<bbaeab101002261455wf89eenf2796bd3d6cc8ddb@mail.gmail.com>
	<4B885AB4.7040004@g.nevcal.com>
	<bbaeab101002261713h5fd3e9f4hab56807e0831c266@mail.gmail.com>
	<4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz>
	<4B89F5FA.2010604@g.nevcal.com> <4B8AFAAB.5020106@canterbury.ac.nz>
	<4B8B3539.5060709@g.nevcal.com>
Message-ID: <20100228224020.46da4e23@msiwind>

Le Sun, 28 Feb 2010 19:32:09 -0800,
Glenn Linderman <v+python at g.nevcal.com> a ?crit :
> 
> If your text editor blows up because it is binary, it is a sad text
> editor.
> 
> If you have .py mapped to a text editor, that's sort of kooky too; I 
> have it mapped to Python.

File extensions exist for a reason, even if you find that "kooky" and
have strong ideas about the psychology of text editors.

Having some binary files named "foobar.py" would certainly annoy a lot
of people, including me.


Antoine.



From greg.ewing at canterbury.ac.nz  Mon Mar  1 06:21:52 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 01 Mar 2010 18:21:52 +1300
Subject: [Python-Dev] __file__
In-Reply-To: <4B8B3539.5060709@g.nevcal.com>
References: <20100130190005.058c8187@freewill.wooz.org>
	<bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<ca471dc21002261429x5674ea95vc52e0255185a254a@mail.gmail.com>
	<bbaeab101002261455wf89eenf2796bd3d6cc8ddb@mail.gmail.com>
	<4B885AB4.7040004@g.nevcal.com>
	<bbaeab101002261713h5fd3e9f4hab56807e0831c266@mail.gmail.com>
	<4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz>
	<4B89F5FA.2010604@g.nevcal.com> <4B8AFAAB.5020106@canterbury.ac.nz>
	<4B8B3539.5060709@g.nevcal.com>
Message-ID: <4B8B4EF0.5020204@canterbury.ac.nz>

Glenn Linderman wrote:

> If your text editor blows up because it is binary, it is a sad text editor.

Blow up is probably an exaggeration, but even just getting
a screen full of gibberish when I think I'm opening a
text file is a jarring experience.

> If you have .py mapped to a text editor, that's sort of kooky too; I 
> have it mapped to Python.

On Windows the action for double-clicking is usually
mapped to running the file, but there's typically another
action such as "Open with IDLE" or whatever available,
and a bytecode file named with ".py" would allow you to
apply that action to it.

-- 
Greg

From larry.bugbee at boeing.com  Mon Mar  1 07:09:00 2010
From: larry.bugbee at boeing.com (Bugbee, Larry)
Date: Sun, 28 Feb 2010 22:09:00 -0800
Subject: [Python-Dev] __file__
In-Reply-To: <mailman.112.1267402861.23597.python-dev@python.org>
References: <mailman.112.1267402861.23597.python-dev@python.org>
Message-ID: <261064E43172924BBADB00DA14BB4B672A18782E13@XCH-NW-14V.nw.nos.boeing.com>

Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Having .py files around that aren't source text could lead 
> to a lot of confusion, given that most platforms these days 
> decide which application to open for a given file based 
> solely on the filename extension. I wouldn't enjoy trying 
> to open a .py file only to have my text editor blow up 
> because it was actually a binary file.
> 
> So on balance I think it's a bit too kooky for my taste.

+1

Add to that the inverse...  I will cleanup directories based on the suffix keeping the .py and deleting .pyc and .pyo.  Overloading a source file suffix is not good.

Larry

From baptiste13z at free.fr  Mon Mar  1 08:25:08 2010
From: baptiste13z at free.fr (Baptiste Carvello)
Date: Mon, 01 Mar 2010 08:25:08 +0100
Subject: [Python-Dev] __file__
In-Reply-To: <bbaeab101002281829v5d25f0f5ra722dbd98128ed25@mail.gmail.com>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com> <hmekmc$1p4$1@dough.gmane.org>
	<bbaeab101002281829v5d25f0f5ra722dbd98128ed25@mail.gmail.com>
Message-ID: <hmfq4t$6c3$1@dough.gmane.org>

Brett Cannon a ?crit :
> 
>     However, making a difference between zipimport and the filesystem
>     importer means the application will stop working if I unzip the
>     library zip file, which is surprising. Unzipping the zip file can be
>     handy when debugging a bug caused by a forgotten module.
> 
> 
> Is it really that hard to unzip a bunch of .pyc files, modify what you 
> need to, and then zip it back up? And if you are given a zip file of 
> only .pyc files you can't really debug anything anyway.
> 

Well, this is a micro-use-case, I admit, I only mention it because it's 
something I've really done. It's only useful for debugging the building process, 
not the application (so I do have the source at hand), and the only reason for 
not rezipping is to test more quickly. I can definitely live without it!

Cheers, Baptiste


From stefan_ml at behnel.de  Mon Mar  1 08:32:02 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Mar 2010 08:32:02 +0100
Subject: [Python-Dev] Update xml.etree.ElementTree for Python 2.7 and 3.2
In-Reply-To: <e43810331002281536i19089b54ka789339cf07393d2@mail.gmail.com>
References: <loom.20100218T101639-70@post.gmane.org>	<4B7DB541.4000604@v.loewis.de>
	<hll0l9$tg3$1@ger.gmane.org>	<4B7E2430.2070804@v.loewis.de>	<fb73205e1002182336p621f5dft96fdae095dd68c23@mail.gmail.com>	<4B7F95FE.20703@v.loewis.de>
	<loom.20100220T120554-850@post.gmane.org>	<4B7FD0C7.4050207@v.loewis.de>
	<hmdid5$eqc$1@dough.gmane.org>
	<e43810331002281536i19089b54ka789339cf07393d2@mail.gmail.com>
Message-ID: <hmfqhj$7br$1@dough.gmane.org>

Florent XICLUNA, 01.03.2010 00:36:
> I exchanged some e-mails with Fredrik last week. Not sure if it will be
> 1.2.8 or 1.3, but now he is positive on the goals of the patch. I've
> commited all the changes and external fixes to a branch of the Mercurial
> repo owned by Fredrik. I'm expecting an answer soon.

Happy to hear that. Thanks for putting so much work into this!


> Branch based on the official etree repository (Mercurial):
> http://bitbucket.org/flox/et-2009-provolone/

Interesting, I didn't even know Fredrik had continued to work on this. It
even looks like lxml.etree has a bit to catch up API-wise before I release 2.3.

Stefan


From robertc at robertcollins.net  Mon Mar  1 09:04:59 2010
From: robertc at robertcollins.net (Robert Collins)
Date: Mon, 01 Mar 2010 19:04:59 +1100
Subject: [Python-Dev] __file__
In-Reply-To: <bbaeab101002281811m6311b5far8f386b0155c921e6@mail.gmail.com>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<4B8AFDB5.8040302@canterbury.ac.nz>
	<1267401380.25053.116.camel@lifeless-64>
	<4B8B0AD5.4090606@canterbury.ac.nz>
	<bbaeab101002281811m6311b5far8f386b0155c921e6@mail.gmail.com>
Message-ID: <1267430699.9222.23.camel@lifeless-64>

On Sun, 2010-02-28 at 18:11 -0800, Brett Cannon wrote:
...

>         So then looking for a .pyc alongside a .py or vice versa
>         should be almost free, and we shouldn't be worrying about
>         it.

> But that is making the assumption that all filesystems operate this
> way (.e.g does NFS have the same performance characteristics?).

NFS doesn't cache pages, rather it caches individual entries. I do not
know if adjacent data is pre-populated into an NFS client cache. I
rather suspect not, but the general point is true - many filesystems
cache by pages, not all. And some have unsorted lists or might have hash
tables rather than b-trees or sorted lists, where locality of reference
doesn't help at all. (VFAT is unsorted list, IIRC).

>                 hot cache:
>                 % time     seconds  usecs/call     calls    errors
>                 syscall
>                 ------ ----------- ----------- --------- ---------
>                 ----------------
>                  45.10    0.000368          92         4
>                 getdents
>                 
>                  0.00    0.000000           0       734       625 stat
>                 
>         
>         Further supporting the idea that stat calls are negligible
>         once the cache is warmed up.
> 
> 
> But that's the point: once it's warmed up. This is not the case when
> executing a script once every once in a while compared to something
> bzr where you are most likely going to execute the command multiple
> times within a small timeframe.

bzr would /love/ cold cache times to come down. They are one of the most
glaring performance differences between a large C program, and a large
python program. Even though the second run is fast, it hurts the first
time you run 'bzr st'. (**)

-Rob

(**) To the extent that I've seriously considered an import hook to
disable normal imports under bzrlib, and special case how the search
works so that we only load pyc's and assume all imports are absolute.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/140c661b/attachment.pgp>

From baptiste13z at free.fr  Mon Mar  1 09:09:09 2010
From: baptiste13z at free.fr (Baptiste Carvello)
Date: Mon, 01 Mar 2010 09:09:09 +0100
Subject: [Python-Dev] __file__
In-Reply-To: <hmencr$9mc$1@dough.gmane.org>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>	<20100225160832.6e7a3063@freewill.wooz.org>	<4B870E2A.8090406@canterbury.ac.nz>
	<4B870CD5.8020105@voidspace.org.uk>	<4B871227.2030707@canterbury.ac.nz>	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>	<4B88526E.7030808@voidspace.org.uk>	<20100227105613.6581a198@freewill.wooz.org>	<20100227173502.GA9522@laurie.devork>	<4B89CC14.5070902@canterbury.ac.nz>	<20100228121907.GA13564@laurie.devork>	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>	<4B8A6A8F.7050008@gmail.com>
	<hmekmc$1p4$1@dough.gmane.org> <hmencr$9mc$1@dough.gmane.org>
Message-ID: <hmfsne$d5s$1@dough.gmane.org>

Antoine Pitrou a ?crit :
> Le Sun, 28 Feb 2010 21:45:56 +0100, Baptiste Carvello a ?crit :
>> bytecode-only in a zip is used by py2exe, cx_freeze and the like, for
>> space reasons. Disabling it would probably hurt them.
> 
> Source code compresses quite well. I'm not sure it would make much of a 
> difference. 

I did a quick check on the stdlib: a zip with .py and .pyc is about 80% bigger 
than one with .pyc only. If you use only the bytecode, this can be seen as 
waisted space. On the other hand, if you ever need to debug the application, 
source is very handy...

Anyway, I'm a bit worried if bytecode-only is disabled from zipimport without 
some input from the developpers of py2exe/cx_freeze/etc, as they are big users 
of it.

Cheers, Baptiste


From fetchinson at googlemail.com  Mon Mar  1 10:02:18 2010
From: fetchinson at googlemail.com (Daniel Fetchinson)
Date: Mon, 1 Mar 2010 10:02:18 +0100
Subject: [Python-Dev] Draft PEP on RSON configuration file format
In-Reply-To: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
References: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
Message-ID: <fbe2e2101003010102s257d9b2g71b71f208559afb9@mail.gmail.com>

> Finding .ini configuration files too limiting, JSON and XML to hard to
> manually edit

[snip]

> I call the new format RSON (for "Readable Serial Object Notation"),
> and it is designed to be a superset of JSON.

Quick question: if JSON is too hard to manually edit, how can RSON be
any easier when it is a *superset* of JSON?

Cheers,
Daniel


-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown

From ncoghlan at gmail.com  Mon Mar  1 14:04:11 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 01 Mar 2010 23:04:11 +1000
Subject: [Python-Dev] __file__
In-Reply-To: <bbaeab101002281827pbf73ecg8a7e5d30726aa093@mail.gmail.com>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com>
	<bbaeab101002281221g608db2bem798361786ba39f24@mail.gmail.com>
	<4B8AD627.60103@gmail.com>
	<bbaeab101002281827pbf73ecg8a7e5d30726aa093@mail.gmail.com>
Message-ID: <4B8BBB4B.7030900@gmail.com>

Brett Cannon wrote:
> So there are a total of five to six depending on the OS (actually, VMS
> goes up to eight!) before a search path is considered not to contain a
> module.

The windows list is actually going to be slightly different (dir, pyd,
py, pyw, py[co]). It looks for .pyd files rather than either flavour of
.so file (we stopped allowing .dll files some time back due to the
sqlite3 DLL naming conflict). So I believe it is always 5 stat calls on
the major platform - dropping the bytecode filename check saves 20% of
them on a miss.

I'm not convinced that saving is worth the hassle of incurring a whole
pile of subtle backward compatibility problems. It seems better to say
"Python doesn't create in-place bytecode files anymore, but if you
arrange to put them there yourself we'll still read them".

I certainly wouldn't support removing the feature without some solid
benchmarks to say that it really is going to significantly speed up
typical import times for non-trivial modules (given that part of the
definition of "non-trivial" is "significant amounts of code to be run
when imported", that bar is likely to be a tough one to clear).

> And thanks to doing this I realized importlib is not stat'ing the
> directory first which should fail faster than checking for the __init__
> files every time.

That would explain why we had different ideas as to what the interpreter
was doing :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Mon Mar  1 15:21:21 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 1 Mar 2010 09:21:21 -0500
Subject: [Python-Dev] __file__
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz>
	<4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com> <hmekmc$1p4$1@dough.gmane.org>
	<hmencr$9mc$1@dough.gmane.org> <hmfsne$d5s$1@dough.gmane.org>
Message-ID: <20100301092121.5882a882@msiwind>

Le Mon, 01 Mar 2010 09:09:09 +0100,
Baptiste Carvello <baptiste13z at free.fr> a ?crit :
> 
> I did a quick check on the stdlib: a zip with .py and .pyc is about
> 80% bigger than one with .pyc only. If you use only the bytecode,
> this can be seen as waisted space. On the other hand, if you ever
> need to debug the application, source is very handy...

My point is that the wasted size compared to the total bundle size
(with interpreter and 3rd party C libs) would probably be small.

> Anyway, I'm a bit worried if bytecode-only is disabled from zipimport
> without some input from the developpers of py2exe/cx_freeze/etc, as
> they are big users of it.

Granted.

Regards

Antoine.



From pmaupin at gmail.com  Mon Mar  1 16:14:16 2010
From: pmaupin at gmail.com (Patrick Maupin)
Date: Mon, 1 Mar 2010 09:14:16 -0600
Subject: [Python-Dev] Draft PEP on RSON configuration file format
In-Reply-To: <fbe2e2101003010102s257d9b2g71b71f208559afb9@mail.gmail.com>
References: <d09829f51002281625m73dfbcbcm22ed3c6c1a4e6d88@mail.gmail.com>
	<fbe2e2101003010102s257d9b2g71b71f208559afb9@mail.gmail.com>
Message-ID: <d09829f51003010714q31d5b533uc60585d0bcb154c6@mail.gmail.com>

On Mon, Mar 1, 2010 at 3:02 AM, Daniel Fetchinson <fetchinson at googlemail.com
> wrote:


> Quick question: if JSON is too hard to manually edit, how can RSON be
> any easier when it is a *superset* of JSON?
>

Well, Python is essentially a superset of JSON, with string escape handling
being ever so slightly different, and using True instead of true, False
instead of false, and None instead of null.  YMMV, but I find it possible,
even probable, to write Python that is far easier to edit than JSON, and in
fact, I have used Python for configuration files that are only to be edited
by programmers or other technical types.

Regards,
Pat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/5444acc2/attachment.html>

From rrr at ronadam.com  Mon Mar  1 17:30:04 2010
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 01 Mar 2010 10:30:04 -0600
Subject: [Python-Dev] __file__
In-Reply-To: <4B8A6A8F.7050008@gmail.com>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>	<20100225160832.6e7a3063@freewill.wooz.org>	<4B870E2A.8090406@canterbury.ac.nz>	<4B870CD5.8020105@voidspace.org.uk>	<4B871227.2030707@canterbury.ac.nz>	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>	<4B88526E.7030808@voidspace.org.uk>	<20100227105613.6581a198@freewill.wooz.org>	<20100227173502.GA9522@laurie.devork>	<4B89CC14.5070902@canterbury.ac.nz>	<20100228121907.GA13564@laurie.devork>	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com>
Message-ID: <hmgq2c$ofd$1@dough.gmane.org>



Nick Coghlan wrote:
> Michael Foord wrote:
>>> Can't it look for a .py file in the source directory first (1st stat)?
>>> When it's there check for the .pyc in the cache directory (2nd stat,
>>> magic number encoded in filename), if it's not check for .pyc in the
>>> source directory (2nd stat + read for magic number check).  Or am I
>>> missing a subtlety?
>> The problem is doing this little dance for every path on sys.path.
> 
> To unpack this a little bit for those not quite as familiar with the
> import system (and to make it clear for my own benefit!): for a
> top-level module/package, each path on sys.path needs to be eliminated
> as a possible location before the interpreter can move on to check the
> next path in the list.
> 
> So the important number is the number of stat calls on a "miss" (i.e.
> when the requested module/package is not present in a directory).
> Currently, with builtin support for bytecode only files, there are 3
> checks (package directory, py source file, pyc/pyo bytecode file) to be
> made for each path entry.
> 
> The PEP proposes to reduce that to only two in the case of a miss, by
> checking for the cached pyc only if the source file is present (there
> would still be three checks for a "hit", but that only happens at most
> once per module lookup).
> 
> While the PEP is right in saying that a bytecode-only import hook could
> be added, I believe it would actually be a little tricky to write one
> that didn't severely degrade the performance of either normal imports or
> bytecode-only imports. Keeping it in the core import, but turning it off
> by default seems much less likely to have unintended performance
> consequences when it is switched back on.
> 
> Another option is to remove bytecode-only support from the default
> filesystem importer, but keep it for zipimport (since the stat call
> savings don't apply in the latter case).

What if ... a bytecode-only mode is triggered by "__main__" loading from a 
bytecode file, otherwise the .py files are needed and are checked to make 
sure the bytecode files are current.

Ron






From jacob at jacobian.org  Mon Mar  1 20:45:17 2010
From: jacob at jacobian.org (Jacob Kaplan-Moss)
Date: Mon, 1 Mar 2010 13:45:17 -0600
Subject: [Python-Dev] Status of PEP 381
Message-ID: <bc8ec34e1003011145y6223ba16y68a209c0bb1c961c@mail.gmail.com>

Hey folks --

Can someone point me to some information on what's going on with PEP
381 (PyPI mirroring)? There's a bunch of XXX'd material, and it
doesn't appear that pypi.python.org implements the statistics or
providing parts.

I'd be willing -- happy, actually -- to fill in the missing pieces,
but only if I'm not stepping on anyone's toes. I'd also need some help
adding code to pypi.python.org if that's a necessary part -- I'm not
very good at finding my way around that codebase.

Thanks,

Jacob

From tjreedy at udel.edu  Mon Mar  1 20:58:33 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 01 Mar 2010 14:58:33 -0500
Subject: [Python-Dev] Status of PEP 381
In-Reply-To: <bc8ec34e1003011145y6223ba16y68a209c0bb1c961c@mail.gmail.com>
References: <bc8ec34e1003011145y6223ba16y68a209c0bb1c961c@mail.gmail.com>
Message-ID: <hmh696$9e8$1@dough.gmane.org>

On 3/1/2010 2:45 PM, Jacob Kaplan-Moss wrote:
> Hey folks --
>
> Can someone point me to some information on what's going on with PEP
> 381 (PyPI mirroring)? There's a bunch of XXX'd material, and it
> doesn't appear that pypi.python.org implements the statistics or
> providing parts.
>
> I'd be willing -- happy, actually -- to fill in the missing pieces,
> but only if I'm not stepping on anyone's toes. I'd also need some help
> adding code to pypi.python.org if that's a necessary part -- I'm not
> very good at finding my way around that codebase.

There is a separate catalog/pypi mailing list, mirrored as 
gmane.comp.python.catalog. Ask there.

tjr


From brett at python.org  Mon Mar  1 21:37:10 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 1 Mar 2010 12:37:10 -0800
Subject: [Python-Dev] __file__
In-Reply-To: <hmgq2c$ofd$1@dough.gmane.org>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com> 
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com> 
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org> 
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz> 
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk> 
	<4B8A6A8F.7050008@gmail.com> <hmgq2c$ofd$1@dough.gmane.org>
Message-ID: <bbaeab101003011237idce4af6s6f177b0b993c50cf@mail.gmail.com>

On Mon, Mar 1, 2010 at 08:30, Ron Adam <rrr at ronadam.com> wrote:

>
>
> Nick Coghlan wrote:
>
>> Michael Foord wrote:
>>
>>> Can't it look for a .py file in the source directory first (1st stat)?
>>>> When it's there check for the .pyc in the cache directory (2nd stat,
>>>> magic number encoded in filename), if it's not check for .pyc in the
>>>> source directory (2nd stat + read for magic number check).  Or am I
>>>> missing a subtlety?
>>>>
>>> The problem is doing this little dance for every path on sys.path.
>>>
>>
>> To unpack this a little bit for those not quite as familiar with the
>> import system (and to make it clear for my own benefit!): for a
>> top-level module/package, each path on sys.path needs to be eliminated
>> as a possible location before the interpreter can move on to check the
>> next path in the list.
>>
>> So the important number is the number of stat calls on a "miss" (i.e.
>> when the requested module/package is not present in a directory).
>> Currently, with builtin support for bytecode only files, there are 3
>> checks (package directory, py source file, pyc/pyo bytecode file) to be
>> made for each path entry.
>>
>> The PEP proposes to reduce that to only two in the case of a miss, by
>> checking for the cached pyc only if the source file is present (there
>> would still be three checks for a "hit", but that only happens at most
>> once per module lookup).
>>
>> While the PEP is right in saying that a bytecode-only import hook could
>> be added, I believe it would actually be a little tricky to write one
>> that didn't severely degrade the performance of either normal imports or
>> bytecode-only imports. Keeping it in the core import, but turning it off
>> by default seems much less likely to have unintended performance
>> consequences when it is switched back on.
>>
>> Another option is to remove bytecode-only support from the default
>> filesystem importer, but keep it for zipimport (since the stat call
>> savings don't apply in the latter case).
>>
>
> What if ... a bytecode-only mode is triggered by "__main__" loading from a
> bytecode file, otherwise the .py files are needed and are checked to make
> sure the bytecode files are current.
>
>
That's way too magical for my tastes, especially if you mess up and
accidentally leave behind a __main__.pyc after moving the __main__.py file.

-Brett




> Ron
>
>
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/e0eb83b8/attachment-0001.html>

From brett at python.org  Mon Mar  1 21:44:33 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 1 Mar 2010 12:44:33 -0800
Subject: [Python-Dev] Who needs to be added to python-committers?
Message-ID: <bbaeab101003011244i4a32dc58pcf5215ef8645a0f8@mail.gmail.com>

Any people who have commit privileges not subscribed to python-committers?
We have gained some new people recently so I just want to make sure everyone
is subscribed to the list.

Also, anyone have an email address for Hirokazu Yamamoto? His mailbox is
full so Mailman unsubscribed him.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/2689c57c/attachment.html>

From ziade.tarek at gmail.com  Mon Mar  1 21:59:11 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 1 Mar 2010 21:59:11 +0100
Subject: [Python-Dev] Status of PEP 381
In-Reply-To: <bc8ec34e1003011145y6223ba16y68a209c0bb1c961c@mail.gmail.com>
References: <bc8ec34e1003011145y6223ba16y68a209c0bb1c961c@mail.gmail.com>
Message-ID: <94bdd2611003011259v56aa6ccaq430fe7ed6e73b504@mail.gmail.com>

On Mon, Mar 1, 2010 at 8:45 PM, Jacob Kaplan-Moss <jacob at jacobian.org> wrote:
> Hey folks --
>
> Can someone point me to some information on what's going on with PEP
> 381 (PyPI mirroring)? There's a bunch of XXX'd material, and it
> doesn't appear that pypi.python.org implements the statistics or
> providing parts.
>
> I'd be willing -- happy, actually -- to fill in the missing pieces,
> but only if I'm not stepping on anyone's toes. I'd also need some help
> adding code to pypi.python.org if that's a necessary part -- I'm not
> very good at finding my way around that codebase.

Hey Jacob,

I have started to implement the server parts (some are already
available at pypi but not activated) and Martin worked on the client
part on his side to provide a library for mirrors. I need to finish
editing the PEP text but any help is welcome of course on any part.

As Terry suggested you can send a mail at Catalog, I'll catch up on
the thread once its started, to list possible tasks.

Regards,
Tarek
-- 
Tarek Ziad? | http://ziade.org

From martin at v.loewis.de  Mon Mar  1 22:40:35 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 01 Mar 2010 22:40:35 +0100
Subject: [Python-Dev] Python and Windows 2000
Message-ID: <4B8C3453.7030507@v.loewis.de>

I don't recall whether we have already decided about continued support
for Windows 2000.

If not, I'd like to propose that we phase out that support: the Windows
2.7 installer should display a warning; 3.2 will stop supporting Windows
2000.

Opinions?

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Mon Mar  1 22:53:35 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 02 Mar 2010 10:53:35 +1300
Subject: [Python-Dev] __file__
In-Reply-To: <hmgq2c$ofd$1@dough.gmane.org>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz> <4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com> <hmgq2c$ofd$1@dough.gmane.org>
Message-ID: <4B8C375F.30508@canterbury.ac.nz>

Ron Adam wrote:

> What if ... a bytecode-only mode is triggered by "__main__" loading from 
> a bytecode file, otherwise the .py files are needed and are checked to 
> make sure the bytecode files are current.

That would preclude having a bytecode-only library that
could be used by a sourceful program.

Such a situation might arise if you have an application
with a scripting interface that is used by importing
stuff from the application's internal libraries.

-- 
Greg

From guido at python.org  Mon Mar  1 22:53:45 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 1 Mar 2010 13:53:45 -0800
Subject: [Python-Dev] Python and Windows 2000
In-Reply-To: <4B8C3453.7030507@v.loewis.de>
References: <4B8C3453.7030507@v.loewis.de>
Message-ID: <ca471dc21003011353l25236db9q69a6074b0d70f5f@mail.gmail.com>

On Mon, Mar 1, 2010 at 1:40 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> I don't recall whether we have already decided about continued support
> for Windows 2000.
>
> If not, I'd like to propose that we phase out that support: the Windows
> 2.7 installer should display a warning; 3.2 will stop supporting Windows
> 2000.
>
> Opinions?

+2000

-- 
--Guido van Rossum (python.org/~guido)

From ssteinerx at gmail.com  Mon Mar  1 23:33:21 2010
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 1 Mar 2010 17:33:21 -0500
Subject: [Python-Dev] Python and Windows 2000
In-Reply-To: <ca471dc21003011353l25236db9q69a6074b0d70f5f@mail.gmail.com>
References: <4B8C3453.7030507@v.loewis.de>
	<ca471dc21003011353l25236db9q69a6074b0d70f5f@mail.gmail.com>
Message-ID: <339E3993-FFF8-4B86-BE17-59AC56540CFB@gmail.com>


On Mar 1, 2010, at 4:53 PM, Guido van Rossum wrote:

> On Mon, Mar 1, 2010 at 1:40 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> I don't recall whether we have already decided about continued support
>> for Windows 2000.
>> 
>> If not, I'd like to propose that we phase out that support: the Windows
>> 2.7 installer should display a warning; 3.2 will stop supporting Windows
>> 2000.
>> 
>> Opinions?
> 
> +2000

I guess i'll finally have to retire my ten year old MSDN cd's and my favorite version of Windows -- the last one I ever used.

S



From collinwinter at google.com  Tue Mar  2 00:35:15 2010
From: collinwinter at google.com (Collin Winter)
Date: Mon, 1 Mar 2010 15:35:15 -0800
Subject: [Python-Dev] Packaging JIT-less versions of Python
Message-ID: <3c8293b61003011535x6190351ald0640855bbfa7c4b@mail.gmail.com>

Hey packaging guys,

We recently committed a change to Unladen Swallow [1] that moves all
the JIT infrastructure into a Python extension module. The theory [2]
behind this patch was that this would make it easier for downstream
packagers to ship a JIT-less Python package, with the JIT compiler
available via an optional add-on package.

Some questions for you, so we're not shooting blind here:
- Have you guys thought about how a JIT-enabled Python 3 installation
would be packaged by your respective distros?
- Would you prefer the default python3.x package to have a JIT, or
would you omit the JIT by default?
- How would you prefer to build the JIT-less package (current options:
via a ./configure flag; or by deleting _llvmjit.so from the
JIT-enabled package)?
- Would the two packages be able to exist side-by-side, or would they
be mutually exclusive?

My strong preference would be to have the JIT included by default so
that it receives as much testing as possible.

Thanks,
Collin Winter

[1] - http://code.google.com/p/unladen-swallow/source/detail?r=1110
[2] - http://code.google.com/p/unladen-swallow/issues/detail?id=136

From barry at python.org  Tue Mar  2 01:32:41 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 1 Mar 2010 19:32:41 -0500
Subject: [Python-Dev] __file__
In-Reply-To: <20100228224020.46da4e23@msiwind>
References: <20100130190005.058c8187@freewill.wooz.org>
	<bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz>
	<4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<ca471dc21002261429x5674ea95vc52e0255185a254a@mail.gmail.com>
	<bbaeab101002261455wf89eenf2796bd3d6cc8ddb@mail.gmail.com>
	<4B885AB4.7040004@g.nevcal.com>
	<bbaeab101002261713h5fd3e9f4hab56807e0831c266@mail.gmail.com>
	<4B889AB1.4050402@g.nevcal.com> <4B89C5FB.40104@canterbury.ac.nz>
	<4B89F5FA.2010604@g.nevcal.com> <4B8AFAAB.5020106@canterbury.ac.nz>
	<4B8B3539.5060709@g.nevcal.com> <20100228224020.46da4e23@msiwind>
Message-ID: <20100301193241.38a81a93@freewill.wooz.org>

On Feb 28, 2010, at 10:40 PM, Antoine Pitrou wrote:

>File extensions exist for a reason, even if you find that "kooky" and
>have strong ideas about the psychology of text editors.
>
>Having some binary files named "foobar.py" would certainly annoy a lot
>of people, including me.

I completely agree.
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/11f54e43/attachment-0001.pgp>

From barry at python.org  Tue Mar  2 01:41:52 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 1 Mar 2010 19:41:52 -0500
Subject: [Python-Dev] __file__
In-Reply-To: <4B89CC14.5070902@canterbury.ac.nz>
References: <20100202231252.77a79b0e@freewill.wooz.org>
	<loom.20100203T140452-568@post.gmane.org>
	<bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz>
	<4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
Message-ID: <20100301194152.766d0de8@freewill.wooz.org>

On Feb 28, 2010, at 02:51 PM, Greg Ewing wrote:

>A solution might be to look for the presence of the
>cache directory, and only look for a .pyc in the source
>directory if there is no cache directory. Testing for
>the cache directory would only have to be done once
>per package and the result remembered, so it would
>add very little overhead.

I think the other thing that bothers me about continuing to support pyc-only
imports, is that people will then want tools to create them.  Right now, it's
probably just as easy as byte-compiling everything, then finding the .py files
and removing them.

After PEP 3147 is implemented, and the default, you'll have to byte-compile
the files, then find the pycs in the __pycache__ directory, move them up a
level and rename them.  Then of course remove the .py files.

It's not insurmountable of course, I think if we support pyc-only imports,
people are rightly going to want us to write and support the tool to create
those imports.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/75dc922a/attachment.pgp>

From barry at python.org  Tue Mar  2 01:49:46 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 1 Mar 2010 19:49:46 -0500
Subject: [Python-Dev] __file__
In-Reply-To: <4B8A6A8F.7050008@gmail.com>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz>
	<4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com>
Message-ID: <20100301194946.188a8855@freewill.wooz.org>

On Feb 28, 2010, at 11:07 PM, Nick Coghlan wrote:

>While the PEP is right in saying that a bytecode-only import hook could
>be added, I believe it would actually be a little tricky to write one
>that didn't severely degrade the performance of either normal imports or
>bytecode-only imports. Keeping it in the core import, but turning it off
>by default seems much less likely to have unintended performance
>consequences when it is switched back on.

Except that even users of bytecode-only imports probably don't want or care
about that for *every* package directory.  So really, all-or-nothing hits them
too.  One option to help with that is to have a flag or marker in a package's
__init__.py that signals pyc-only imports for that package directory.  It's
getting complicated again. ;)

>Another option is to remove bytecode-only support from the default
>filesystem importer, but keep it for zipimport (since the stat call
>savings don't apply in the latter case).

I'd be okay with this, but even here I'd argue that it would be fine to
require the source files by default.  The primary use case I've seen mentioned
for pyc-only imports is to make it more difficult for users to accidentally
shoot themselves in the foot.  I think the very presence of a zip file for
importing is enough without the extra step of removing the source.  But that's
just me. :)

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/f49e437a/attachment.pgp>

From barry at python.org  Tue Mar  2 01:59:55 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 1 Mar 2010 19:59:55 -0500
Subject: [Python-Dev] __file__
In-Reply-To: <20100301194946.188a8855@freewill.wooz.org>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100225160832.6e7a3063@freewill.wooz.org>
	<4B870E2A.8090406@canterbury.ac.nz>
	<4B870CD5.8020105@voidspace.org.uk>
	<4B871227.2030707@canterbury.ac.nz>
	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>
	<4B88526E.7030808@voidspace.org.uk>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com>
	<20100301194946.188a8855@freewill.wooz.org>
Message-ID: <20100301195955.3091dc19@freewill.wooz.org>

Thanks everybody for providing great input on this aspect of the PEP.  I've
updated the open issues section to include a list of the possible resolutions
for bytecode-only imports.  Unless anybody has more ideas, it might just be
time to get a BDFL pronouncement.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/107edee4/attachment.pgp>

From brett at python.org  Tue Mar  2 03:02:57 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 1 Mar 2010 18:02:57 -0800
Subject: [Python-Dev] Python and Windows 2000
In-Reply-To: <4B8C3453.7030507@v.loewis.de>
References: <4B8C3453.7030507@v.loewis.de>
Message-ID: <bbaeab101003011802q202df896ub44c2c240d96b62b@mail.gmail.com>

On Mon, Mar 1, 2010 at 13:40, "Martin v. L?wis" <martin at v.loewis.de> wrote:

> I don't recall whether we have already decided about continued support
> for Windows 2000.
>
> If not, I'd like to propose that we phase out that support: the Windows
> 2.7 installer should display a warning; 3.2 will stop supporting Windows
> 2000.
>
> Opinions?
>

Considering MS is retiring support for Win2K in July (
http://support.microsoft.com/lifecycle/?p1=3071) I think we can be faster
than them in dropping it. =)  +1 from me.


>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/2d7e1943/attachment.html>

From meadori at gmail.com  Tue Mar  2 03:05:49 2010
From: meadori at gmail.com (Meador Inge)
Date: Mon, 1 Mar 2010 20:05:49 -0600
Subject: [Python-Dev] PEP 3188: Implementation Questions
In-Reply-To: <hmbk81$b99$1@dough.gmane.org>
References: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com>
	<4B88464E.9080700@canterbury.ac.nz>
	<4095897c1002270858j5e14579dhe01c1fb4958f087b@mail.gmail.com>
	<hmbk81$b99$1@dough.gmane.org>
Message-ID: <4095897c1003011805r72f33112j1891e3dac13c7707@mail.gmail.com>

On Sat, Feb 27, 2010 at 11:20 AM, Thomas Heller <theller at ctypes.org> wrote:

> See issue 887237:
>
> http://bugs.python.org/issue887237
>

Thanks for the link Thomas.  Since there is already interest in adding
arithmetic to ctypes, perhaps that is an option.  One question that raises
in my mind, though, is should only 'long double' unpack to a ctype in that
case?  Or should all items unpack to ctypes now?  It seems to me that you
would want everything to unpack to types from the same family (e.g. Python
builtins or ctypes).  This seems conceptually cleaner and the
interoperability between types in the same "family" are (or can be in the
case of modifying ctypes) more clearly defined.

Thanks,

-- Meador
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100301/22a241e6/attachment-0001.html>

From nyamatongwe at gmail.com  Tue Mar  2 03:11:28 2010
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Tue, 2 Mar 2010 13:11:28 +1100
Subject: [Python-Dev] Python and Windows 2000
In-Reply-To: <4B8C3453.7030507@v.loewis.de>
References: <4B8C3453.7030507@v.loewis.de>
Message-ID: <50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com>

Martin v. L?wis:

> I don't recall whether we have already decided about continued support
> for Windows 2000.
>
> If not, I'd like to propose that we phase out that support: the Windows
> 2.7 installer should display a warning; 3.2 will stop supporting Windows
> 2000.

   Is there any reason for this? I can understand dropping Windows 9x
due to the lack of Unicode support but is there anything missing from
Windows 2000 that makes supporting it difficult?

   Neil

From dmalcolm at redhat.com  Tue Mar  2 04:29:32 2010
From: dmalcolm at redhat.com (David Malcolm)
Date: Mon, 01 Mar 2010 22:29:32 -0500
Subject: [Python-Dev] Packaging JIT-less versions of Python
In-Reply-To: <3c8293b61003011535x6190351ald0640855bbfa7c4b@mail.gmail.com>
References: <3c8293b61003011535x6190351ald0640855bbfa7c4b@mail.gmail.com>
Message-ID: <1267500572.8882.351.camel@brick>

On Mon, 2010-03-01 at 15:35 -0800, Collin Winter wrote:
> Hey packaging guys,
> 
> We recently committed a change to Unladen Swallow [1] that moves all
> the JIT infrastructure into a Python extension module. The theory [2]
> behind this patch was that this would make it easier for downstream
> packagers to ship a JIT-less Python package, with the JIT compiler
> available via an optional add-on package.
> 
> Some questions for you, so we're not shooting blind here:
> - Have you guys thought about how a JIT-enabled Python 3 installation
> would be packaged by your respective distros?
I suspect that there may be architecture-specific issues both in getting
LLVM to work, and in getting the Python JIT code running (e.g. I've no
idea how well it works on, say, s390).  The answers may vary between CPU
architecture (e.g. "on by default for i386 and x86_64, off by default on
ppc", or whatever).


> - Would you prefer the default python3.x package to have a JIT, or
> would you omit the JIT by default?

I'd be inclined to turn it on by default in Fedora's "rawhide"
development branch as soon as it's meaningful to test it (with a
suitable warning on our mailing list); we like to try this kind of
thing!  If no major issues are found, I'd leave it turned on.  I'd
expect to run into "fun" CPU-arch-specific bugs, though.  Multicore
64-bit big-endian springs to mind.

I can't speak for what I'd do for RHEL at this time.  It's likely to be
affected by how experience in Fedora goes, but goes through more formal
QA, and on more "enterprisey" hardware (e.g. s390).


> - How would you prefer to build the JIT-less package (current options:
> via a ./configure flag; or by deleting _llvmjit.so from the
> JIT-enabled package)?
> - Would the two packages be able to exist side-by-side, or would they
> be mutually exclusive?

I have a particular interest in ABI compatibility: if turning JIT on and
off is going to change the ABI of extension modules, that would be a
major pain, as I hope that we will have dozens of C extension modules
available via RPM for our Python 3 stack by the time of the "great
unladen merger".

So I'm keen for the ability to toggle the JIT code in the face of bugs
and have it not affect ABI.  "-Xjit" will do this at runtime (once
that's renamed), but I think it would be useful to be able to toggle the
JIT on/off default during the build, so that I can fix a broken
architecture for non-technical users, but have individual testers opt
back in with -Xjit whilst tracking down a major bug.

In either case, I don't want to have to recompile 30 extension modules
to try with/without JIT; that would introduce too much change during
bug-hunts, and be no fun at all.


(In the blue-sky nirvana future, I'd love to be able to ship
ahead-of-time compiled versions of the stdlib, pre-optimized based on
realworld workloads.  Back in my reality, though, I have bugs to fix
before I can work on _that_ patch :( )


> My strong preference would be to have the JIT included by default so
> that it receives as much testing as possible.

Sounds reasonable.  Hope the above made sense and is useful.
Dave

> 
> Thanks,
> Collin Winter
> 
> [1] - http://code.google.com/p/unladen-swallow/source/detail?r=1110
> [2] - http://code.google.com/p/unladen-swallow/issues/detail?id=136



From martin at v.loewis.de  Tue Mar  2 06:17:04 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 02 Mar 2010 06:17:04 +0100
Subject: [Python-Dev] Python and Windows 2000
In-Reply-To: <50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com>
References: <4B8C3453.7030507@v.loewis.de>
	<50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com>
Message-ID: <4B8C9F50.7010709@v.loewis.de>

>> If not, I'd like to propose that we phase out that support: the Windows
>> 2.7 installer should display a warning; 3.2 will stop supporting Windows
>> 2000.
> 
>    Is there any reason for this? I can understand dropping Windows 9x
> due to the lack of Unicode support but is there anything missing from
> Windows 2000 that makes supporting it difficult?

See http://bugs.python.org/issue6926

The SDK currently hides symbolic constants from us that people are
asking for.

In addition, we could simplify the code in dl_nt.c around
GetCurrentActCtx and friends, by linking to these functions directly.

Regards,
Martni

From stefan_ml at behnel.de  Tue Mar  2 10:06:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Mar 2010 10:06:06 +0100
Subject: [Python-Dev] ElementTree.tostring() returning str by default in
	Py3.2
Message-ID: <hmikdu$bi0$1@dough.gmane.org>

Hi,

I just noticed that the xml.etree.ElementTree.tostring() function in Py3.2
returns a str object by default, unless an encoding is specified. This is a
backwards incompatible change compared to ET 1.2. For one, it breaks tons
of tests in lxml's compatibility test suite. Previously, the default
encoding was 'ASCII', and tostring() was guaranteed to always return a byte
string.

According to svn blame, this was changed by GvR in rev. 56841.

Is there any reasoning behind this?

Stefan


From nyamatongwe at gmail.com  Tue Mar  2 10:08:06 2010
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Tue, 2 Mar 2010 20:08:06 +1100
Subject: [Python-Dev] Python and Windows 2000
In-Reply-To: <4B8C9F50.7010709@v.loewis.de>
References: <4B8C3453.7030507@v.loewis.de>
	<50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com>
	<4B8C9F50.7010709@v.loewis.de>
Message-ID: <50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com>

Martin v. L?wis:

> See http://bugs.python.org/issue6926
>
> The SDK currently hides symbolic constants from us that people are
> asking for.

   Setting the version to 0x501 (XP) doesn't actively try to stop
running on version 0x500 (2K), it just reveals the symbols and APIs
from 0x501. Including a call to an 0x501-specific API will then fail
at load.

    IPPROTO_IPV6 (the cause of issue 6926) isn't a new symbol that
started working in Windows XP - it was present in older SDKs without a
version guard so was visible when compiling for any version of
Windows.

> In addition, we could simplify the code in dl_nt.c around
> GetCurrentActCtx and friends, by linking to these functions directly.

   It would be simpler but its not as if this code needs any changes
at this point.

   I don't really have a strong need for Windows 2000 although I keep
an instance for checking compatibility of my code and I do still get
queries from people using old versions of Windows, including 9x. There
is the question of whether to force failure on Windows 2000 or just
remove it from the list of known-working platforms while still
allowing it to run.

   Neil

From greg.ewing at canterbury.ac.nz  Tue Mar  2 10:40:47 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 02 Mar 2010 22:40:47 +1300
Subject: [Python-Dev] Python and Windows 2000
In-Reply-To: <50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com>
References: <4B8C3453.7030507@v.loewis.de>
	<50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com>
	<4B8C9F50.7010709@v.loewis.de>
	<50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com>
Message-ID: <4B8CDD1F.5020501@canterbury.ac.nz>

Neil Hodgson wrote:
> There
> is the question of whether to force failure on Windows 2000 or just
> remove it from the list of known-working platforms while still
> allowing it to run.

I'd be grateful if you could refrain from doing anything to
actively break it. Win 2000 was the last version to be free
of Microsoft DRM crap, and I'd be unhappy if I were forced
to downgrade to a later version just so I could use Py3 on
it.

-- 
Greg

From stefan_ml at behnel.de  Tue Mar  2 10:47:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Mar 2010 10:47:06 +0100
Subject: [Python-Dev] ElementTree.tostring() returning str by default in
	Py3.2
In-Reply-To: <hmikdu$bi0$1@dough.gmane.org>
References: <hmikdu$bi0$1@dough.gmane.org>
Message-ID: <hmimqr$jq8$1@dough.gmane.org>

Stefan Behnel, 02.03.2010 10:06:
> I just noticed that the xml.etree.ElementTree.tostring() function in Py3.2
> returns a str object by default, unless an encoding is specified. This is a
> backwards incompatible change compared to ET 1.2. For one, it breaks tons
> of tests in lxml's compatibility test suite. Previously, the default
> encoding was 'ASCII', and tostring() was guaranteed to always return a byte
> string.
> 
> According to svn blame, this was changed by GvR in rev. 56841.
> 
> Is there any reasoning behind this?

BTW, sorry for not reporting this earlier (the change is from 2007). It
seems that bug 7173 (or some other bug) prevented me from running lxml's
test suite in Py3 for quite a while now. This is the first time in ages
that I get a complete run in Py3.x (x>0).

Stefan


From steve at pearwood.info  Tue Mar  2 10:53:26 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 2 Mar 2010 20:53:26 +1100
Subject: [Python-Dev] __file__
In-Reply-To: <20100301194152.766d0de8@freewill.wooz.org>
References: <20100202231252.77a79b0e@freewill.wooz.org>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100301194152.766d0de8@freewill.wooz.org>
Message-ID: <201003022053.27049.steve@pearwood.info>

On Tue, 2 Mar 2010 11:41:52 am Barry Warsaw wrote:

> After PEP 3147 is implemented, and the default, you'll have to
> byte-compile the files, then find the pycs in the __pycache__
> directory, move them up a level and rename them.  Then of course
> remove the .py files.
>
> It's not insurmountable of course, I think if we support pyc-only
> imports, people are rightly going to want us to write and support the
> tool to create those imports.

Surely that's a job for a tiny Python script, or even a shell script? It 
doesn't sound hard, not from the description given. I imagine there 
will be recipes on ActiveState quite quickly, and if there isn't, that 
would be good evidence that demand for the feature is low.



-- 
Steven D'Aprano

From steve at pearwood.info  Tue Mar  2 11:06:16 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 2 Mar 2010 21:06:16 +1100
Subject: [Python-Dev] __file__
In-Reply-To: <20100301195955.3091dc19@freewill.wooz.org>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100301194946.188a8855@freewill.wooz.org>
	<20100301195955.3091dc19@freewill.wooz.org>
Message-ID: <201003022106.17179.steve@pearwood.info>

On Tue, 2 Mar 2010 11:59:55 am Barry Warsaw wrote:
> Thanks everybody for providing great input on this aspect of the PEP.
>  I've updated the open issues section to include a list of the
> possible resolutions for bytecode-only imports.  Unless anybody has
> more ideas, it might just be time to get a BDFL pronouncement.

Please excuse me if these minor points have already been discussed, but 
I couldn't see them in the PEP.

(1) What happens if the __cache__ directory doesn't exist and the 
enclosing directory is unwriteable, or if it does exist, but is 
unreadable?

I expect that the byte code files will simply not be created, and 
everything will continue without them.

(2) Presumably this only effects imports, not running python source code 
as a script. If I do this:

python myscript.py

from the shell, I would expect that no __cache__ directory will be 
created, just like today. 

BTW, you have some sort of automated warning in the PEP:

System Message: WARNING/2 (pep-3147.txt, line 237)
Title underline too short.

http://www.python.org/dev/peps/pep-3147/#id47



-- 
Steven D'Aprano

From dickinsm at gmail.com  Tue Mar  2 13:38:47 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 2 Mar 2010 12:38:47 +0000
Subject: [Python-Dev] PEP 3188: Implementation Questions
In-Reply-To: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com>
References: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com>
Message-ID: <5c6f2a5d1003020438g3d67e4falb02f9501e1b535fd@mail.gmail.com>

On Fri, Feb 26, 2010 at 4:51 AM, Meador Inge <meadori at gmail.com> wrote:
>
> Recently some discussion began in the issue 3132 thread (http://bugs.python.org/issue3132) regarding
> implementation of the new struct string syntax for PEP 3118.? Mark Dickinson
> suggested that I bring the discussion on over to Python Dev.? Below is a summary
> of the questions\comments from the thread.

Thanks for bringing this up here!

> Unpacking a long-double
> ===================
> ...

I don't want to dwell on this too much, since this is just one small
part of the proposed struct module additions, and I'd like to get
answers to some of the other questions you raised.? If interested
parties (Carl?, Travis?) have time to comment on the other questions
Meador raised, I'd *really* appreciate it!

For long-double: I'm essentially -1 on any Decimal involvement here,
on the basis that the semantics are messy and I doubt that the
resulting behaviour would match users' needs or expectations.? I'm
also not too keen on introducing a long-double-with-arithmetic type
(wherever it ends up living);? I think having a single (binary)
floating-point type has served Python very well, and adding other
binary float precisions would risk either adding significant
complexity, or leaving the extended precision type somewhat crippled.
If there were a long double type that supported arithmetic, how should
mixed-mode double + long double operations behave?  What about
conversions of long doubles to and from decimal strings?  Should math
module and cmath module functions accept long double arguments, and if
so, what type result should they produce?  The fact that there are at
least 3 different common formats, before considering padding and byte
order, for long double (IEEE 80-bit extended used on x86, IEEE
binary128, and the so-called double-double format) isn't going to make
things any easier here.

I guess, in my currently not-very-informed state, I'd vote for:

  - packing and unpacking with the long double format expects and
produces a ctypes.long_double in native mode;  the long-double format
would match that of the platform.
  - non-native mode packing and unpacking aren't permitted, and raise
an exception.
  - to do arithmetic with long doubles, users would simply have to
convert to and from Python floats, accepting the loss of precision
that such conversion entails.

But I think I'm failing to understand the intended use-cases for this
(and other) additions.  Would it be possible for Carl or Travis, or
anyone else, to give some examples of situations where this would be
useful?  I don't doubt that such situations exist;  I'm just not sure
what they are.

> Pointers
> ======
> [...]
> 2. How would unpacking a pointer to a Python Object work out?? Given an
> ??? address how would the appropriate object to be unpacked be determined?

Again, I'd really like to see some examples of how/when packing and
unpacking pointers to Python objects would be used;  I'm currently
feeling too stupid to understand how this might work in practice.

> 4. For the 'X{}' format (pointer to a function), is this supposed to mean a
> ??? Python function or a C function?

Ditto for this;  assuming that we're talking about a Python function here.

> String Syntax
> ==========
>
> The syntax seems to have transcended verbal description.? I think we
> need to put forth a grammar.

Agreed:  a clear specification is needed.

> Use Cases
> ========
>
> ?? 1. What are the real world use cases for these struct string extensions?
> ????? These should be fleshed out and documented.

+many.

Mark

From dickinsm at gmail.com  Tue Mar  2 13:50:43 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 2 Mar 2010 12:50:43 +0000
Subject: [Python-Dev] PEP 3188: Implementation Questions
In-Reply-To: <4B89C946.6010403@canterbury.ac.nz>
References: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com>
	<4B88464E.9080700@canterbury.ac.nz>
	<4095897c1002270858j5e14579dhe01c1fb4958f087b@mail.gmail.com>
	<4B89C946.6010403@canterbury.ac.nz>
Message-ID: <5c6f2a5d1003020450r3d4af2cdi1e29bf7f1a5254ad@mail.gmail.com>

On Sun, Feb 28, 2010 at 1:39 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Meador Inge wrote:
>
>> Even with the user-defined precision capabilities of the 'Decimal' class?
>> ?In other words, can I create an instance of a 'Decimal' that behaves (in
>> all operations: arithmetic, comparison, etc...) exactly as the extended
>> double precision type offered by a given machine?
>
> It's not precision that's the issue, it's that the number
> base is different. That affects which numbers can
> be represented exactly, and how results that can't be
> represented exactly are rounded.
>
> I would be very surprised if there is a way of configuring
> the Decimal type so that it gives identical results to that
> of any IEEE binary floating point type, including rounding
> behaviour, denormalisation, etc.

I'd be astonished. :)

Mark

From ncoghlan at gmail.com  Tue Mar  2 14:03:19 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 02 Mar 2010 23:03:19 +1000
Subject: [Python-Dev] Breaking bytecode only imports (was Re:  __file__)
In-Reply-To: <20100301195955.3091dc19@freewill.wooz.org>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>	<20100225160832.6e7a3063@freewill.wooz.org>	<4B870E2A.8090406@canterbury.ac.nz>	<4B870CD5.8020105@voidspace.org.uk>	<4B871227.2030707@canterbury.ac.nz>	<bbaeab101002261409q77c048f9x9bd5c06d18e8943e@mail.gmail.com>	<4B88526E.7030808@voidspace.org.uk>	<20100227105613.6581a198@freewill.wooz.org>	<20100227173502.GA9522@laurie.devork>	<4B89CC14.5070902@canterbury.ac.nz>	<20100228121907.GA13564@laurie.devork>	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>	<4B8A6A8F.7050008@gmail.com>	<20100301194946.188a8855@freewill.wooz.org>
	<20100301195955.3091dc19@freewill.wooz.org>
Message-ID: <4B8D0C97.4020002@gmail.com>

Barry Warsaw wrote:
> Thanks everybody for providing great input on this aspect of the PEP.  I've
> updated the open issues section to include a list of the possible resolutions
> for bytecode-only imports.  Unless anybody has more ideas, it might just be
> time to get a BDFL pronouncement.

I think the benchmarking in the bytecode-only section is still too weak.
"evidence shows that the extra stats can be fairly costly to start up
time" isn't a valid justification for breaking working code. Doing 4
stat calls instead of 5 on a directory miss just doesn't excite me very
much without some genuine benchmarks across different operating systems
and filesystems showing that reducing the number of stat calls by at
best 20% will result in a measurable reduction in import times for real
modules (where we can expect the import time to be dominated by the
execution of the actual module code rather than the time needed to find
that code in the first place).

Using the sample numbers Robert Collins posted:

# Startup time for bzr (cold cache):
$ drop-caches
$ time bzr --no-plugins revno
5061

real    0m8.875s
user    0m0.210s
sys     0m0.140s

# Hot cache
$ time bzr --no-plugins revno
5061

real    0m0.307s
user    0m0.250s
sys     0m0.040s

$ strace -c bzr --no-plugins revno
5061
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 56.34    0.040000          76       527           read
 28.98    0.020573           9      2273      1905 open
 14.43    0.010248          14       734       625 stat
  0.15    0.000107           0       533           fstat


hot cache:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 45.10    0.000368          92         4           getdents
 19.49    0.000159           0       527           read
 16.91    0.000138           1       163           munmap
 10.05    0.000082           2        54           mprotect
  8.46    0.000069           0      2273      1905 open
  0.00    0.000000           0         8           write
  0.00    0.000000           0       367           close
  0.00    0.000000           0       734       625 stat

Assuming all those stat errors are misses from the import system, we're
looking at reducing that 625 figure down to 500: 125 fewer failed calls.
With a hot cache, the impact is too small for strace to even measure.
With a cold cache, it is 1.75 milliseconds: only 1.25% of the system
time consumed in the script's execution, and not even registering
relative to the 9 second wall clock time.

Without significant measurable performance gains, a mere aesthetic
preference isn't enough to justify inflicting subtle breakage on even a
small subset of our users.

Even aside from the issue of a lack of benchmarks to justify the
breakage, bytecode only imports *cannot* legitimately be broken without
at least one release where they generate Deprecation Warnings.

Cheers,
Nick.

P.S. I actually started this thread as a +0 to the idea of dropping
bytecode only imports. Over the course of the discussion I've shifted to
a firm -1 in the absence of some proper comparative benchmarks to
justify the change in semantics.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Tue Mar  2 16:32:12 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 2 Mar 2010 10:32:12 -0500
Subject: [Python-Dev] code.python.org dead?
Message-ID: <20100302103212.69669f0f@msiwind>


Hello,

It seems someone has decided that code.python.org doesn't resolve
anymore.
This is annoying since there were Mercurial mirrors there
( at http://code.python.org/hg ). Until we make the official switch,
these mirrors are still useful.

Regards

Antoine.



From guido at python.org  Tue Mar  2 18:34:26 2010
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Mar 2010 09:34:26 -0800
Subject: [Python-Dev] Breaking bytecode only imports (was Re: __file__)
In-Reply-To: <4B8D0C97.4020002@gmail.com>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com> 
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork> 
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork> 
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com> 
	<20100301194946.188a8855@freewill.wooz.org>
	<20100301195955.3091dc19@freewill.wooz.org> 
	<4B8D0C97.4020002@gmail.com>
Message-ID: <ca471dc21003020934o3713c86cs2c9ec19f4db33046@mail.gmail.com>

On Tue, Mar 2, 2010 at 5:03 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> P.S. I actually started this thread as a +0 to the idea of dropping
> bytecode only imports. Over the course of the discussion I've shifted to
> a firm -1 in the absence of some proper comparative benchmarks to
> justify the change in semantics.

FWIW, I started at -1 and am still -1. I think the PEP is overreaching
in this aspect; it does not serve the stated purpose of the PEP to
make life easier for distros that share code between Python versions.

-- 
--Guido van Rossum (python.org/~guido)

From barry at python.org  Tue Mar  2 20:52:59 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 2 Mar 2010 14:52:59 -0500
Subject: [Python-Dev] Breaking bytecode only imports (was Re: __file__)
In-Reply-To: <ca471dc21003020934o3713c86cs2c9ec19f4db33046@mail.gmail.com>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100227105613.6581a198@freewill.wooz.org>
	<20100227173502.GA9522@laurie.devork>
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork>
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com>
	<20100301194946.188a8855@freewill.wooz.org>
	<20100301195955.3091dc19@freewill.wooz.org>
	<4B8D0C97.4020002@gmail.com>
	<ca471dc21003020934o3713c86cs2c9ec19f4db33046@mail.gmail.com>
Message-ID: <20100302145259.1442a324@heresy.wooz.org>

On Mar 02, 2010, at 09:34 AM, Guido van Rossum wrote:

>On Tue, Mar 2, 2010 at 5:03 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> P.S. I actually started this thread as a +0 to the idea of dropping
>> bytecode only imports. Over the course of the discussion I've shifted to
>> a firm -1 in the absence of some proper comparative benchmarks to
>> justify the change in semantics.
>
>FWIW, I started at -1 and am still -1. I think the PEP is overreaching
>in this aspect; it does not serve the stated purpose of the PEP to
>make life easier for distros that share code between Python versions.

I think that's fair, and just the guidance I'm looking for.  By now you
understand the pros and cons, so if this is a pronouncement, I will cement it
into the PEP.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100302/f740fd0e/attachment.pgp>

From brett at python.org  Tue Mar  2 21:06:53 2010
From: brett at python.org (Brett Cannon)
Date: Tue, 2 Mar 2010 12:06:53 -0800
Subject: [Python-Dev] __file__
In-Reply-To: <201003022106.17179.steve@pearwood.info>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com> 
	<20100301194946.188a8855@freewill.wooz.org>
	<20100301195955.3091dc19@freewill.wooz.org> 
	<201003022106.17179.steve@pearwood.info>
Message-ID: <bbaeab101003021206l3051c7afm6a7c9281b678d0f@mail.gmail.com>

On Tue, Mar 2, 2010 at 02:06, Steven D'Aprano <steve at pearwood.info> wrote:

> On Tue, 2 Mar 2010 11:59:55 am Barry Warsaw wrote:
> > Thanks everybody for providing great input on this aspect of the PEP.
> >  I've updated the open issues section to include a list of the
> > possible resolutions for bytecode-only imports.  Unless anybody has
> > more ideas, it might just be time to get a BDFL pronouncement.
>
> Please excuse me if these minor points have already been discussed, but
> I couldn't see them in the PEP.
>
> (1) What happens if the __cache__ directory doesn't exist and the
> enclosing directory is unwriteable, or if it does exist, but is
> unreadable?
>
> I expect that the byte code files will simply not be created, and
> everything will continue without them.
>
> (2) Presumably this only effects imports, not running python source code
> as a script. If I do this:
>
> python myscript.py
>
> from the shell, I would expect that no __cache__ directory will be
> created, just like today.
>
> BTW, you have some sort of automated warning in the PEP:
>
> System Message: WARNING/2 (pep-3147.txt, line 237)
> Title underline too short.
>
> http://www.python.org/dev/peps/pep-3147/#id47


It's now fixed. Barry forgot to run the Makefile for the PEPs before
checking in.  Shame! =)

-Brett




>
>
>
>
> --
> Steven D'Aprano
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100302/7de93ac8/attachment-0001.html>

From martin at v.loewis.de  Tue Mar  2 22:33:56 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 02 Mar 2010 22:33:56 +0100
Subject: [Python-Dev] Python and Windows 2000
In-Reply-To: <50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com>
References: <4B8C3453.7030507@v.loewis.de>	
	<50862ebd1003011811t5ce65f23x1a89d65faac76997@mail.gmail.com>	
	<4B8C9F50.7010709@v.loewis.de>
	<50862ebd1003020108i6ebeef69l2e7987a887438058@mail.gmail.com>
Message-ID: <4B8D8444.106@v.loewis.de>

Neil Hodgson wrote:
> Martin v. L?wis:
> 
>> See http://bugs.python.org/issue6926
>>
>> The SDK currently hides symbolic constants from us that people are
>> asking for.
> 
>    Setting the version to 0x501 (XP) doesn't actively try to stop
> running on version 0x500 (2K), it just reveals the symbols and APIs
> from 0x501. Including a call to an 0x501-specific API will then fail
> at load.
> 
>     IPPROTO_IPV6 (the cause of issue 6926) isn't a new symbol that
> started working in Windows XP - it was present in older SDKs without a
> version guard so was visible when compiling for any version of
> Windows.

I'm not proposing to actively break Windows 2000 support. I propose to
stop supporting it, which means that we will not strive for Python
running on Windows 2000, and that bug reports that report issues
specific with Windows 2000 will be closed as "won't fix".

I would not want to bump the SDK version while still aiming at Windows
2000 support - as that gets rarely tested, this API version is actually
a mechanism to simplify ongoing support for Windows 2000.

>    I don't really have a strong need for Windows 2000 although I keep
> an instance for checking compatibility of my code and I do still get
> queries from people using old versions of Windows, including 9x. There
> is the question of whether to force failure on Windows 2000 or just
> remove it from the list of known-working platforms while still
> allowing it to run.

I would propose the latter; see above. However, patches that do actively
break it would then be acceptable.

In any case, we would need a warning in the installer that Windows 2000
supports is going away.

Regards,
Martin


From martin at v.loewis.de  Tue Mar  2 22:54:25 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 02 Mar 2010 22:54:25 +0100
Subject: [Python-Dev] code.python.org dead?
In-Reply-To: <20100302103212.69669f0f@msiwind>
References: <20100302103212.69669f0f@msiwind>
Message-ID: <4B8D8911.1070003@v.loewis.de>

> It seems someone has decided that code.python.org doesn't resolve
> anymore.

Correct - I deleted the DNS record, and the site, as it didn't work
anymore (and because Barry confirmed that it is ok to delete the site)

> This is annoying since there were Mercurial mirrors there
> ( at http://code.python.org/hg ). Until we make the official switch,
> these mirrors are still useful.

Do you know who is in charge of creating the site? I could restore it,
but I would need help to make it actually work (such as giving it a
working DocumentRoot).

Regards,
Martin

From barry at python.org  Tue Mar  2 23:20:53 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 2 Mar 2010 17:20:53 -0500
Subject: [Python-Dev] code.python.org dead?
In-Reply-To: <4B8D8911.1070003@v.loewis.de>
References: <20100302103212.69669f0f@msiwind>
	<4B8D8911.1070003@v.loewis.de>
Message-ID: <20100302172053.422d058c@heresy.wooz.org>

On Mar 02, 2010, at 10:54 PM, Martin v. L?wis wrote:

>Correct - I deleted the DNS record, and the site, as it didn't work
>anymore (and because Barry confirmed that it is ok to delete the site)

Yep, sorry.  I didn't realize it was still in use.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100302/01c4870a/attachment.pgp>

From guido at python.org  Wed Mar  3 00:27:05 2010
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Mar 2010 15:27:05 -0800
Subject: [Python-Dev] Breaking bytecode only imports (was Re: __file__)
In-Reply-To: <20100302145259.1442a324@heresy.wooz.org>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com> 
	<4B89CC14.5070902@canterbury.ac.nz>
	<20100228121907.GA13564@laurie.devork> 
	<690612F5-13B4-4538-BABA-D24FB432A8CF@voidspace.org.uk>
	<4B8A6A8F.7050008@gmail.com> 
	<20100301194946.188a8855@freewill.wooz.org>
	<20100301195955.3091dc19@freewill.wooz.org> 
	<4B8D0C97.4020002@gmail.com>
	<ca471dc21003020934o3713c86cs2c9ec19f4db33046@mail.gmail.com> 
	<20100302145259.1442a324@heresy.wooz.org>
Message-ID: <ca471dc21003021527j4e445d4lca693b71d63a4ca7@mail.gmail.com>

On Tue, Mar 2, 2010 at 11:52 AM, Barry Warsaw <barry at python.org> wrote:
> On Mar 02, 2010, at 09:34 AM, Guido van Rossum wrote:
>
>>On Tue, Mar 2, 2010 at 5:03 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> P.S. I actually started this thread as a +0 to the idea of dropping
>>> bytecode only imports. Over the course of the discussion I've shifted to
>>> a firm -1 in the absence of some proper comparative benchmarks to
>>> justify the change in semantics.
>>
>>FWIW, I started at -1 and am still -1. I think the PEP is overreaching
>>in this aspect; it does not serve the stated purpose of the PEP to
>>make life easier for distros that share code between Python versions.
>
> I think that's fair, and just the guidance I'm looking for. ?By now you
> understand the pros and cons, so if this is a pronouncement, I will cement it
> into the PEP.

Yes, and thanks!

-- 
--Guido van Rossum (python.org/~guido)

From florent.xicluna at gmail.com  Wed Mar  3 00:55:25 2010
From: florent.xicluna at gmail.com (Florent Xicluna)
Date: Tue, 2 Mar 2010 23:55:25 +0000 (UTC)
Subject: [Python-Dev] Deprecation warnings in Python 2.7
Message-ID: <loom.20100303T005231-38@post.gmane.org>

Hello,

I would like to open a discussion on the meaning of deprecation warnings in 2.7.
I assume, but I may be wrong, that 2.7 will be the last version in the 2.x
branch.  With this assumption, we should not find many things deprecated in 2.7
final.

On the other hand, the list of py3k deprecation warnings is increasing a lot.
And more users will probably start to move from 2.7 to the 3.1 release.

While working on ticket #7092 and #7849, we started to discuss some proposals
to simplify the deprecation warnings for 2.7.

We discussed these 2 proposals:

 1) in 2.6 there's no distinction between py3k and normal deprecations: they
    share the same category, DeprecationWarning.  The "-3" switch enables the
    py3k DeprecationWarning and SyntaxWarning.
    Do we need to introduce a subclass Py3kDeprecationWarning in 2.7?  It will
    make it easier to separate both kinds of deprecations (2.x and 3.x), and
    to filter them.
    
 2) a different idea is to deprecate the "-3" switch and consider all py3k
    deprecations as normal deprecations.  They will be hidden by default
    thanks to #7309.  Since there's no future for the 2.x branch, it seems
    normal to consider the migration from 2.7 to 3.1 and show these warnings
    when the developer uses "-Wd" switch.

What do you expect as DeprecationWarning in 2.7?

I will post a list of the DeprecationWarnings in the python trunk, in a
followup message, for a review, and to help the discussion.

Best regards,

-- 
Florent Xicluna



From collinwinter at google.com  Wed Mar  3 01:12:48 2010
From: collinwinter at google.com (Collin Winter)
Date: Tue, 2 Mar 2010 16:12:48 -0800
Subject: [Python-Dev] Packaging JIT-less versions of Python
In-Reply-To: <1267500572.8882.351.camel@brick>
References: <3c8293b61003011535x6190351ald0640855bbfa7c4b@mail.gmail.com> 
	<1267500572.8882.351.camel@brick>
Message-ID: <3c8293b61003021612j2b76e7bcw98cc54995168c82d@mail.gmail.com>

Hey David,

On Mon, Mar 1, 2010 at 7:29 PM, David Malcolm <dmalcolm at redhat.com> wrote:
> On Mon, 2010-03-01 at 15:35 -0800, Collin Winter wrote:
[snip]
>> - How would you prefer to build the JIT-less package (current options:
>> via a ./configure flag; or by deleting _llvmjit.so from the
>> JIT-enabled package)?
>> - Would the two packages be able to exist side-by-side, or would they
>> be mutually exclusive?
>
> I have a particular interest in ABI compatibility: if turning JIT on and
> off is going to change the ABI of extension modules, that would be a
> major pain, as I hope that we will have dozens of C extension modules
> available via RPM for our Python 3 stack by the time of the "great
> unladen merger".

Do you have a good way of testing ABI compatibility, or is it just
"build modules, see if they work"? Some general way of testing ABI
compatibility would be really useful for PEP 384, too.

> So I'm keen for the ability to toggle the JIT code in the face of bugs
> and have it not affect ABI. ?"-Xjit" will do this at runtime (once
> that's renamed), but I think it would be useful to be able to toggle the
> JIT on/off default during the build, so that I can fix a broken
> architecture for non-technical users, but have individual testers opt
> back in with -Xjit whilst tracking down a major bug.

That's something we can definitely do: you'd just change the default
value for the -Xjit flag from "whenhot" to "never". Those individual
testers would pass -Xjit=whenhot to opt back in. We could make that a
./configure flag if it would be useful to you and the other distros.

> In either case, I don't want to have to recompile 30 extension modules
> to try with/without JIT; that would introduce too much change during
> bug-hunts, and be no fun at all.

That would suck indeed; I want to avoid that. I think that kind of
thing falls under PEP 384, which we will have to obey once it is
accepted.

> (In the blue-sky nirvana future, I'd love to be able to ship
> ahead-of-time compiled versions of the stdlib, pre-optimized based on
> realworld workloads. ?Back in my reality, though, I have bugs to fix
> before I can work on _that_ patch :( )

Reid Kleckner may be looking at that for his Master's project. It's
definitely doable.

>> My strong preference would be to have the JIT included by default so
>> that it receives as much testing as possible.
>
> Sounds reasonable. ?Hope the above made sense and is useful.

Thanks for your perspective,
Collin Winter

From florent.xicluna at gmail.com  Wed Mar  3 01:44:52 2010
From: florent.xicluna at gmail.com (Florent XICLUNA)
Date: Wed, 3 Mar 2010 01:44:52 +0100
Subject: [Python-Dev] Deprecation warnings in Python 2.7
In-Reply-To: <loom.20100303T005231-38@post.gmane.org>
References: <loom.20100303T005231-38@post.gmane.org>
Message-ID: <e43810331003021644h5f554715m6d72d45bf25fb66@mail.gmail.com>

2010/3/3 Florent Xicluna wrote:
>
> I will post a list of the DeprecationWarnings in the python trunk, in a
> followup message, for a review, and to help the discussion.
>

Here is the list of the "Classic" DeprecationWarnings:
http://paste.pocoo.org/show/184931/

And the list of the "Py3k" DeprecationWarnings and SyntaxWarnings:
http://paste.pocoo.org/show/184932/

I expect most of the things in the first list to be removed in 2.7
instead of being only deprecated.

Then what is the best approach:
 1) to separate better the Py3k warnings giving them a specific
subclass Py3kDeprecationWarnings
 2) to merge both lists because we consider deprecation related to 3.1
(and newer) ?
 3) to keep things unchanged?


-- 
Florent

From brett at python.org  Wed Mar  3 01:51:37 2010
From: brett at python.org (Brett Cannon)
Date: Tue, 2 Mar 2010 16:51:37 -0800
Subject: [Python-Dev] Deprecation warnings in Python 2.7
In-Reply-To: <loom.20100303T005231-38@post.gmane.org>
References: <loom.20100303T005231-38@post.gmane.org>
Message-ID: <bbaeab101003021651p3dbe4e3axcf9ecb01d43825ae@mail.gmail.com>

On Tue, Mar 2, 2010 at 15:55, Florent Xicluna <florent.xicluna at gmail.com>wrote:

> Hello,
>
> I would like to open a discussion on the meaning of deprecation warnings in
> 2.7.
> I assume, but I may be wrong, that 2.7 will be the last version in the 2.x
> branch.  With this assumption, we should not find many things deprecated in
> 2.7
> final.
>
> On the other hand, the list of py3k deprecation warnings is increasing a
> lot.
> And more users will probably start to move from 2.7 to the 3.1 release.
>
> While working on ticket #7092 and #7849, we started to discuss some
> proposals
> to simplify the deprecation warnings for 2.7.
>
> We discussed these 2 proposals:
>
>  1) in 2.6 there's no distinction between py3k and normal deprecations:
> they
>    share the same category, DeprecationWarning.  The "-3" switch enables
> the
>    py3k DeprecationWarning and SyntaxWarning.
>    Do we need to introduce a subclass Py3kDeprecationWarning in 2.7?  It
> will
>    make it easier to separate both kinds of deprecations (2.x and 3.x), and
>    to filter them.
>
>  2) a different idea is to deprecate the "-3" switch and consider all py3k
>    deprecations as normal deprecations.  They will be hidden by default
>    thanks to #7309.  Since there's no future for the 2.x branch, it seems
>    normal to consider the migration from 2.7 to 3.1 and show these warnings
>    when the developer uses "-Wd" switch.
>
> What do you expect as DeprecationWarning in 2.7?
>

I like option 2; we are serious about trying to make 2.7 the last in the 2.x
series, which means if you care about deprecations then you are caring about
Python 3. So I say make -3 the equivalent of -Wd in Python 2.7

-Brett



>
> I will post a list of the DeprecationWarnings in the python trunk, in a
> followup message, for a review, and to help the discussion.
>
> Best regards,
>
> --
> Florent Xicluna
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100302/6a6a4370/attachment-0001.html>

From raymond.hettinger at gmail.com  Wed Mar  3 02:28:11 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 2 Mar 2010 17:28:11 -0800
Subject: [Python-Dev] Deprecation warnings in Python 2.7
In-Reply-To: <bbaeab101003021651p3dbe4e3axcf9ecb01d43825ae@mail.gmail.com>
References: <loom.20100303T005231-38@post.gmane.org>
	<bbaeab101003021651p3dbe4e3axcf9ecb01d43825ae@mail.gmail.com>
Message-ID: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com>


> 
> On Tue, Mar 2, 2010 at 15:55, Florent Xicluna <florent.xicluna at gmail.com> wrote:
> Hello,
> 
> I would like to open a discussion on the meaning of deprecation warnings in 2.7.
> I assume, but I may be wrong, that 2.7 will be the last version in the 2.x
> branch.  With this assumption, we should not find many things deprecated in 2.7
> final.
> 
> On the other hand, the list of py3k deprecation warnings is increasing a lot.
> And more users will probably start to move from 2.7 to the 3.1 release.
> 
> While working on ticket #7092 and #7849, we started to discuss some proposals
> to simplify the deprecation warnings for 2.7.
> 
> We discussed these 2 proposals:
> 
>  1) in 2.6 there's no distinction between py3k and normal deprecations: they
>    share the same category, DeprecationWarning.  The "-3" switch enables the
>    py3k DeprecationWarning and SyntaxWarning.
>    Do we need to introduce a subclass Py3kDeprecationWarning in 2.7?  It will
>    make it easier to separate both kinds of deprecations (2.x and 3.x), and
>    to filter them.
> 
>  2) a different idea is to deprecate the "-3" switch and consider all py3k
>    deprecations as normal deprecations. 

-1 for several reasons.  

1) We've advertised -3 as part of TheOneTrueWay(tm).  It's a core part of our transition story, so we should keep that as clean/consistent as possible.  Deprecating the -3 switch seems like shooting ourselves in the foot.

2) There is some chance that there will be a 2.8, so it isn't helpful to burn down our bridges.

ISTM, nothing is currently broken and in need of "fixing" in 2.7.
I don't see any advantage to conflating py3k warnings with other deprecations.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100302/98db9096/attachment.html>

From daniel at stutzbachenterprises.com  Wed Mar  3 03:54:20 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Tue, 2 Mar 2010 20:54:20 -0600
Subject: [Python-Dev] Caching function pointers in type objects
Message-ID: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com>

In CPython, is it safe to cache function pointers that are in type objects?


For example, if I know that some_type->tp_richcompare is non-NULL, and I
call it (which may execute arbitrary user code), can I assume that
some_type->tp_richcompare is still non-NULL?
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100302/d4bb381b/attachment.html>

From benjamin at python.org  Wed Mar  3 03:57:31 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 2 Mar 2010 20:57:31 -0600
Subject: [Python-Dev] Caching function pointers in type objects
In-Reply-To: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com>
References: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com>
Message-ID: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com>

2010/3/2 Daniel Stutzbach <daniel at stutzbachenterprises.com>:
> In CPython, is it safe to cache function pointers that are in type objects?
>
> For example, if I know that some_type->tp_richcompare is non-NULL, and I
> call it (which may execute arbitrary user code), can I assume that
> some_type->tp_richcompare is still non-NULL?

Not unless it's builtin. Somebody could have deleted the rich
comparison methods.



-- 
Regards,
Benjamin

From rnk at mit.edu  Wed Mar  3 04:06:41 2010
From: rnk at mit.edu (Reid Kleckner)
Date: Tue, 2 Mar 2010 22:06:41 -0500
Subject: [Python-Dev] Caching function pointers in type objects
In-Reply-To: <1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com>
References: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com> 
	<1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com>
Message-ID: <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com>

I don't think this will help you solve your problem, but one thing
we've done in unladen swallow is to hack PyType_Modified to invalidate
our own descriptor caches.  We may eventually want to extend that into
a callback interface, but it probably will never be considered an API
that outside code should depend on.

Reid

On Tue, Mar 2, 2010 at 9:57 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2010/3/2 Daniel Stutzbach <daniel at stutzbachenterprises.com>:
>> In CPython, is it safe to cache function pointers that are in type objects?
>>
>> For example, if I know that some_type->tp_richcompare is non-NULL, and I
>> call it (which may execute arbitrary user code), can I assume that
>> some_type->tp_richcompare is still non-NULL?
>
> Not unless it's builtin. Somebody could have deleted the rich
> comparison methods.
>
>
>
> --
> Regards,
> Benjamin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/reid.kleckner%40gmail.com
>

From barry at python.org  Wed Mar  3 04:17:44 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 2 Mar 2010 22:17:44 -0500
Subject: [Python-Dev] Python 2.6.5 release candidate 1 now available
Message-ID: <20100302221744.7ee4d3dd@heresy.wooz.org>

Hello everyone,

The source tarballs and Windows installer for Python 2.6.5 release candidate 1
are now available:

http://www.python.org/download/releases/2.6.5/

Please download them, install them, and try to use them with your favorite
projects and environments.  If no regressions are found, we will do the final
release on Monday March 15, 2010.  Please test the release candidate as
much as possible in the meantime, and help make 2.6.5 a rock solid release!

Thanks,
-Barry

P.S. The Mac installer will hopefully be available soon.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100302/467fcbe0/attachment.pgp>

From solipsis at pitrou.net  Wed Mar  3 04:17:13 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 2 Mar 2010 22:17:13 -0500
Subject: [Python-Dev] code.python.org dead?
In-Reply-To: <4B8D8911.1070003@v.loewis.de>
References: <20100302103212.69669f0f@msiwind>
	<4B8D8911.1070003@v.loewis.de>
Message-ID: <20100302221713.5882549b@msiwind>

Le Tue, 02 Mar 2010 22:54:25 +0100,
"Martin v. L?wis" <martin at v.loewis.de> a ?crit :
> 
> > This is annoying since there were Mercurial mirrors there
> > ( at http://code.python.org/hg ). Until we make the official switch,
> > these mirrors are still useful.
> 
> Do you know who is in charge of creating the site? I could restore it,
> but I would need help to make it actually work (such as giving it a
> working DocumentRoot).

I actually sent a template of the actual Apache configuration long ago,
to Neil Schemenauer I think (my memory may be failing on that).

I normally have ssh access to the machine using the "svn2hg" account,
but I don't think I can actually use that account until I return from
vacation - that is, until mid-March. The setup was a quite standard
hgwebdir wsgi script hooked into mod_wsgi.

Also, if the mirrors themselves have been wiped as well, they will have
to be regenerated (hopefully yielding the same changeset ids :-)).

Regards

Antoine.

From solipsis at pitrou.net  Wed Mar  3 04:24:00 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 2 Mar 2010 22:24:00 -0500
Subject: [Python-Dev] Deprecation warnings in Python 2.7
References: <loom.20100303T005231-38@post.gmane.org>
	<bbaeab101003021651p3dbe4e3axcf9ecb01d43825ae@mail.gmail.com>
	<2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com>
Message-ID: <20100302222400.2e18ccef@msiwind>

Le Tue, 2 Mar 2010 17:28:11 -0800,
Raymond Hettinger <raymond.hettinger at gmail.com> a ?crit :
> -1 for several reasons.  
> 
> 1) We've advertised -3 as part of TheOneTrueWay(tm).  It's a core
> part of our transition story, so we should keep that as
> clean/consistent as possible.  Deprecating the -3 switch seems like
> shooting ourselves in the foot.
> 
> 2) There is some chance that there will be a 2.8, so it isn't helpful
> to burn down our bridges.

That's my feeling too. I think introducing a Py3kDeprecationWarning
subclass is the right thing to do.

Regards

Antoine.



From benjamin at python.org  Wed Mar  3 05:07:36 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 2 Mar 2010 22:07:36 -0600
Subject: [Python-Dev] Deprecation warnings in Python 2.7
In-Reply-To: <20100302222400.2e18ccef@msiwind>
References: <loom.20100303T005231-38@post.gmane.org>
	<bbaeab101003021651p3dbe4e3axcf9ecb01d43825ae@mail.gmail.com>
	<2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com>
	<20100302222400.2e18ccef@msiwind>
Message-ID: <1afaf6161003022007q7f723630n555f0834dea33096@mail.gmail.com>

2010/3/2 Antoine Pitrou <solipsis at pitrou.net>:
> Le Tue, 2 Mar 2010 17:28:11 -0800,
> Raymond Hettinger <raymond.hettinger at gmail.com> a ?crit :
>> -1 for several reasons.
>>
>> 1) We've advertised -3 as part of TheOneTrueWay(tm). ?It's a core
>> part of our transition story, so we should keep that as
>> clean/consistent as possible. ?Deprecating the -3 switch seems like
>> shooting ourselves in the foot.
>>
>> 2) There is some chance that there will be a 2.8, so it isn't helpful
>> to burn down our bridges.
>
> That's my feeling too. I think introducing a Py3kDeprecationWarning
> subclass is the right thing to do.

Not to bikeshed, but I think it should be called Py3DeprecationWarning
in light of the fact that Python 3 is no longer a mythical beast. :)



-- 
Regards,
Benjamin

From stefan_ml at behnel.de  Wed Mar  3 08:18:08 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Mar 2010 08:18:08 +0100
Subject: [Python-Dev] ElementTree.tostring() returning str by default in
	Py3.2
In-Reply-To: <hmikdu$bi0$1@dough.gmane.org>
References: <hmikdu$bi0$1@dough.gmane.org>
Message-ID: <hml2fh$c3e$1@dough.gmane.org>

Stefan Behnel, 02.03.2010 10:06:
> I just noticed that the xml.etree.ElementTree.tostring() function in Py3.2
> returns a str object by default, unless an encoding is specified. This is a
> backwards incompatible change compared to ET 1.2. For one, it breaks tons
> of tests in lxml's compatibility test suite. Previously, the default
> encoding was 'ASCII', and tostring() was guaranteed to always return a byte
> string.
>
> According to svn blame, this was changed by GvR in rev. 56841.
>
> Is there any reasoning behind this?

I filed a bug report for now.

http://bugs.python.org/issue8047

Stefan


From ncoghlan at gmail.com  Wed Mar  3 11:35:33 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 03 Mar 2010 20:35:33 +1000
Subject: [Python-Dev] Deprecation warnings in Python 2.7
In-Reply-To: <20100302222400.2e18ccef@msiwind>
References: <loom.20100303T005231-38@post.gmane.org>	<bbaeab101003021651p3dbe4e3axcf9ecb01d43825ae@mail.gmail.com>	<2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com>
	<20100302222400.2e18ccef@msiwind>
Message-ID: <4B8E3B75.1000004@gmail.com>

Antoine Pitrou wrote:
> Le Tue, 2 Mar 2010 17:28:11 -0800,
> Raymond Hettinger <raymond.hettinger at gmail.com> a ?crit :
>> -1 for several reasons.  
>>
>> 1) We've advertised -3 as part of TheOneTrueWay(tm).  It's a core
>> part of our transition story, so we should keep that as
>> clean/consistent as possible.  Deprecating the -3 switch seems like
>> shooting ourselves in the foot.
>>
>> 2) There is some chance that there will be a 2.8, so it isn't helpful
>> to burn down our bridges.
> 
> That's my feeling too. I think introducing a Py3kDeprecationWarning
> subclass is the right thing to do.

Agreed for the reasons Raymond cited. As far as the subclass goes, I
agree that's a good idea to allow people to readily detect whether they
tripped over a normal deprecation or a Py3 only deprecation when running
with -3 enabled.

As far as naming goes, I agree with Py3kDeprecationWarning since the
flag to enable them is called sys.py3kwarning.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From florent.xicluna at gmail.com  Wed Mar  3 11:39:01 2010
From: florent.xicluna at gmail.com (Florent Xicluna)
Date: Wed, 3 Mar 2010 11:39:01 +0100
Subject: [Python-Dev] Deprecation warnings in Python 2.7
In-Reply-To: <2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com>
References: <loom.20100303T005231-38@post.gmane.org>
	<bbaeab101003021651p3dbe4e3axcf9ecb01d43825ae@mail.gmail.com>
	<2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com>
Message-ID: <e43810331003030239x34fcd7e7j1f5270d7ebffd67a@mail.gmail.com>

2010/3/3 Raymond Hettinger <raymond.hettinger at gmail.com>:
>
> -1 for several reasons.
> 1) We've advertised -3 as part of TheOneTrueWay(tm). ?It's a core part of
> our transition story, so we should keep that as clean/consistent as
> possible. ?Deprecating the -3 switch seems like shooting ourselves in the
> foot.

Instead of deprecating the -3 switch, we can change it to a useful
alias (see below).

> 2) There is some chance that there will be a 2.8, so it isn't helpful to
> burn down our bridges.
> ISTM, nothing is currently broken and in need of "fixing" in 2.7.
> I don't see any advantage to conflating py3k warnings with other
> deprecations.
>

IMHO, the current deprecation and warning mechanism is not far from a
Rube Goldberg machine.
There's many options available, and it's not obvious both for the core
developer and for the end user.

On the python interpreter side, you have many options for the careful developer:
 -Wdefault: display all warnings
 -3:        display all warnings except ImportWarning and
PendingDeprecationWarning
 -Qwarn     does nothing if -Wd is omitted, else warn about 2/1 or 2L/1
 -Qwarnall  does nothing if -Wd is omitted, else warn about 2.0/1 or 2j/1
 -b/-bb     (undocumented), warns about u'' == bytearray('')
 -t/-tt     warns about tab inconsistencies

Why -Qwarn needs -Wd to show its warnings?
And you should know that -3 implies -Qwarn, but it still mask
PendingDeprecationWarnings.
So you will not see the PendingDeprecationWarning issued by
cgi.parse_qs or shutil module (for example).

At the same time, you could notice that the mhlib module yields a
non-py3k DeprecationWarning about its removal in 3.0.

And now you want to compare unicode with bytes (ok, it's a very bad
idea, but it may happen when you sort dictionary keys, for example):

~ $ ./python

>>> u'\xff' == bytearray('\xff')    # No warning?
False
>>> u'\xff' == '\xff'
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert
both arguments to Unicode - interpreting them as being unequal
False

~ $ ./python -Wd -bb

>>> u'\xff' == bytearray('\xff')    # It should raise an error because of '-bb'
__main__:1: BytesWarning: Comparison between bytearray and string
False
>>> u'\xff' == '\xff'
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert
both arguments to Unicode - interpreting them as being unequal
False

Yeah, it may be confusing... "Comparison between bytearray and string"
is not really the correct message in 2.7. And why to keep these 2
warnings categories which are not used anywhere else in Python source
code?  The behavior should be consistent between these 2 warnings, and
they may share the same category. Why we don't use a RuntimeWarning
here?

I still support the simplification of the warnings mechanism:
 * -Qwarn/-Qwarnall should imply
    -Wdefault -Wignore::ImportWarning -Wignore::PendingDeprecationWarning
    (same as current -3 behavior)
 * BytesWarning should be replaced by UnicodeWarning or RuntimeWarning
(and -b deprecated)
 * -Wdefault should show all warnings (including py3k warnings)
 * -3 should show the PendingDeprecationWarning in addition to its
current behavior.
   (i.e. an alias for -Wdefault -Wignore::ImportWarning)

-- 
Florent

From cournape at gmail.com  Wed Mar  3 11:51:54 2010
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 3 Mar 2010 19:51:54 +0900
Subject: [Python-Dev] PEP 3188: Implementation Questions
In-Reply-To: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com>
References: <4095897c1002252051w24377ec2tc60d0b04adee8c5d@mail.gmail.com>
Message-ID: <5b8d13221003030251r52ecc55do3ae7e9c773b8c4c9@mail.gmail.com>

On Fri, Feb 26, 2010 at 1:51 PM, Meador Inge <meadori at gmail.com> wrote:
> Hi All,
>
> Recently some discussion began in the issue 3132 thread
> (http://bugs.python.org/issue3132) regarding
> implementation of the new struct string syntax for PEP 3118.? Mark Dickinson
> suggested that I bring the discussion on over to Python Dev.? Below is a
> summary
> of the questions\comments from the thread.
>
> Unpacking a long-double
> ===================
>
> 1. Should this return a Decimal object or a ctypes 'long double'?
> 2. Using ctypes 'long double' is easier to implement, but precision is
> ??? lost when needing to do arithmetic, since the value for cytpes 'long
> double'
> ??? is converted to a Python float.
> 3. Using Decimal keeps the desired precision, but the implementation would
> ??? be non-trivial and architecture specific (unless we just picked a
> ??? fixed number of bytes regardless of the architecture).
> 4. What representation should be used for standard size and alignment?
> ??? IEEE 754 extended double precision?

I think supporting even basic arithmetic correctly for long double
would be a tremendous amount of work in python. First, as you know,
there are many different formats which depend not only on the CPU but
also on the OS and the compiler, but there are quite a few issues
which are specific to long double (like converting to an integer which
cannot fit in any C integer type on most implementations).

Also, IEEE 754  does not define any alignment as far as I know, that's
up to the CPU implementer I believe. In Numpy, long double usually
maps to either 12 bytes (np.float96) or 16 bytes (np.float128).

I would expect the long double to be mostly useful for data exchange -
if you want to do arithmetic on long double, then the user of the
buffer protocol would have to implement it by himself (like NumPy does
ATM). So the important thing is to have enough information to use the
long double: alignment and size are not enough.

cheers,

David

From florent.xicluna at gmail.com  Wed Mar  3 12:00:05 2010
From: florent.xicluna at gmail.com (Florent Xicluna)
Date: Wed, 3 Mar 2010 12:00:05 +0100
Subject: [Python-Dev] Deprecation warnings in Python 2.7
In-Reply-To: <e43810331003030239x34fcd7e7j1f5270d7ebffd67a@mail.gmail.com>
References: <loom.20100303T005231-38@post.gmane.org>
	<bbaeab101003021651p3dbe4e3axcf9ecb01d43825ae@mail.gmail.com>
	<2DFA312B-72C0-4EC6-B703-4076F7C945F7@gmail.com>
	<e43810331003030239x34fcd7e7j1f5270d7ebffd67a@mail.gmail.com>
Message-ID: <e43810331003030300tf025114lf0e9e2caed790ae4@mail.gmail.com>

2010/3/3 Florent Xicluna <florent.xicluna at gmail.com>:
>
>>>> u'\xff' == bytearray('\xff') ? ?# It should raise an error because of '-bb'
> __main__:1: BytesWarning: Comparison between bytearray and string
> False
>>>> u'\xff' == '\xff'
> __main__:1: UnicodeWarning: Unicode equal comparison failed to convert
> both arguments to Unicode - interpreting them as being unequal
> False

I see that the BytesWarning (and -b option) is a 3.x feature.
I don't see the reason to keep it in 2.x, though....

On the other side, UnicodeWarning is unused in 3.x, why we keep it
around? We may phase it out in 3.2?

-- 
Florent

From martin at v.loewis.de  Wed Mar  3 20:43:58 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 03 Mar 2010 20:43:58 +0100
Subject: [Python-Dev] code.python.org dead?
In-Reply-To: <20100302103212.69669f0f@msiwind>
References: <20100302103212.69669f0f@msiwind>
Message-ID: <4B8EBBFE.40401@v.loewis.de>

> It seems someone has decided that code.python.org doesn't resolve
> anymore.

I have now restored it.

Regards,
Martin

From henning.vonbargen at arcor.de  Wed Mar  3 21:30:34 2010
From: henning.vonbargen at arcor.de (Henning von Bargen)
Date: Wed, 03 Mar 2010 21:30:34 +0100
Subject: [Python-Dev] __file__
In-Reply-To: <mailman.91.1267392743.23597.python-dev@python.org>
References: <mailman.91.1267392743.23597.python-dev@python.org>
Message-ID: <4B8EC6EA.3050603@arcor.de>

Nick Coghlan wrote:
> Another option is to remove bytecode-only support from the default
> filesystem importer, but keep it for zipimport (since the stat call
> savings don't apply in the latter case).

+1

Baptiste Carvello wrote:

> However, making a difference between zipimport and the filesystem importer means 
> the application will stop working if I unzip the library zip file, which is 
> surprising.
 > Unzipping the zip file can be handy when debugging a bug caused by a
> forgotten module.

If the ZIP contains only bytecode files, it is just not intended
for changing any code, so I don't think this is an argument.
If you have access to the source code, you still can use that instead
of messing around with byte code.

Henning


From daniel at stutzbachenterprises.com  Wed Mar  3 22:24:08 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Wed, 3 Mar 2010 15:24:08 -0600
Subject: [Python-Dev] Caching function pointers in type objects
In-Reply-To: <9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com>
References: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com>
	<1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com>
	<9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com>
Message-ID: <eae285401003031324vf7bef21x6fcaa626e6412378@mail.gmail.com>

On Tue, Mar 2, 2010 at 9:06 PM, Reid Kleckner <rnk at mit.edu> wrote:

> I don't think this will help you solve your problem, but one thing
> we've done in unladen swallow is to hack PyType_Modified to invalidate
> our own descriptor caches.  We may eventually want to extend that into
> a callback interface, but it probably will never be considered an API
> that outside code should depend on.
>

Thanks Reid and Benjamin for the information.

I think I see a way to dramatically speed up PyObject_RichCompareBool when
comparing immutable, built-in, non-container objects (int, float, str,
etc.).  It would speed up list.sort when the key is one of those types, as
well as most operations on the ubiquitous dictionary with str keys.

Is that a worthwhile avenue to pursue, or is it likely to be redundant with
Unladen Swallow's optimizations?

If I can find time to pursue it, would it be best for me to implement it as
a patch to Unladen Swallow, CPython trunk, or CPython py3k?
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100303/5d0ffed2/attachment.html>

From benjamin at python.org  Wed Mar  3 22:29:25 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 3 Mar 2010 15:29:25 -0600
Subject: [Python-Dev] Caching function pointers in type objects
In-Reply-To: <eae285401003031324vf7bef21x6fcaa626e6412378@mail.gmail.com>
References: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com>
	<1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com>
	<9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com>
	<eae285401003031324vf7bef21x6fcaa626e6412378@mail.gmail.com>
Message-ID: <1afaf6161003031329t2574e8a9qf5fef6fb4b0f33e4@mail.gmail.com>

2010/3/3 Daniel Stutzbach <daniel at stutzbachenterprises.com>:
> I think I see a way to dramatically speed up PyObject_RichCompareBool when
> comparing immutable, built-in, non-container objects (int, float, str,
> etc.).? It would speed up list.sort when the key is one of those types, as
> well as most operations on the ubiquitous dictionary with str keys.
>
> Is that a worthwhile avenue to pursue, or is it likely to be redundant with
> Unladen Swallow's optimizations?

Perhaps you could explain what exactly you want to do. :) That would
help us make a judgment.

>
> If I can find time to pursue it, would it be best for me to implement it as
> a patch to Unladen Swallow, CPython trunk, or CPython py3k?

Your choice.


-- 
Regards,
Benjamin

From python at mrabarnett.plus.com  Wed Mar  3 22:34:14 2010
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 03 Mar 2010 21:34:14 +0000
Subject: [Python-Dev] Matching multiple regexes
Message-ID: <4B8ED5D6.4060206@mrabarnett.plus.com>

I've thought of a possible additional feature for my regex module and
I'm looking for opinions.

Occasionally there's a question about matching multiple regexes and
someone might suggest combining them into one regex using "|".

The feature would be to allow regex.compile, etc, to accept a list or
tuple of regex strings. These would be combined into a single regex
internally. The match object would get a new attribute .index which
would give the index of the regex string which matched.

An additional feature could be to allow regex.sub to accept as a
replacement any object which has the __getitem__ method and call that
method with the match index, if multiple regexes were used. (A
replacement string would still behave the same.)

This could mean that the following code would become possible:

     >>> subs = [("foo", "bar"), ("baz", "quux"), ("quuux", "foo")]
     >>> old_string, new_string = zip(*subs)
     >>> s = "fooxxxbazyyyquuux"
     >>> regex.sub(old_string, new_string, s)
     "barxxxquuxyyyfoo"


From daniel at stutzbachenterprises.com  Wed Mar  3 22:57:54 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Wed, 3 Mar 2010 15:57:54 -0600
Subject: [Python-Dev] Caching function pointers in type objects
In-Reply-To: <1afaf6161003031329t2574e8a9qf5fef6fb4b0f33e4@mail.gmail.com>
References: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com>
	<1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com>
	<9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com>
	<eae285401003031324vf7bef21x6fcaa626e6412378@mail.gmail.com>
	<1afaf6161003031329t2574e8a9qf5fef6fb4b0f33e4@mail.gmail.com>
Message-ID: <eae285401003031357t73a1c17fp4028e644cd7fbe7a@mail.gmail.com>

On Wed, Mar 3, 2010 at 3:29 PM, Benjamin Peterson <benjamin at python.org>wrote:

> 2010/3/3 Daniel Stutzbach <daniel at stutzbachenterprises.com>:
> > I think I see a way to dramatically speed up PyObject_RichCompareBool
> when
> > comparing immutable, built-in, non-container objects (int, float, str,
> > etc.).  It would speed up list.sort when the key is one of those types,
> as
> > well as most operations on the ubiquitous dictionary with str keys.
>

(correcting myself)  I just noticed that CPython already optimizes
dictionaries with str-only keys and skips PyObject_RichCompareBool, so no
speed up there.  I think it would be redundant with optimization I have in
mind, so it could perhaps be taken out which would simplify the dict code a
bit and save a pointer in the dict structure.


> Perhaps you could explain what exactly you want to do. :) That would
> help us make a judgment.
>

It'd actually be a pretty small patch, so I think I should just explain it
by actually writing it. ;-)
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100303/2408026d/attachment-0001.html>

From fdrake at acm.org  Wed Mar  3 23:15:37 2010
From: fdrake at acm.org (Fred Drake)
Date: Wed, 3 Mar 2010 17:15:37 -0500
Subject: [Python-Dev] Release blocker: urllib2 issue 7540
Message-ID: <9cee7ab81003031415p717420f3j89ebf1116a76f790@mail.gmail.com>

Regarding:  http://bugs.python.org/issue7540

The change made in response to this issue report breaks existing uses
of Python 2.6.4.

The mechanize library frequently re-initializes the data in the
request without re-using the request.  This worked fine in the past,
and does not trigger the problem reported without the request being
re-used.  Applications (including tests) that use mechanize now break
with this TypeError.  This includes much (if not all) of the Zope
community, which uses mechanize via zope.testbrowser.

The proper response to this issue for Python 2.6 is to make no code
changes (though a documentation enhancement may be in order).

This change should be reverted from all branches.

Python 2.6.5 should be blocked until this is done.


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller

From benjamin at python.org  Wed Mar  3 23:23:11 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 3 Mar 2010 16:23:11 -0600
Subject: [Python-Dev] [Pydotorg] FWD: Broken link in 2.6.5 rc 1 download
	page.
In-Reply-To: <4B8EC488.10104@python.org>
References: <4B8EC488.10104@python.org>
Message-ID: <1afaf6161003031423q66d9babel67bac9b9282b79ca@mail.gmail.com>

2010/3/3 Michael Foord <mfoord at python.org>
>
> As far as I can tell there is no Python 2.6.5 documentation online and the link to 3.1 docs is actually for 3.1.1 whilst the link for 3.1.1 docs is dead.

I've now fixed the 3.1.1 docs link.



--
Regards,
Benjamin

From collinwinter at google.com  Wed Mar  3 23:34:44 2010
From: collinwinter at google.com (Collin Winter)
Date: Wed, 3 Mar 2010 14:34:44 -0800
Subject: [Python-Dev] Caching function pointers in type objects
In-Reply-To: <eae285401003031324vf7bef21x6fcaa626e6412378@mail.gmail.com>
References: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com> 
	<1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> 
	<9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> 
	<eae285401003031324vf7bef21x6fcaa626e6412378@mail.gmail.com>
Message-ID: <3c8293b61003031434t5de492f7n368c92767e28376e@mail.gmail.com>

Hey Daniel,

On Wed, Mar 3, 2010 at 1:24 PM, Daniel Stutzbach
<daniel at stutzbachenterprises.com> wrote:
> On Tue, Mar 2, 2010 at 9:06 PM, Reid Kleckner <rnk at mit.edu> wrote:
>>
>> I don't think this will help you solve your problem, but one thing
>> we've done in unladen swallow is to hack PyType_Modified to invalidate
>> our own descriptor caches. ?We may eventually want to extend that into
>> a callback interface, but it probably will never be considered an API
>> that outside code should depend on.
>
> Thanks Reid and Benjamin for the information.
>
> I think I see a way to dramatically speed up PyObject_RichCompareBool when
> comparing immutable, built-in, non-container objects (int, float, str,
> etc.).? It would speed up list.sort when the key is one of those types, as
> well as most operations on the ubiquitous dictionary with str keys.

That definitely sounds worth pursuing.

> Is that a worthwhile avenue to pursue, or is it likely to be redundant with
> Unladen Swallow's optimizations?

I don't believe it will be redundant with the optimizations in Unladen Swallow.

> If I can find time to pursue it, would it be best for me to implement it as
> a patch to Unladen Swallow, CPython trunk, or CPython py3k?

I would recommend patching py3k, with a backport to trunk.

Thanks,
Collin Winter

From daniel at stutzbachenterprises.com  Wed Mar  3 23:41:50 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Wed, 3 Mar 2010 16:41:50 -0600
Subject: [Python-Dev] Caching function pointers in type objects
In-Reply-To: <3c8293b61003031434t5de492f7n368c92767e28376e@mail.gmail.com>
References: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com>
	<1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com>
	<9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com>
	<eae285401003031324vf7bef21x6fcaa626e6412378@mail.gmail.com>
	<3c8293b61003031434t5de492f7n368c92767e28376e@mail.gmail.com>
Message-ID: <eae285401003031441tcc0122ex4b3fa467068c2ba9@mail.gmail.com>

On Wed, Mar 3, 2010 at 4:34 PM, Collin Winter <collinwinter at google.com>wrote:

> I would recommend patching py3k, with a backport to trunk.
>

After thinking it over, I'm inclined to patch trunk, so I can run the
Unladen Swallow macro-benchmarks, then forward-port to py3k.

I'm correct in understanding that it will be a while before the Unladen
Swallow benchmarks can support Python 3, right?
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100303/81308009/attachment.html>

From mfoord at python.org  Wed Mar  3 21:20:24 2010
From: mfoord at python.org (Michael Foord)
Date: Wed, 03 Mar 2010 20:20:24 +0000
Subject: [Python-Dev] FWD: Broken link in 2.6.5 rc 1 download page.
Message-ID: <4B8EC488.10104@python.org>

As far as I can tell there is no Python 2.6.5 documentation online and 
the link to 3.1 docs is actually for 3.1.1 whilst the link for 3.1.1 
docs is dead.

The actual documentation doesn't seem to be in the pydotorg svn 
repository so I guess this a release manager issue?

Michael Foord

-------- Original Message --------
Subject: 	[Pydotorg] FWD: Broken link in 2.6.5 rc 1 download page.
Date: 	Wed, 3 Mar 2010 08:45:36 -0800
From: 	Aahz <aahz at pythoncraft.com>
Organization: 	The Cat & Dragon
To: 	pydotorg at python.org



----- Forwarded message from Simon de Vlieger<simon at ikanobori.jp>  -----

>  From: Simon de Vlieger<simon at ikanobori.jp>
>  To: webmaster at python.org
>  Subject: Broken link in 2.6.5 rc 1 download page.
>  Date: Wed, 3 Mar 2010 09:57:29 +0100
>
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>
>  Dear webmaster (I know no name),
>
>  on this page: http://www.python.org/download/releases/2.6.5/ both links
>  for the documentation are incorrect. On this page:
>  http://www.python.org/doc/versions/ the link for the Python 3.1.1 is
>  broken.
>
>  Hope to have informed you well enough.
>
>  Regards,
>
>  Simon de Vlieger
>  -----BEGIN PGP SIGNATURE-----
>  Version: GnuPG v1.4.9 (Darwin)
>
>  iQIcBAEBAgAGBQJLjiR5AAoJEBBSHP7i+JXfidsP/0fLCK0sn5+mj9CPsobDant7
>  1tkXK7pwrbujXhpAyhjhE/kB5BhNI/h9XpmmUkLSZzz7AoOpE6KzlqCzOmwBmdAk
>  6LxO2c4ZojICVB5zgNYXibDtnSwbBhcefbEsN6BSh1kI/3ThcIdWRYFavBAlxj1T
>  6P7y8MOHUn3INLEw8foxV20geivcqzmMhRtfglMVs9KcB/PZWrK8fWw8K2Ov0B+U
>  ihxaywfiJWcdkFKTTLNvQCbVr6ZGke728/7/lcWLvdPlwRPfVteKlLDglWX0beBy
>  tgQyxmtxFx+AeCiLeyybnxEnxRVgU1nYxsdWluJuwFT4l+/BsJG3CkM522rQ3MWZ
>  +DYB6xfONlvfCSXaPn3fOYesEvXek9BxKLiXpjWlJsmXvDcdFN2sr24aR1cJEXL8
>  ZRkiJO5VFzngIxytk0Oy/SeMkxCUtsq4TMlMT2OyxblxxRoLikPSsfgKRuDkQ5s+
>  QK2yNxTgKs2yA5ux9kmZscmVgHcWqJFjkE72dJoARyHMra9Jvcr2U7GVE2zqcOCS
>  rObaqq61TpZmvlPUXlkLUueTUI9Djz01hAQUp+Vcwu8omVlIgZC0GbuXK8JDfQJd
>  60WFI1BvUQg7mDB1ThUU6HUE9rF71921lx02YyZFzGFn1K2/w/bRhxrrzD5XwJDQ
>  fFwYKj+pwgHO5AU5hMCR
>  =rTPq
>  -----END PGP SIGNATURE-----

----- End forwarded message -----

-- 
Aahz (aahz at pythoncraft.com)<*>          http://www.pythoncraft.com/

"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer
_______________________________________________
Pydotorg mailing list
Pydotorg at python.org
http://mail.python.org/mailman/listinfo/pydotorg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100303/9e4e71bf/attachment-0001.html>

From collinwinter at google.com  Wed Mar  3 23:50:02 2010
From: collinwinter at google.com (Collin Winter)
Date: Wed, 3 Mar 2010 14:50:02 -0800
Subject: [Python-Dev] Caching function pointers in type objects
In-Reply-To: <eae285401003031441tcc0122ex4b3fa467068c2ba9@mail.gmail.com>
References: <eae285401003021854i3f77364fg59ad06fc813bf697@mail.gmail.com> 
	<1afaf6161003021857k11bf4109r90e85f36dd97c49b@mail.gmail.com> 
	<9a9942201003021906m22728b35p57608f199ee1f79d@mail.gmail.com> 
	<eae285401003031324vf7bef21x6fcaa626e6412378@mail.gmail.com> 
	<3c8293b61003031434t5de492f7n368c92767e28376e@mail.gmail.com> 
	<eae285401003031441tcc0122ex4b3fa467068c2ba9@mail.gmail.com>
Message-ID: <3c8293b61003031450p4525e308qb7fdca502d90728a@mail.gmail.com>

On Wed, Mar 3, 2010 at 2:41 PM, Daniel Stutzbach
<daniel at stutzbachenterprises.com> wrote:
> On Wed, Mar 3, 2010 at 4:34 PM, Collin Winter <collinwinter at google.com>
> wrote:
>>
>> I would recommend patching py3k, with a backport to trunk.
>
> After thinking it over, I'm inclined to patch trunk, so I can run the
> Unladen Swallow macro-benchmarks, then forward-port to py3k.
>
> I'm correct in understanding that it will be a while before the Unladen
> Swallow benchmarks can support Python 3, right?

That's correct; porting the full benchmark suite to Python 3 will
require projects like Django to support Python 3.

Collin Winter

From barry at python.org  Thu Mar  4 00:04:51 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 3 Mar 2010 18:04:51 -0500
Subject: [Python-Dev] __file__
In-Reply-To: <201003022106.17179.steve@pearwood.info>
References: <bbaeab101002031242g2b3d410aqbb790859006cf11b@mail.gmail.com>
	<20100301194946.188a8855@freewill.wooz.org>
	<20100301195955.3091dc19@freewill.wooz.org>
	<201003022106.17179.steve@pearwood.info>
Message-ID: <20100303180451.61929d09@heresy.wooz.org>

On Mar 02, 2010, at 09:06 PM, Steven D'Aprano wrote:

>(1) What happens if the __cache__ directory doesn't exist and the 
>enclosing directory is unwriteable, or if it does exist, but is 
>unreadable?
>
>I expect that the byte code files will simply not be created, and 
>everything will continue without them.

s/__cache__/__pycache__

but yes, just as it does today.

>(2) Presumably this only effects imports, not running python source code 
>as a script. If I do this:
>
>python myscript.py
>
>from the shell, I would expect that no __cache__ directory will be 
>created, just like today. 

Correct.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100303/d449cb49/attachment.pgp>

From ziade.tarek at gmail.com  Thu Mar  4 00:54:02 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 4 Mar 2010 00:54:02 +0100
Subject: [Python-Dev] The fate of Distutils in Python 2.7
In-Reply-To: <94bdd2611002261344q361e702av10fdf99ea5eb9f71@mail.gmail.com>
References: <94bdd2611002261344q361e702av10fdf99ea5eb9f71@mail.gmail.com>
Message-ID: <94bdd2611003031554o495fcfd8kb24b61c3b6f63427@mail.gmail.com>

Just FYI : I am post-poning the revert of Distutils to 2.6.x right
after 2.7a4 has been tagged to avoid any problems (this is in 3 days)

The revert is ready but 3 days is not long enough to make sure
everything is going smooth.

On Fri, Feb 26, 2010 at 10:44 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> Hello,
>
> This is a follow-up of the Pycon summit + sprints on packaging.
>
> This is what we have planned to do:
>
> 1. refactor distutils in a new standalone version called distutils2
> [this is done already and we are actively working in the code]
> 2. completely revert distutils in Lib/ and Doc/ so the code + doc is
> the same than the current 2.6.x branch
> 3. leave the new sysconfig module, that is used by the Makefile and
> the site module
>
> The rest of the work will happen in distutils2 and we will try to
> release a version asap for Python 2.x and 3.x (2.4 to 3.2), and the
> goal
> is to put it back in the stdlib in Python 3.3
>
> Distutils in Python will be feature-frozen and I will only do bug
> fixes there. ?All feature requests will be redirected to Distutils2.
>
> I think the easiest way to manage this for me and for the feedback of
> the community is to add in bugs.python.org a "Distutils2" component,
> so I can
> start to reorganize the issues in there and reassign new issues to
> Distutils2 when it applies.
>
> Regards
> Tarek
>
> --
> Tarek Ziad? | http://ziade.org
>



-- 
Tarek Ziad? | http://ziade.org

From jimjjewett at gmail.com  Thu Mar  4 01:37:33 2010
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 3 Mar 2010 19:37:33 -0500
Subject: [Python-Dev] __file__ and bytecode-only
Message-ID: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>

I understand the need to ship without source -- but why does that
require supporting .pyc (or .pyo) -only?

Couldn't vendors just replace the real .py files with empty files?

Then no one would need the extra stat call, and no one would be bitten
by orphaned .pyc files after a rename.

[Yes, zips could still allow unmatched names; yes, it would be helpful
if a tool were available to sync the last-modification time; yes a
deprecation release should still be needed.]

-jJ

From solipsis at pitrou.net  Thu Mar  4 02:21:08 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Mar 2010 20:21:08 -0500
Subject: [Python-Dev] code.python.org dead?
In-Reply-To: <4B8EBBFE.40401@v.loewis.de>
References: <20100302103212.69669f0f@msiwind>
	<4B8EBBFE.40401@v.loewis.de>
Message-ID: <20100303202108.1f86979d@msiwind>

Le Wed, 03 Mar 2010 20:43:58 +0100,
"Martin v. L?wis" <martin at v.loewis.de> a ?crit :

> > It seems someone has decided that code.python.org doesn't resolve
> > anymore.
> 
> I have now restored it.

Thanks a lot!

Antoine.

From barry at python.org  Thu Mar  4 02:49:23 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 3 Mar 2010 20:49:23 -0500
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
Message-ID: <20100303204923.6d597f03@heresy.wooz.org>

On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote:

>I understand the need to ship without source -- but why does that
>require supporting .pyc (or .pyo) -only?
>
>Couldn't vendors just replace the real .py files with empty files?

Yes, I think that's a possibility.  What would people think about that?

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100303/b33adaaf/attachment.pgp>

From brett at python.org  Thu Mar  4 02:51:04 2010
From: brett at python.org (Brett Cannon)
Date: Wed, 3 Mar 2010 17:51:04 -0800
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
Message-ID: <bbaeab101003031751o1ce1bc75h1333865c023efd52@mail.gmail.com>

On Wed, Mar 3, 2010 at 16:37, Jim Jewett <jimjjewett at gmail.com> wrote:

> I understand the need to ship without source -- but why does that
> require supporting .pyc (or .pyo) -only?
>
> Couldn't vendors just replace the real .py files with empty files?
>

Because if someone screws up the mod time on the source files the .pyc files
will get recreated silently.

-Brett




>
> Then no one would need the extra stat call, and no one would be bitten
> by orphaned .pyc files after a rename.
>
> [Yes, zips could still allow unmatched names; yes, it would be helpful
> if a tool were available to sync the last-modification time; yes a
> deprecation release should still be needed.]
>
> -jJ
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100303/38201107/attachment-0001.html>

From v+python at g.nevcal.com  Thu Mar  4 03:33:21 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 03 Mar 2010 18:33:21 -0800
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100303204923.6d597f03@heresy.wooz.org>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
Message-ID: <4B8F1BF1.6060608@g.nevcal.com>

On approximately 3/3/2010 5:49 PM, came the following characters from 
the keyboard of Barry Warsaw:
> On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote:
>    
>> >I understand the need to ship without source -- but why does that
>> >require supporting .pyc (or .pyo) -only?
>> >
>> >Couldn't vendors just replace the real .py files with empty files?
>>      
> Yes, I think that's a possibility.  What would people think about that?
>
> -Barry
>    

That's kooky, but not as kooky as my idea.  As mentioned elsewhere, 
timestamps would have to be treated carefully.

In this scenario, the .pyc files would still live in __pycache__ ?  
Complete with the foo.<token>.pyc naming ?

So one could actually create a "fat .zip" application that could work 
with a variety of installed Python versions ???  Hmm...

-- 
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking


From greg.ewing at canterbury.ac.nz  Thu Mar  4 04:18:09 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 04 Mar 2010 16:18:09 +1300
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100303204923.6d597f03@heresy.wooz.org>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
Message-ID: <4B8F2671.1030601@canterbury.ac.nz>

Barry Warsaw wrote:
> On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote:
>
>> Couldn't vendors just replace the real .py files with empty files?
> 
> Yes, I think that's a possibility.  What would people think about that?

Seems like a perverse thing to have to do to me.

Also a bit fragile, since you would have to make
sure that the empty .py files were dated older
than the .pyc files and stayed that way, lest
Python try to recompile them and wipe out your
code.

You would also have to be careful to build
installers that didn't recompile .py files on
installation. (Haven't had much experience building
installers using distutils, so I'm not sure
how much of a problem that would be.)

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Mar  4 04:22:13 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 04 Mar 2010 16:22:13 +1300
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B8F1BF1.6060608@g.nevcal.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8F1BF1.6060608@g.nevcal.com>
Message-ID: <4B8F2765.4030401@canterbury.ac.nz>

Glenn Linderman wrote:

> In this scenario, the .pyc files would still live in __pycache__ ?  
> Complete with the foo.<token>.pyc naming ?

It might be neater to have a separate cache directory
for each bytecode version, named __cache.<token>__ or
some such.

-- 
Greg

From glyph at twistedmatrix.com  Thu Mar  4 04:28:45 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Wed, 3 Mar 2010 22:28:45 -0500
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B8F2765.4030401@canterbury.ac.nz>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8F1BF1.6060608@g.nevcal.com> <4B8F2765.4030401@canterbury.ac.nz>
Message-ID: <ECC9E075-2D7A-429F-BFE5-59EB26C47718@twistedmatrix.com>


On Mar 3, 2010, at 10:22 PM, Greg Ewing wrote:

> Glenn Linderman wrote:
> 
>> In this scenario, the .pyc files would still live in __pycache__ ?  Complete with the foo.<token>.pyc naming ?
> 
> It might be neater to have a separate cache directory
> for each bytecode version, named __cache.<token>__ or
> some such.

Okay, this is probably some pretty silly bikeshedding, but: if we're going to have it be something.something-else, can we please make sure that .something-else is a common extension that means "python bytecode cache"?  It would be good to keep the file-manager and shell operations required to say "blow away bytecode cache directories" as simple as possible.


From solipsis at pitrou.net  Thu Mar  4 04:31:15 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Mar 2010 22:31:15 -0500
Subject: [Python-Dev] __file__ and bytecode-only
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<bbaeab101003031751o1ce1bc75h1333865c023efd52@mail.gmail.com>
Message-ID: <20100303223115.51fe7276@msiwind>

Le Wed, 3 Mar 2010 17:51:04 -0800,
Brett Cannon <brett at python.org> a ?crit :
> On Wed, Mar 3, 2010 at 16:37, Jim Jewett <jimjjewett at gmail.com> wrote:
> 
> > I understand the need to ship without source -- but why does that
> > require supporting .pyc (or .pyo) -only?
> >
> > Couldn't vendors just replace the real .py files with empty files?
> >
> 
> Because if someone screws up the mod time on the source files
> the .pyc files will get recreated silently.

Unless the .py files arrange to raise a syntax error on compiling.




From solipsis at pitrou.net  Thu Mar  4 04:32:53 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Mar 2010 22:32:53 -0500
Subject: [Python-Dev] __file__ and bytecode-only
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8F1BF1.6060608@g.nevcal.com> <4B8F2765.4030401@canterbury.ac.nz>
Message-ID: <20100303223253.502e1fb0@msiwind>

Le Thu, 04 Mar 2010 16:22:13 +1300,
Greg Ewing <greg.ewing at canterbury.ac.nz> a ?crit :

> Glenn Linderman wrote:
> 
> > In this scenario, the .pyc files would still live in __pycache__ ?  
> > Complete with the foo.<token>.pyc naming ?
> 
> It might be neater to have a separate cache directory
> for each bytecode version, named __cache.<token>__ or
> some such.
> 

Actually, I find it neater to have a single cache directory. It makes
for much less clutter, and simpler ignore rules.



From greg.ewing at canterbury.ac.nz  Thu Mar  4 04:37:04 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 04 Mar 2010 16:37:04 +1300
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <ECC9E075-2D7A-429F-BFE5-59EB26C47718@twistedmatrix.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8F1BF1.6060608@g.nevcal.com> <4B8F2765.4030401@canterbury.ac.nz>
	<ECC9E075-2D7A-429F-BFE5-59EB26C47718@twistedmatrix.com>
Message-ID: <4B8F2AE0.8010400@canterbury.ac.nz>

Glyph Lefkowitz wrote:
> if we're going to have it be something.something-else, can we please 
> make sure that .something-else is a common extension that means 
> "python bytecode cache"?

Maybe something like

   __bytecode-<version>__.pycache

?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Mar  4 04:39:03 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 04 Mar 2010 16:39:03 +1300
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100303223115.51fe7276@msiwind>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<bbaeab101003031751o1ce1bc75h1333865c023efd52@mail.gmail.com>
	<20100303223115.51fe7276@msiwind>
Message-ID: <4B8F2B57.2020404@canterbury.ac.nz>

Antoine Pitrou wrote:

> Unless the .py files arrange to raise a syntax error on compiling.

I guess that prevents a total disaster, but the
program is still broken and you have to hunt down
the offending files and fix the timestamps -- if
it's even evident what the problem is and how to
fix it.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Mar  4 04:48:36 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 04 Mar 2010 16:48:36 +1300
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100303223253.502e1fb0@msiwind>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8F1BF1.6060608@g.nevcal.com>
	<4B8F2765.4030401@canterbury.ac.nz> <20100303223253.502e1fb0@msiwind>
Message-ID: <4B8F2D94.8020703@canterbury.ac.nz>

Antoine Pitrou wrote:

> Actually, I find it neater to have a single cache directory. It makes
> for much less clutter, and simpler ignore rules.

Another possibility would be to have a single top-level
cache directory with a subdirectory for each version:

   __bytecode__.pycache/<version>/<filename>.pyc

I don't think this ought to increase the stat call
count, since the interpreter will always be looking
for a particular version of bytecode, so it can
just stat for the appropriate subdirectory
directly.

-- 
Greg

From solipsis at pitrou.net  Thu Mar  4 04:54:22 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Mar 2010 22:54:22 -0500
Subject: [Python-Dev] __file__ and bytecode-only
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<bbaeab101003031751o1ce1bc75h1333865c023efd52@mail.gmail.com>
	<20100303223115.51fe7276@msiwind>
	<4B8F2B57.2020404@canterbury.ac.nz>
Message-ID: <20100303225422.39b030a7@msiwind>

Le Thu, 04 Mar 2010 16:39:03 +1300,
Greg Ewing <greg.ewing at canterbury.ac.nz> a ?crit :
> Antoine Pitrou wrote:
> 
> > Unless the .py files arrange to raise a syntax error on compiling.
> 
> I guess that prevents a total disaster, but the
> program is still broken and you have to hunt down
> the offending files and fix the timestamps -- if
> it's even evident what the problem is and how to
> fix it.
> 

Bytecode-only distributions are supposed to be created by the
vendor/distributor. It's up to him/her to get things right.

Regards

Antoine.



From orsenthil at gmail.com  Thu Mar  4 05:06:23 2010
From: orsenthil at gmail.com (Senthil Kumaran)
Date: Thu, 4 Mar 2010 09:36:23 +0530
Subject: [Python-Dev] Release blocker: urllib2 issue 7540
In-Reply-To: <9cee7ab81003031415p717420f3j89ebf1116a76f790@mail.gmail.com>
References: <9cee7ab81003031415p717420f3j89ebf1116a76f790@mail.gmail.com>
Message-ID: <7c42eba11003032006v182315bfn7efb62ebdd632de1@mail.gmail.com>

On Thu, Mar 4, 2010 at 3:45 AM, Fred Drake <fdrake at acm.org> wrote:
> Regarding: ?http://bugs.python.org/issue7540
>
> The proper response to this issue for Python 2.6 is to make no code
> changes (though a documentation enhancement may be in order).
>
> This change should be reverted from all branches.
>
> Python 2.6.5 should be blocked until this is done.

I have commented on that issue.
Ready to revert the change from the branches and keep a doc fix and a warning.




-- 
Senthil

From ncoghlan at gmail.com  Thu Mar  4 14:25:36 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 04 Mar 2010 23:25:36 +1000
Subject: [Python-Dev] __file__
In-Reply-To: <4B8EC6EA.3050603@arcor.de>
References: <mailman.91.1267392743.23597.python-dev@python.org>
	<4B8EC6EA.3050603@arcor.de>
Message-ID: <4B8FB4D0.5060008@gmail.com>

Henning von Bargen wrote:
> If the ZIP contains only bytecode files, it is just not intended
> for changing any code, so I don't think this is an argument.
> If you have access to the source code, you still can use that instead
> of messing around with byte code.

That doesn't apply when it is the app developer trying to figure out why
their app is breaking when they throw it in a bytecode only zip :)

It's a moot point anyway - Guido has pronounced that the bytecode-only
support will be left alone by PEP 3147 even if the other caching changes
are eventually accepted.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Mar  4 14:34:10 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 04 Mar 2010 23:34:10 +1000
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100303204923.6d597f03@heresy.wooz.org>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
Message-ID: <4B8FB6D2.4040501@gmail.com>

Barry Warsaw wrote:
> On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote:
> 
>> I understand the need to ship without source -- but why does that
>> require supporting .pyc (or .pyo) -only?
>>
>> Couldn't vendors just replace the real .py files with empty files?
> 
> Yes, I think that's a possibility.  What would people think about that?

I actually thought of this, but didn't post it because it defeats the
point of byte-code only distribution.

- putting text in the .py file will break the application
- touching the .py file in any way will break the application

If someone wants to break the bytecode only support it should be its own
PEP and not coupled with PEP 3147.

The remaining open question to my mind is whether or not there should be
a -X option to control the bytecode generation. E.g.:

-Xcache_bytecode=no (don't write bytecode files at all)
-Xcache_bytecode=file (write a classic "foo.pyc" file)
-Xcache_bytecode=dir (write to the "__pycache__" directory)

With cache_bytecode=dir being the default for future releases.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From barry at python.org  Thu Mar  4 16:33:24 2010
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Mar 2010 10:33:24 -0500
Subject: [Python-Dev] __file__
In-Reply-To: <4B8FB4D0.5060008@gmail.com>
References: <mailman.91.1267392743.23597.python-dev@python.org>
	<4B8EC6EA.3050603@arcor.de> <4B8FB4D0.5060008@gmail.com>
Message-ID: <20100304103324.219704b9@heresy.wooz.org>

On Mar 04, 2010, at 11:25 PM, Nick Coghlan wrote:

>It's a moot point anyway - Guido has pronounced that the bytecode-only
>support will be left alone by PEP 3147 even if the other caching changes
>are eventually accepted.

Right.  I should have stopped while I was ahead.  We'll just keep what's in
the PEP.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100304/2c3e8b25/attachment.pgp>

From brett at python.org  Thu Mar  4 21:20:00 2010
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Mar 2010 12:20:00 -0800
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
Message-ID: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>

1) I miss not having the affected files listed in the subject line.

2) The To field is set to hg at python.org which gets rejected as an invalid
email address if you reply. Would be better to set it to python-checkins so
that the habitual reply to a checkin won't get rejected.

At least I would like to see these changes happen.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100304/71828b40/attachment.html>

From fdrake at acm.org  Thu Mar  4 21:37:20 2010
From: fdrake at acm.org (Fred Drake)
Date: Thu, 4 Mar 2010 15:37:20 -0500
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
Message-ID: <9cee7ab81003041237y19b42105y73728db31384964e@mail.gmail.com>

On Thu, Mar 4, 2010 at 3:20 PM, Brett Cannon <brett at python.org> wrote:
> 1) I miss not having the affected files listed in the subject line.
> 2) The To field is set to hg at python.org which gets rejected as an invalid
> email address if you reply. Would be better to set it to python-checkins so
> that the habitual reply to a checkin won't get rejected.

I concur on both points.  You are not alone.  :-)


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller

From ncoghlan at gmail.com  Thu Mar  4 21:39:50 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 05 Mar 2010 06:39:50 +1000
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
Message-ID: <4B901A96.2000100@gmail.com>

Brett Cannon wrote:
> 1) I miss not having the affected files listed in the subject line.
> 
> 2) The To field is set to hg at python.org <mailto:hg at python.org> which
> gets rejected as an invalid email address if you reply. Would be better
> to set it to python-checkins so that the habitual reply to a checkin
> won't get rejected.
> 
> At least I would like to see these changes happen.

+1 (and it's OK if the file list gets truncated for big checkins - it's
still handy in typical cases)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From eric at trueblade.com  Thu Mar  4 21:43:06 2010
From: eric at trueblade.com (Eric Smith)
Date: Thu, 04 Mar 2010 15:43:06 -0500
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
Message-ID: <4B901B5A.5040300@trueblade.com>

Are the diffs gone for some deliberate reason?

I realize the link tells me the changes, but I'll review a lot more code 
if the diffs show up in my inbox than if I have to fire up a browser, 
especially from my phone.

Eric.

Brett Cannon wrote:
> 1) I miss not having the affected files listed in the subject line.
> 
> 2) The To field is set to hg at python.org <mailto:hg at python.org> which 
> gets rejected as an invalid email address if you reply. Would be better 
> to set it to python-checkins so that the habitual reply to a checkin 
> won't get rejected.
> 
> At least I would like to see these changes happen.
> 
> -Brett
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com


-- 
Eric.

From ncoghlan at gmail.com  Thu Mar  4 21:46:35 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 05 Mar 2010 06:46:35 +1000
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
Message-ID: <4B901C2B.9010203@gmail.com>

Brett Cannon wrote:
> 1) I miss not having the affected files listed in the subject line.
> 
> 2) The To field is set to hg at python.org <mailto:hg at python.org> which
> gets rejected as an invalid email address if you reply. Would be better
> to set it to python-checkins so that the habitual reply to a checkin
> won't get rejected.
> 
> At least I would like to see these changes happen.

I'd like to keep the inline diffs as well - much easier to skim patches
as I click through messages, whereas links to changesets, while useful
if I decide to look at a change in more detail, don't tell me a lot at a
glance.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From barry at python.org  Thu Mar  4 21:46:38 2010
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Mar 2010 15:46:38 -0500
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <4B901B5A.5040300@trueblade.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
	<4B901B5A.5040300@trueblade.com>
Message-ID: <20100304154638.0623468e@heresy.wooz.org>

On Mar 04, 2010, at 03:43 PM, Eric Smith wrote:

>Are the diffs gone for some deliberate reason?
>
>I realize the link tells me the changes, but I'll review a lot more code 
>if the diffs show up in my inbox than if I have to fire up a browser, 
>especially from my phone.

Agreed, please restore the diffs!
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100304/e4973229/attachment.pgp>

From dirkjan at ochtman.nl  Thu Mar  4 22:37:59 2010
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 4 Mar 2010 22:37:59 +0100
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
Message-ID: <ea2499da1003041337y2259f0d1g76237abb24158238@mail.gmail.com>

First off, this was just a first sample, it was by no means the
definitive format.

Second, the diffs are already back, in the next message (this first
one was per changegroup, not per changeset -- new ones will be per
changeset).

On Thu, Mar 4, 2010 at 21:20, Brett Cannon <brett at python.org> wrote:
> 1) I miss not having the affected files listed in the subject line.

Well, I think it looks rather ugly, and Tarek said he'd like the first
line of the commit message in the subject instead, so that's what I've
implemented right now. But then if there's a majority who want the
files, I'm happy to put them back.

> 2) The To field is set to hg at python.org which gets rejected as an invalid
> email address if you reply. Would be better to set it to python-checkins so
> that the habitual reply to a checkin won't get rejected.

Will fix.

Cheers,

Dirkjan

From benjamin at python.org  Thu Mar  4 22:44:11 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 4 Mar 2010 15:44:11 -0600
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <ea2499da1003041337y2259f0d1g76237abb24158238@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
	<ea2499da1003041337y2259f0d1g76237abb24158238@mail.gmail.com>
Message-ID: <1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com>

2010/3/4 Dirkjan Ochtman <dirkjan at ochtman.nl>:
> Well, I think it looks rather ugly, and Tarek said he'd like the first
> line of the commit message in the subject instead, so that's what I've
> implemented right now. But then if there's a majority who want the
> files, I'm happy to put them back.

+1 to bringing back affected files. It helps tell at a glance whether
a changeset is interesting or not.



-- 
Regards,
Benjamin

From dirkjan at ochtman.nl  Thu Mar  4 22:55:35 2010
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Thu, 4 Mar 2010 22:55:35 +0100
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com> 
	<ea2499da1003041337y2259f0d1g76237abb24158238@mail.gmail.com> 
	<1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com>
Message-ID: <ea2499da1003041355k1715e276qd13eed20687381cb@mail.gmail.com>

On Thu, Mar 4, 2010 at 22:44, Benjamin Peterson <benjamin at python.org> wrote:
> +1 to bringing back affected files. It helps tell at a glance whether
> a changeset is interesting or not.

Do they need to be in the subject, or would it be fine to have them in
the message?

Cheers,

Dirkjan

From benjamin at python.org  Thu Mar  4 22:56:58 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 4 Mar 2010 15:56:58 -0600
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <ea2499da1003041355k1715e276qd13eed20687381cb@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
	<ea2499da1003041337y2259f0d1g76237abb24158238@mail.gmail.com>
	<1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com>
	<ea2499da1003041355k1715e276qd13eed20687381cb@mail.gmail.com>
Message-ID: <1afaf6161003041356h6e9a971ex3b7c57c597c29226@mail.gmail.com>

2010/3/4 Dirkjan Ochtman <dirkjan at ochtman.nl>:
> On Thu, Mar 4, 2010 at 22:44, Benjamin Peterson <benjamin at python.org> wrote:
>> +1 to bringing back affected files. It helps tell at a glance whether
>> a changeset is interesting or not.
>
> Do they need to be in the subject, or would it be fine to have them in
> the message?

Both is good actually.


-- 
Regards,
Benjamin

From brett at python.org  Thu Mar  4 22:58:34 2010
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Mar 2010 13:58:34 -0800
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <1afaf6161003041356h6e9a971ex3b7c57c597c29226@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com> 
	<ea2499da1003041337y2259f0d1g76237abb24158238@mail.gmail.com> 
	<1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com> 
	<ea2499da1003041355k1715e276qd13eed20687381cb@mail.gmail.com> 
	<1afaf6161003041356h6e9a971ex3b7c57c597c29226@mail.gmail.com>
Message-ID: <bbaeab101003041358ga91c388h6ebd3594716e7d8a@mail.gmail.com>

On Thu, Mar 4, 2010 at 13:56, Benjamin Peterson <benjamin at python.org> wrote:

> 2010/3/4 Dirkjan Ochtman <dirkjan at ochtman.nl>:
> > On Thu, Mar 4, 2010 at 22:44, Benjamin Peterson <benjamin at python.org>
> wrote:
> >> +1 to bringing back affected files. It helps tell at a glance whether
> >> a changeset is interesting or not.
> >
> > Do they need to be in the subject, or would it be fine to have them in
> > the message?
>
> Both is good actually.
>

I prefer the subject so that I can easily skim them to see if someone edited
a file I really care about, but if that is not possible then the body is
acceptable, especially if it is the first thing in the body (that would let
me at least see some of it in the initial snippet Gmail shows).

-Brett



>
>
> --
> Regards,
> Benjamin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100304/6ed766dc/attachment-0001.html>

From stephen at xemacs.org  Fri Mar  5 01:43:09 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 05 Mar 2010 09:43:09 +0900
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <bbaeab101003041358ga91c388h6ebd3594716e7d8a@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
	<ea2499da1003041337y2259f0d1g76237abb24158238@mail.gmail.com>
	<1afaf6161003041344g20791bddpf13d66aa0c4d9438@mail.gmail.com>
	<ea2499da1003041355k1715e276qd13eed20687381cb@mail.gmail.com>
	<1afaf6161003041356h6e9a971ex3b7c57c597c29226@mail.gmail.com>
	<bbaeab101003041358ga91c388h6ebd3594716e7d8a@mail.gmail.com>
Message-ID: <87vddbvbs2.fsf@uwakimon.sk.tsukuba.ac.jp>

Brett Cannon writes:

 > I prefer the subject so that I can easily skim them to see if someone edited
 > a file I really care about, but if that is not possible then the body is
 > acceptable, especially if it is the first thing in the body (that would let
 > me at least see some of it in the initial snippet Gmail shows).

I don't know if it's pedantically RFC-correct, but occasionally you'll
see a Summary field in the header.  I think using the first line of
the log in the Subject field and the file list as the Summary field
would be a good way to do it, *if* people's MUAs would display
Summary.  I suspect they don't (mine does, but mine does a lot of
nifty things that cellphone MUAs don't :-( ).

From martin at v.loewis.de  Fri Mar  5 05:11:02 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 05 Mar 2010 05:11:02 +0100
Subject: [Python-Dev] doctest, unicode repr, and 2to3
Message-ID: <4B908456.2070505@v.loewis.de>

Johan Harjano ran into an interesting problem when trying to run the
Django test suite under Python 3.1.

Django has doctests of the form

>>> a6.headline
u'Default headline'

Even when converting the doctest with 2to3, the expected output is
unmodified. However, in 3.x, the expected output will change (i.e. not
produce an u"" prefix anymore).

Now, it might be possible to reformulate the test case (e.g. use print()
instead of relying on repr), however, this is undesirable as a) the test
should continue to test in 2.x that the result object is a unicode
string, and b) it makes the test less readable.

I would like to find a solution where this gets automatically corrected,
e.g. through 2to3, or through changes to doctest, or through changes of
str.__repr__.

Any proposal appreciated.

Regards,
Martin

From barry at python.org  Fri Mar  5 05:30:12 2010
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Mar 2010 23:30:12 -0500
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <4B908456.2070505@v.loewis.de>
References: <4B908456.2070505@v.loewis.de>
Message-ID: <20100304233012.18ac0b41@freewill.wooz.org>

On Mar 05, 2010, at 05:11 AM, Martin v. L?wis wrote:

>Johan Harjano ran into an interesting problem when trying to run the
>Django test suite under Python 3.1.
>
>Django has doctests of the form
>
>>>> a6.headline
>u'Default headline'
>
>Even when converting the doctest with 2to3, the expected output is
>unmodified. However, in 3.x, the expected output will change (i.e. not
>produce an u"" prefix anymore).

For this reason, I always recommend using print, even though...

>Now, it might be possible to reformulate the test case (e.g. use print()
>instead of relying on repr), however, this is undesirable as a) the test
>should continue to test in 2.x that the result object is a unicode
>string, and b) it makes the test less readable.

If you really want to test that it's a unicode, shouldn't you actually test
its type?  (I'm not sure what would happen with that under 2to3.)  Besides,
the type of the string is very rarely important, so I think the u-prefix and
quotes is mostly just noise.

>I would like to find a solution where this gets automatically corrected,
>e.g. through 2to3, or through changes to doctest, or through changes of
>str.__repr__.

I think Michael was also talking about changes to doctest that would
automatically sort dictionaries and sets.  Again, it's not hard to write
doctests correctly, but it's surprisingly common to implicitly rely on sort
order.

I think the right place to change these is in doctest.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100304/b5f770e9/attachment.pgp>

From glyph at twistedmatrix.com  Fri Mar  5 05:33:16 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Thu, 4 Mar 2010 23:33:16 -0500
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <20100304233012.18ac0b41@freewill.wooz.org>
References: <4B908456.2070505@v.loewis.de>
	<20100304233012.18ac0b41@freewill.wooz.org>
Message-ID: <E590AEFB-0EB0-4498-82D5-1201E8D7FD8D@twistedmatrix.com>


On Mar 4, 2010, at 11:30 PM, Barry Warsaw wrote:

> If you really want to test that it's a unicode, shouldn't you actually test
> its type?  (I'm not sure what would happen with that under 2to3.)

Presumably 2to3 will be smart enough to translate 'unicode' to 'str' and 'bytes' to... 'bytes'.  Just don't use 'str' in 2.x and you should be okay :).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100304/f7c01267/attachment.html>

From brian at sweetapp.com  Fri Mar  5 07:03:02 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Fri, 5 Mar 2010 17:03:02 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
Message-ID: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>

Hi all,

I recently submitted a daft PEP for a package designed to make it  
easier to execute Python functions asynchronously using threads and  
processes. It lets the user focus on their computational problem  
without having to build explicit thread/process pools and work queues.

The package has been discussed on stdlib-sig but now I'd like this  
group's feedback.

The PEP lives here:
http://python.org/dev/peps/pep-3148/

Here are two examples to whet your appetites:

"""Determine if several numbers are prime."""
import futures
import math

PRIMES = [
     112272535095293,
     112582705942171,
     112272535095293,
     115280095190773,
     115797848077099,
     1099726899285419]

def is_prime(n):
     if n % 2 == 0:
         return False

     sqrt_n = int(math.floor(math.sqrt(n)))
     for i in range(3, sqrt_n + 1, 2):
         if n % i == 0:
             return False
     return True

# Uses as many CPUs as your machine has.
with futures.ProcessPoolExecutor() as executor:
     for number, is_prime in zip(PRIMES, executor.map(is_prime,  
PRIMES)):
         print('%d is prime: %s' % (number, is_prime))


"""Print out the size of the home pages of various new sites (and Fox  
News)."""
import futures
import urllib.request

URLS = ['http://www.foxnews.com/',
         'http://www.cnn.com/',
         'http://europe.wsj.com/',
         'http://www.bbc.co.uk/',
         'http://some-made-up-domain.com/']

def load_url(url, timeout):
     return urllib.request.urlopen(url, timeout=timeout).read()

with futures.ThreadPoolExecutor(max_workers=5) as executor:
     # Create a future for each URL load.
     future_to_url = dict((executor.submit(load_url, url, 60), url)
                          for url in URLS)

     # Iterate over the futures in the order that they complete.
     for future in futures.as_completed(future_to_url):
         url = future_to_url[future]
         if future.exception() is not None:
             print('%r generated an exception: %s' % (url,
                                                       
future.exception()))
         else:
             print('%r page is %d bytes' % (url, len(future.result())))

Cheers,
Brian

From ncoghlan at gmail.com  Fri Mar  5 10:29:28 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 05 Mar 2010 19:29:28 +1000
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <E590AEFB-0EB0-4498-82D5-1201E8D7FD8D@twistedmatrix.com>
References: <4B908456.2070505@v.loewis.de>	<20100304233012.18ac0b41@freewill.wooz.org>
	<E590AEFB-0EB0-4498-82D5-1201E8D7FD8D@twistedmatrix.com>
Message-ID: <4B90CEF8.6000403@gmail.com>

Glyph Lefkowitz wrote:
> 
> On Mar 4, 2010, at 11:30 PM, Barry Warsaw wrote:
> 
>> If you really want to test that it's a unicode, shouldn't you actually
>> test
>> its type?  (I'm not sure what would happen with that under 2to3.)
> 
> Presumably 2to3 will be smart enough to translate 'unicode' to 'str' and
> 'bytes' to... 'bytes'.  Just don't use 'str' in 2.x and you should be
> okay :).

In code (including doctest input lines) 2to3 can make those
translations, but I believe the doctest output lines are a different
story (since they're just arbitrary strings as far as the translator is
concerned).

I would expect 2to3 to be able to translate the following tests
correctly because the expected output stays the same even though the
commands change:
>>> print a6.headline
'Default headline'
>>> type(a6.headline) is type(u'')
True

But I don't see a ready way to support doctests where the expected
*output* changes between versions.

Cheers,
Nick.
-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From andrew at bemusement.org  Fri Mar  5 12:53:06 2010
From: andrew at bemusement.org (Andrew Bennetts)
Date: Fri, 5 Mar 2010 22:53:06 +1100
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <4B908456.2070505@v.loewis.de>
References: <4B908456.2070505@v.loewis.de>
Message-ID: <20100305115306.GA9002@steerpike.home.puzzling.org>

"Martin v. L?wis" wrote:
[...]
> Any proposal appreciated.

I propose screaming ?help me, I have written a test suite using nothing
but string matching assertions, what is wrong with me?!?

-Andrew.


From ironfroggy at gmail.com  Fri Mar  5 13:45:46 2010
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Fri, 5 Mar 2010 07:45:46 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
Message-ID: <76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com>

A young library solving an old problem in a way that conflicts with
many of the other implementations available for years and with zero
apparent users in the wild is not an appropriate candidate for a PEP.

On Fri, Mar 5, 2010 at 1:03 AM, Brian Quinlan <brian at sweetapp.com> wrote:
> Hi all,
>
> I recently submitted a daft PEP for a package designed to make it easier to
> execute Python functions asynchronously using threads and processes. It lets
> the user focus on their computational problem without having to build
> explicit thread/process pools and work queues.
>
> The package has been discussed on stdlib-sig but now I'd like this group's
> feedback.
>
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/
>
> Here are two examples to whet your appetites:
>
> """Determine if several numbers are prime."""
> import futures
> import math
>
> PRIMES = [
> ? ?112272535095293,
> ? ?112582705942171,
> ? ?112272535095293,
> ? ?115280095190773,
> ? ?115797848077099,
> ? ?1099726899285419]
>
> def is_prime(n):
> ? ?if n % 2 == 0:
> ? ? ? ?return False
>
> ? ?sqrt_n = int(math.floor(math.sqrt(n)))
> ? ?for i in range(3, sqrt_n + 1, 2):
> ? ? ? ?if n % i == 0:
> ? ? ? ? ? ?return False
> ? ?return True
>
> # Uses as many CPUs as your machine has.
> with futures.ProcessPoolExecutor() as executor:
> ? ?for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
> ? ? ? ?print('%d is prime: %s' % (number, is_prime))
>
>
> """Print out the size of the home pages of various new sites (and Fox
> News)."""
> import futures
> import urllib.request
>
> URLS = ['http://www.foxnews.com/',
> ? ? ? ?'http://www.cnn.com/',
> ? ? ? ?'http://europe.wsj.com/',
> ? ? ? ?'http://www.bbc.co.uk/',
> ? ? ? ?'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
> ? ?return urllib.request.urlopen(url, timeout=timeout).read()
>
> with futures.ThreadPoolExecutor(max_workers=5) as executor:
> ? ?# Create a future for each URL load.
> ? ?future_to_url = dict((executor.submit(load_url, url, 60), url)
> ? ? ? ? ? ? ? ? ? ? ? ? for url in URLS)
>
> ? ?# Iterate over the futures in the order that they complete.
> ? ?for future in futures.as_completed(future_to_url):
> ? ? ? ?url = future_to_url[future]
> ? ? ? ?if future.exception() is not None:
> ? ? ? ? ? ?print('%r generated an exception: %s' % (url,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? future.exception()))
> ? ? ? ?else:
> ? ? ? ? ? ?print('%r page is %d bytes' % (url, len(future.result())))
>
> Cheers,
> Brian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

From jnoller at gmail.com  Fri Mar  5 15:50:28 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 5 Mar 2010 09:50:28 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com>
Message-ID: <4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com>

On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman <ironfroggy at gmail.com> wrote:
> A young library solving an old problem in a way that conflicts with
> many of the other implementations available for years and with zero
> apparent users in the wild is not an appropriate candidate for a PEP.
>

Baloney. A young library providing some syntactic sugar which uses
primitives in the standard library to implement a common pattern is
fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig
list prior to bringing it here. By the same argument, we should shunt
all of the recent unittest changes and improvements into space, since
golly, other people did it, why should we.

This is something relatively simple, which I would gladly add in an
instant to the multiprocessing package - but Brian's one-upped me in
that regard and is providing something which works with both threads
and processes handily. Take a look at multiprocessing.Pool for example
- all that is some sugar on top of the primitives, but it's good
sugar, and is used by a fair number of people.

Let me also state - "my" vision of where futures would live would be
in a concurrent package - for example:

from concurrent import futures

The reason *why* is that I would like to also move the abstractions I
have in multiprocessing *out* of that module, make them work with both
threads and processes (if it makes sense) and reduce the
multiprocessing module to the base primitive Process object. A
concurrent package which implements common patterns built on top of
the primitives we support is an objectively Good Thing.

For example, how many of us have sat down and implemented a thread
pool on top of threading, I would hazard to say that most of us who
use threading have done this, and probably more than once. It stands
to reason that this is a common enough pattern to include in the
standard library.

In any case; consider me a strong +1 to adding it.

jesse

From ironfroggy at gmail.com  Fri Mar  5 16:50:38 2010
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Fri, 5 Mar 2010 10:50:38 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com> 
	<4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com>
Message-ID: <76fd5acf1003050750l5019e7deud2b5891647c5d7ac@mail.gmail.com>

I revert my objections. I still would like to see this in use "in the
wild" and I might even use it thusly, myself.

On Fri, Mar 5, 2010 at 9:50 AM, Jesse Noller <jnoller at gmail.com> wrote:
> On Fri, Mar 5, 2010 at 7:45 AM, Calvin Spealman <ironfroggy at gmail.com> wrote:
>> A young library solving an old problem in a way that conflicts with
>> many of the other implementations available for years and with zero
>> apparent users in the wild is not an appropriate candidate for a PEP.
>>
>
> Baloney. A young library providing some syntactic sugar which uses
> primitives in the standard library to implement a common pattern is
> fine for a PEP. We've hashed this out pretty heavily on the stdlib-sig
> list prior to bringing it here. By the same argument, we should shunt
> all of the recent unittest changes and improvements into space, since
> golly, other people did it, why should we.
>
> This is something relatively simple, which I would gladly add in an
> instant to the multiprocessing package - but Brian's one-upped me in
> that regard and is providing something which works with both threads
> and processes handily. Take a look at multiprocessing.Pool for example
> - all that is some sugar on top of the primitives, but it's good
> sugar, and is used by a fair number of people.
>
> Let me also state - "my" vision of where futures would live would be
> in a concurrent package - for example:
>
> from concurrent import futures
>
> The reason *why* is that I would like to also move the abstractions I
> have in multiprocessing *out* of that module, make them work with both
> threads and processes (if it makes sense) and reduce the
> multiprocessing module to the base primitive Process object. A
> concurrent package which implements common patterns built on top of
> the primitives we support is an objectively Good Thing.
>
> For example, how many of us have sat down and implemented a thread
> pool on top of threading, I would hazard to say that most of us who
> use threading have done this, and probably more than once. It stands
> to reason that this is a common enough pattern to include in the
> standard library.
>
> In any case; consider me a strong +1 to adding it.
>
> jesse
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

From ncoghlan at gmail.com  Fri Mar  5 16:53:27 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 06 Mar 2010 01:53:27 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<76fd5acf1003050445r1f8720f6w89599b8adb234666@mail.gmail.com>
	<4222a8491003050650j1f6be276tab614016564fe10d@mail.gmail.com>
Message-ID: <4B9128F7.3020301@gmail.com>

Jesse Noller wrote:
> The reason *why* is that I would like to also move the abstractions I
> have in multiprocessing *out* of that module, make them work with both
> threads and processes (if it makes sense) and reduce the
> multiprocessing module to the base primitive Process object. A
> concurrent package which implements common patterns built on top of
> the primitives we support is an objectively Good Thing.

Yes, I've often thought we should have a pool model that covers threads
as well as processes.

The reason I think "futures" works as a PEP and potential standard
library addition is that it is small enough to be readily reviewed and
maintained, and serves as a useful building block for more complex usage.

For a developer to get anything similar from a third party library is
almost certainly going to require buying into a much heavier framework.
A simple futures module provides a way to farm out worker tasks in a
standard fashion without having to build as much of your own
infrastructure every time.

I've read the various PEP checkins as they went by on the checkins list
- it gets a +0 from me (the only reason it isn't a +1 is because I
personally tend to write with a Thread+Queue style. However, I could
easily become a futures convert if they were readily available in the
standard library)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Fri Mar  5 16:56:09 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 5 Mar 2010 10:56:09 -0500
Subject: [Python-Dev] doctest, unicode repr, and 2to3
References: <4B908456.2070505@v.loewis.de>
	<20100304233012.18ac0b41@freewill.wooz.org>
Message-ID: <20100305105609.16766c2c@msiwind>

Le Thu, 4 Mar 2010 23:30:12 -0500,
Barry Warsaw <barry at python.org> a ?crit :
> 
> If you really want to test that it's a unicode, shouldn't you
> actually test its type?  (I'm not sure what would happen with that
> under 2to3.)  Besides, the type of the string is very rarely
> important, so I think the u-prefix and quotes is mostly just noise.

String type is actually very important, if you don't want your
application/library to fail in the face of non-ASCII data.

That's why we did all this thing in py3k, after all :)

Regards

Antoine.



From fuzzyman at voidspace.org.uk  Fri Mar  5 17:11:59 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 05 Mar 2010 16:11:59 +0000
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <20100305105609.16766c2c@msiwind>
References: <4B908456.2070505@v.loewis.de>	<20100304233012.18ac0b41@freewill.wooz.org>
	<20100305105609.16766c2c@msiwind>
Message-ID: <4B912D4F.7020700@voidspace.org.uk>

On 05/03/2010 15:56, Antoine Pitrou wrote:
> Le Thu, 4 Mar 2010 23:30:12 -0500,
> Barry Warsaw<barry at python.org>  a ?crit :
>    
>> If you really want to test that it's a unicode, shouldn't you
>> actually test its type?  (I'm not sure what would happen with that
>> under 2to3.)  Besides, the type of the string is very rarely
>> important, so I think the u-prefix and quotes is mostly just noise.
>>      
> String type is actually very important, if you don't want your
> application/library to fail in the face of non-ASCII data.
>
> That's why we did all this thing in py3k, after all :)
>    

Well, I'd like to see a 'unicode agnostic' mode for different reasons - 
although given that it all changes in Python 3 and it is almost 
certainly too late to get changes to doctest before the 2.7 release the 
point is probably moot.

IronPython has very different unicode behaviour to Python 2. IronPython 
is like Python 3 in this regard, all strings are unicode (str is 
unicode). This means that it rarely puts the u prefix on strings. So 
doctests that use string repr also fail on IronPython - the django 
doctest suite is a culprit in this regard too.

Dictionary iteration order is also different in IronPython - so dict 
reprs also fail. For dict reprs it would be nice if doctest compared in 
an order agnostic way. (Internally reconstruct the dict as a dict of 
strings from the repr and then compare using dict equality.)

I know it is easily *possible* to construct doctests that are robust 
against these differences, but the fact that the "obvious-thing" often 
fails in subtle ways on other platforms are *one* of the reasons I don't 
consider doctest to be a suitable tool for unit testing [1]. :-)

Michael

[1] It isn't the main reason and I realise that is entirely orthoganal 
anway... but doctest makes every line an assertion, and you have to jump 
through hoops if you don't want that behaviour. All besides the point. 
Doctest simply *rocks* for testing documentation examples - especially 
in conjunction with Sphinx.

> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From barry at python.org  Fri Mar  5 17:13:32 2010
From: barry at python.org (Barry Warsaw)
Date: Fri, 5 Mar 2010 11:13:32 -0500
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <20100305105609.16766c2c@msiwind>
References: <4B908456.2070505@v.loewis.de>
	<20100304233012.18ac0b41@freewill.wooz.org>
	<20100305105609.16766c2c@msiwind>
Message-ID: <20100305111332.18650d1e@heresy.wooz.org>

On Mar 05, 2010, at 10:56 AM, Antoine Pitrou wrote:

>String type is actually very important, if you don't want your
>application/library to fail in the face of non-ASCII data.
>
>That's why we did all this thing in py3k, after all :)

That's not actually what I mean.  I meant that in doctests, you probably don't
need to be confronted by the string type every single time.  It's just not
that interesting for documentation, most of the time.  I have no problem
adding (unit)tests that ensure your outputs and values are of the expected
type though!

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100305/f3b6acc5/attachment.pgp>

From daniel at stutzbachenterprises.com  Fri Mar  5 17:21:03 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Fri, 5 Mar 2010 10:21:03 -0600
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
Message-ID: <eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>

On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com> wrote:

> import futures
>

+1 on the idea, -1 on the name.  It's too similar to "from __future__ import
...".

Also, the PEP should probably link to the discussions on stdlib-sig?
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100305/34e86815/attachment.html>

From jnoller at gmail.com  Fri Mar  5 17:35:54 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 5 Mar 2010 11:35:54 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
Message-ID: <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>

On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
<daniel at stutzbachenterprises.com> wrote:
> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>>
>> import futures
>
> +1 on the idea, -1 on the name.? It's too similar to "from __future__ import
> ...".

Futures is a common term for this, and implemented named this in other
languages. I don't think we should be adopting things that are common,
and found elsewhere and then renaming them.

From skip at pobox.com  Fri Mar  5 17:56:52 2010
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 5 Mar 2010 10:56:52 -0600
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
Message-ID: <19345.14292.321112.787896@montanaro.dyndns.org>

    >>> import futures
    >> 
    >> +1 on the idea, -1 on the name.? It's too similar to "from __future__ import
    >> ...".

    Jesse> Futures is a common term for this, and implemented named this in
    Jesse> other languages. I don't think we should be adopting things that
    Jesse> are common, and found elsewhere and then renaming them.

Perhaps, but is it a common term for Python programmers (or the target
population for Python)?  I've never heard of it.  "futures" to me are
futures contracts in a trading environment (that's the industry I work in).
No matter how well known the term is in the environment where it's used
today you have to be sensitive to other meanings of the term.

Skip


From jnoller at gmail.com  Fri Mar  5 18:03:58 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 5 Mar 2010 12:03:58 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <19345.14292.321112.787896@montanaro.dyndns.org>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
Message-ID: <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>

On Fri, Mar 5, 2010 at 11:56 AM,  <skip at pobox.com> wrote:
> ? ?>>> import futures
> ? ?>>
> ? ?>> +1 on the idea, -1 on the name.? It's too similar to "from __future__ import
> ? ?>> ...".
>
> ? ?Jesse> Futures is a common term for this, and implemented named this in
> ? ?Jesse> other languages. I don't think we should be adopting things that
> ? ?Jesse> are common, and found elsewhere and then renaming them.
>
> Perhaps, but is it a common term for Python programmers (or the target
> population for Python)? ?I've never heard of it. ?"futures" to me are
> futures contracts in a trading environment (that's the industry I work in).
> No matter how well known the term is in the environment where it's used
> today you have to be sensitive to other meanings of the term.
>

It's a common programming term. I don't think we should make a new
name - I mean, how many different names for green threads / coroutines
or "flavors" of those concepts are out there because someone painted
the bike shed a different color? I mean, there's prior art here:

http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html

jesse

From curt at hagenlocher.org  Fri Mar  5 18:06:44 2010
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Fri, 5 Mar 2010 09:06:44 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
Message-ID: <d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com>

On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller <jnoller at gmail.com> wrote:
>
> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
> <daniel at stutzbachenterprises.com> wrote:
> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com> wrote:
> >>
> >> import futures
> >
> > +1 on the idea, -1 on the name.? It's too similar to "from __future__ import
> > ...".
>
> Futures is a common term for this, and implemented named this in other
> languages. I don't think we should be adopting things that are common,
> and found elsewhere and then renaming them.

Another common term for this is a "promise".

--
Curt Hagenlocher
curt at hagenlocher.org

From daniel at stutzbachenterprises.com  Fri Mar  5 18:28:42 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Fri, 5 Mar 2010 11:28:42 -0600
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
Message-ID: <eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com>

On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller <jnoller at gmail.com> wrote:

> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
>

 According to that link, Java has a module named "Concurrent" with an
interface named "Future".  You're proposing a module named "Futures" with a
class named "Future".

Why not name your module "concurrent"?  That would eliminate the confusion
with "from __future__".  I don't see a problem with keeping the class name.

Plus, a "concurrent" module might be useful for things other than Futures,
in the future. ;-)

Just my 0.02 cents,
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100305/db13a15d/attachment.html>

From skip at pobox.com  Fri Mar  5 18:38:10 2010
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 5 Mar 2010 11:38:10 -0600
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
Message-ID: <19345.16770.613942.887721@montanaro.dyndns.org>


    Jesse> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html

Without reading that I can assure you that not everybody has drunk the Java
Kool-Aid.  Just because Sun thought it was a fine term doesn't mean everyone
else will.  I've been a professional programmer for about 30 years, have
never touched Java, and the only times I've seen its effect in Python I have
been unimpressed (logging and unittest).

Google for "future".  The Java doc is aways down on the front page.  Now
Google for "futures". It's nowhere to be found.  As far as I can tell, all
but one hit on the first page are for my definition of "futures".  At the
bottom of the page the "searches related to _futures_" are all related to
trading futures or other derivatives except one.  That appears to be related
to the LPGA futures tour.  Java and its definition of the term is nowhere to
be found.

I suppose you can argue that it should be called "future".  I still contend
that the name is *not* intuitive, no matter how well known it happens to be
within the Java world.  Don't name it "futures" though.  That has nothing to
do with common definitions of the term.

Skip

From jnoller at gmail.com  Fri Mar  5 18:55:04 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 5 Mar 2010 12:55:04 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
	<eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com>
Message-ID: <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>

On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach
<daniel at stutzbachenterprises.com> wrote:
> On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller <jnoller at gmail.com> wrote:
>>
>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
>
> ?According to that link, Java has a module named "Concurrent" with an
> interface named "Future".? You're proposing a module named "Futures" with a
> class named "Future".
>
> Why not name your module "concurrent"?? That would eliminate the confusion
> with "from __future__".? I don't see a problem with keeping the class name.
>
> Plus, a "concurrent" module might be useful for things other than Futures,
> in the future. ;-)
>

Brian's module is named futures; I am +1'ing his proposal, and also
suggesting we put it under concurrent/ package name. This means you
would do the following:

from concurrent import futures

and in the future:
from concurrent import pool

And so on.

From exarkun at twistedmatrix.com  Fri Mar  5 19:30:35 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Fri, 05 Mar 2010 18:30:35 -0000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com>
Message-ID: <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>

On 05:06 pm, curt at hagenlocher.org wrote:
>On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller <jnoller at gmail.com> wrote:
>>
>>On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>><daniel at stutzbachenterprises.com> wrote:
>> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com> 
>>wrote:
>> >>
>> >> import futures
>> >
>> > +1 on the idea, -1 on the name.? It's too similar to "from 
>>__future__ import
>> > ...".
>>
>>Futures is a common term for this, and implemented named this in other
>>languages. I don't think we should be adopting things that are common,
>>and found elsewhere and then renaming them.
>
>Another common term for this is a "promise".

Promises aren't exactly the same.  This would be a particularly bad name 
to apply here.

Jean-Paul

From guido at python.org  Fri Mar  5 19:31:03 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 Mar 2010 10:31:03 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com> 
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> 
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> 
	<eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com> 
	<4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>
Message-ID: <ca471dc21003051031i716237b1pc7cf06be255c51e3@mail.gmail.com>

On Fri, Mar 5, 2010 at 9:55 AM, Jesse Noller <jnoller at gmail.com> wrote:
> On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach
> <daniel at stutzbachenterprises.com> wrote:
>> On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller <jnoller at gmail.com> wrote:
>>>
>>> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
>>
>> ?According to that link, Java has a module named "Concurrent" with an
>> interface named "Future".? You're proposing a module named "Futures" with a
>> class named "Future".
>>
>> Why not name your module "concurrent"?? That would eliminate the confusion
>> with "from __future__".? I don't see a problem with keeping the class name.
>>
>> Plus, a "concurrent" module might be useful for things other than Futures,
>> in the future. ;-)
>>
>
> Brian's module is named futures; I am +1'ing his proposal, and also
> suggesting we put it under concurrent/ package name. This means you
> would do the following:
>
> from concurrent import futures
>
> and in the future:
> from concurrent import pool
>
> And so on.

"Future" is a pretty standard CS term for this concept (as noted
"promise" is another), and it wasn't invented by Java. I am not
worried at all about confusion with __future__ (which is after all
surrounded by dunders and only appears in one specific context).

FWIW, there are two different ways of designing the API for such a
concept. In one, the "future" or "promise" object masquerades as an
instance of the eventual result, like a proxy, and accessing any
attribute would block until the result is available. I don't like this
design much; the masquerading is never perfect, and attribute accesses
become risks of exceptions related to the promised work. In the other
design, which this PEP proposes, there is an explicit call to get the
value, which blocks as needed.

I found a wikipedia article
(http://en.wikipedia.org/wiki/Futures_and_promises#Implicit_vs_explicit)
that calls the former "implicit" and the latter "explicit". According
to this article, the term "promise" was coined slightly older (by a
year or so). I have no strong preference for either term; I suspect
that if this were the only objection to the PEP Brian would be happy
to rename futures to promises.

-- 
--Guido van Rossum (python.org/~guido)

From janssen at parc.com  Fri Mar  5 19:45:50 2010
From: janssen at parc.com (Bill Janssen)
Date: Fri, 5 Mar 2010 10:45:50 PST
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <19345.16770.613942.887721@montanaro.dyndns.org>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
	<19345.16770.613942.887721@montanaro.dyndns.org>
Message-ID: <26632.1267814750@parc.com>

skip at pobox.com wrote:

> 
>     Jesse> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
> 
> Without reading that I can assure you that not everybody has drunk the Java
> Kool-Aid.  Just because Sun thought it was a fine term doesn't mean everyone
> else will.  I've been a professional programmer for about 30 years, have
> never touched Java, and the only times I've seen its effect in Python I have
> been unimpressed (logging and unittest).
> 
> Google for "future".  The Java doc is aways down on the front page.  Now
> Google for "futures". It's nowhere to be found.  As far as I can tell, all
> but one hit on the first page are for my definition of "futures".  At the
> bottom of the page the "searches related to _futures_" are all related to
> trading futures or other derivatives except one.  That appears to be related
> to the LPGA futures tour.  Java and its definition of the term is nowhere to
> be found.

This term, the "future", has been in use at least since 1985, even if
the Web hasn't, and even if Google can't seem to separate this use from
drivel about stock trading.  See Halstead's 1985 TOPLAS paper on
MultiLisp.

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.97.1841

Bill

From janssen at parc.com  Fri Mar  5 19:49:27 2010
From: janssen at parc.com (Bill Janssen)
Date: Fri, 5 Mar 2010 10:49:27 PST
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <19345.16770.613942.887721@montanaro.dyndns.org>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
	<19345.16770.613942.887721@montanaro.dyndns.org>
Message-ID: <26666.1267814967@parc.com>

http://home.pipeline.com/~hbaker1/Futures.html (1977)

From ndbecker2 at gmail.com  Fri Mar  5 19:51:15 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 05 Mar 2010 13:51:15 -0500
Subject: [Python-Dev] argparse ugliness
Message-ID: <hmrjr4$kkd$1@dough.gmane.org>

I generally enjoy argparse, but one thing I find rather
ugly and unpythonic.

    parser.add_argument ('--plot', action='store_true')

Specifying the argument 'action' as a string is IMO ugly.


From guido at python.org  Fri Mar  5 20:10:33 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 Mar 2010 11:10:33 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com> 
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> 
	<d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com> 
	<20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>
Message-ID: <ca471dc21003051110pa728102j15c61e0c04d0ff1c@mail.gmail.com>

On Fri, Mar 5, 2010 at 10:30 AM,  <exarkun at twistedmatrix.com> wrote:
> On 05:06 pm, curt at hagenlocher.org wrote:
>>
>> On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller <jnoller at gmail.com> wrote:
>>>
>>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>> <daniel at stutzbachenterprises.com> wrote:
>>> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com>
>>> > wrote:
>>> >>
>>> >> import futures
>>> >
>>> > +1 on the idea, -1 on the name.? It's too similar to "from __future__
>>> > import
>>> > ...".
>>>
>>> Futures is a common term for this, and implemented named this in other
>>> languages. I don't think we should be adopting things that are common,
>>> and found elsewhere and then renaming them.
>>
>> Another common term for this is a "promise".
>
> Promises aren't exactly the same. ?This would be a particularly bad name to
> apply here.

Please explain. Even the Wikipedia article
(http://en.wikipedia.org/wiki/Futures_and_promises), despite promising
to explain the difference, didn't explain it.

-- 
--Guido van Rossum (python.org/~guido)

From brian.curtin at gmail.com  Fri Mar  5 20:15:58 2010
From: brian.curtin at gmail.com (Brian Curtin)
Date: Fri, 5 Mar 2010 13:15:58 -0600
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <hmrjr4$kkd$1@dough.gmane.org>
References: <hmrjr4$kkd$1@dough.gmane.org>
Message-ID: <cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>

On Fri, Mar 5, 2010 at 12:51, Neal Becker <ndbecker2 at gmail.com> wrote:

> I generally enjoy argparse, but one thing I find rather
> ugly and unpythonic.
>
>    parser.add_argument ('--plot', action='store_true')
>
> Specifying the argument 'action' as a string is IMO ugly.
>

What else would you propose?
FWIW, this is the same in optparse.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100305/7a4e0cc6/attachment.html>

From tjreedy at udel.edu  Fri Mar  5 20:37:13 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 05 Mar 2010 14:37:13 -0500
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <4B908456.2070505@v.loewis.de>
References: <4B908456.2070505@v.loewis.de>
Message-ID: <hmrmh8$hc$1@dough.gmane.org>

On 3/4/2010 11:11 PM, "Martin v. L?wis" wrote:
> Johan Harjano ran into an interesting problem when trying to run the
> Django test suite under Python 3.1.
>
> Django has doctests of the form
>
>>>> a6.headline
> u'Default headline'
>
> Even when converting the doctest with 2to3, the expected output is
> unmodified. However, in 3.x, the expected output will change (i.e. not
> produce an u"" prefix anymore).
>
> Now, it might be possible to reformulate the test case (e.g. use print()
> instead of relying on repr), however, this is undesirable as a) the test
> should continue to test in 2.x that the result object is a unicode
> string, and b) it makes the test less readable.
>
> I would like to find a solution where this gets automatically corrected,
> e.g. through 2to3, or through changes to doctest, or through changes of
> str.__repr__.
>
> Any proposal appreciated.

What is the easiest thing that works?

If 2to3 can fix string literals and fix code within doc strings, would 
it be difficult to fix expected strings within doc strings?

On the otherhand, as Foord pointed out, the 'u' prefix is something of a 
CPythonism not required by the language def, so an 'ignore leading u on 
expected output' flag would be useful. But 'correcting'

 >>> a,b
u'ah', u'hah'

is too much to expect. Doctest is both useful and fragile because it 
*almost* imitates human testing methods.

If tests like the above were bunched, and my primary goal were to make 
Django tests work, and work now, without changing the style, I would be 
tempted to use something like the following:

def u(s): #import as needed from a testutil module
   if type(s) is unicode: # str in 3.x
     print(s)
   else print('') #or any 'nevermatch' string

 >>> u(a6.headline)
'Default headline'

which both retains the type test and readability. This would be a 
project specific solution, but then, the other comments suggest that 
wanting to combine a type and value test this way seems to be someone 
project specific.

Terry Jan Reedy



From martin at v.loewis.de  Fri Mar  5 20:49:54 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 05 Mar 2010 20:49:54 +0100
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <20100305115306.GA9002@steerpike.home.puzzling.org>
References: <4B908456.2070505@v.loewis.de>
	<20100305115306.GA9002@steerpike.home.puzzling.org>
Message-ID: <4B916062.90908@v.loewis.de>

Andrew Bennetts wrote:
> "Martin v. L?wis" wrote:
> [...]
>> Any proposal appreciated.
> 
> I propose screaming ?help me, I have written a test suite using nothing
> but string matching assertions, what is wrong with me?!?

Thanks a lot.

Regards,
Martin

From exarkun at twistedmatrix.com  Fri Mar  5 21:18:22 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Fri, 05 Mar 2010 20:18:22 -0000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <ca471dc21003051110pa728102j15c61e0c04d0ff1c@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com>
	<20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>
	<ca471dc21003051110pa728102j15c61e0c04d0ff1c@mail.gmail.com>
Message-ID: <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain>

On 07:10 pm, guido at python.org wrote:
>On Fri, Mar 5, 2010 at 10:30 AM,  <exarkun at twistedmatrix.com> wrote:
>>On 05:06 pm, curt at hagenlocher.org wrote:
>>>
>>>On Fri, Mar 5, 2010 at 8:35 AM, Jesse Noller <jnoller at gmail.com> 
>>>wrote:
>>>>
>>>>On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>>><daniel at stutzbachenterprises.com> wrote:
>>>> > On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan 
>>>><brian at sweetapp.com>
>>>> > wrote:
>>>> >>
>>>> >> import futures
>>>> >
>>>> > +1 on the idea, -1 on the name.? It's too similar to "from 
>>>>__future__
>>>> > import
>>>> > ...".
>>>>
>>>>Futures is a common term for this, and implemented named this in 
>>>>other
>>>>languages. I don't think we should be adopting things that are 
>>>>common,
>>>>and found elsewhere and then renaming them.
>>>
>>>Another common term for this is a "promise".
>>
>>Promises aren't exactly the same. ?This would be a particularly bad 
>>name to
>>apply here.
>
>Please explain. Even the Wikipedia article
>(http://en.wikipedia.org/wiki/Futures_and_promises), despite promising
>to explain the difference, didn't explain it.

The "explicit" futures on the wikipedia page seems to cover what is 
commonly referred to as a future.  For example, Java's futures look like 
this.

The "implicit" futures are what is generally called a promise.  For 
example, E's promises look like this.

Though the difference is mainly one of API, it turns out to make a 
significant difference in what you can accomplish.  Promises are much 
more amenable to the pipelining optimization, for example.  They're also 
much harder to implement in Python without core language changes.

Jean-Paul

From brett at python.org  Fri Mar  5 21:31:29 2010
From: brett at python.org (Brett Cannon)
Date: Fri, 5 Mar 2010 12:31:29 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com> 
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> 
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com> 
	<eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com> 
	<4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>
Message-ID: <bbaeab101003051231u76783e0eh5ec08731af70a521@mail.gmail.com>

On Fri, Mar 5, 2010 at 09:55, Jesse Noller <jnoller at gmail.com> wrote:

> On Fri, Mar 5, 2010 at 12:28 PM, Daniel Stutzbach
> <daniel at stutzbachenterprises.com> wrote:
> > On Fri, Mar 5, 2010 at 11:03 AM, Jesse Noller <jnoller at gmail.com> wrote:
> >>
> >> http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html
> >
> >  According to that link, Java has a module named "Concurrent" with an
> > interface named "Future".  You're proposing a module named "Futures" with
> a
> > class named "Future".
> >
> > Why not name your module "concurrent"?  That would eliminate the
> confusion
> > with "from __future__".  I don't see a problem with keeping the class
> name.
> >
> > Plus, a "concurrent" module might be useful for things other than
> Futures,
> > in the future. ;-)
> >
>
> Brian's module is named futures; I am +1'ing his proposal, and also
> suggesting we put it under concurrent/ package name. This means you
> would do the following:
>
> from concurrent import futures
>
> and in the future:
> from concurrent import pool
>
> And so on.



So I don't quite get what you are after here. Are you wanting to eventually
have a generic pool class that you can simply import and use that is always
set to the best option for the platform?

And as for moving stuff from multiprocessing into the concurrent namespace,
are you thinking like concurrent.multiprocessing? I guess I am just trying
to figure out what the abstraction is you are after in the package
namespace.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100305/abab7673/attachment.html>

From tseaver at palladion.com  Fri Mar  5 21:33:25 2010
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 05 Mar 2010 15:33:25 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
Message-ID: <hmrpql$b86$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesse Noller wrote:
> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
> <daniel at stutzbachenterprises.com> wrote:
>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>>> import futures
>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>> ...".
> 
> Futures is a common term for this, and implemented named this in other
> languages. I don't think we should be adopting things that are common,
> and found elsewhere and then renaming them.

- -1 to the name from me as well:  it isn't "scoped" properly to make it
clear what the module is about.  If they were inside a pacakge named
'concurrency' or some such (as hinted by Jesse Noller, I think), the
clash would go away.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuRapAACgkQ+gerLs4ltQ7dBwCfRmMuq6X9VE8usYgSScXEA1D0
1PsAoI8MR5hjPjq9C7MFPTZhcO/T+NM4
=7wpK
-----END PGP SIGNATURE-----


From brett at python.org  Fri Mar  5 21:38:43 2010
From: brett at python.org (Brett Cannon)
Date: Fri, 5 Mar 2010 12:38:43 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
Message-ID: <bbaeab101003051238xa8c8c45ka2ebb4ad0eabe3a2@mail.gmail.com>

The PEP says that futures.wait() should only use keyword arguments past its
first positional argument, but the PEP has the function signature as
``wait(fs, timeout=None, return_when=ALL_COMPLETED)``.  Should it be
``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``?

On Thu, Mar 4, 2010 at 22:03, Brian Quinlan <brian at sweetapp.com> wrote:

> Hi all,
>
> I recently submitted a daft PEP for a package designed to make it easier to
> execute Python functions asynchronously using threads and processes. It lets
> the user focus on their computational problem without having to build
> explicit thread/process pools and work queues.
>
> The package has been discussed on stdlib-sig but now I'd like this group's
> feedback.
>
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/
>
> Here are two examples to whet your appetites:
>
> """Determine if several numbers are prime."""
> import futures
> import math
>
> PRIMES = [
>    112272535095293,
>    112582705942171,
>    112272535095293,
>    115280095190773,
>    115797848077099,
>    1099726899285419]
>
> def is_prime(n):
>    if n % 2 == 0:
>        return False
>
>    sqrt_n = int(math.floor(math.sqrt(n)))
>    for i in range(3, sqrt_n + 1, 2):
>        if n % i == 0:
>            return False
>    return True
>
> # Uses as many CPUs as your machine has.
> with futures.ProcessPoolExecutor() as executor:
>    for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
>        print('%d is prime: %s' % (number, is_prime))
>
>
> """Print out the size of the home pages of various new sites (and Fox
> News)."""
> import futures
> import urllib.request
>
> URLS = ['http://www.foxnews.com/',
>        'http://www.cnn.com/',
>        'http://europe.wsj.com/',
>        'http://www.bbc.co.uk/',
>        'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
>    return urllib.request.urlopen(url, timeout=timeout).read()
>
> with futures.ThreadPoolExecutor(max_workers=5) as executor:
>    # Create a future for each URL load.
>    future_to_url = dict((executor.submit(load_url, url, 60), url)
>                         for url in URLS)
>
>    # Iterate over the futures in the order that they complete.
>    for future in futures.as_completed(future_to_url):
>        url = future_to_url[future]
>        if future.exception() is not None:
>            print('%r generated an exception: %s' % (url,
>                                                     future.exception()))
>        else:
>            print('%r page is %d bytes' % (url, len(future.result())))
>
> Cheers,
> Brian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100305/1fd342c9/attachment-0001.html>

From jnoller at gmail.com  Fri Mar  5 21:51:59 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 5 Mar 2010 15:51:59 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <bbaeab101003051231u76783e0eh5ec08731af70a521@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
	<eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com>
	<4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>
	<bbaeab101003051231u76783e0eh5ec08731af70a521@mail.gmail.com>
Message-ID: <4222a8491003051251o4d467f37m81ad74aa1fff4a5c@mail.gmail.com>

On Fri, Mar 5, 2010 at 3:31 PM, Brett Cannon <brett at python.org> wrote:

>
> So I don't quite get what you are after here. Are you wanting to eventually
> have a generic pool class that you can simply import and use that is always
> set to the best option for the platform?
> And as for moving stuff from multiprocessing into the concurrent namespace,
> are you thinking like concurrent.multiprocessing? I guess I am just trying
> to figure out what the abstraction is you are after in the package
> namespace.
> -Brett

My goal would be to put futures into a "concurrent" package - as it is
an abstraction that allows for threads, and processes to be used. By
default; I don't think we'd make a guess based on the platform, but
rather pick a sane default (such as threads).

After that; I would begin to remove chunks of multiprocessing (such as
pool) and adapt it to the same type of "pick threads or processes"
that futures uses. For example:

from concurrent import Pool

 x = Pool(4, worker_primitive=Thread())

And so on. The end-goal would be to make concurrent.* a package
containing common abstractions/patterns which can use threads or
processes interchangeably.

jesse

From guido at python.org  Fri Mar  5 21:55:46 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 Mar 2010 12:55:46 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com> 
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> 
	<d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com> 
	<20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>
	<ca471dc21003051110pa728102j15c61e0c04d0ff1c@mail.gmail.com> 
	<20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain>
Message-ID: <ca471dc21003051255t6dda3bb8hd35a4f590dfa7138@mail.gmail.com>

On Fri, Mar 5, 2010 at 12:18 PM,  <exarkun at twistedmatrix.com> wrote:
> The "explicit" futures on the wikipedia page seems to cover what is commonly
> referred to as a future. ?For example, Java's futures look like this.
>
> The "implicit" futures are what is generally called a promise. ?For example,
> E's promises look like this.

Fair enough, though the article confuses the matter by using the words
more or less interchangeably.

> Though the difference is mainly one of API, it turns out to make a
> significant difference in what you can accomplish. ?Promises are much more
> amenable to the pipelining optimization, for example. ?They're also much
> harder to implement in Python without core language changes.

That's why implicit futures (by any name) aren't on the table.

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Fri Mar  5 22:22:36 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 Mar 2010 13:22:36 -0800
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <4B908456.2070505@v.loewis.de>
References: <4B908456.2070505@v.loewis.de>
Message-ID: <ca471dc21003051322j489d0580kb8bbcf1a8ec06752@mail.gmail.com>

On Thu, Mar 4, 2010 at 8:11 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Johan Harjano ran into an interesting problem when trying to run the
> Django test suite under Python 3.1.
>
> Django has doctests of the form
>
>>>> a6.headline
> u'Default headline'
>
> Even when converting the doctest with 2to3, the expected output is
> unmodified. However, in 3.x, the expected output will change (i.e. not
> produce an u"" prefix anymore).
>
> Now, it might be possible to reformulate the test case (e.g. use print()
> instead of relying on repr), however, this is undesirable as a) the test
> should continue to test in 2.x that the result object is a unicode
> string, and b) it makes the test less readable.
>
> I would like to find a solution where this gets automatically corrected,
> e.g. through 2to3, or through changes to doctest, or through changes of
> str.__repr__.
>
> Any proposal appreciated.

How about a heuristic rule (which you have to explicitly select) that
changes u'XXX' into 'XXX' inside triply-quoted strings given certain
contexts, e.g. only at the start of the line, only if there is a
nearby preceding line starting with '>>>'? Exactly what context is of
the right strength will have to be determined experimentally; if there
are a lot of tests outputting things like [u'...'] or {u'...': u'...'}
the context may have to be made more liberal. Possibly \bu('.*'|".*")
would do it?

The issue shows (yet again) a general problem with doctests being
overspecified -- the test shouldn't care that the output starts with
'u', it should only care that the value is unicode, but there's no
easy way to express this in doctests. But since these doctests exist I
suggest that the practical way forward is to stick with them rather
than trying to reformulate all the tests.

-- 
--Guido van Rossum (python.org/~guido)

From martin at v.loewis.de  Fri Mar  5 22:39:30 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 05 Mar 2010 22:39:30 +0100
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <ca471dc21003051322j489d0580kb8bbcf1a8ec06752@mail.gmail.com>
References: <4B908456.2070505@v.loewis.de>
	<ca471dc21003051322j489d0580kb8bbcf1a8ec06752@mail.gmail.com>
Message-ID: <4B917A12.3030003@v.loewis.de>

> The issue shows (yet again) a general problem with doctests being
> overspecified -- the test shouldn't care that the output starts with
> 'u', it should only care that the value is unicode, but there's no
> easy way to express this in doctests. But since these doctests exist I
> suggest that the practical way forward is to stick with them rather
> than trying to reformulate all the tests.

Ok, I think I would prefer that. One approach I considered was to
override sys.displayhook, so that I can determine what parts of the
output are "single" repr results. Of course, this wouldn't catch repr
strings that were ultimately print()ed (not sure whether that
restriction would affect Django).

The other (more aggressive) approach is the heuristics you propose,
which may end up with false negatives. doctest already has a wildcard
matching flag, so it would grow another one.

Regards,
Martin


From jnoller at gmail.com  Fri Mar  5 22:42:51 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 5 Mar 2010 16:42:51 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <hmrpql$b86$1@dough.gmane.org>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<hmrpql$b86$1@dough.gmane.org>
Message-ID: <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com>

On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jesse Noller wrote:
>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>> <daniel at stutzbachenterprises.com> wrote:
>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>>>> import futures
>>> +1 on the idea, -1 on the name. ?It's too similar to "from __future__ import
>>> ...".
>>
>> Futures is a common term for this, and implemented named this in other
>> languages. I don't think we should be adopting things that are common,
>> and found elsewhere and then renaming them.
>
> - -1 to the name from me as well: ?it isn't "scoped" properly to make it
> clear what the module is about. ?If they were inside a pacakge named
> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
> clash would go away.

If people agree with this; do you feel the proposal of said namespace
should be a separate PEP, or piggy back on this? I don't want to piggy
back on Brian's hard work.

Jesse

From g.brandl at gmx.net  Fri Mar  5 23:01:23 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 05 Mar 2010 23:01:23 +0100
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <hmrmh8$hc$1@dough.gmane.org>
References: <4B908456.2070505@v.loewis.de> <hmrmh8$hc$1@dough.gmane.org>
Message-ID: <hmruuc$t9o$1@dough.gmane.org>

Am 05.03.2010 20:37, schrieb Terry Reedy:
> On 3/4/2010 11:11 PM, "Martin v. L?wis" wrote:
>> Johan Harjano ran into an interesting problem when trying to run the
>> Django test suite under Python 3.1.
>>
>> Django has doctests of the form
>>
>>>>> a6.headline
>> u'Default headline'
>>
>> Even when converting the doctest with 2to3, the expected output is
>> unmodified. However, in 3.x, the expected output will change (i.e. not
>> produce an u"" prefix anymore).
>>
>> Now, it might be possible to reformulate the test case (e.g. use print()
>> instead of relying on repr), however, this is undesirable as a) the test
>> should continue to test in 2.x that the result object is a unicode
>> string, and b) it makes the test less readable.
>>
>> I would like to find a solution where this gets automatically corrected,
>> e.g. through 2to3, or through changes to doctest, or through changes of
>> str.__repr__.
>>
>> Any proposal appreciated.
> 
> What is the easiest thing that works?
> 
> If 2to3 can fix string literals and fix code within doc strings, would 
> it be difficult to fix expected strings within doc strings?

Yes.  Expected output from doctests can be anything.  Doing conversions
on it that are correct only for Python code is potentially producing many
false positives.  Heuristics need to be applied that can get very intricate.

> On the otherhand, as Foord pointed out,

Prefect?

> the 'u' prefix is something of a 
> CPythonism not required by the language def, so an 'ignore leading u on 
> expected output' flag would be useful. But 'correcting'
> 
>  >>> a,b
> u'ah', u'hah'
> 
> is too much to expect. Doctest is both useful and fragile because it 
> *almost* imitates human testing methods.

Yes, it is documentation and tests at the same time :)

Georg


From guido at python.org  Fri Mar  5 23:01:04 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 Mar 2010 14:01:04 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com> 
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com> 
	<hmrpql$b86$1@dough.gmane.org>
	<4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com>
Message-ID: <ca471dc21003051401q7bd60f91jc04e26cc9a04277c@mail.gmail.com>

On Fri, Mar 5, 2010 at 1:42 PM, Jesse Noller <jnoller at gmail.com> wrote:
> On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver <tseaver at palladion.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Jesse Noller wrote:
>>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>> <daniel at stutzbachenterprises.com> wrote:
>>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>>>>> import futures
>>>> +1 on the idea, -1 on the name. ?It's too similar to "from __future__ import
>>>> ...".
>>>
>>> Futures is a common term for this, and implemented named this in other
>>> languages. I don't think we should be adopting things that are common,
>>> and found elsewhere and then renaming them.
>>
>> - -1 to the name from me as well: ?it isn't "scoped" properly to make it
>> clear what the module is about. ?If they were inside a pacakge named
>> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
>> clash would go away.
>
> If people agree with this; do you feel the proposal of said namespace
> should be a separate PEP, or piggy back on this? I don't want to piggy
> back on Brian's hard work.

A simple renaming of futures to concurrency.futures seems easy enough
to swallow. (Though I haven't kept track of what other modules the PEP
proposes.)

-- 
--Guido van Rossum (python.org/~guido)

From g.brandl at gmx.net  Fri Mar  5 23:06:33 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 05 Mar 2010 23:06:33 +0100
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
Message-ID: <hmrv82$t9o$2@dough.gmane.org>

Am 04.03.2010 21:20, schrieb Brett Cannon:
> 1) I miss not having the affected files listed in the subject line.

I actually like the (first line of the) commit message better.  It
communicates much better what the change is about (and for me, if I
want to look at/review it).

However, what I miss is the summary of affected files at the top of
the message.  Having to scroll down all of the diffs to find out if e.g.
a change to a documentation file is included is tedious, even if the
diffs are colored like they are for me.

> 2) The To field is set to hg at python.org <mailto:hg at python.org> which
> gets rejected as an invalid email address if you reply. Would be better
> to set it to python-checkins so that the habitual reply to a checkin
> won't get rejected.

+1.

Georg


From solipsis at pitrou.net  Fri Mar  5 23:54:04 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 5 Mar 2010 17:54:04 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
Message-ID: <20100305175404.7daeca3e@msiwind>

Le Fri, 5 Mar 2010 17:03:02 +1100,
Brian Quinlan <brian at sweetapp.com> a ?crit :
> 
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/

Ok, here is my take on it:

> cancel()
> 
> Attempt to cancel the call. If the call is currently being executed
> then it cannot be cancelled and the method will return False,
> otherwise the call will be cancelled and the method will return
> True.

I think it shouldn't return anything, and raise an exception if
cancelling failed. It is really an error condition, and ignoring the
result doesn't seem right.

> Future.running()
> 
> Return True if the call is currently being executed and cannot be
> cancelled.
> 
> Future.done()
> 
> Return True if the call was successfully cancelled or finished
> running.

These don't really make sense since the future is executing
concurrently. By the time the result is returned, it can already be
wrong. I advocate removing those two methods.

> The following Future methods are meant for use in unit tests and
> Executor implementations.

Their names should then be preceded by an underscore '_'. We don't want
people to think they are public APIs and start relying on them.

> wait(fs, timeout=None, return_when=ALL_COMPLETED)
> [...]
> 
> This method should always be called using keyword arguments

I don't think this is right. Keyword arguments are nice, but mandating
them too often is IMO a nuisance (after all, it makes things longer to
type and requires you to remember the exact parameter names).
Especially when the method only takes at most 3 arguments.

IMO, keyword-only arguments are mostly useful when there are a lot of
positional arguments before, and you want to help the user use the
right calling signature.

Regards

Antoine.



From ziga.seilnacht at gmail.com  Sat Mar  6 00:08:14 2010
From: ziga.seilnacht at gmail.com (=?windows-1252?Q?=8Eiga_Seilnacht?=)
Date: Sat, 06 Mar 2010 00:08:14 +0100
Subject: [Python-Dev] doctest, unicode repr, and 2to3
In-Reply-To: <4B908456.2070505@v.loewis.de>
References: <4B908456.2070505@v.loewis.de>
Message-ID: <4B918EDE.6070506@gmail.com>

Martin v. L?wis wrote:
> Johan Harjano ran into an interesting problem when trying to run the
> Django test suite under Python 3.1.
> 
> Django has doctests of the form
> 
> >>> a6.headline
> u'Default headline'
> 
> Even when converting the doctest with 2to3, the expected output is
> unmodified. However, in 3.x, the expected output will change (i.e. not
> produce an u"" prefix anymore).
> 
> Now, it might be possible to reformulate the test case (e.g. use print()
> instead of relying on repr), however, this is undesirable as a) the test
> should continue to test in 2.x that the result object is a unicode
> string, and b) it makes the test less readable.
> 
> I would like to find a solution where this gets automatically corrected,
> e.g. through 2to3, or through changes to doctest, or through changes of
> str.__repr__.
> 
> Any proposal appreciated.

You can use a custom DocTestRunner that replaces sys.displayhook in
its run() method and records the changed output. Something like the
attached seems to do the trick.

Regards,
Ziga
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: convert_doctest.py
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100306/381e9d33/attachment.ksh>

From tseaver at palladion.com  Sat Mar  6 00:12:17 2010
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 05 Mar 2010 18:12:17 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>	<hmrpql$b86$1@dough.gmane.org>
	<4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com>
Message-ID: <hms34h$cg1$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesse Noller wrote:
> On Fri, Mar 5, 2010 at 3:33 PM, Tres Seaver <tseaver at palladion.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Jesse Noller wrote:
>>> On Fri, Mar 5, 2010 at 11:21 AM, Daniel Stutzbach
>>> <daniel at stutzbachenterprises.com> wrote:
>>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com> wrote:
>>>>> import futures
>>>> +1 on the idea, -1 on the name.  It's too similar to "from __future__ import
>>>> ...".
>>> Futures is a common term for this, and implemented named this in other
>>> languages. I don't think we should be adopting things that are common,
>>> and found elsewhere and then renaming them.
>> - -1 to the name from me as well:  it isn't "scoped" properly to make it
>> clear what the module is about.  If they were inside a pacakge named
>> 'concurrency' or some such (as hinted by Jesse Noller, I think), the
>> clash would go away.
> 
> If people agree with this; do you feel the proposal of said namespace
> should be a separate PEP, or piggy back on this? I don't want to piggy
> back on Brian's hard work.

I'm just expressiong a preference for scoping the name, and don't want
to preempt the process.  If your proposed work on factoring common stuff
out of multiprocessing would sit in the same conceptual space, then
sharing the package name seems like a good plan to me.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuRj74ACgkQ+gerLs4ltQ5RNACeKYku88A9PBuQR46QTl7GrEwo
mPEAoLdYyi+TLGYFw4SRAIM8zBsNvwxr
=iPkb
-----END PGP SIGNATURE-----


From steven.bethard at gmail.com  Sat Mar  6 02:37:15 2010
From: steven.bethard at gmail.com (Steven Bethard)
Date: Fri, 5 Mar 2010 17:37:15 -0800
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <hmrjr4$kkd$1@dough.gmane.org>
References: <hmrjr4$kkd$1@dough.gmane.org>
Message-ID: <d11dcfba1003051737h514c44f6hddfb2afad07e250e@mail.gmail.com>

On Fri, Mar 5, 2010 at 10:51 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> I generally enjoy argparse, but one thing I find rather
> ugly and unpythonic.
>
> ? ?parser.add_argument ('--plot', action='store_true')
>
> Specifying the argument 'action' as a string is IMO ugly.

If it really bothers you, you can use:

    action=argparse._StoreTrueAction

instead. Undocumented (purposefully) but that's all argparse is doing
under the covers.

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

From jyasskin at gmail.com  Sat Mar  6 03:00:56 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Fri, 5 Mar 2010 18:00:56 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100305175404.7daeca3e@msiwind>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100305175404.7daeca3e@msiwind>
Message-ID: <5d44f72f1003051800n70e57782i7589ad584db5ee9f@mail.gmail.com>

On Fri, Mar 5, 2010 at 2:54 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Fri, 5 Mar 2010 17:03:02 +1100,
> Brian Quinlan <brian at sweetapp.com> a ?crit :
>>
>> The PEP lives here:
>> http://python.org/dev/peps/pep-3148/
>
> Ok, here is my take on it:
>
>> cancel()
>>
>> Attempt to cancel the call. If the call is currently being executed
>> then it cannot be cancelled and the method will return False,
>> otherwise the call will be cancelled and the method will return
>> True.
>
> I think it shouldn't return anything, and raise an exception if
> cancelling failed. It is really an error condition, and ignoring the
> result doesn't seem right.

The caller can't avoid the error here by querying the future, because
of the problem you point out below, so I'm inclined to think that "the
future was already started" should be a return value rather than an
exception (although that may be my C++ background showing through).
Would calling the method try_cancel() work better?

>> Future.running()
>>
>> Return True if the call is currently being executed and cannot be
>> cancelled.
>>
>> Future.done()
>>
>> Return True if the call was successfully cancelled or finished
>> running.
>
> These don't really make sense since the future is executing
> concurrently. By the time the result is returned, it can already be
> wrong. I advocate removing those two methods.

"done()" can only be wrong in one direction though. If it returns
True, it'll never again return False, which can be useful (although
perhaps only for polling, which might be an argument for removing it
anyway).

"running()" becomes True and then False again, so I agree with your
objection. A "started()" function would only go from False to True
once. Maybe that's a better function?

Jeffrey

From solipsis at pitrou.net  Sat Mar  6 03:28:44 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 5 Mar 2010 21:28:44 -0500
Subject: [Python-Dev] argparse ugliness
References: <hmrjr4$kkd$1@dough.gmane.org>
Message-ID: <20100305212844.6edbb0b7@msiwind>

Le Fri, 05 Mar 2010 13:51:15 -0500,
Neal Becker <ndbecker2 at gmail.com> a ?crit :
> I generally enjoy argparse, but one thing I find rather
> ugly and unpythonic.
> 
>     parser.add_argument ('--plot', action='store_true')

I would argue that a string is actually more Pythonic than
integers or anonymous objects repurposed as magic constants.
(I'm looking at things such as SEEK_SET and friends)



From stephen at xemacs.org  Sat Mar  6 06:47:03 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 06 Mar 2010 14:47:03 +0900
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <ca471dc21003051031i716237b1pc7cf06be255c51e3@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
	<eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com>
	<4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>
	<ca471dc21003051031i716237b1pc7cf06be255c51e3@mail.gmail.com>
Message-ID: <87eijyuhm0.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:

 > "Future" is a pretty standard CS term for this concept (as noted
 > "promise" is another),

I like the term "promise" better.  "Future" is very generic ("not now,
but later"), whereas a "promise" is something I don't get from you
now, but you will give me later.

The wikipedia article is not very helpful on the implicit vs. explicit
distinction.  As far as I can tell from it, that distinction isn't
really attached to "future" vs "promise."  The only distinction the
article described was in the context of the Alice language, where a
future = promise (read-only) plus resolver (mutator).  IMO that's not
a compelling reason for adopting "future" in Python.

From stephen at xemacs.org  Sat Mar  6 06:55:25 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 06 Mar 2010 14:55:25 +0900
Subject: [Python-Dev] [PEP 3148] futures -
	execute	computations	asynchronously
In-Reply-To: <20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com>
	<20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>
	<ca471dc21003051110pa728102j15c61e0c04d0ff1c@mail.gmail.com>
	<20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain>
Message-ID: <87d3ziuh82.fsf@uwakimon.sk.tsukuba.ac.jp>

exarkun at twistedmatrix.com writes:

 > The "explicit" futures on the wikipedia page seems to cover what is 
 > commonly referred to as a future.  For example, Java's futures look like 
 > this.
 > 
 > The "implicit" futures are what is generally called a promise.  For 
 > example, E's promises look like this.

*sigh*  All I can say is "it's a damned shame that there are no native
speakers of English working in computer science."<wink>

I have to admit Jean-Paul's explanation a pretty convincing reason for
adopting "future" rather than "promise".  But I'm with Skip, I would
prefer that the module be named "future" rather than "futures".
(Especially when wearing my Professional Economist sweatshirt. :-)

From pje at telecommunity.com  Sat Mar  6 07:11:14 2010
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 06 Mar 2010 01:11:14 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
Message-ID: <20100306061134.5AEE23A40A7@sparrow.telecommunity.com>

At 01:03 AM 3/5/2010, Brian Quinlan wrote:
>Hi all,
>
>I recently submitted a daft PEP for a package designed to make it
>easier to execute Python functions asynchronously using threads and
>processes. It lets the user focus on their computational problem
>without having to build explicit thread/process pools and work queues.
>
>The package has been discussed on stdlib-sig but now I'd like this
>group's feedback.

My immediate reaction is that this would be a lot more useful if it 
built on an API for coroutine yielding/interaction, similar to what's 
in say, Eventlet.  That would seem to make it easier to write 
synchronous-looking code that operates on futures, and allow futures 
to be composed more cleanly.

ISTM that if futures were standardized before a coroutine API, it 
would lead to effective orphaning ala what happened with asyncore, 
especially since the new library is, well, new.

I'm somewhat concerned that, as described, the proposed API adds 
little over what's relatively easy to do with a mature coroutine 
framework like Eventlet, while at the same time creating yet another 
alternative (and mutually incompatible) event loop system in the 
stdlib, beyond the ones that are already in asyncore, tkinter, and 
the various SocketServer subclasses.

As far as naming goes, Twisted uses the term "Deferred" for this 
concept (and also has a very mature API for handling them).

And speaking of Twisted, it seems to me that the PEP would be much 
improved in general by learning from some of the lessons of other 
systems.  I don't think that Java's example is really the best one to 
follow in this instance, compared to the many existing async 
frameworks that have Python-specific experience and APIs to learn from.

Or, to put it another way, something that worries me about this PEP 
is that nearly all of its Python-related citations are for 
*discussions* of futures, with the only previous Python 
implementation cited being a crude sketch of a cookbook recipe.  The 
PEP also doesn't address questions of interoperability with existing 
solutions, compare features with them, or even so much as say, "There 
are other production implementations of this concept in Python, but 
we are going to pretend they don't exist."  ;-)


From jyasskin at gmail.com  Sat Mar  6 07:19:30 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Fri, 5 Mar 2010 22:19:30 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
Message-ID: <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>

On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> I'm somewhat concerned that, as described, the proposed API ... [creates] yet another alternative (and
> mutually incompatible) event loop system in the stdlib ...

Futures are a blocking construct; they don't involve an event loop.

From pje at telecommunity.com  Sat Mar  6 07:50:12 2010
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 06 Mar 2010 01:50:12 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com
 >
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
Message-ID: <20100306065016.DBDF33A40A7@sparrow.telecommunity.com>

At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote:
>On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> > I'm somewhat concerned that, as described, the proposed API ... 
> [creates] yet another alternative (and
> > mutually incompatible) event loop system in the stdlib ...
>
>Futures are a blocking construct; they don't involve an event loop.

And where they block is in a loop, waiting for events (completed 
promises) coming back from other threads or processes.

The Motivation section of the PEP also stresses avoiding reinvention 
of such loops, and points to the complication of using more than one 
at a time as a justification for the mechanism.  It seems relevant to 
at least address why wrapping multiprocessing and multithreading is 
appropriate, but *not* dealing with any other form of sync/async 
boundary, *or* composition of futures.

On which subject, I might add, the PEP is silent on whether executors 
are reentrant to the called code.  That is, can I call a piece of 
code that uses futures, using the futures API?  How will the called 
code know what executor to use?  Must I pass it one explicitly?  Will 
that work across threads and processes, without explicit support from the API?

IOW, as far as I can tell from the PEP, it doesn't look like you can 
compose futures without *global* knowledge of the application...  and 
in and of itself, this seems to negate the PEP's own motivation to 
prevent duplication of parallel execution handling!

That is, if I use code from module A and module B that both want to 
invoke tasks asynchronously, and I want to invoke A and B 
asynchronously, what happens?  Based on the design of the API, it 
appears there is nothing you can do except refactor A and B to take 
an executor in a parameter, instead of creating their own.

It seems therefore to me that either the proposal does not define its 
scope/motivation very well, or it is not well-equipped to address the 
problem it's setting out to solve.  If it's meant to be something 
less ambitious -- more like a recipe or example -- it should properly 
motivate that scope.  If it's intended to be a robust tool for 
composing different pieces of code, OTOH, it should absolutely 
address the issue of writing composable code...  since, that seems to 
be what it says the purpose of the API is.  (I.e., composing code to 
use a common waiting loop.)

And, existing Python async APIs (such as Twisted's Deferreds) 
actually *address* this issue of composition; the PEP does 
not.  Hence my comments about not looking at existing implementations 
for API and implementation guidance.  (With respect to what the API 
needs, and how it needs to do it, not necessarily directly copying 
actual APIs or implementations.  Certainly some of the Deferred API 
naming has a rather, um, "twisted" vocabulary.)


From jyasskin at gmail.com  Sat Mar  6 08:14:36 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Fri, 5 Mar 2010 23:14:36 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <87eijyuhm0.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
	<eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com>
	<4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>
	<ca471dc21003051031i716237b1pc7cf06be255c51e3@mail.gmail.com>
	<87eijyuhm0.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <5d44f72f1003052314l178f1f94sd90f55967ba83b43@mail.gmail.com>

On Fri, Mar 5, 2010 at 9:47 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Guido van Rossum writes:
>
> ?> "Future" is a pretty standard CS term for this concept (as noted
> ?> "promise" is another),
>
> I like the term "promise" better. ?"Future" is very generic ("not now,
> but later"), whereas a "promise" is something I don't get from you
> now, but you will give me later.
>
> The wikipedia article is not very helpful on the implicit vs. explicit
> distinction. ?As far as I can tell from it, that distinction isn't
> really attached to "future" vs "promise." ?The only distinction the
> article described was in the context of the Alice language, where a
> future = promise (read-only) plus resolver (mutator). ?IMO that's not
> a compelling reason for adopting "future" in Python.

It seems like a good idea to follow the choice other languages have
used for the name (if they tend to agree) regardless of whether the
evil Java followed it too. So let's take a poll:

Io: Uses "future" to refer to the implicit kind
(http://www.iolanguage.com/scm/io/docs/IoGuide.html#Concurrency-Futures)
Alice ML: Uses "future" to refer to the implicit kind, and "promise"
to refer to a handle that can fill in the future
(http://www.ps.uni-saarland.de/alice/manual/futures.html)
Java: Uses "future" to refer to the explicit kind.
(http://java.sun.com/javase/6/docs/api/java/util/concurrent/Future.html)
AmbientTalk: Uses "future" to refer to something more like a deferred:
you register callbacks to run when the future is resolved.
(http://soft.vub.ac.be/amop/at/tutorial/actors#futures)
C++0x: Uses "future" to refer to the explicit kind; "promise"
similarly to AliceML, and "packaged_task" to get a future from a
callable. (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3035.pdf
section 30.6)
E: Uses "promise" to refer to the implicit kind, and "resolver" to
refer to a handle that can fill in the promise.
(http://wiki.erights.org/wiki/Promise)
Oz: A "future" is a read-only logic variable. I'm not entirely sure
what that means.
(http://www.mozart-oz.org/documentation/dstutorial/node2.html#label61)
PLT Scheme: Uses "future" to refer to the explicit kind.
(http://docs.plt-scheme.org/futures/index.html)
C#: Uses "Task<TResult>" to refer to the explicit kind.
(http://msdn.microsoft.com/en-us/library/dd321424(VS.100).aspx)
Id, from 1991: Used "I-structure" to refer to the implicit kind.
(http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.4920)
Scala: Uses "Future" to refer to the explicit kind.
(http://www.scala-lang.org/docu/files/api/scala/actors/Actor.html#%21%21%28Any%2CPartialFunction%5BAny%2CA%5D%29)

What languages did I miss? From this list, "future" seems to be the
most popular choice, and it doesn't seem to distinguish between the
implicit and explicit kinds.

Jeffrey

From brian at sweetapp.com  Sat Mar  6 10:20:39 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 6 Mar 2010 20:20:39 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
Message-ID: <7EFBC33B-06AD-4BAD-8173-005D6F08240E@sweetapp.com>


On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote:

> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com>  
> wrote:
> import futures
>
> +1 on the idea, -1 on the name.  It's too similar to "from  
> __future__ import ...".
>
> Also, the PEP should probably link to the discussions on stdlib-sig?

I thought about that but this discussion is spread over many threads  
and many months.

Cheers,
Brian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100306/be7647fb/attachment.html>

From brian at sweetapp.com  Sat Mar  6 10:32:24 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 6 Mar 2010 20:32:24 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <bbaeab101003051238xa8c8c45ka2ebb4ad0eabe3a2@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<bbaeab101003051238xa8c8c45ka2ebb4ad0eabe3a2@mail.gmail.com>
Message-ID: <258205C5-979D-416E-8D54-35438C909CFE@sweetapp.com>

On 6 Mar 2010, at 07:38, Brett Cannon wrote:

> The PEP says that futures.wait() should only use keyword arguments  
> past its first positional argument, but the PEP has the function  
> signature as ``wait(fs, timeout=None, return_when=ALL_COMPLETED)``.   
> Should it be ``wait(fs, *, timeout=None, return_when=ALL_COMPLETED)``?

Hi Brett,

That recommendation was designed to make it easy to change the API  
without breaking code.

I'd don't think that recommendation makes sense anymore any I'll  
update the PEP.

Cheers,
Brian

> On Thu, Mar 4, 2010 at 22:03, Brian Quinlan <brian at sweetapp.com>  
> wrote:
> Hi all,
>
> I recently submitted a daft PEP for a package designed to make it  
> easier to execute Python functions asynchronously using threads and  
> processes. It lets the user focus on their computational problem  
> without having to build explicit thread/process pools and work queues.
>
> The package has been discussed on stdlib-sig but now I'd like this  
> group's feedback.
>
> The PEP lives here:
> http://python.org/dev/peps/pep-3148/
>
> Here are two examples to whet your appetites:
>
> """Determine if several numbers are prime."""
> import futures
> import math
>
> PRIMES = [
>    112272535095293,
>    112582705942171,
>    112272535095293,
>    115280095190773,
>    115797848077099,
>    1099726899285419]
>
> def is_prime(n):
>    if n % 2 == 0:
>        return False
>
>    sqrt_n = int(math.floor(math.sqrt(n)))
>    for i in range(3, sqrt_n + 1, 2):
>        if n % i == 0:
>            return False
>    return True
>
> # Uses as many CPUs as your machine has.
> with futures.ProcessPoolExecutor() as executor:
>    for number, is_prime in zip(PRIMES, executor.map(is_prime,  
> PRIMES)):
>        print('%d is prime: %s' % (number, is_prime))
>
>
> """Print out the size of the home pages of various new sites (and  
> Fox News)."""
> import futures
> import urllib.request
>
> URLS = ['http://www.foxnews.com/',
>        'http://www.cnn.com/',
>        'http://europe.wsj.com/',
>        'http://www.bbc.co.uk/',
>        'http://some-made-up-domain.com/']
>
> def load_url(url, timeout):
>    return urllib.request.urlopen(url, timeout=timeout).read()
>
> with futures.ThreadPoolExecutor(max_workers=5) as executor:
>    # Create a future for each URL load.
>    future_to_url = dict((executor.submit(load_url, url, 60), url)
>                         for url in URLS)
>
>    # Iterate over the futures in the order that they complete.
>    for future in futures.as_completed(future_to_url):
>        url = future_to_url[future]
>        if future.exception() is not None:
>            print('%r generated an exception: %s' % (url,
>                                                      
> future.exception()))
>        else:
>            print('%r page is %d bytes' % (url, len(future.result())))
>
> Cheers,
> Brian
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100306/bd4b87eb/attachment.html>

From brian at sweetapp.com  Sat Mar  6 10:37:16 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 6 Mar 2010 20:37:16 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<hmrpql$b86$1@dough.gmane.org>
	<4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com>
Message-ID: <E2339DFA-FE8A-4554-97DF-D4BF29B04055@sweetapp.com>


On 6 Mar 2010, at 08:42, Jesse Noller wrote:
> If people agree with this; do you feel the proposal of said namespace
> should be a separate PEP, or piggy back on this? I don't want to piggy
> back on Brian's hard work.

It doesn't really matter to me.

We can either update this PEP to propose the concurrent.futures name  
or you can draft a more complete PEP that describes what other  
functionality should live in the concurrent package.

Cheers,
Brian

From brian at sweetapp.com  Sat Mar  6 10:52:46 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 6 Mar 2010 20:52:46 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100305175404.7daeca3e@msiwind>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100305175404.7daeca3e@msiwind>
Message-ID: <7E1039CA-2768-47BF-AD8C-8B12D3D6C20B@sweetapp.com>


On 6 Mar 2010, at 09:54, Antoine Pitrou wrote:

> Le Fri, 5 Mar 2010 17:03:02 +1100,
> Brian Quinlan <brian at sweetapp.com> a ?crit :
>>
>> The PEP lives here:
>> http://python.org/dev/peps/pep-3148/
>
> Ok, here is my take on it:
>
>> cancel()
>>
>> Attempt to cancel the call. If the call is currently being executed
>> then it cannot be cancelled and the method will return False,
>> otherwise the call will be cancelled and the method will return
>> True.
>
> I think it shouldn't return anything, and raise an exception if
> cancelling failed. It is really an error condition, and ignoring the
> result doesn't seem right.

In my experience with futures, canceling them is a best-effort  
optimization that people use when another future fails. For example:

futures = [executor.submit(CopyDirectory, src, dest) for dest in ...]
finished, unfinished = wait(futures, return_when=FIRST_EXCEPTION)
# If there are unfinished futures then there must have been a failure
for f in unfinished:
   # No reason to waste bandwidth copying files if the operation has  
already failed.
   f.cancel()
for f in finished():
   if f.exception():
     raise f.exception()

>> Future.running()
>>
>> Return True if the call is currently being executed and cannot be
>> cancelled.
>>
>> Future.done()
>>
>> Return True if the call was successfully cancelled or finished
>> running.
>
> These don't really make sense since the future is executing
> concurrently. By the time the result is returned, it can already be
> wrong. I advocate removing those two methods.

There methods are useful for logging - by displaying the count of  
pending, running and completed futures you can estimate the progress  
of the system.

>> The following Future methods are meant for use in unit tests and
>> Executor implementations.
>
> Their names should then be preceded by an underscore '_'. We don't  
> want
> people to think they are public APIs and start relying on them.

Actually, as discussed on the stdlib-sig, these methods are designed  
to make it possible for users to implement their own Executors so  
we'll have keep the interface stable.

>> wait(fs, timeout=None, return_when=ALL_COMPLETED)
>> [...]
>>
>> This method should always be called using keyword arguments
>
> I don't think this is right. Keyword arguments are nice, but mandating
> them too often is IMO a nuisance (after all, it makes things longer to
> type and requires you to remember the exact parameter names).
> Especially when the method only takes at most 3 arguments.
>
> IMO, keyword-only arguments are mostly useful when there are a lot of
> positional arguments before, and you want to help the user use the
> right calling signature.

I agree, I'll change this.

Cheers,
Brian


From brian at sweetapp.com  Sat Mar  6 11:32:50 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sat, 6 Mar 2010 21:32:50 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
Message-ID: <AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>


On 6 Mar 2010, at 17:50, Phillip J. Eby wrote:

> At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote:
>> On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby <pje at telecommunity.com 
>> > wrote:
>> > I'm somewhat concerned that, as described, the proposed API ...  
>> [creates] yet another alternative (and
>> > mutually incompatible) event loop system in the stdlib ...
>>
>> Futures are a blocking construct; they don't involve an event loop.
>
> And where they block is in a loop, waiting for events (completed  
> promises) coming back from other threads or processes.
>
> The Motivation section of the PEP also stresses avoiding reinvention  
> of such loops, and points to the complication of using more than one  
> at a time as a justification for the mechanism.  It seems relevant  
> to at least address why wrapping multiprocessing and multithreading  
> is appropriate, but *not* dealing with any other form of sync/async  
> boundary, *or* composition of futures.
>
> On which subject, I might add, the PEP is silent on whether  
> executors are reentrant to the called code.  That is, can I call a  
> piece of code that uses futures, using the futures API?  How will  
> the called code know what executor to use?  Must I pass it one  
> explicitly?  Will that work across threads and processes, without  
> explicit support from the API?

Executors are reentrant but deadlock is possible. There are two  
deadlock examples in the PEP.

>
> IOW, as far as I can tell from the PEP, it doesn't look like you can  
> compose futures without *global* knowledge of the application...   
> and in and of itself, this seems to negate the PEP's own motivation  
> to prevent duplication of parallel execution handling!
>
> That is, if I use code from module A and module B that both want to  
> invoke tasks asynchronously, and I want to invoke A and B  
> asynchronously, what happens?  Based on the design of the API, it  
> appears there is nothing you can do except refactor A and B to take  
> an executor in a parameter, instead of creating their own.

A and B could both use their own executor instances. You would need to  
refactor A and B if you wanted to manage thread and process counts  
globally.

> It seems therefore to me that either the proposal does not define  
> its scope/motivation very well, or it is not well-equipped to  
> address the problem it's setting out to solve.  If it's meant to be  
> something less ambitious -- more like a recipe or example -- it  
> should properly motivate that scope.  If it's intended to be a  
> robust tool for composing different pieces of code, OTOH, it should  
> absolutely address the issue of writing composable code...  since,  
> that seems to be what it says the purpose of the API is.  (I.e.,  
> composing code to use a common waiting loop.)

My original motivation when designing this module was having to deal  
with a lot of code that looks like this:

def get_some_user_info(user):
   x = make_ldap_call1(user)
   y = make_ldap_call2(user)
   z = [make_db_call(user, i) for i in something]

   # Do some processing with x, y, z and return a result

Doing these operations serially is too slow. So how do I parallelize  
them? Using the threading module is the obvious choice but having to  
create my own work/result queue every time I encounter this pattern is  
annoying. The futures module lets you write this as:

def get_some_user_info(user):
   with ThreadPoolExecutor(max_threads=10) as executor:
     x_future = executor.submit(make_ldap_call1, user)
     y_future = executor.submit(make_ldap_call2, user)
     z_futures = [executor.submit(make_db_call, user, i) for i in  
something]
    finished, _ = wait([x_future, y_future] + z_futures,  
return_when=FIRST_EXCEPTION)
  for f in finished:
    if f.exception():
      raise f.exception()
  x = x_future.result()
  y = y_future.result()
  z = [f.result() for f in z_futures]

   # Do some processing with x, y, z and return a result

> And, existing Python async APIs (such as Twisted's Deferreds)  
> actually *address* this issue of composition; the PEP does not.   
> Hence my comments about not looking at existing implementations for  
> API and implementation guidance.  (With respect to what the API  
> needs, and how it needs to do it, not necessarily directly copying  
> actual APIs or implementations.  Certainly some of the Deferred API  
> naming has a rather, um, "twisted" vocabulary.)

Using twisted (or any other asynchronous I/O framework) forces you to  
rewrite your I/O code. Futures do not.

Cheers,
Brian

From ncoghlan at gmail.com  Sat Mar  6 13:28:08 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 06 Mar 2010 22:28:08 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <E2339DFA-FE8A-4554-97DF-D4BF29B04055@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>	<hmrpql$b86$1@dough.gmane.org>	<4222a8491003051342y762fe18bi253cd1f81d8d36f6@mail.gmail.com>
	<E2339DFA-FE8A-4554-97DF-D4BF29B04055@sweetapp.com>
Message-ID: <4B924A58.4030000@gmail.com>

Brian Quinlan wrote:
> 
> On 6 Mar 2010, at 08:42, Jesse Noller wrote:
>> If people agree with this; do you feel the proposal of said namespace
>> should be a separate PEP, or piggy back on this? I don't want to piggy
>> back on Brian's hard work.
> 
> It doesn't really matter to me.
> 
> We can either update this PEP to propose the concurrent.futures name or
> you can draft a more complete PEP that describes what other
> functionality should live in the concurrent package.

I think a "concurrent.futures" name works - it gives the scoping desired
by the folks with an finance background and gives us a bucket for future
thread/process agnostic concurrency tools (such as a pools and message
queues).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From stephen at xemacs.org  Sat Mar  6 13:46:23 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 06 Mar 2010 21:46:23 +0900
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <5d44f72f1003052314l178f1f94sd90f55967ba83b43@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<19345.14292.321112.787896@montanaro.dyndns.org>
	<4222a8491003050903u569e76e2u590123ff7bcf2900@mail.gmail.com>
	<eae285401003050928l6bb8446frc1d9d94241344afe@mail.gmail.com>
	<4222a8491003050955o224c30f0gb2505f439a6cbf2b@mail.gmail.com>
	<ca471dc21003051031i716237b1pc7cf06be255c51e3@mail.gmail.com>
	<87eijyuhm0.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5d44f72f1003052314l178f1f94sd90f55967ba83b43@mail.gmail.com>
Message-ID: <871vfxvcrk.fsf@uwakimon.sk.tsukuba.ac.jp>

Jeffrey Yasskin writes:

 > It seems like a good idea to follow the choice other languages have
 > used for the name (if they tend to agree)

For the record, I've conceded that point.


From ncoghlan at gmail.com  Sat Mar  6 13:43:40 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 06 Mar 2010 22:43:40 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
Message-ID: <4B924DFC.9070207@gmail.com>

Brian Quinlan wrote:
>> IOW, as far as I can tell from the PEP, it doesn't look like you can
>> compose futures without *global* knowledge of the application...  and
>> in and of itself, this seems to negate the PEP's own motivation to
>> prevent duplication of parallel execution handling!
>>
>> That is, if I use code from module A and module B that both want to
>> invoke tasks asynchronously, and I want to invoke A and B
>> asynchronously, what happens?  Based on the design of the API, it
>> appears there is nothing you can do except refactor A and B to take an
>> executor in a parameter, instead of creating their own.
> 
> A and B could both use their own executor instances. You would need to
> refactor A and B if you wanted to manage thread and process counts
> globally.

You may want to consider providing global thread and process executors
in the futures module itself. Code which just wants to say "do this in
the background" without having to manage the lifecycle of its own
executor instance is then free to do so. I've had a lot of experience
with a framework that provides this and it is *very* convenient (it's
also a good way to avoid deadlocks due to synchronous notification APIs).


On PJE's broader point, async event loops with non-blocking I/O and
messages passed back to the event loop to indicate completion of
operations and relying on threads and processes to farm out tasks (which
you will later block on in order to retrieve the results) are completely
different programming models. This PEP doesn't change that - it just
makes certain aspects of the latter approach easier to handle.

Trying to design an API that can cope with either model strikes me as a
fool's errand. They differ at such a fundamental level that I don't see
how a hybrid API could be particularly optimal for either approach.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sat Mar  6 13:50:47 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 06 Mar 2010 22:50:47 +1000
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <hmrjr4$kkd$1@dough.gmane.org>
References: <hmrjr4$kkd$1@dough.gmane.org>
Message-ID: <4B924FA7.4050807@gmail.com>

Neal Becker wrote:
> I generally enjoy argparse, but one thing I find rather
> ugly and unpythonic.
> 
>     parser.add_argument ('--plot', action='store_true')
> 
> Specifying the argument 'action' as a string is IMO ugly.

It isn't ideal, but having to grab constants from the module isn't ideal
either.

Using a string here allows argparse (and optparse before it) to spit out
an explicit "unknown action" message. Using named constants a typo would
just get you a generic NameError instead. With variable names only being
checked at runtime, you wouldn't get the static name checking that is
the primary benefit of using named constants in less dynamic languages.

So, agreed that using strings isn't pretty, but there aren't any great
alternatives available.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From daniel at stutzbachenterprises.com  Sat Mar  6 16:35:53 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Sat, 6 Mar 2010 09:35:53 -0600
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B924DFC.9070207@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<4B924DFC.9070207@gmail.com>
Message-ID: <eae285401003060735j628edb3dv35937b9ea0335cae@mail.gmail.com>

On Sat, Mar 6, 2010 at 6:43 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> You may want to consider providing global thread and process executors
> in the futures module itself. Code which just wants to say "do this in
> the background" without having to manage the lifecycle of its own
> executor instance is then free to do so.


+1
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100306/14947a56/attachment.html>

From pje at telecommunity.com  Sat Mar  6 17:04:26 2010
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 06 Mar 2010 11:04:26 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
Message-ID: <20100306160434.39FBD3A4077@sparrow.telecommunity.com>

At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>Using twisted (or any other asynchronous I/O framework) forces you to
>rewrite your I/O code. Futures do not.

Twisted's "Deferred" API has nothing to do with I/O. 


From gregg.lind at gmail.com  Sat Mar  6 17:27:46 2010
From: gregg.lind at gmail.com (Gregg Lind)
Date: Sat, 6 Mar 2010 10:27:46 -0600
Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts
Message-ID: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com>

Python-devs,

I'm writing to you for some help in understanding the Python grammar.  As an
excuse to deep dive into Python's tokenizer / grammar, I decided (as a
hideous, hideous joke) to want to allow braces where colons are allowed (as
flow control).

Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer

As a first example:

funcdef: ('def' NAME parameters ['->' test] ':' suite |
          'def' NAME parameters ['->' test] '{' suite '}' )

I reran Parser/pgen and the dfa changes, but python (3.1) when recompiled,
throws errors on things like:

def a() { None }

Strangely enough:

lambdef: ( 'lambda' [varargslist] ':' test  |
               'lambda' [varargslist] '{' test '}' )

works fine!  I this simplely some difference between "test" and "suite".

I have tried tackling this with gdb, looking at err_input clearly isn't
enough.

    (gdb) break err_input
    (gdb) break PyParser_ASTFromString
    import sys
    b = compile("def a() {pass}","sys.stdout","single")
    # yet a simple grammar fix is enough for this!
    c = compile("lambda x {None}","sys.stdout","single")

I'm in over my head!

Any insights / help would be appreciated.  Full-on flaming is also
appropriate, but would be less appreciated.

Specific questions

1.)  I assume the Grammar/grammar is read top to bottom.  Confirm?
2.)  More help figuring out how to debug what python *thinks* it's seeing
when it see "def a() {pass}".  It's not getting to the ast construction
stage, as near as I can tell.  What additional breakpoints can I set to see
where it's failing.

Gregg L.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100306/1f13f9fd/attachment-0001.html>

From martin at v.loewis.de  Sat Mar  6 17:56:20 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 06 Mar 2010 17:56:20 +0100
Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts
In-Reply-To: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com>
References: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com>
Message-ID: <4B928934.2070103@v.loewis.de>

> 1.)  I assume the Grammar/grammar is read top to bottom.  Confirm?

Confirm - but this is not surprising: *any* source file is typically
read from top to bottom. Randoma access reading is typically done for
binary files, only.

So you must be asking something else, but I can't guess what that might be.

> 2.)  More help figuring out how to debug what python *thinks* it's
> seeing when it see "def a() {pass}".  It's not getting to the ast
> construction stage, as near as I can tell.  What additional breakpoints
> can I set to see where it's failing. 

Enable the D() debugging in parser.c (configure with --with-pydebug,
and set PYTHONDEBUG).

Regards,
Martin


From python-dev at masklinn.net  Sat Mar  6 18:26:00 2010
From: python-dev at masklinn.net (Xavier Morel)
Date: Sat, 6 Mar 2010 18:26:00 +0100
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <4B924FA7.4050807@gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org> <4B924FA7.4050807@gmail.com>
Message-ID: <C9209D7C-5C84-48AD-AA2A-BC971C471133@masklinn.net>

I don't believe argparse's action specification scheme is bad either, but

On 6 Mar 2010, at 13:50 , Nick Coghlan wrote:
> 
> you wouldn't get the static name checking that is
> the primary benefit of using named constants in less dynamic languages.

There are quite a few tools which do handle static checking of that kind
of stuff, so while Python itself doesn't provide it (as a builtin) it's
not like you can't get it some other way.

From gregg.lind at gmail.com  Sat Mar  6 19:26:25 2010
From: gregg.lind at gmail.com (Gregg Lind)
Date: Sat, 6 Mar 2010 12:26:25 -0600
Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts
In-Reply-To: <4B928934.2070103@v.loewis.de>
References: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com>
	<4B928934.2070103@v.loewis.de>
Message-ID: <ff30f7581003061026k2f39185aq553effc966597631@mail.gmail.com>

Sorry, re: question one, forgive the ill-formed question.  I meant more, are
the parser rules applied "first matching".   Essentially trying to confirm
that the parser is "top down" or "bottom up" or whether or not it even
matters.

Thanks for the tip -- it seems to be exactly what I want.  To make it
explicit, this seems to be fuller (unix) recipe for how to make this style
of debugging happen.

    $ ./configure --with-pydebug
    $ make
    $ set PYTHONDEBUG=1
    $ ./python -d   # then this shows the parsing info


On Sat, Mar 6, 2010 at 10:56 AM, "Martin v. L?wis" <martin at v.loewis.de>wrote:

> > 1.)  I assume the Grammar/grammar is read top to bottom.  Confirm?
>
> Confirm - but this is not surprising: *any* source file is typically
> read from top to bottom. Randoma access reading is typically done for
> binary files, only.
>
> So you must be asking something else, but I can't guess what that might be.
>
> > 2.)  More help figuring out how to debug what python *thinks* it's
> > seeing when it see "def a() {pass}".  It's not getting to the ast
> > construction stage, as near as I can tell.  What additional breakpoints
> > can I set to see where it's failing.
>
> Enable the D() debugging in parser.c (configure with --with-pydebug,
> and set PYTHONDEBUG).
>
> Regards,
> Martin
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100306/27598e33/attachment.html>

From brett at python.org  Sat Mar  6 20:43:20 2010
From: brett at python.org (Brett Cannon)
Date: Sat, 6 Mar 2010 11:43:20 -0800
Subject: [Python-Dev] Silencing DeprecationWarning by default in Python 2.7
	is silencing division warnings
Message-ID: <bbaeab101003061143s4095b346h1e973db3fe78d6ee@mail.gmail.com>

I see two possible fixes for this. One is to not silence DeprecationWarning
if Py_DivisionWarningFlag is set to >= 1. The other is to introduce a new
subclass of DeprecationWarning called IntegerDivisionWarning and have that
added to the warnings filter so that if it is triggered it is handled
separately from what DeprecationWarning triggers.

The former means that you might get more than you bargained for in terms of
warnings as you are suddenly switching on all DeprecationWarnings on top of
division warnings. The latter means that you now have to explicit care about
IntegerDivisionWarning on top of DeprecationWarning (to minimize this I
could have IntegerDivisionWarning added to the warnings filter only in the
case of when Py_DivisionWarningFlag is set instead of blindly adding it).

Thoughts?

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100306/359432b1/attachment.html>

From benjamin at python.org  Sat Mar  6 20:51:05 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 6 Mar 2010 13:51:05 -0600
Subject: [Python-Dev] Silencing DeprecationWarning by default in Python
	2.7 is silencing division warnings
In-Reply-To: <bbaeab101003061143s4095b346h1e973db3fe78d6ee@mail.gmail.com>
References: <bbaeab101003061143s4095b346h1e973db3fe78d6ee@mail.gmail.com>
Message-ID: <1afaf6161003061151i5415116ex3d13c713f46a3ab4@mail.gmail.com>

2010/3/6 Brett Cannon <brett at python.org>:
> I see two possible fixes for this. One is to not silence DeprecationWarning
> if Py_DivisionWarningFlag is set to >= 1. The other is to introduce a new
> subclass of DeprecationWarning called IntegerDivisionWarning and have that
> added to the warnings filter so that if it is triggered it is handled
> separately from what DeprecationWarning triggers.
> The former means that you might get more than you bargained for in terms of
> warnings as you are suddenly switching on all DeprecationWarnings on top of
> division warnings. The latter means that you now have to explicit care about
> IntegerDivisionWarning on top of DeprecationWarning (to minimize this I
> could have IntegerDivisionWarning added to the warnings filter only in the
> case of when Py_DivisionWarningFlag is set instead of blindly adding it).
> Thoughts?

How about just setting the warnings filter based on the integer
division warning message? Might be a little brittle, but I don't much
third party code is warning about classic division.



-- 
Regards,
Benjamin

From digitalxero at gmail.com  Sat Mar  6 22:20:10 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Sat, 6 Mar 2010 14:20:10 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100306160434.39FBD3A4077@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> 
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
Message-ID: <e9764b731003061320o69449787pbec117866a9f95b4@mail.gmail.com>

I have been playing with the feedback branch of this package for py3
and there seems to be a rather serious bug in the Process version.
Using the code @ http://dpaste.com/hold/168795/

When I was running in debug mode I found that as soon as

            p = multiprocessing.Process(
                    target=_process_worker,
                    args=(self._call_queue,
                          self._result_queue,
                          self._shutdown_process_event))

was called (yes even before p.start() was called) the processes just
started launching all by themselves.

I am also wondering why you are launching the process directly instead
of using a Pool since you are limiting the number of processes always
wouldnt it be better to launch the worker processes up front then just
add worker items to the queue?

From digitalxero at gmail.com  Sat Mar  6 22:40:34 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Sat, 6 Mar 2010 14:40:34 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <e9764b731003061320o69449787pbec117866a9f95b4@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> 
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<e9764b731003061320o69449787pbec117866a9f95b4@mail.gmail.com>
Message-ID: <e9764b731003061340iec149d3t6ad53c645d5d8e5c@mail.gmail.com>

I am also getting an odd error on odd error on exit

Error in atexit._run_exitfuncs:
TypeError: print_exception(): Exception expected for value, str found

From guido at python.org  Sat Mar  6 22:41:01 2010
From: guido at python.org (Guido van Rossum)
Date: Sat, 6 Mar 2010 13:41:01 -0800
Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts
In-Reply-To: <ff30f7581003061026k2f39185aq553effc966597631@mail.gmail.com>
References: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com> 
	<4B928934.2070103@v.loewis.de>
	<ff30f7581003061026k2f39185aq553effc966597631@mail.gmail.com>
Message-ID: <ca471dc21003061341t7d98184by42acd9b85e682c09@mail.gmail.com>

On Sat, Mar 6, 2010 at 10:26 AM, Gregg Lind <gregg.lind at gmail.com> wrote:
> Sorry, re: question one, forgive the ill-formed question.? I meant more, are
> the parser rules applied "first matching".?? Essentially trying to confirm
> that the parser is "top down" or "bottom up" or whether or not it even
> matters.

That's not how it works at all. I can't explain it in a few words --
but any text on LL(1) parsing should clarify this. The parser uses no
backtracking and a 1-token lookahead. The only unusual thing is that
individual rules use a regex-like notation, but that is all converted
to a DFA. If one token is not enough to know which path to take
through the DFA (this may invoke another rule -- but you always know
which one) you're hosed.

I suspect you've introduced ambiguities, though I don't immediately
see where (they could be in the combination of different rules).

Another possibility is that you may be running into problems where the
parser expects a newline at the end of a suite.

(FWIW since you're not proposing a language change, this is
technically off-topic for python-dev. :-)

--Guido

> Thanks for the tip -- it seems to be exactly what I want.? To make it
> explicit, this seems to be fuller (unix) recipe for how to make this style
> of debugging happen.
>
> ??? $ ./configure --with-pydebug
> ??? $ make
> ??? $ set PYTHONDEBUG=1
> ??? $ ./python -d?? # then this shows the parsing info
>
>
> On Sat, Mar 6, 2010 at 10:56 AM, "Martin v. L?wis" <martin at v.loewis.de>
> wrote:
>>
>> > 1.) ?I assume the Grammar/grammar is read top to bottom. ?Confirm?
>>
>> Confirm - but this is not surprising: *any* source file is typically
>> read from top to bottom. Randoma access reading is typically done for
>> binary files, only.
>>
>> So you must be asking something else, but I can't guess what that might
>> be.
>>
>> > 2.) ?More help figuring out how to debug what python *thinks* it's
>> > seeing when it see "def a() {pass}". ?It's not getting to the ast
>> > construction stage, as near as I can tell. ?What additional breakpoints
>> > can I set to see where it's failing.
>>
>> Enable the D() debugging in parser.c (configure with --with-pydebug,
>> and set PYTHONDEBUG).
>>
>> Regards,
>> Martin
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>



-- 
--Guido van Rossum (python.org/~guido)

From jnoller at gmail.com  Sat Mar  6 22:58:19 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 6 Mar 2010 16:58:19 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <e9764b731003061320o69449787pbec117866a9f95b4@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<e9764b731003061320o69449787pbec117866a9f95b4@mail.gmail.com>
Message-ID: <52B6BD6E-5678-4F74-89DE-EF3A47AD5C6A@gmail.com>



On Mar 6, 2010, at 4:20 PM, Dj Gilcrease <digitalxero at gmail.com> wrote:

> I have been playing with the feedback branch of this package for py3
> and there seems to be a rather serious bug in the Process version.
> Using the code @ http://dpaste.com/hold/168795/
>
> When I was running in debug mode I found that as soon as
>
>            p = multiprocessing.Process(
>                    target=_process_worker,
>                    args=(self._call_queue,
>                          self._result_queue,
>                          self._shutdown_process_event))
>
> was called (yes even before p.start() was called) the processes just
> started launching all by themselves.
>

Did you run the provided example code on windows by chance? If so,  
look at the multiprocessing docs, there are restrictions on windows  
(see the __main__ note) - not following the guidelines can result in  
lots of processes spawning.



From benjamin at python.org  Sat Mar  6 23:13:46 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 6 Mar 2010 16:13:46 -0600
Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate
Message-ID: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com>

On behalf of the Python development team, I'm pleased to announce a release
candidate for the second bugfix release of the Python 3.1 series, Python 3.1.2.

This bug fix release fixes numerous issues found in 3.1.1.  This release
candidate has been released to solicit testing and feedback over an possible
regressions from 3.1.1.  Please consider testing it with your library or
application and reporting an bugs you encounter.  This will help make the final
3.1.2 release, planned in 2 weeks time, all the more stable.

The Python 3.1 version series focuses on the stabilization and optimization of
the features and changes that Python 3.0 introduced.  For example, the new I/O
system has been rewritten in C for speed.  File system APIs that use unicode
strings now handle paths with undecodable bytes in them. Other features include
an ordered dictionary implementation, a condensed syntax for nested with
statements, and support for ttk Tile in Tkinter.  For a more extensive list of
changes in 3.1, see http://doc.python.org/3.1/whatsnew/3.1.html or Misc/NEWS in
the Python distribution.

To download Python 3.1.2rc1 visit:

     http://www.python.org/download/releases/3.1.2/

A list of changes in 3.1.2rc1 can be found here:

     http://svn.python.org/projects/python/tags/r312rc1/Misc/NEWS

The 3.1 documentation can be found at:

     http://docs.python.org/3.1

Bugs can always be reported to:

     http://bugs.python.org


Enjoy!

--
Benjamin Peterson
Release Manager
benjamin at python.org
(on behalf of the entire python-dev team and 3.1.2's contributors)

From benjamin at python.org  Sat Mar  6 23:41:33 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 6 Mar 2010 16:41:33 -0600
Subject: [Python-Dev] [RELEASED] Python 2.7 alpha 4
Message-ID: <1afaf6161003061441j5621bbe7jdb8ba4c22f81f46e@mail.gmail.com>

On behalf of the Python development team, I'm overjoyed to announce the fourth
alpha release of Python 2.7.

Python 2.7 is scheduled (by Guido and Python-dev) to be the last major version
in the 2.x series.  Though more major releases have not been absolutely ruled
out, it's likely that the 2.7 release will an extended period of maintenance for
the 2.x series.

2.7 includes many features that were first released in Python 3.1.  The faster
io module, the new nested with statement syntax, improved float repr, set
literals, dictionary views, and the memoryview object have been backported from
3.1. Other features include an ordered dictionary implementation, unittests
improvements, a new sysconfig module, and support for ttk Tile in Tkinter.  For
a more extensive list of changes in 2.7, see
http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python
distribution.

To download Python 2.7 visit:

     http://www.python.org/download/releases/2.7/

Please note that this is a development release, intended as a preview of new
features for the community, and is thus not suitable for production use.

The 2.7 documentation can be found at:

     http://docs.python.org/2.7

Please consider trying Python 2.7 with your code and reporting any bugs you may
notice to:

     http://bugs.python.org


Enjoy!

--
Benjamin Peterson
2.7 Release Manager
benjamin at python.org
(on behalf of the entire python-dev team and 2.7's contributors)

From ben+python at benfinney.id.au  Sat Mar  6 23:47:57 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 07 Mar 2010 09:47:57 +1100
Subject: [Python-Dev] [PEP 3148] futures -
	execute	computations	asynchronously
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com>
	<20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>
	<ca471dc21003051110pa728102j15c61e0c04d0ff1c@mail.gmail.com>
	<20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain>
	<87d3ziuh82.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <87hbot12zm.fsf@benfinney.id.au>

"Stephen J. Turnbull" <stephen at xemacs.org> writes:

> I have to admit Jean-Paul's explanation a pretty convincing reason for
> adopting "future" rather than "promise". But I'm with Skip, I would
> prefer that the module be named "future" rather than "futures".

Has anyone in this very long thread raised the issue that Python
*already* uses this term for the name of a module with a totally
unrelated purpose; the ?__future__? pseudo-module?

That alone seems a pretty strong reason to avoid the word ?future?
(singular or plural) for some other module name.

-- 
 \     ?Creativity can be a social contribution, but only in so far as |
  `\            society is free to use the results.? ?Richard Stallman |
_o__)                                                                  |
Ben Finney


From jnoller at gmail.com  Sat Mar  6 23:55:08 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 6 Mar 2010 17:55:08 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute	computations
	asynchronously
In-Reply-To: <87hbot12zm.fsf@benfinney.id.au>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com>
	<20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>
	<ca471dc21003051110pa728102j15c61e0c04d0ff1c@mail.gmail.com>
	<20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain>
	<87d3ziuh82.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87hbot12zm.fsf@benfinney.id.au>
Message-ID: <07376A42-ECA8-483D-BDDA-EEE6864EA6E8@gmail.com>



On Mar 6, 2010, at 5:47 PM, Ben Finney <ben+python at benfinney.id.au>  
wrote:

> "Stephen J. Turnbull" <stephen at xemacs.org> writes:
>
>> I have to admit Jean-Paul's explanation a pretty convincing reason  
>> for
>> adopting "future" rather than "promise". But I'm with Skip, I would
>> prefer that the module be named "future" rather than "futures".
>
> Has anyone in this very long thread raised the issue that Python
> *already* uses this term for the name of a module with a totally
> unrelated purpose; the ?__future__? pseudo-module?
>
> That alone seems a pretty strong reason to avoid the word ?future?
> (singular or plural) for some other module name.
>

Yes, they have, and putting it in a sub namespace has also come up. In  
the thread.

From greg.ewing at canterbury.ac.nz  Sun Mar  7 00:19:42 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 07 Mar 2010 12:19:42 +1300
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <5d44f72f1003051800n70e57782i7589ad584db5ee9f@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100305175404.7daeca3e@msiwind>
	<5d44f72f1003051800n70e57782i7589ad584db5ee9f@mail.gmail.com>
Message-ID: <4B92E30E.9070400@canterbury.ac.nz>

Jeffrey Yasskin wrote:

> The caller can't avoid the error here by querying the future, because
> of the problem you point out below, so I'm inclined to think that "the
> future was already started" should be a return value rather than an
> exception (although that may be my C++ background showing through).

I think it's your C++ background showing. In Python philosophy,
there's no particlular connection between whether something can
be tested for and whether it should raise an exception.

The thing to consider, I think, is whether it makes sense in
a large proportion of use cases to ignore the fact that the
cancel failed and carry on regardless. If not, then raising an
exception makes it much harder to accidentally ignore the
situation.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun Mar  7 00:28:17 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 07 Mar 2010 12:28:17 +1300
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <20100305212844.6edbb0b7@msiwind>
References: <hmrjr4$kkd$1@dough.gmane.org> <20100305212844.6edbb0b7@msiwind>
Message-ID: <4B92E511.3070304@canterbury.ac.nz>

Antoine Pitrou wrote:

> I would argue that a string is actually more Pythonic than
> integers or anonymous objects repurposed as magic constants.
> (I'm looking at things such as SEEK_SET and friends)

Strings are certainly preferable to ints, one reason being that
they display as something meaningful in debugging output.

But it also doesn't hurt to export named constants for them,
in the interests of catching typos.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun Mar  7 00:37:26 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 07 Mar 2010 12:37:26 +1300
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
Message-ID: <4B92E736.2010005@canterbury.ac.nz>

Phillip J. Eby wrote:
> while at the same time creating yet another alternative (and 
> mutually incompatible) event loop system in the stdlib, beyond the ones 
> that are already in asyncore, tkinter, and the various SocketServer 
> subclasses.

Aaargh... that's the *last* thing we need!

I've been thinking for a while that it would be a big help
if there were one, standardised module in the stdlib for
handling async events, and all the other gui toolkits
etc. were made to use it.

-- 
Greg

From fuzzyman at voidspace.org.uk  Sun Mar  7 00:38:51 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 06 Mar 2010 23:38:51 +0000
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B92E736.2010005@canterbury.ac.nz>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<4B92E736.2010005@canterbury.ac.nz>
Message-ID: <4B92E78B.6060503@voidspace.org.uk>

On 06/03/2010 23:37, Greg Ewing wrote:
> Phillip J. Eby wrote:
>> while at the same time creating yet another alternative (and mutually 
>> incompatible) event loop system in the stdlib, beyond the ones that 
>> are already in asyncore, tkinter, and the various SocketServer 
>> subclasses.
>
> Aaargh... that's the *last* thing we need!
>
> I've been thinking for a while that it would be a big help
> if there were one, standardised module in the stdlib for
> handling async events, and all the other gui toolkits
> etc. were made to use it.
>
Wouldn't it have to be the Tcl event loop then?

Michael

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From greg.ewing at canterbury.ac.nz  Sun Mar  7 00:59:20 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 07 Mar 2010 12:59:20 +1300
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <258205C5-979D-416E-8D54-35438C909CFE@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<bbaeab101003051238xa8c8c45ka2ebb4ad0eabe3a2@mail.gmail.com>
	<258205C5-979D-416E-8D54-35438C909CFE@sweetapp.com>
Message-ID: <4B92EC58.4090501@canterbury.ac.nz>

Brian Quinlan wrote:

> That recommendation was designed to make it easy to change the API 
> without breaking code.
> 
> I'd don't think that recommendation makes sense anymore any I'll update 
> the PEP.

I don't think there's anything wrong with stating that the
order of the arguments is not a guaranteed part of the API.
There isn't necessarily any need to enforce that, though.

-- 
Greg

From daniel at stutzbachenterprises.com  Sun Mar  7 00:52:01 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Sat, 6 Mar 2010 17:52:01 -0600
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B92E78B.6060503@voidspace.org.uk>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<4B92E736.2010005@canterbury.ac.nz>
	<4B92E78B.6060503@voidspace.org.uk>
Message-ID: <eae285401003061552w6b96a7a3ib40f3d60fff948f7@mail.gmail.com>

On Sat, Mar 6, 2010 at 5:38 PM, Michael Foord <fuzzyman at voidspace.org.uk>wrote:

> On 06/03/2010 23:37, Greg Ewing wrote:
>
>> I've been thinking for a while that it would be a big help
>> if there were one, standardised module in the stdlib for
>> handling async events, and all the other gui toolkits
>> etc. were made to use it.
>>
>>  Wouldn't it have to be the Tcl event loop then?
>

I image he means a standardized Abstract Base Class, which each GUI toolkit
would subclass.  That's more or less how Twisted's "reactors" work.  That
way non-GUI async code can just use the ABC and not worry about what event
loop is running underneath (be it TCL, GTK, or just poll()).
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100306/06829ac9/attachment.html>

From greg.ewing at canterbury.ac.nz  Sun Mar  7 01:19:41 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 07 Mar 2010 13:19:41 +1300
Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts
In-Reply-To: <ff30f7581003061026k2f39185aq553effc966597631@mail.gmail.com>
References: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com>
	<4B928934.2070103@v.loewis.de>
	<ff30f7581003061026k2f39185aq553effc966597631@mail.gmail.com>
Message-ID: <4B92F11D.30402@canterbury.ac.nz>

Gregg Lind wrote:
> Sorry, re: question one, forgive the ill-formed question.  I meant more, 
> are the parser rules applied "first matching".   Essentially trying to 
> confirm that the parser is "top down" or "bottom up" or whether or not 
> it even matters.

I think pgen would complain if you had two rules that could
both match at the same time, so the question doesn't apply.

BTW, be careful with terminology here -- the terms "top down"
and "bottom up" have a different meaning in parsing theory
(they refer to the way the parse tree is built, not the order
of matching in a list of rules).

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun Mar  7 01:37:42 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 07 Mar 2010 13:37:42 +1300
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <4B92E78B.6060503@voidspace.org.uk>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<4B92E736.2010005@canterbury.ac.nz> <4B92E78B.6060503@voidspace.org.uk>
Message-ID: <4B92F556.1080309@canterbury.ac.nz>

Michael Foord wrote:

> Wouldn't it have to be the Tcl event loop then?

No, tcl/tk would have to be threatened with the comfy chair
until it allowed itself to be spliced into the official
event loop somehow.

-- 
Greg

From skip at pobox.com  Sun Mar  7 02:02:48 2010
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 6 Mar 2010 19:02:48 -0600
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <87hbot12zm.fsf@benfinney.id.au>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<4222a8491003050835g117cf193k988cdb1772b28fa3@mail.gmail.com>
	<d2155e361003050906n3b8578cao7224b9e241f7c0c8@mail.gmail.com>
	<20100305183035.2792.884989761.divmod.xquotient.934@localhost.localdomain>
	<ca471dc21003051110pa728102j15c61e0c04d0ff1c@mail.gmail.com>
	<20100305201822.2792.1965005768.divmod.xquotient.976@localhost.localdomain>
	<87d3ziuh82.fsf@uwakimon.sk.tsukuba.ac.jp>
	<87hbot12zm.fsf@benfinney.id.au>
Message-ID: <19346.64312.597466.830641@montanaro.dyndns.org>


    Ben> Has anyone in this very long thread raised the issue that Python
    Ben> *already* uses this term for the name of a module with a totally
    Ben> unrelated purpose; the ???__future__??? pseudo-module?

Yes, it's already come up.  While not quite the same it does remind me of
the __builtin__/__builtins__ confusion.

Skip

From digitalxero at gmail.com  Sun Mar  7 02:47:46 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Sat, 6 Mar 2010 18:47:46 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <52B6BD6E-5678-4F74-89DE-EF3A47AD5C6A@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> 
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<e9764b731003061320o69449787pbec117866a9f95b4@mail.gmail.com> 
	<52B6BD6E-5678-4F74-89DE-EF3A47AD5C6A@gmail.com>
Message-ID: <e9764b731003061747m76255d7ofa8ff863b6301d60@mail.gmail.com>

On Sat, Mar 6, 2010 at 2:58 PM, Jesse Noller <jnoller at gmail.com> wrote:
> Did you run the provided example code on windows by chance? If so, look at
> the multiprocessing docs, there are restrictions on windows (see the
> __main__ note) - not following the guidelines can result in lots of
> processes spawning.


Yes, on win7. I might recommend then that the examples in the PEP be
restructured to work correctly on windows by including the

if __name__ == '__main__':

From brian at sweetapp.com  Sun Mar  7 03:10:26 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Sun, 7 Mar 2010 13:10:26 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100306160434.39FBD3A4077@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
Message-ID: <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>

On 7 Mar 2010, at 03:04, Phillip J. Eby wrote:

> At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>> Using twisted (or any other asynchronous I/O framework) forces you to
>> rewrite your I/O code. Futures do not.
>
> Twisted's "Deferred" API has nothing to do with I/O.

I see, you just mean the API and not the underlying model.

We discussed the Deferred API on the stdlib-sig and I don't think that  
anyone expressed a preference for it over the one described in the PEP.

Do you have any concrete criticism?

Cheers,
Brian

From exarkun at twistedmatrix.com  Sun Mar  7 03:34:54 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Sun, 07 Mar 2010 02:34:54 -0000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
Message-ID: <20100307023454.2792.1549472893.divmod.xquotient.1332@localhost.localdomain>

On 02:10 am, brian at sweetapp.com wrote:
>On 7 Mar 2010, at 03:04, Phillip J. Eby wrote:
>>At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>>>Using twisted (or any other asynchronous I/O framework) forces you to
>>>rewrite your I/O code. Futures do not.
>>
>>Twisted's "Deferred" API has nothing to do with I/O.
>
>I see, you just mean the API and not the underlying model.
>
>We discussed the Deferred API on the stdlib-sig and I don't think that 
>anyone expressed a preference for it over the one described in the PEP.
>
>Do you have any concrete criticism?

 From reading some of the stdlib-sig archives, it sounds like there is 
general agreement that Deferreds and Futures can be used to complement 
each other, and that getting code that is primarily Deferred-based to 
integrate with Future-based code or vice versa should eventually be 
possible.

Do I have the right sense of people's feelings?

And relatedly, once Futures are accepted and implemented, are people 
going to use them as an argument to exclude Deferreds from the stdlib 
(or be swayed by other people making such arguments)?  Hopefully not, 
given what I read on stdlib-sig, but it doesn't hurt to check...

Jean-Paul

From jnoller at gmail.com  Sun Mar  7 03:40:54 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 6 Mar 2010 21:40:54 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100307023454.2792.1549472893.divmod.xquotient.1332@localhost.localdomain>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307023454.2792.1549472893.divmod.xquotient.1332@localhost.localdomain>
Message-ID: <4222a8491003061840j78f84ecfs4dbb7997d1c27d21@mail.gmail.com>

On Sat, Mar 6, 2010 at 9:34 PM,  <exarkun at twistedmatrix.com> wrote:
> On 02:10 am, brian at sweetapp.com wrote:
>>
>> On 7 Mar 2010, at 03:04, Phillip J. Eby wrote:
>>>
>>> At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>>>>
>>>> Using twisted (or any other asynchronous I/O framework) forces you to
>>>> rewrite your I/O code. Futures do not.
>>>
>>> Twisted's "Deferred" API has nothing to do with I/O.
>>
>> I see, you just mean the API and not the underlying model.
>>
>> We discussed the Deferred API on the stdlib-sig and I don't think that
>> anyone expressed a preference for it over the one described in the PEP.
>>
>> Do you have any concrete criticism?
>
> From reading some of the stdlib-sig archives, it sounds like there is
> general agreement that Deferreds and Futures can be used to complement each
> other, and that getting code that is primarily Deferred-based to integrate
> with Future-based code or vice versa should eventually be possible.
>
> Do I have the right sense of people's feelings?
>
> And relatedly, once Futures are accepted and implemented, are people going
> to use them as an argument to exclude Deferreds from the stdlib (or be
> swayed by other people making such arguments)? ?Hopefully not, given what I
> read on stdlib-sig, but it doesn't hurt to check...
>
> Jean-Paul

Generally speaking; I don't see futures as an exclusion to Deferreds,
or other asynchronous doodads. I just see it as a useful construct on
top of threads and processes primarily. So in my mind, no.

jesse

From digitalxero at gmail.com  Sun Mar  7 04:09:59 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Sat, 6 Mar 2010 20:09:59 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> 
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
Message-ID: <e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>

After playing with the API for a while & running into many issues with
the examples & tests crashing windows I decided to modify the API a
little and fix up the examples so they dont crash windows based
computers.

http://code.google.com/p/pythonfutures/issues/detail?id=1

API Change that changes the current Executor to ExecutorBase and
adds a new Executor class that is used like

futures.Executor() # creates an executor that uses threading and a
max_workers = to the number of cpus

futures.Executor(use='process') # Creates an executor that uses
multiprocessing and a max_workers = to the number of cpus

futures.Executor(max_workers=5) # threading again, just specifying the
number of workers

futures.Executor(use='process', max_workers=5) # back to multiprocessing,
but with the max_workers specified

From pje at telecommunity.com  Sun Mar  7 04:56:57 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 06 Mar 2010 22:56:57 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
Message-ID: <20100307035657.111563A4077@sparrow.telecommunity.com>

At 01:10 PM 3/7/2010 +1100, Brian Quinlan wrote:
>On 7 Mar 2010, at 03:04, Phillip J. Eby wrote:
>
>>At 05:32 AM 3/6/2010, Brian Quinlan wrote:
>>>Using twisted (or any other asynchronous I/O framework) forces you to
>>>rewrite your I/O code. Futures do not.
>>
>>Twisted's "Deferred" API has nothing to do with I/O.
>
>I see, you just mean the API and not the underlying model.
>
>We discussed the Deferred API on the stdlib-sig and I don't think that
>anyone expressed a preference for it over the one described in the PEP.
>
>Do you have any concrete criticism?

Of the PEP, yes, absolutely, and I've already stated much of it.  My 
quibbles are with the PEP *itself*, not so much the API or implementation.

I think that said API and implementation is fine, but FAR too 
narrowly scoped to claim to be "futures" or "execute computations 
asynchronously", as the PEP calls it.  It's really just a nice task 
queuing system.

Now, if the PEP were *scoped* as such, i.e., "hey, let's just have a 
nice multithread/multiprocess task queuing implementation in the 
stdlib", I would be +1.  It's a handy utility to have.

But I think that the scope given by the PEP appears overly ambitious 
compared to what is actually being delivered; this seems less of a 
"futures API" and more like a couple of utility functions for waiting 
on threads and processes.

To rise to the level of an API, it seems to me that it would need to 
address interop with coroutines and async frameworks, where the idea 
of "futures" seems much more relevant than simple 
synchronous-but-parallel scripts.  (It should also have better tools 
for working with futures asynchronously, because, hey, it says right 
there in the title, "execute computations asynchronously".)

Anyway, I'd like to see the answers to (at *least*) the following 
issues fleshed out in the PEP, if you want it to really be a "futures 
API", vs. "nice task queue in the stdlib":

* Compare/contrast alternatives now available
* Address the issue of competing event loops and sharing/passing 
executors among code
* Either offer a way for executed code to re-enter its own executor 
(e.g. via an optional parameter), or explain why this was considered 
and rejected
* Address interoperability with coroutines and async frameworks, or 
clearly explain why such is out of scope

(Personally, I think it would be better to just drop the ambitious 
title and scope, and go for the "nice task queue" scope.  I imagine, 
too, that in that case Jean-Paul wouldn't need to worry about it 
being raised as a future objection to Deferreds or some such getting 
into the stdlib.)


From ncoghlan at gmail.com  Sun Mar  7 05:43:15 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 07 Mar 2010 14:43:15 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B92E30E.9070400@canterbury.ac.nz>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<20100305175404.7daeca3e@msiwind>	<5d44f72f1003051800n70e57782i7589ad584db5ee9f@mail.gmail.com>
	<4B92E30E.9070400@canterbury.ac.nz>
Message-ID: <4B932EE3.80908@gmail.com>

Greg Ewing wrote:
> The thing to consider, I think, is whether it makes sense in
> a large proportion of use cases to ignore the fact that the
> cancel failed and carry on regardless. If not, then raising an
> exception makes it much harder to accidentally ignore the
> situation.

Cancelling operations is generally a best effort activity - having to
wrap it an exception handler would be annoying.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sun Mar  7 05:49:02 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 07 Mar 2010 14:49:02 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100307035657.111563A4077@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
Message-ID: <4B93303E.9050301@gmail.com>

P.J. Eby wrote:
> (Personally, I think it would be better to just drop the ambitious title
> and scope, and go for the "nice task queue" scope.  I imagine, too, that
> in that case Jean-Paul wouldn't need to worry about it being raised as a
> future objection to Deferreds or some such getting into the stdlib.)

This may be a terminology thing - to me futures *are* just a nice way to
handle farming tasks out to worker threads or processes. You seem to see
them as something more comprehensive than that.

I agree the PEP should just target what the current implementation
provides and put whatever scope limitations are needed in the preamble
text to make that clear.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From g.brandl at gmx.net  Sun Mar  7 10:10:08 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 07 Mar 2010 10:10:08 +0100
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <20100305212844.6edbb0b7@msiwind>
References: <hmrjr4$kkd$1@dough.gmane.org> <20100305212844.6edbb0b7@msiwind>
Message-ID: <hmvqjt$mq6$1@dough.gmane.org>

Am 06.03.2010 03:28, schrieb Antoine Pitrou:
> Le Fri, 05 Mar 2010 13:51:15 -0500,
> Neal Becker <ndbecker2 at gmail.com> a ?crit :
>> I generally enjoy argparse, but one thing I find rather
>> ugly and unpythonic.
>> 
>>     parser.add_argument ('--plot', action='store_true')
> 
> I would argue that a string is actually more Pythonic than
> integers or anonymous objects repurposed as magic constants.
> (I'm looking at things such as SEEK_SET and friends)

Me too.  In absence of a separate symbol type, Python itself
uses strings for this purpose (hence the intern()ing business).

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From stefan_ml at behnel.de  Sun Mar  7 11:10:52 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 07 Mar 2010 11:10:52 +0100
Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate
In-Reply-To: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com>
References: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com>
Message-ID: <hmvu3c$she$1@dough.gmane.org>

Benjamin Peterson, 06.03.2010 23:13:
> A list of changes in 3.1.2rc1 can be found here:
>
>       http://svn.python.org/projects/python/tags/r312rc1/Misc/NEWS

Any reason the fix for bug 7173 didn't make it in? The test case that 
Amaury came up with seems to indicate that this a pretty serious issue.

http://bugs.python.org/issue7173

Stefan


From ndbecker2 at gmail.com  Sun Mar  7 13:29:25 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 07 Mar 2010 07:29:25 -0500
Subject: [Python-Dev] argparse ugliness
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
Message-ID: <hn0676$br0$1@dough.gmane.org>

Brian Curtin wrote:

> On Fri, Mar 5, 2010 at 12:51, Neal Becker <ndbecker2 at gmail.com> wrote:
> 
>> I generally enjoy argparse, but one thing I find rather
>> ugly and unpythonic.
>>
>>    parser.add_argument ('--plot', action='store_true')
>>
>> Specifying the argument 'action' as a string is IMO ugly.
>>
> 
> What else would you propose?
> FWIW, this is the same in optparse.

I would have thought use the object itself, instead of a string that spells 
the object's name.


From jnoller at gmail.com  Sun Mar  7 14:50:03 2010
From: jnoller at gmail.com (Jesse Noller)
Date: Sun, 7 Mar 2010 08:50:03 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
Message-ID: <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>

On Sat, Mar 6, 2010 at 10:09 PM, Dj Gilcrease <digitalxero at gmail.com> wrote:
> After playing with the API for a while & running into many issues with
> the examples & tests crashing windows I decided to modify the API a
> little and fix up the examples so they dont crash windows based
> computers.
>
> http://code.google.com/p/pythonfutures/issues/detail?id=1
>
> API Change that changes the current Executor to ExecutorBase and
> adds a new Executor class that is used like
>
> futures.Executor() # creates an executor that uses threading and a
> max_workers = to the number of cpus
>
> futures.Executor(use='process') # Creates an executor that uses
> multiprocessing and a max_workers = to the number of cpus
>
> futures.Executor(max_workers=5) # threading again, just specifying the
> number of workers
>
> futures.Executor(use='process', max_workers=5) # back to multiprocessing,
> but with the max_workers specified

Making the tests and examples happy on windows is fine; but some
explanation is needed for the API changes.

From solipsis at pitrou.net  Sun Mar  7 15:54:21 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 7 Mar 2010 09:54:21 -0500
Subject: [Python-Dev] Silencing DeprecationWarning by default in Python
 2.7 is silencing division warnings
References: <bbaeab101003061143s4095b346h1e973db3fe78d6ee@mail.gmail.com>
Message-ID: <20100307095421.6cf6c596@msiwind>

Le Sat, 6 Mar 2010 11:43:20 -0800,
Brett Cannon <brett at python.org> a ?crit :
> I see two possible fixes for this. One is to not silence
> DeprecationWarning if Py_DivisionWarningFlag is set to >= 1. The
> other is to introduce a new subclass of DeprecationWarning called
> IntegerDivisionWarning and have that added to the warnings filter so
> that if it is triggered it is handled separately from what
> DeprecationWarning triggers.

The latter would have my preference. Being able to discriminate between
various classes of deprecations is good. Especially when what you want
is to test for them :)




From solipsis at pitrou.net  Sun Mar  7 15:57:16 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 7 Mar 2010 09:57:16 -0500
Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate
References: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com>
	<hmvu3c$she$1@dough.gmane.org>
Message-ID: <20100307095716.53afe4cb@msiwind>

Le Sun, 07 Mar 2010 11:10:52 +0100,
Stefan Behnel <stefan_ml at behnel.de> a ?crit :
> 
> Any reason the fix for bug 7173 didn't make it in? The test case that 
> Amaury came up with seems to indicate that this a pretty serious
> issue.

The patch needs to properly include an unit test reproducing the test
case.
Also, a committer must step up and apply it. I am currently on vacation
and refrain from doing commits myself.

Regards

Antoine.



From pje at telecommunity.com  Sun Mar  7 16:48:09 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 07 Mar 2010 10:48:09 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <4B93303E.9050301@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
Message-ID: <20100307154809.742E73A4062@sparrow.telecommunity.com>

At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
>P.J. Eby wrote:
> > (Personally, I think it would be better to just drop the ambitious title
> > and scope, and go for the "nice task queue" scope.  I imagine, too, that
> > in that case Jean-Paul wouldn't need to worry about it being raised as a
> > future objection to Deferreds or some such getting into the stdlib.)
>
>This may be a terminology thing - to me futures *are* just a nice way to
>handle farming tasks out to worker threads or processes. You seem to see
>them as something more comprehensive than that.

Actual futures are, yes.  Specifically, futures are a mechanism for 
asynchronous computation, whereas the PEP seems to be all about 
synchronously managing parallel tasks.  That's a huge difference.

Technically, the things in the PEP (and by extension, Java's futures) 
match the letter of the definition of a future, but not (IMO) the 
spirit.  There's no clean way to compose them, and at base they're 
more about parallelism than asynchrony.


>I agree the PEP should just target what the current implementation
>provides and put whatever scope limitations are needed in the preamble
>text to make that clear.

Yep.  I'm just saying "parallel task queueing" is a much better 
description of what the implementation is/does, and would suggest 
renaming Future -> Task and Executor -> WorkerPool or some 
such.  These names would be *much* clearer to people who've never 
heard of futures, as well as more appropriate to the actual scope of 
what this does.


From jyasskin at gmail.com  Sun Mar  7 17:39:55 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sun, 7 Mar 2010 08:39:55 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100307154809.742E73A4062@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
Message-ID: <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>

On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
>>
>> P.J. Eby wrote:
>> > (Personally, I think it would be better to just drop the ambitious title
>> > and scope, and go for the "nice task queue" scope. ?I imagine, too, that
>> > in that case Jean-Paul wouldn't need to worry about it being raised as a
>> > future objection to Deferreds or some such getting into the stdlib.)
>>
>> This may be a terminology thing - to me futures *are* just a nice way to
>> handle farming tasks out to worker threads or processes. You seem to see
>> them as something more comprehensive than that.
>
> Actual futures are, yes. ?Specifically, futures are a mechanism for
> asynchronous computation, whereas the PEP seems to be all about
> synchronously managing parallel tasks. ?That's a huge difference.
>
> Technically, the things in the PEP (and by extension, Java's futures) match
> the letter of the definition of a future, but not (IMO) the spirit. ?There's
> no clean way to compose them, and at base they're more about parallelism
> than asynchrony.

Do you have an example of a language or library that uses the term
"future" to refer to what you're talking about? I'm curious to see
what it looks like.

From millbr02 at luther.edu  Sun Mar  7 18:36:17 2010
From: millbr02 at luther.edu (Brad Miller)
Date: Sun, 7 Mar 2010 11:36:17 -0600
Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate
In-Reply-To: <20100307095716.53afe4cb@msiwind>
References: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com>
	<hmvu3c$she$1@dough.gmane.org> <20100307095716.53afe4cb@msiwind>
Message-ID: <e45086041003070936i58cf0f00ne7b233c1f5590581@mail.gmail.com>

I download the source snapshot

./configure --enable-framework
make

...

/usr/bin/install -c -d -m 755 Python.framework/Versions/3.1
if test ""; then \
                gcc -o Python.framework/Versions/3.1/Python  -dynamiclib \
                        -isysroot "" \
                        -all_load libpython3.1.a -Wl,-single_module \
                        -install_name
/Library/Frameworks/Python.framework/Versions/3.1/Python \
                        -compatibility_version 3.1 \
                        -current_version 3.1 \
                        -framework CoreFoundation -ldl  -framework
CoreFoundation; \
        else \
                /usr/bin/libtool -o Python.framework/Versions/3.1/Python
-dynamic  libpython3.1.a \
                         -lSystem -lSystemStubs -arch_only x86_64
-install_name /Library/Frameworks/Python.framework/Versions/3.1/Python
-compatibility_version 3.1 -current_version 3.1 -framework CoreFoundation
-ldl  -framework CoreFoundation;\
        fi

Undefined symbols:
  "_libintl_bindtextdomain", referenced from:
      _PyIntl_bindtextdomain in libpython3.1.a(_localemodule.o)
  "_libintl_textdomain", referenced from:
      _PyIntl_textdomain in libpython3.1.a(_localemodule.o)
  "_libintl_dgettext", referenced from:
      _PyIntl_dgettext in libpython3.1.a(_localemodule.o)
  "_libintl_gettext", referenced from:
      _PyIntl_gettext in libpython3.1.a(_localemodule.o)
  "_libintl_dcgettext", referenced from:
      _PyIntl_dcgettext in libpython3.1.a(_localemodule.o)
ld: symbol(s) not found
/usr/bin/libtool: internal link edit command failed

I'm on 10.6.2, gcc version 4.2.1

Any hints on where to go from here?


On Sun, Mar 7, 2010 at 8:57 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Sun, 07 Mar 2010 11:10:52 +0100,
> Stefan Behnel <stefan_ml at behnel.de> a ?crit :
> >
> > Any reason the fix for bug 7173 didn't make it in? The test case that
> > Amaury came up with seems to indicate that this a pretty serious
> > issue.
>
> The patch needs to properly include an unit test reproducing the test
> case.
> Also, a committer must step up and apply it. I am currently on vacation
> and refrain from doing commits myself.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/bmiller%40luther.edu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100307/41d21169/attachment-0001.html>

From pje at telecommunity.com  Sun Mar  7 18:57:31 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 07 Mar 2010 12:57:31 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.co
 m>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
Message-ID: <20100307175732.307513A4062@sparrow.telecommunity.com>

At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby <pje at telecommunity.com> wrote:
> > At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
> >>
> >> P.J. Eby wrote:
> >> > (Personally, I think it would be better to just drop the ambitious title
> >> > and scope, and go for the "nice task queue" scope. ? I 
> imagine, too, that
> >> > in that case Jean-Paul wouldn't need to worry about it being raised as a
> >> > future objection to Deferreds or some such getting into the stdlib.)
> >>
> >> This may be a terminology thing - to me futures *are* just a nice way to
> >> handle farming tasks out to worker threads or processes. You seem to see
> >> them as something more comprehensive than that.
> >
> > Actual futures are, yes. ? Specifically, futures are a mechanism for
> > asynchronous computation, whereas the PEP seems to be all about
> > synchronously managing parallel tasks. ? That's a huge difference.
> >
> > Technically, the things in the PEP (and by extension, Java's futures) match
> > the letter of the definition of a future, but not (IMO) the 
> spirit. ? There's
> > no clean way to compose them, and at base they're more about parallelism
> > than asynchrony.
>
>Do you have an example of a language or library that uses the term
>"future" to refer to what you're talking about? I'm curious to see
>what it looks like.

The wikipedia page menetioned earlier in the discussion has several 
examples.  Twisted's Deferreds are probably the best example of such 
a system in Python, though, as far as I know.  The features proposed 
in the PEP are basically akin to Twisted's deferToThread(), i.e. 
"return a future for running this code in some other thread".

However, you could easily make your own thread or process pool for 
executing tasks, using Deferreds as your "future" objects; 
deferToThread() is not particularly special, it's just one possible 
"executor", to use the PEP's terminology.  Likewise, deferreds are 
not limited to representing parallel task execution - they can also 
represent asynchronous events such as incoming I/O or a waited-for mouse click.

Deferreds allow composing calculation by stacking callbacks (in 
continuation-passing style), but they're most Pythonic (IMO) when 
used with a coroutine wrapping system like Eventlet or my earlier 
peak.events framework, so that you can use them in "yield" 
expressions.  That way you can write synchronous-looking code that 
nonetheless executes asynchronously.

Twisted uses Deferreds extensively, both for I/O and IPC, and of 
course for database access.  However, the 'defer' module itself is 
conceptually distinct from both the I/O system, event loop, and 
threads: it's a largely-generic framework for managing asynchronous 
computation.  Parallel task queueing is simply *one* possible 
application of Deferreds; their actual purpose is to provide a common 
API for working with values that aren't yet known.

That is to say, futures.

In contrast, the PEP manages only a very limited sort of "future" - 
the completion of a single task, which must be synchronously waited 
for.  If one "future" in the PEP needs to wait for another, you tie 
up that thread or process in a busy loop...  which rather limits how 
much parallelism you can have.

Deferreds also allow you to write code that doesn't *care* whether 
the value has been determined or not...  you can simply stack on what 
code you want to have run when the value is available, and it runs 
immediately if the value is there.

Likewise, a deferred is not tied to the execution of a single piece 
of code - it's more of a rendezvous or synchronization point, that 
can be given a value to "fire" with.  (In contrast, the PEP's 
"futures" are tightly coupled to a task execution model: they have a 
notion of running and being cancelled, for example.)

For these reasons, I'd say that Deferreds are much more deserving of 
the title "futures", especially since there is no need to call a 
simple task queue a "futures" system.  (Unless you're Java, I 
suppose, in which case you might want the name for marketing 
reasons.  ;-) But in Python I assume we'd like to call a spade a 
spade, and a task queue a task queue.)


From rdmurray at bitdance.com  Sun Mar  7 19:45:52 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 07 Mar 2010 13:45:52 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100307154809.742E73A4062@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
Message-ID: <20100307184552.BF15E1AB54F@kimball.webabinitio.net>

On Sun, 07 Mar 2010 10:48:09 -0500, "P.J. Eby" <pje at telecommunity.com> wrote:
> At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote:
> >I agree the PEP should just target what the current implementation
> >provides and put whatever scope limitations are needed in the preamble
> >text to make that clear.
> 
> Yep.  I'm just saying "parallel task queueing" is a much better 
> description of what the implementation is/does, and would suggest 
> renaming Future -> Task and Executor -> WorkerPool or some 
> such.  These names would be *much* clearer to people who've never 
> heard of futures, as well as more appropriate to the actual scope of 
> what this does.

For what it's worth: I don't have any particular knowledge in this area.
I did loosely follow the stdlib-sig discussion.  I wasn't really sure
exactly what the module was about or what a 'future' was, or why I would
want to use one.  I did get that it was about parallel execution of tasks,
but it seemed like there had to be more to it than that.  Hearing it
called a 'worker pool' makes a lightbulb go off and I can now understand
why this would be a useful facility to have in the standard library.

--
R. David Murray                                      www.bitdance.com

From jyasskin at gmail.com  Sun Mar  7 19:59:50 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sun, 7 Mar 2010 10:59:50 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100307175732.307513A4062@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
Message-ID: <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>

On Sun, Mar 7, 2010 at 9:57 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>> Do you have an example of a language or library that uses the term
>> "future" to refer to what you're talking about? I'm curious to see
>> what it looks like.
>
> The wikipedia page menetioned earlier in the discussion has several
> examples. ?Twisted's Deferreds are probably the best example of such a
> system in Python, though, as far as I know. ?The features proposed in the
> PEP are basically akin to Twisted's deferToThread(), i.e. "return a future
> for running this code in some other thread".

Nearly all of the wikipedia page's examples refer to exactly this kind
of system, but with syntax to support it instead of putting it in a
library. The distinction between futures and deferreds is that futures
are designed to block, while deferreds are designed to continue
through callbacks. We could say that futures are to deferreds as
threading.Lock is to mutex.mutex.

>?That way you
> can write synchronous-looking code that nonetheless executes asynchronously.

 "Asynchronous" in the PEP refers to how the task behind the future
runs, not how users wait for the future. Is there a better way to say
that?

> Twisted uses Deferreds extensively, both for I/O and IPC, and of course for
> database access. ?However, the 'defer' module itself is conceptually
> distinct from both the I/O system, event loop, and threads: it's a
> largely-generic framework for managing asynchronous computation. ?Parallel
> task queueing is simply *one* possible application of Deferreds; their
> actual purpose is to provide a common API for working with values that
> aren't yet known.
>
> That is to say, futures.
>
> In contrast, the PEP manages only a very limited sort of "future" - the
> completion of a single task, which must be synchronously waited for. ?If one
> "future" in the PEP needs to wait for another, you tie up that thread or
> process in a busy loop... ?which rather limits how much parallelism you can
> have.

So is it that you just don't like the idea of blocking, and want to
stop anything that relies on it from getting into the standard
library?

Given the set_result and set_exception methods, it's pretty
straightforward to fill in the value of a future from something that
isn't purely computational. Given a way to register "on-done"
callbacks with the future, it would be straightforward to wait for a
future without blocking, too.

Jeffrey

From guido at python.org  Sun Mar  7 20:49:29 2010
From: guido at python.org (Guido van Rossum)
Date: Sun, 7 Mar 2010 11:49:29 -0800
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <hn0676$br0$1@dough.gmane.org>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com> 
	<hn0676$br0$1@dough.gmane.org>
Message-ID: <ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>

On Sun, Mar 7, 2010 at 4:29 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> Brian Curtin wrote:
>
>> On Fri, Mar 5, 2010 at 12:51, Neal Becker <ndbecker2 at gmail.com> wrote:
>>
>>> I generally enjoy argparse, but one thing I find rather
>>> ugly and unpythonic.
>>>
>>> ? ?parser.add_argument ('--plot', action='store_true')
>>>
>>> Specifying the argument 'action' as a string is IMO ugly.
>>>
>>
>> What else would you propose?
>> FWIW, this is the same in optparse.
>
> I would have thought use the object itself, instead of a string that spells
> the object's name.

What object? How would you write the example instead then?

-- 
--Guido van Rossum (python.org/~guido)

From pje at telecommunity.com  Sun Mar  7 20:56:13 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 07 Mar 2010 14:56:13 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.co
 m>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
Message-ID: <20100307195614.BBD693A4062@sparrow.telecommunity.com>

At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>So is it that you just don't like the idea of blocking, and want to 
>stop anything that relies on it from getting into the standard library?

Um, no.  As I said before, call it a "parallel task queue" or 
"parallel task manager" or something to that general effect and I'm on board.

It may not be in the Zen of Python, but ISTM that names should 
generally follow use cases.  It is something of a corollary to "one 
obvious way to do it", in that if you see something whose name 
matches what you want to do, then it should be obvious that that's 
the way in question.  ;-)

The use cases for "parallel task queues", however, are a subset of 
those for "futures" in the general case.  Since the proposed module 
addresses most of the former but very little of the latter, calling 
it futures is inappropriate.

Specifically, it's:

1. Confusing to people who don't know what futures are (see e.g R.D. 
Murray's post), and

2. Underpowered for people who expect/want a more fully-featured 
futures system along the lines of E or Deferreds.

It seems that the only people for whom it's an intuitively correct 
description are people who've only had experience with more limited 
futures models (like Java's).  However, these people should not have 
a problem understanding the notion of parallel task queueing or task 
management, so changing the name isn't really a loss for them, and 
it's a gain for everybody else.


>  Given the set_result and set_exception methods, it's pretty 
> straightforward to fill in the value of a future from something 
> that isn't purely computational.

Those are described as "internal" methods in the PEP; by contrast, 
the Deferred equivalents are part of the public API.


>  Given a way to register "on-done" callbacks with the future, it 
> would be straightforward to wait for a future without blocking, too.

Yes, and with a few more additions besides that one, you might be on 
the way to an actual competitor for Deferreds.  For example: retry 
support, chaining, logging, API for transparent result processing, 
coroutine support, co-ordination tools like locks, sempaphores and queues, etc.

These are all things you would very likely want or need if you 
actually wanted to write a program using futures as *your main 
computational model*, vs. just needing to toss out some parallel 
tasks in a primarily synchronous program.

Of course, Deferreds are indeed overkill if all you're ever going to 
want is a few parallel tasks, unless you're already skilled in using 
Twisted or some wrapper for it.

So, I totally support having a simple task queue in the stdlib, as 
there are definitely times I would've used such a thing for a quick 
script, if it were available.

However, I've *also* had use cases for using futures as a 
computational model, and so that's what I originally thought this PEP 
was about.  After the use cases were clarified, though, it seems to 
me that *calling* it futures is a bad idea, because it's really just 
a nice task queuing system.

I'm +1 on adding a nice task queuing system, -1 on calling it by any 
other name.  ;-)


From brett at python.org  Sun Mar  7 21:35:02 2010
From: brett at python.org (Brett Cannon)
Date: Sun, 7 Mar 2010 12:35:02 -0800
Subject: [Python-Dev] [RELEASED] Python 3.1.2 release candidate
In-Reply-To: <e45086041003070936i58cf0f00ne7b233c1f5590581@mail.gmail.com>
References: <1afaf6161003061413k646e972eta515d9322793e208@mail.gmail.com> 
	<hmvu3c$she$1@dough.gmane.org> <20100307095716.53afe4cb@msiwind> 
	<e45086041003070936i58cf0f00ne7b233c1f5590581@mail.gmail.com>
Message-ID: <bbaeab101003071235y20321571lb491d7e0424a21ac@mail.gmail.com>

Do you have MacPorts or Fink installed? If you do then you have a version of
gettext installed that is not compiled properly to work with Python (usually
its a 32-bit/64-bit thing). Best piece of advice is to not use MacPorts/Fink
and use Homebrew instead. =)

On Sun, Mar 7, 2010 at 09:36, Brad Miller <millbr02 at luther.edu> wrote:

>
> I download the source snapshot
>
> ./configure --enable-framework
> make
>
> ...
>
> /usr/bin/install -c -d -m 755 Python.framework/Versions/3.1
> if test ""; then \
>                 gcc -o Python.framework/Versions/3.1/Python  -dynamiclib \
>                         -isysroot "" \
>                         -all_load libpython3.1.a -Wl,-single_module \
>                         -install_name
> /Library/Frameworks/Python.framework/Versions/3.1/Python \
>                         -compatibility_version 3.1 \
>                         -current_version 3.1 \
>                         -framework CoreFoundation -ldl  -framework
> CoreFoundation; \
>         else \
>                 /usr/bin/libtool -o Python.framework/Versions/3.1/Python
> -dynamic  libpython3.1.a \
>                          -lSystem -lSystemStubs -arch_only x86_64
> -install_name /Library/Frameworks/Python.framework/Versions/3.1/Python
> -compatibility_version 3.1 -current_version 3.1 -framework CoreFoundation
> -ldl  -framework CoreFoundation;\
>         fi
>
> Undefined symbols:
>   "_libintl_bindtextdomain", referenced from:
>       _PyIntl_bindtextdomain in libpython3.1.a(_localemodule.o)
>   "_libintl_textdomain", referenced from:
>       _PyIntl_textdomain in libpython3.1.a(_localemodule.o)
>   "_libintl_dgettext", referenced from:
>       _PyIntl_dgettext in libpython3.1.a(_localemodule.o)
>   "_libintl_gettext", referenced from:
>       _PyIntl_gettext in libpython3.1.a(_localemodule.o)
>   "_libintl_dcgettext", referenced from:
>       _PyIntl_dcgettext in libpython3.1.a(_localemodule.o)
> ld: symbol(s) not found
> /usr/bin/libtool: internal link edit command failed
>
> I'm on 10.6.2, gcc version 4.2.1
>
> Any hints on where to go from here?
>
>
> On Sun, Mar 7, 2010 at 8:57 AM, Antoine Pitrou <solipsis at pitrou.net>wrote:
>
>> Le Sun, 07 Mar 2010 11:10:52 +0100,
>> Stefan Behnel <stefan_ml at behnel.de> a ?crit :
>> >
>> > Any reason the fix for bug 7173 didn't make it in? The test case that
>> > Amaury came up with seems to indicate that this a pretty serious
>> > issue.
>>
>> The patch needs to properly include an unit test reproducing the test
>> case.
>> Also, a committer must step up and apply it. I am currently on vacation
>> and refrain from doing commits myself.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/bmiller%40luther.edu
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100307/3fa5390b/attachment.html>

From barry at python.org  Sun Mar  7 21:56:12 2010
From: barry at python.org (Barry Warsaw)
Date: Sun, 7 Mar 2010 15:56:12 -0500
Subject: [Python-Dev] Python 2.6.5 rc 2
Message-ID: <20100307155612.59d9db94@heresy.wooz.org>

We are definitely going to need a 2.6.5 rc 2.  We've had a number of critical
issues fixed since rc1 and we have two more critical patches approved for
landing which fix issues on OS X.  I've given Ronald approval in the tracker
to land these two patches.

8066 OS X installer: readline module breaks when targeting on 10.5 or 10.6
8067 OS X Installer: build errors on 10.6 when targeting 10.4 and earlier

These are the only two release blockers left for 2.6.5 (at the moment).  I
would like to cut the rc2 release tomorrow, Monday 2010-03-08, so if anybody
knows of any other bugs you think must go into 2.6.5, mark them release
blockers now so that I can evaluate them.

I really don't want there to be an rc3, and I'd still like to get 2.6.5 final
out on Monday 2010-03-15.  We'll see.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100307/494c8294/attachment.pgp>

From digitalxero at gmail.com  Sun Mar  7 22:44:31 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Sun, 7 Mar 2010 14:44:31 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> 
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> 
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com> 
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
Message-ID: <e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>

On Sun, Mar 7, 2010 at 6:50 AM, Jesse Noller <jnoller at gmail.com> wrote:
> Making the tests and examples happy on windows is fine; but some
> explanation is needed for the API changes.
>

My primary motivation behind the API change is so there is just a
single public Executor class that you tell what system to use instead
of two separate classes. The use case I was thinking about is when a
user is is unsure which system (threads or processes) they want to use
so they just build the system with the defaults (which is threads)
then it is a little easier to switch it to processes in the future
instead of having to change imports and all instances of the class you
just change the use keyword to switch between systems

From guido at python.org  Mon Mar  8 01:47:50 2010
From: guido at python.org (Guido van Rossum)
Date: Sun, 7 Mar 2010 16:47:50 -0800
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100307195614.BBD693A4062@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> 
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com> 
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com> 
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com> 
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
Message-ID: <ca471dc21003071647t3bd97c01p6da7a3d3f748f9cf@mail.gmail.com>

On Sun, Mar 7, 2010 at 11:56 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 10:59 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:
>>
>> So is it that you just don't like the idea of blocking, and want to stop
>> anything that relies on it from getting into the standard library?
>
> Um, no. ?As I said before, call it a "parallel task queue" or "parallel task
> manager" or something to that general effect and I'm on board.
>
> It may not be in the Zen of Python, but ISTM that names should generally
> follow use cases. ?It is something of a corollary to "one obvious way to do
> it", in that if you see something whose name matches what you want to do,
> then it should be obvious that that's the way in question. ?;-)
>
> The use cases for "parallel task queues", however, are a subset of those for
> "futures" in the general case. ?Since the proposed module addresses most of
> the former but very little of the latter, calling it futures is
> inappropriate.
>
> Specifically, it's:
>
> 1. Confusing to people who don't know what futures are (see e.g R.D.
> Murray's post), and

This is not a problem. We will document what we consider a future.

> 2. Underpowered for people who expect/want a more fully-featured futures
> system along the lines of E or Deferreds.

This sounds like an underhanded slur towards the PEP.

> It seems that the only people for whom it's an intuitively correct
> description are people who've only had experience with more limited futures
> models (like Java's). ?However, these people should not have a problem
> understanding the notion of parallel task queueing or task management, so
> changing the name isn't really a loss for them, and it's a gain for
> everybody else.

I expect that the majority of Python users fall either in camp #1
(never heard of futures, will be happy to learn about what Python
calls futures) or camp #3 (have used Java futures). The users of E can
be counted on a few hands. Deferreds are used heavily in some Python
circles but most Python users (myself included) have at most a very
vague idea of them. Also, as you clarify below, Deferreds are so much
more powerful that they can't possibly be mistaken for futures (as
defined by this PEP). Plus they already have a name.

>> ?Given the set_result and set_exception methods, it's pretty
>> straightforward to fill in the value of a future from something that isn't
>> purely computational.
>
> Those are described as "internal" methods in the PEP; by contrast, the
> Deferred equivalents are part of the public API.
>
>
>> ?Given a way to register "on-done" callbacks with the future, it would be
>> straightforward to wait for a future without blocking, too.
>
> Yes, and with a few more additions besides that one, you might be on the way
> to an actual competitor for Deferreds. ?For example: retry support,
> chaining, logging, API for transparent result processing, coroutine support,
> co-ordination tools like locks, sempaphores and queues, etc.
>
> These are all things you would very likely want or need if you actually
> wanted to write a program using futures as *your main computational model*,
> vs. just needing to toss out some parallel tasks in a primarily synchronous
> program.
>
> Of course, Deferreds are indeed overkill if all you're ever going to want is
> a few parallel tasks, unless you're already skilled in using Twisted or some
> wrapper for it.
>
> So, I totally support having a simple task queue in the stdlib, as there are
> definitely times I would've used such a thing for a quick script, if it were
> available.
>
> However, I've *also* had use cases for using futures as a computational
> model, and so that's what I originally thought this PEP was about. ?After
> the use cases were clarified, though, it seems to me that *calling* it
> futures is a bad idea, because it's really just a nice task queuing system.
>
> I'm +1 on adding a nice task queuing system, -1 on calling it by any other
> name. ?;-)

So let's focus on the functionality of the task queuing system, and
stick to roughly the functionality proposed in the PEP.

The name is a non-issue and further discussion ought to be sent to
null-dev at python.org.

-- 
--Guido van Rossum (python.org/~guido)

From fuzzyman at voidspace.org.uk  Mon Mar  8 01:51:02 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 08 Mar 2010 00:51:02 +0000
Subject: [Python-Dev] unittest2 plus class and module level fixtures in
	unittest
Message-ID: <4B9449F6.9080206@voidspace.org.uk>

Hello all,

unittest has seen quite a few new features since Python 2.6. For those 
of you who might have missed the announcements in other places, 
unittest2 is a backport of the new features in unittest to work with 
Python 2.4-2.6. It is already being used for the development of distutils2:

http://pypi.python.org/pypi/unittest2/ 
<http://pypi.python.org/pypi/unittest2/>

In other news, class and module fixtures (setUpClass / tearDownClass / 
setUpModule / tearDownModule) are now implemented in unittest (in trunk 
- not yet merged to py3k). These features are tested but I'm sure there 
are some lurking bugs or oddities, so please try them out. I have not 
yet added documentation for them; I'll pull it out from this email as a 
starting point.

I'd rather this thread didn't become *another* debate on the merit of 
these features, but perhaps that is too much to hope for.

Below are some notes on how class and module fixtures work.

Class and module level fixtures are implemented in unittest.TestSuite. 
When the test suite encounters a test from a new class then 
tearDownClass from the previous class (if there is one) is called, 
followed by setUpClass from the new class.

Similarly if a test is from a different module from the previous test 
then tearDownModule from the previous module is run, followed by 
setUpModule from the new module.

After all the tests have run the final tearDownClass and tearDownModule 
are run.

Note that shared fixtures do not play well with [potential] features 
like test parallelization and they break test isolation. They should be 
used with care.


setUpClass and tearDownClass
--------------------------------------

These must be implemented as class methods.

     import unittest

     class Test(unittest.TestCase):
         @classmethod
         def setUpClass(cls):
             cls._connection = createExpensiveConnectionObject()

         @classmethod
         def tearDownClass(cls):
             cls._connection.destroy()

If you want the setUpClass and tearDownClass on base classes called then 
you must call up to them yourself. The implementations in 
unittest2.TestCase are empty.

If an exception is raised during a setUpClass then the tests in the 
class are not run and the tearDownClass is not run. Skipped classes will 
not have setUpClass or tearDownClass run.

A setUpClass that raises a unittest2.SkipTest exception will currently 
be reported as an error rather than a skip (although the effect is the 
same). This will be fixed at some point in the future.


setUpModule and tearDownModule
--------------------------------------------

These should be implemented as functions.

     def setUpModule():
         createConnection()

     def tearDownModule():
         closeConnection()

If an exception is raised in a setUpModule then none of the tests in the 
module will be run and the tearDownModule will not be run.

A setUpModule that raises a unittest.SkipTest exception will currently 
be reported as an error rather than a skip (although the effect is the 
same). This will be fixed at some point in the future.


The Gory Details
----------------------

The default ordering of tests created by the unittest test loaders is to 
group all tests from the same modules and classes together. This will 
lead to setUpClass / setUpModule (etc) being called exactly once per 
class and module. If you randomize the order, so that tests from 
different modules and classes are adjacent to each other, then these 
shared fixture functions may be called multiple times in a single test run.

This particular issue was a point of some disussion. Shared fixtures are 
not intended to work with suites with non-standard (incompatible) 
ordering (a BaseTestSuite still exists for frameworks that don't want to 
support shared fixtures), so this is already a pathological situation. 
There are a couple of things unittest could do in this situation:

     1) Work out the best order to run the setUp and tearDowns, based on 
the position of the first and last test from each class / module in the 
suite.
     2) Run setUp and tearDowns appropriately if consecutive tests are 
from different classes and modules.

If your setUps do unrelated things then strategy 1 will be the best one, 
but (so long as your setups are repeatable) 2 will be fine just slower. 
If your setups do related things, for example pushing large test 
datasets into the same database, then having multiple setups active 
before the corresponding teardowns have been run will actually do the 
wrong thing and your tests will be broken. In addition strategy 1 relies 
on unrolling the test suite and introspecting all tests prior to the 
run, not all custom test suites implementations support this.

Given that strategy 2 was substantially easier to implement, and less 
likely to cause broken test runs, that was what I went with. It is of 
course possible to add alternative strategies, plus the metadata to 
support them, if there proves to be a substantial need for this. One of 
the major use cases is for test order randomization. It would be easy to 
add a 'randomize' method to the suite that randomizes the order of test 
modules in a suite, the order of test classes within modules and the 
order of tests within a class. This would allow for tests to be run in a 
random order whilst remaining compatible with shared fixtures.

If there are any exceptions raised during one of the shared fixture 
functions the test is reported as an error. Because there is no 
corresponding test instance an _ErrorHolder object (that has the same 
interface as a TestCase) is created to represent the error. If you are 
just using the standard unittest test runner then this detail doesn't 
matter, but if you are a framework author it may be relevant.



I have a few minor features I'd like to add to unittest before 2.7 beta 
- standard out capturing, fail fast and ctrl-c handling (first ctrl-c 
will wait for current test to exit, second will force an exit - results 
will still be reported). After that the *really* big problem for 
unittest is the extensibility story.

I'll be working on extensibility points for unittest that allow people 
to implement extensions without having to override functionality. This 
will take a fair bit of discussion and experimentation, so I will 
probably trial this in unittest2 first. It would be nice to have 
something in place for the release of 3.2, but that may be ambitious.

All the best,

Michael Foord

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100308/0e816607/attachment.html>

From solipsis at pitrou.net  Mon Mar  8 02:11:43 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 7 Mar 2010 20:11:43 -0500
Subject: [Python-Dev] unittest2 plus class and module level fixtures in
	unittest
References: <4B9449F6.9080206@voidspace.org.uk>
Message-ID: <20100307201143.48f0fd3e@msiwind>

Le Mon, 08 Mar 2010 00:51:02 +0000,
Michael Foord <fuzzyman at voidspace.org.uk> a ?crit :
> 
> In other news, class and module fixtures (setUpClass /
> tearDownClass / setUpModule / tearDownModule) are now implemented in
> unittest (in trunk 
> - not yet merged to py3k). These features are tested but I'm sure
> there are some lurking bugs or oddities, so please try them out. I
> have not yet added documentation for them; I'll pull it out from this
> email as a starting point.
> 
> I'd rather this thread didn't become *another* debate on the merit of 
> these features, but perhaps that is too much to hope for.

Just for the record, could you sum up the reasons why you/we decided on
this route rather than e.g. adding a test-resources-like facility?

(I'm not saying I disagree, it's just that I didn't remember a decision
had already been made on this point)

Thanks for all your work on this!

cheers

Antoine.



From markroddy at gmail.com  Mon Mar  8 05:04:15 2010
From: markroddy at gmail.com (Mark Roddy)
Date: Sun, 7 Mar 2010 23:04:15 -0500
Subject: [Python-Dev] unittest2 plus class and module level fixtures in
	unittest
In-Reply-To: <20100307201143.48f0fd3e@msiwind>
References: <4B9449F6.9080206@voidspace.org.uk>
	<20100307201143.48f0fd3e@msiwind>
Message-ID: <7930f1201003072004q25b57535q86e4e0c9cc04736c@mail.gmail.com>

 On Sun, Mar 7, 2010 at 8:11 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Mon, 08 Mar 2010 00:51:02 +0000,
> Michael Foord <fuzzyman at voidspace.org.uk> a ?crit :
>>
>> In other news, class and module fixtures (setUpClass /
>> tearDownClass / setUpModule / tearDownModule) are now implemented in
>> unittest (in trunk
>> - not yet merged to py3k). These features are tested but I'm sure
>> there are some lurking bugs or oddities, so please try them out. I
>> have not yet added documentation for them; I'll pull it out from this
>> email as a starting point.
>>
>> I'd rather this thread didn't become *another* debate on the merit of
>> these features, but perhaps that is too much to hope for.
>
> Just for the record, could you sum up the reasons why you/we decided on
> this route rather than e.g. adding a test-resources-like facility?
>
> (I'm not saying I disagree, it's just that I didn't remember a decision
> had already been made on this point)
>
> Thanks for all your work on this!
>
> cheers
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/markroddy%40gmail.com
>


There were some lengthy threads on python-ideas
(http://mail.python.org/pipermail/python-ideas/2010-January/006736.html)
and python-dev (http://mail.python.org/pipermail/python-dev/2010-February/097521.html).

I'm sure Michael has a set of factors that are most important to him
for the decision (which I don't blame him for not wanting to get into,
the two previous threads were contentious), but for me personally I
don't view class/module setup and test-resources as necessarily
competing options.  For the simplest use case ( create some expensive
connection and/or object which is essentially stateless and won't need
to be torn down), I personally find a test-resources solution to
require an unnecessary amount of work to implement.  After exploring
it a bit in the course of those discussions I hope that test-resources
(or something similar, maybe something like junit rules which was also
brought up) makes it into the stdlib at some point as some of the more
complicated integration/scenario tests I support would be made much
simpler with such functionality.

-Mark

From mark.russell at zen.co.uk  Mon Mar  8 11:25:16 2010
From: mark.russell at zen.co.uk (Mark Russell)
Date: Mon, 8 Mar 2010 10:25:16 +0000
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
Message-ID: <67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk>

On 7 Mar 2010, at 19:49, Guido van Rossum wrote:
> How would you write the example instead then?

Boolean flags are a common enough case that I'd be inclined to add a wrapper method, so you could just say:

	parser.add_bool_argument('--plot')

As you can always fall back to the more general add_argument() call, the wrapper could just handle the usual case (default false, presence of option sets flag to True).  So the signature would be pretty simple - something like:

	def add_bool_argument(self, help=None, dest=None):

Mark Russell

From ncoghlan at gmail.com  Mon Mar  8 12:14:56 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 08 Mar 2010 21:14:56 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100307195614.BBD693A4062@sparrow.telecommunity.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
Message-ID: <4B94DC30.2030904@gmail.com>

P.J. Eby wrote:
> I'm +1 on adding a nice task queuing system, -1 on calling it by any
> other name.  ;-)

As Guido said, let's call the nice task queuing system "futures" and
point people wanting a full-power asynchronous process model to Twisted
- while the Deferred API may technically be independent of the rest of
the framework, you need at least some of the other tools for
asynchronous operations to make it really shine.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon Mar  8 12:25:10 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 08 Mar 2010 21:25:10 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306061134.5AEE23A40A7@sparrow.telecommunity.com>	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
Message-ID: <4B94DE96.8080100@gmail.com>

Dj Gilcrease wrote:
> On Sun, Mar 7, 2010 at 6:50 AM, Jesse Noller <jnoller at gmail.com> wrote:
>> Making the tests and examples happy on windows is fine; but some
>> explanation is needed for the API changes.
>>
> 
> My primary motivation behind the API change is so there is just a
> single public Executor class that you tell what system to use instead
> of two separate classes. The use case I was thinking about is when a
> user is is unsure which system (threads or processes) they want to use
> so they just build the system with the defaults (which is threads)
> then it is a little easier to switch it to processes in the future
> instead of having to change imports and all instances of the class you
> just change the use keyword to switch between systems

Wouldn't a factory function serve that purpose just as well? Or even
just "from concurrent.futures import ProcessPoolExecutor as TaskExecutor".

That last form has the virtue that you can retrieve your executor from
anywhere rather than being limited to the two provided by the
concurrent.futures model.

I think the string based approach actually unduly constrains the API
despite superficially appearing to make it more flexible.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From victor.stinner at haypocalc.com  Mon Mar  8 14:12:51 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 8 Mar 2010 14:12:51 +0100
Subject: [Python-Dev] Catch SIGINT at Python startup
Message-ID: <201003081412.51255.victor.stinner@haypocalc.com>

Hi,

I wrote patches for the issue #3137 and I would like your opinion about my 
solution (not directly about the patches): I propose to consider a SIGINT 
during startup as a fatal error. If you press CTRL+c while Python is starting, 
Python displays an error and exit with the code 1.

Currently, if you press CTRL+c while Python is starting: the SIGINT signal 
will be *ignored* somewhere between Python initialization and the first Python 
instruction. First problem is that if site import is interrupted, Python is 
not completly initialized (if a SIGINT is sent just after the call to 
initsigs(), initsite() is completly skipped).

Example:
------------------
$ python
^C
Traceback (most recent call last):                                                                                                     
  File "/home/SHARE/SVN/python-trunk/Lib/site.py", line 62, in <module>                                                                
    import os                                                                                                                          
  File "/home/SHARE/SVN/python-trunk/Lib/os.py", line 44, in <module>                                                                  
    from posix import *                                                                                                                
KeyboardInterrupt         
$ echo $?
1
------------------

It's not easy to reproduce this issue because Python initialization is really 
fast (yeah really! or at least to fast for me ;-)). I'm using "valgrind --log-
file=/dev/null ./python -v" to simulate a very slow computer :-)

-- 
Victor Stinner
http://www.haypocalc.com/

From steven.bethard at gmail.com  Mon Mar  8 16:40:36 2010
From: steven.bethard at gmail.com (Steven Bethard)
Date: Mon, 8 Mar 2010 07:40:36 -0800
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
Message-ID: <d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>

On Sun, Mar 7, 2010 at 11:49 AM, Guido van Rossum <guido at python.org> wrote:
> On Sun, Mar 7, 2010 at 4:29 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> Brian Curtin wrote:
>>
>>> On Fri, Mar 5, 2010 at 12:51, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>
>>>> I generally enjoy argparse, but one thing I find rather
>>>> ugly and unpythonic.
>>>>
>>>> ? ?parser.add_argument ('--plot', action='store_true')
>>>>
>>>> Specifying the argument 'action' as a string is IMO ugly.
>>>>
>>>
>>> What else would you propose?
>>> FWIW, this is the same in optparse.
>>
>> I would have thought use the object itself, instead of a string that spells
>> the object's name.
>
> What object? How would you write the example instead then?

In argparse, unlike optparse, actions are actually defined by objects
with a particular API, and the string is just a shorthand for
referring to that. So:

  parser.add_argument ('--plot', action='store_true')

is equivalent to:

  parser.add_argument('--plot', argparse._StoreTrueAction)

Because the names are so long and you'd have to import them, I've left
them as private attributes of the module, but if there's really
demand, we could rename them to argparse.StoreTrueAction, etc.

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

From steven.bethard at gmail.com  Mon Mar  8 16:41:09 2010
From: steven.bethard at gmail.com (Steven Bethard)
Date: Mon, 8 Mar 2010 07:41:09 -0800
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
Message-ID: <d11dcfba1003080741u367a4167u2d0eeb300c20a37a@mail.gmail.com>

On Mon, Mar 8, 2010 at 7:40 AM, Steven Bethard <steven.bethard at gmail.com> wrote:
> On Sun, Mar 7, 2010 at 11:49 AM, Guido van Rossum <guido at python.org> wrote:
>> On Sun, Mar 7, 2010 at 4:29 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>> Brian Curtin wrote:
>>>
>>>> On Fri, Mar 5, 2010 at 12:51, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>>
>>>>> I generally enjoy argparse, but one thing I find rather
>>>>> ugly and unpythonic.
>>>>>
>>>>> ? ?parser.add_argument ('--plot', action='store_true')
>>>>>
>>>>> Specifying the argument 'action' as a string is IMO ugly.
>>>>>
>>>>
>>>> What else would you propose?
>>>> FWIW, this is the same in optparse.
>>>
>>> I would have thought use the object itself, instead of a string that spells
>>> the object's name.
>>
>> What object? How would you write the example instead then?
>
> In argparse, unlike optparse, actions are actually defined by objects
> with a particular API, and the string is just a shorthand for
> referring to that. So:
>
> ?parser.add_argument ('--plot', action='store_true')
>
> is equivalent to:
>
> ?parser.add_argument('--plot', argparse._StoreTrueAction)

Sorry, that should have been:

  parser.add_argument('--plot', action=argparse._StoreTrueAction)

>
> Because the names are so long and you'd have to import them, I've left
> them as private attributes of the module, but if there's really
> demand, we could rename them to argparse.StoreTrueAction, etc.
>
> Steve
> --
> Where did you get that preposterous hypothesis?
> Did Steve tell you that?
> ? ? ? ?--- The Hiphopopotamus
>



-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

From dstanek at dstanek.com  Mon Mar  8 16:53:43 2010
From: dstanek at dstanek.com (David Stanek)
Date: Mon, 8 Mar 2010 10:53:43 -0500
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
Message-ID: <de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>

On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard
<steven.bethard at gmail.com> wrote:
>
> In argparse, unlike optparse, actions are actually defined by objects
> with a particular API, and the string is just a shorthand for
> referring to that. So:
>
> ?parser.add_argument ('--plot', action='store_true')
>
> is equivalent to:
>
> ?parser.add_argument('--plot', argparse._StoreTrueAction)
>
> Because the names are so long and you'd have to import them, I've left
> them as private attributes of the module, but if there's really
> demand, we could rename them to argparse.StoreTrueAction, etc.
>

Any reason not to do something like:

  from argparse import actions
  ...
  parser.add_argument('--plot', actions.store_true)

Basically a small namespace for the constants.

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek

From tjreedy at udel.edu  Mon Mar  8 17:16:33 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 08 Mar 2010 11:16:33 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B94DC30.2030904@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>	<20100307035657.111563A4077@sparrow.telecommunity.com>	<4B93303E.9050301@gmail.com>	<20100307154809.742E73A4062@sparrow.telecommunity.com>	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>	<20100307175732.307513A4062@sparrow.telecommunity.com>	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
	<4B94DC30.2030904@gmail.com>
Message-ID: <hn37su$tq9$1@dough.gmane.org>

On 3/8/2010 6:14 AM, Nick Coghlan wrote:
> P.J. Eby wrote:
>> I'm +1 on adding a nice task queuing system, -1 on calling it by any
>> other name.  ;-)
>
> As Guido said, let's call the nice task queuing system "futures" and

I was confused by 'futures' also until Philip explained it as task-queue 
or task-pool, and hence also do not like it.

Since the examples in the PEP do *NOT* give example output, it was not 
clear to me whether execution or the termination thereof is ordered 
(queue) or not (pool). Looking more close, I gather that the prime 
results will be printed 'in order' (waiting on each even if others are 
done) while the url results will be printed 'as available'. Adding 'will 
print ...' and 'might print ...' outputs would help.

> point people wanting a full-power asynchronous process model to Twisted

That could be done in the PEP to clarify its scope.

Terry Jan Reedy


From tjreedy at udel.edu  Mon Mar  8 17:21:42 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 08 Mar 2010 11:21:42 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <7EFBC33B-06AD-4BAD-8173-005D6F08240E@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<7EFBC33B-06AD-4BAD-8173-005D6F08240E@sweetapp.com>
Message-ID: <hn386j$uuk$1@dough.gmane.org>

On 3/6/2010 4:20 AM, Brian Quinlan wrote:
>
> On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote:
>
>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com
>> <mailto:brian at sweetapp.com>> wrote:
>>
>>     import futures
>>
>>
>> +1 on the idea, -1 on the name. It's too similar to "from __future__
>> import ...".
>>
>> Also, the PEP should probably link to the discussions on stdlib-sig?
>
> I thoug ht about that but this discussion is spread over many threads
> and many months.

This is pretty typical. I would say just that, and link to the first.
"This PEP was discussed over many months in many threads in the 
stdlib-sig list. The first was .... . Python-dev discussion occured in 
<this thread>.

tjr




From ndbecker2 at gmail.com  Mon Mar  8 17:44:15 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 8 Mar 2010 11:44:15 -0500
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
	<de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>
Message-ID: <201003081144.16370.ndbecker2@gmail.com>

On Monday 08 March 2010, David Stanek wrote:
> On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard
> 
> <steven.bethard at gmail.com> wrote:
> > In argparse, unlike optparse, actions are actually defined by objects
> > with a particular API, and the string is just a shorthand for
> > referring to that. So:
> > 
> >  parser.add_argument ('--plot', action='store_true')
> > 
> > is equivalent to:
> > 
> >  parser.add_argument('--plot', argparse._StoreTrueAction)
> > 
> > Because the names are so long and you'd have to import them, I've left
> > them as private attributes of the module, but if there's really
> > demand, we could rename them to argparse.StoreTrueAction, etc.
> 
> Any reason not to do something like:
> 
>   from argparse import actions
>   ...
>   parser.add_argument('--plot', actions.store_true)
> 
> Basically a small namespace for the constants.
+1

From python-dev at masklinn.net  Mon Mar  8 17:49:42 2010
From: python-dev at masklinn.net (Xavier Morel)
Date: Mon, 8 Mar 2010 17:49:42 +0100
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
	<de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>
Message-ID: <A2F055BA-ECBC-474E-A076-2AF571C4B582@masklinn.net>

On 8 Mar 2010, at 16:53 , David Stanek wrote:
> 
> On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard
> <steven.bethard at gmail.com> wrote:
>> 
>> In argparse, unlike optparse, actions are actually defined by objects
>> with a particular API, and the string is just a shorthand for
>> referring to that. So:
>> 
>>  parser.add_argument ('--plot', action='store_true')
>> 
>> is equivalent to:
>> 
>>  parser.add_argument('--plot', argparse._StoreTrueAction)
>> 
>> Because the names are so long and you'd have to import them, I've left
>> them as private attributes of the module, but if there's really
>> demand, we could rename them to argparse.StoreTrueAction, etc.
>> 
> 
> Any reason not to do something like:
> 
>  from argparse import actions
>  ...
>  parser.add_argument('--plot', actions.store_true)
> 
> Basically a small namespace for the constants.

action is taken from **kwargs, not from a positional argument as *args
is a sequence of option strings (so you can write
 add_argument('-p', '/p', '--plot', '--land-mass')). So you'd have to write
add_argument('--plot', action=actions.store_true) which is straight from
the department of redundant redundancies.

An option would be 

  parser.add(actions.StoreTrue('--plot'))

but I'm not sure this makes any sense API-wise, and it would probably make
the code a lot messier as the parser would have to reach into the action
to get the information it needs. Either that, or the action would be an *arg
and argparse would have to walk all of its *args type-testing each one to find
if there's an action anywhere.

From dstanek at dstanek.com  Mon Mar  8 17:55:09 2010
From: dstanek at dstanek.com (David Stanek)
Date: Mon, 8 Mar 2010 11:55:09 -0500
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <A2F055BA-ECBC-474E-A076-2AF571C4B582@masklinn.net>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
	<de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>
	<A2F055BA-ECBC-474E-A076-2AF571C4B582@masklinn.net>
Message-ID: <de32cc031003080855h3c01fdc0y2d2cd12c4a256770@mail.gmail.com>

On Mon, Mar 8, 2010 at 11:49 AM, Xavier Morel <python-dev at masklinn.net> wrote:
> On 8 Mar 2010, at 16:53 , David Stanek wrote:
>>
>> On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard
>> <steven.bethard at gmail.com> wrote:
>>>
>>> In argparse, unlike optparse, actions are actually defined by objects
>>> with a particular API, and the string is just a shorthand for
>>> referring to that. So:
>>>
>>> ?parser.add_argument ('--plot', action='store_true')
>>>
>>> is equivalent to:
>>>
>>> ?parser.add_argument('--plot', argparse._StoreTrueAction)
>>>
>>> Because the names are so long and you'd have to import them, I've left
>>> them as private attributes of the module, but if there's really
>>> demand, we could rename them to argparse.StoreTrueAction, etc.
>>>
>>
>> Any reason not to do something like:
>>
>> ?from argparse import actions
>> ?...
>> ?parser.add_argument('--plot', actions.store_true)
>>
>> Basically a small namespace for the constants.
>
> action is taken from **kwargs, not from a positional argument as *args
> is a sequence of option strings (so you can write
> ?add_argument('-p', '/p', '--plot', '--land-mass')). So you'd have to write
> add_argument('--plot', action=actions.store_true) which is straight from
> the department of redundant redundancies.
>
> An option would be
>
> ?parser.add(actions.StoreTrue('--plot'))
>
> but I'm not sure this makes any sense API-wise, and it would probably make
> the code a lot messier as the parser would have to reach into the action
> to get the information it needs. Either that, or the action would be an *arg
> and argparse would have to walk all of its *args type-testing each one to find
> if there's an action anywhere.

You could just change my example to:
 ?from argparse.actions import store_true
 ?...
 ?parser.add_argument('--plot', action=store_true)

I would probably still import the actions namespace directly, but
wouldn't have to.


-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek

From fuzzyman at voidspace.org.uk  Mon Mar  8 18:14:35 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 08 Mar 2010 17:14:35 +0000
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <201003081412.51255.victor.stinner@haypocalc.com>
References: <201003081412.51255.victor.stinner@haypocalc.com>
Message-ID: <4B95307B.7060202@voidspace.org.uk>

On 08/03/2010 13:12, Victor Stinner wrote:
> Hi,
>
> I wrote patches for the issue #3137 and I would like your opinion about my
> solution (not directly about the patches): I propose to consider a SIGINT
> during startup as a fatal error. If you press CTRL+c while Python is starting,
> Python displays an error and exit with the code 1.
>
>    
That sounds like a good solution to me. +1

Michael

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From rrr at ronadam.com  Mon Mar  8 18:17:13 2010
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 08 Mar 2010 11:17:13 -0600
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <d11dcfba1003080741u367a4167u2d0eeb300c20a37a@mail.gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>	<hn0676$br0$1@dough.gmane.org>	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
	<d11dcfba1003080741u367a4167u2d0eeb300c20a37a@mail.gmail.com>
Message-ID: <hn3beq$csa$1@dough.gmane.org>



Steven Bethard wrote:
> On Mon, Mar 8, 2010 at 7:40 AM, Steven Bethard <steven.bethard at gmail.com> wrote:
>> On Sun, Mar 7, 2010 at 11:49 AM, Guido van Rossum <guido at python.org> wrote:
>>> On Sun, Mar 7, 2010 at 4:29 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>> Brian Curtin wrote:
>>>>
>>>>> On Fri, Mar 5, 2010 at 12:51, Neal Becker <ndbecker2 at gmail.com> wrote:
>>>>>
>>>>>> I generally enjoy argparse, but one thing I find rather
>>>>>> ugly and unpythonic.
>>>>>>
>>>>>>    parser.add_argument ('--plot', action='store_true')
>>>>>>
>>>>>> Specifying the argument 'action' as a string is IMO ugly.
>>>>>>
>>>>> What else would you propose?
>>>>> FWIW, this is the same in optparse.
>>>> I would have thought use the object itself, instead of a string that spells
>>>> the object's name.
>>> What object? How would you write the example instead then?
>> In argparse, unlike optparse, actions are actually defined by objects
>> with a particular API, and the string is just a shorthand for
>> referring to that. So:
>>
>>  parser.add_argument ('--plot', action='store_true')
>>
>> is equivalent to:
>>
>>  parser.add_argument('--plot', argparse._StoreTrueAction)
> 
> Sorry, that should have been:
> 
>   parser.add_argument('--plot', action=argparse._StoreTrueAction)
> 
>> Because the names are so long and you'd have to import them, I've left
>> them as private attributes of the module, but if there's really
>> demand, we could rename them to argparse.StoreTrueAction, etc.

I like the strings.  They are simple and easy to use/read and they don't 
have to be created or imported before the parser is defined.  That allows 
me to put the parser setup right after the 'if __name__ == '__main__':' so 
it's easy to find and read.  It also allows me to not import or create 
objects that may be expensive in either time or resources if I'm not going 
to use them.

Also the strings help separate the parser from the rest of your program in 
a much cleaner way. This can make your programs more modular and easier to 
modify and maintain.

Ron






From guido at python.org  Mon Mar  8 18:47:18 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Mar 2010 09:47:18 -0800
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <4B95307B.7060202@voidspace.org.uk>
References: <201003081412.51255.victor.stinner@haypocalc.com> 
	<4B95307B.7060202@voidspace.org.uk>
Message-ID: <ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com>

Actually it sounds like there's some overly general except clause
somewhere that should be adjusted to catch just "Exception" instead of
"*".

On Mon, Mar 8, 2010 at 9:14 AM, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> On 08/03/2010 13:12, Victor Stinner wrote:
>>
>> Hi,
>>
>> I wrote patches for the issue #3137 and I would like your opinion about my
>> solution (not directly about the patches): I propose to consider a SIGINT
>> during startup as a fatal error. If you press CTRL+c while Python is
>> starting,
>> Python displays an error and exit with the code 1.
>>
>>
>
> That sounds like a good solution to me. +1
>
> Michael
>
> --
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
> READ CAREFULLY. By accepting and reading this email you agree, on behalf of
> your employer, to release me from all obligations and waivers arising from
> any and all NON-NEGOTIATED agreements, licenses, terms-of-service,
> shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure,
> non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have
> entered into with your employer, its partners, licensors, agents and
> assigns, in perpetuity, without prejudice to my ongoing rights and
> privileges. You further represent that you have the authority to release me
> from any BOGUS AGREEMENTS on behalf of your employer.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Mon Mar  8 18:49:55 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Mar 2010 09:49:55 -0800
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <hn3beq$csa$1@dough.gmane.org>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com> 
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com> 
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com> 
	<d11dcfba1003080741u367a4167u2d0eeb300c20a37a@mail.gmail.com> 
	<hn3beq$csa$1@dough.gmane.org>
Message-ID: <ca471dc21003080949p7dff7701v94c5d9db760b323b@mail.gmail.com>

On Mon, Mar 8, 2010 at 9:17 AM, Ron Adam <rrr at ronadam.com> wrote:
> I like the strings. ?They are simple and easy to use/read and they don't
> have to be created or imported before the parser is defined.

I like them too. I don't see anything unpythonic about them.

> That allows me
> to put the parser setup right after the 'if __name__ == '__main__':' so it's
> easy to find and read. ?It also allows me to not import or create objects
> that may be expensive in either time or resources if I'm not going to use
> them.
>
> Also the strings help separate the parser from the rest of your program in a
> much cleaner way. This can make your programs more modular and easier to
> modify and maintain.

Also in this particular case a typo in the string will be caught just
about at the same time as a typo in the name would be caught.

-- 
--Guido van Rossum (python.org/~guido)

From fijall at gmail.com  Mon Mar  8 19:21:20 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 8 Mar 2010 11:21:20 -0700
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com> 
	<4B95307B.7060202@voidspace.org.uk>
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com>
Message-ID: <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com>

On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido at python.org> wrote:
> Actually it sounds like there's some overly general except clause
> somewhere that should be adjusted to catch just "Exception" instead of
> "*".
>

There is at least one that prints "import 'site' failed" and continues
to run your program.

Cheers,
fijal

From guido at python.org  Mon Mar  8 19:34:12 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Mar 2010 10:34:12 -0800
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com> 
	<4B95307B.7060202@voidspace.org.uk>
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com> 
	<693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com>
Message-ID: <ca471dc21003081034q2893a369me5cef22bcbff494@mail.gmail.com>

On Mon, Mar 8, 2010 at 10:21 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido at python.org> wrote:
>> Actually it sounds like there's some overly general except clause
>> somewhere that should be adjusted to catch just "Exception" instead of
>> "*".
>>
>
> There is at least one that prints "import 'site' failed" and continues
> to run your program.

Yeah, it shouldn't do that. The ^C should fall through.

-- 
--Guido van Rossum (python.org/~guido)

From digitalxero at gmail.com  Mon Mar  8 20:04:39 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Mon, 8 Mar 2010 12:04:39 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B94DE96.8080100@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<5d44f72f1003052219m2d270846x48e3cc0da77b80f@mail.gmail.com> 
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> 
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com> 
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> 
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com> 
	<4B94DE96.8080100@gmail.com>
Message-ID: <e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>

On Mon, Mar 8, 2010 at 4:25 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Wouldn't a factory function serve that purpose just as well? Or even
> just "from concurrent.futures import ProcessPoolExecutor as TaskExecutor".
>
> That last form has the virtue that you can retrieve your executor from
> anywhere rather than being limited to the two provided by the
> concurrent.futures model.
>
> I think the string based approach actually unduly constrains the API
> despite superficially appearing to make it more flexible.

mm you are correct, I went with the string approach because I was
experimenting with 3 additional executor types and wanted to be able
to switch between or intermix them them without having to change
imports and didnt feel like writing a register class with a factory
method.

A style I have used in my own code in the past is a Singleton class
with register and create methods, where the register takes a
name(string) and the class and the create method takes the name and
*args, **kwargs and acts as a factory.

Would this style be better or would it be better to just leave it with
the two executor classes? I tend to dislike multiple classes for a
what is essentially a Strategy of a concept and factories are
something I tend to forget about until well after my initial idea has
formed into a proof of concept.

From fuzzyman at voidspace.org.uk  Mon Mar  8 20:20:50 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 08 Mar 2010 19:20:50 +0000
Subject: [Python-Dev] unittest2 plus class and module level fixtures in
 unittest
In-Reply-To: <20100307201143.48f0fd3e@msiwind>
References: <4B9449F6.9080206@voidspace.org.uk>
	<20100307201143.48f0fd3e@msiwind>
Message-ID: <4B954E12.40901@voidspace.org.uk>

On 08/03/2010 01:11, Antoine Pitrou wrote:
> Le Mon, 08 Mar 2010 00:51:02 +0000,
> Michael Foord<fuzzyman at voidspace.org.uk>  a ?crit :
>    
>> In other news, class and module fixtures (setUpClass /
>> tearDownClass / setUpModule / tearDownModule) are now implemented in
>> unittest (in trunk
>> - not yet merged to py3k). These features are tested but I'm sure
>> there are some lurking bugs or oddities, so please try them out. I
>> have not yet added documentation for them; I'll pull it out from this
>> email as a starting point.
>>
>> I'd rather this thread didn't become *another* debate on the merit of
>> these features, but perhaps that is too much to hope for.
>>      
> Just for the record, could you sum up the reasons why you/we decided on
> this route rather than e.g. adding a test-resources-like facility?
>
> (I'm not saying I disagree, it's just that I didn't remember a decision
> had already been made on this point)
>    

Well, Guido had already stated on python-ideas that he wanted these 
features (class and module fixtures) in unittest. I (and others) 
suggested testresources as an alternative but it became clear that it 
didn't offer a suitable abstraction for module level fixtures, instead 
you would have to manually attach the resources to all of the classes in 
the module yourself.

In addition to the lack of a module level fixture abstraction Guido felt 
that the testresources API itself was not yet mature or polished enough 
to come into the standard library. At that point I figured we had 
reverted to the decision that setUpClass / setUpModule should go in.

testresources still works of course, and is a better general solution 
for shared fixtures where people are pushing beyond what setUpClass and 
setUpModule allow. I'll probably link to testresources in the shared 
fixture documentation - and it has the great advantage of *not* being in 
the standard library and is therefore free to evolve faster.

All the best,

Michael

> Thanks for all your work on this!
>
> cheers
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From darren at ontrenet.com  Mon Mar  8 21:06:07 2010
From: darren at ontrenet.com (Darren Govoni)
Date: Mon, 08 Mar 2010 15:06:07 -0500
Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1
Message-ID: <1268078767.5175.4.camel@kratos>

Hi,
  I'm trying to get the thread-safe enabled for sqlite3 in my build of
2.6.5rc1, but its not taking and throws the old thread access error.
I am running Ubuntu with other versions of Python, but I successfully
built and installed sqlite 3.6.22, pysqlite 2.6.0.
Do I have to set something explicit when building python 2.6.5 to find
the updated sqlite3?

thanks for any tips.
Darren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100308/ea964b19/attachment.html>

From victor.stinner at haypocalc.com  Mon Mar  8 21:50:02 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 8 Mar 2010 21:50:02 +0100
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com>
	<4B95307B.7060202@voidspace.org.uk>
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com>
Message-ID: <201003082150.02469.victor.stinner@haypocalc.com>

Hi,

Le lundi 08 mars 2010 18:47:18, Guido van Rossum a ?crit :
> Actually it sounds like there's some overly general except clause
> somewhere that should be adjusted to catch just "Exception" instead of
> "*".

Most of my patches (for SIGINT) are exactly doing that: check the exception 
type instead of "if (result == NULL) PyErr_Clear();" (try/except: pass). About 
initsite(): I consider any error as fatal. If initsite() failed, something 
goes wrong.

I think that my patches are enough to catch SIGINT during the Python 
initialization. I used different tricks to test Python: add kill(getpid(), 
SIGINT), use "signal SIGINT" in gdb, add 
PyErr_SetNone(PyExc_KeyboardInterrupt), ... valgrind is a very good tool to 
slow down any process :-D

-- 
Victor Stinner
http://www.haypocalc.com/

From skip at pobox.com  Mon Mar  8 21:53:58 2010
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 8 Mar 2010 14:53:58 -0600
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <4B94DC30.2030904@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
	<4B94DC30.2030904@gmail.com>
Message-ID: <19349.25574.880099.668647@montanaro.dyndns.org>


    >> I'm +1 on adding a nice task queuing system, -1 on calling it by any
    >> other name.  ;-)

    Nick> As Guido said, let's call the nice task queuing system "futures"
    Nick> and point people wanting a full-power asynchronous process model
    Nick> to Twisted

Can this module at least be pushed down into a package?  I think
"concurrent" or "concurrency" were both suggested at one point.

Skip

From digitalxero at gmail.com  Mon Mar  8 21:56:24 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Mon, 8 Mar 2010 13:56:24 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> 
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com> 
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> 
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com> 
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
Message-ID: <e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>

On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease <digitalxero at gmail.com> wrote:
> A style I have used in my own code in the past is a Singleton class
> with register and create methods, where the register takes a
> name(string) and the class and the create method takes the name and
> *args, **kwargs and acts as a factory.


So I decided to play with this design a little and since I made it a
singleton I decided to place all the thread/process tracking and exit
handle code in it instead of having the odd semi-global scoped
_shutdown, _thread_references, _remove_dead_thread_references and
_python_exit objects floating around in each executor file, seems to
work well. The API would be

from concurrent.futures import executors

executor = executors.create(NAME, *args, **kwargs) # NAME is 'process'
or 'thread' by default


To create your own executor you create your executor class and add the
following at the end

from concurrent.futures import executors, ExecutorBase
class MyExecutor(ExecutorBase): ...
executors.register(NAME, MyExecutor)

It checks to see if your executor is a subclass of ExecutorBase, but
only does a UserWarning if it is not since you should know what
methods are required to be an executor like object, so if you are not
subclassing ExecutorBase you should suppress the UserWarning before
you register you class and un-suppress it after


Some Helper Methods/Properties on the executors Singleton
add_joinable_ref - This replaces the _thread_references.add system of
tracking threads, and it allows for adding processes as well hence the
joinable_ref name. It does check to make sure the ref being passes has
a join method then creates a weakref to it and adds it to a set. For
every thread or process your executor creates you should call this
with so it will be tracked properly

cleanup_joinable_refs - This replaces the
_remove_dead_thread_references that had to be written for each
executor individually. This should be called periodically, currently
it is only called when you create a new executor since it is a
blocking method (it uses a thread lock to make sure the set of
references does not change while it is discarding old ones)

shutdown - is a readonly property and replaces the _shutdown global
var that had to be created for each executor individually, it is set
in the executors destructor

__del__ - replaces the _python_exit method that had to be written for
each executor individually



If this API change isnt accepted its no big deal, since I am only
changing it due to personal pet peeves and the only real issue I had
with the package was scoping which has already been addressed by
adding it to a concurrent package

From guido at python.org  Mon Mar  8 22:06:36 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Mar 2010 13:06:36 -0800
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <201003082150.02469.victor.stinner@haypocalc.com>
References: <201003081412.51255.victor.stinner@haypocalc.com> 
	<4B95307B.7060202@voidspace.org.uk>
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com> 
	<201003082150.02469.victor.stinner@haypocalc.com>
Message-ID: <ca471dc21003081306s7b229146mb48e283e57052b0e@mail.gmail.com>

On Mon, Mar 8, 2010 at 12:50 PM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Le lundi 08 mars 2010 18:47:18, Guido van Rossum a ?crit :
>> Actually it sounds like there's some overly general except clause
>> somewhere that should be adjusted to catch just "Exception" instead of
>> "*".
>
> Most of my patches (for SIGINT) are exactly doing that: check the exception
> type instead of "if (result == NULL) PyErr_Clear();" (try/except: pass). About
> initsite(): I consider any error as fatal. If initsite() failed, something
> goes wrong.
>
> I think that my patches are enough to catch SIGINT during the Python
> initialization. I used different tricks to test Python: add kill(getpid(),
> SIGINT), use "signal SIGINT" in gdb, add
> PyErr_SetNone(PyExc_KeyboardInterrupt), ... valgrind is a very good tool to
> slow down any process :-D

I see. I thought for a moment that by "fatal" you meant something like
calling Py_FatalError().

I am trying to remember why I made site.py failures non-fatal in the
first place. I don't have any specific recollection but it must've
been either from before the separation between site.py (part of the
stdlib) and sitecustomize.py (site-specific) or out of a worry that if
some external cause broke site.py (which does a lot of I/O) it would
be a fatal breakdown of all Python execution.

I am fine with making an exception for KeyboardInterrupt (and other
exceptions not deriving from Exception but only from BaseException)
but I don't think I want to go as far as making all errors coming out
of site.py fatal. (At least not without more understanding of the use
case for making these non-fatal in the first place.)

OTOH I think the try/except in site.py around the execution of
sitecustomize.py might be changed so that it prints a full traceback
whenever it raises an exception other than ImportError or, again,
exceptions inheriting from BaseException but not from Exception. IOW I
think that exceptions coming out of sitecustomize.py should continued
to be treated non-fatally, apart from ^C and friends, but I think that
exceptions coming out of site.py *might* be considered more fatally
once the change to exceptions coming out of sitecustomize.py is made.

-- 
--Guido van Rossum (python.org/~guido)

From greg.ewing at canterbury.ac.nz  Mon Mar  8 22:20:38 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 09 Mar 2010 10:20:38 +1300
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk>
Message-ID: <4B956A26.6010204@canterbury.ac.nz>

Mark Russell wrote:
> Boolean flags are a common enough case that I'd be inclined to add a wrapper method,
> 
> 	parser.add_bool_argument('--plot')

+1, this looks good.

-- 
Greg

From exarkun at twistedmatrix.com  Mon Mar  8 22:11:45 2010
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Mon, 08 Mar 2010 21:11:45 -0000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
	<e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
Message-ID: <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>

On 08:56 pm, digitalxero at gmail.com wrote:
>On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease <digitalxero at gmail.com> 
>wrote:
>>A style I have used in my own code in the past is a Singleton class
>>with register and create methods, where the register takes a
>>name(string) and the class and the create method takes the name and
>>*args, **kwargs and acts as a factory.
>
>
>So I decided to play with this design a little and since I made it a
>singleton I decided to place all the thread/process tracking and exit
>handle code in it instead of having the odd semi-global scoped
>_shutdown, _thread_references, _remove_dead_thread_references and
>_python_exit objects floating around in each executor file, seems to
>work well. The API would be
>
>from concurrent.futures import executors
>
>executor = executors.create(NAME, *args, **kwargs) # NAME is 'process'
>or 'thread' by default
>
>
>To create your own executor you create your executor class and add the
>following at the end

Getting rid of the process-global state like this simplifies testing 
(both testing of the executors themselves and of application code which 
uses them).  It also eliminates the unpleasant interpreter 
shutdown/module globals interactions that have plagued a number of 
stdlib systems that keep global state.

Jean-Paul

From glyph at twistedmatrix.com  Mon Mar  8 22:18:34 2010
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Mon, 8 Mar 2010 16:18:34 -0500
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <ca471dc21003081306s7b229146mb48e283e57052b0e@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com>
	<4B95307B.7060202@voidspace.org.uk>
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com>
	<201003082150.02469.victor.stinner@haypocalc.com>
	<ca471dc21003081306s7b229146mb48e283e57052b0e@mail.gmail.com>
Message-ID: <C9306408-D304-49C5-AD19-65EF5D1BAE5B@twistedmatrix.com>


On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote:

> I am trying to remember why I made site.py failures non-fatal in the
> first place. I don't have any specific recollection but it must've
> been either from before the separation between site.py (part of the
> stdlib) and sitecustomize.py (site-specific) or out of a worry that if
> some external cause broke site.py (which does a lot of I/O) it would
> be a fatal breakdown of all Python execution.


The thing that occurs to me is that one might want to write an administrative tool in Python to manipulate site.py, or even just some data that something in site.py would load.  If exceptions from site.py were fatal, then bugs in such a tool would be completely unrecoverable; in trying to run it to un-do the buggy operation, it would crash immediately.

On the other hand, such a tool should *really* be invoked with the -S option anyway, so... maybe not that pressing of a concern.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100308/b265ae51/attachment.html>

From greg.ewing at canterbury.ac.nz  Mon Mar  8 22:31:16 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 09 Mar 2010 10:31:16 +1300
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
Message-ID: <4B956CA4.4050108@canterbury.ac.nz>

Steven Bethard wrote:

> Because the names are so long and you'd have to import them, I've left
> them as private attributes of the module, but if there's really
> demand, we could rename them to argparse.StoreTrueAction, etc.

What's wrong with just StoreTrue?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon Mar  8 22:39:14 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 09 Mar 2010 10:39:14 +1300
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <hn37su$tq9$1@dough.gmane.org>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
	<4B94DC30.2030904@gmail.com> <hn37su$tq9$1@dough.gmane.org>
Message-ID: <4B956E82.4080707@canterbury.ac.nz>

Terry Reedy wrote:
> Looking more close, I gather that the prime 
> results will be printed 'in order' (waiting on each even if others are 
> done) while the url results will be printed 'as available'.

Seems to me that if you care about the order of the results,
you should be able to just wait for each result separately
in the order you want them. Something like

   task1 = start_task(proc1)
   task2 = start_task(proc2)
   task3 = start_task(proc3)
   result1 = task1.wait_for_result()
   result2 = task2.wait_for_result()
   result3 = task3.wait_for_result()

This would also be a natural way to write things even if
you don't care about the order, but you need all the results
before proceeding. You're going to be held up until the
longest-running task completes anyway, so it doesn't matter
if some of them finish earlier and have to sit around
waiting for you to collect the result.

-- 
Greg


From robert.kern at gmail.com  Mon Mar  8 22:35:46 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 08 Mar 2010 15:35:46 -0600
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <4B956A26.6010204@canterbury.ac.nz>
References: <hmrjr4$kkd$1@dough.gmane.org>	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>	<hn0676$br0$1@dough.gmane.org>	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>	<67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk>
	<4B956A26.6010204@canterbury.ac.nz>
Message-ID: <hn3qji$ba6$1@dough.gmane.org>

On 2010-03-08 15:20 PM, Greg Ewing wrote:
> Mark Russell wrote:
>> Boolean flags are a common enough case that I'd be inclined to add a
>> wrapper method,
>>
>> parser.add_bool_argument('--plot')
>
> +1, this looks good.

I've added it to the argparse bugtracker, along with my suggested spelling 
add_flag():

   http://code.google.com/p/argparse/issues/detail?id=62

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From greg.ewing at canterbury.ac.nz  Mon Mar  8 22:46:56 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 09 Mar 2010 10:46:56 +1300
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <A2F055BA-ECBC-474E-A076-2AF571C4B582@masklinn.net>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
	<de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>
	<A2F055BA-ECBC-474E-A076-2AF571C4B582@masklinn.net>
Message-ID: <4B957050.90808@canterbury.ac.nz>

Xavier Morel wrote:
> So you'd have to write
> add_argument('--plot', action=actions.store_true) which is straight from
> the department of redundant redundancies.

This could easily be fixed with

   from argparse.actions import store_true

> An option would be 
> 
>   parser.add(actions.StoreTrue('--plot'))
> 
> but I'm not sure this makes any sense API-wise, and it would probably make
> the code a lot messier as the parser would have to reach into the action
> to get the information it needs.

It would make more sense if the object concerned were called
an Option or some such instead of an Action:

   parser.add(options.Bool('--plot'))

-- 
Greg

From guido at python.org  Mon Mar  8 22:40:33 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Mar 2010 13:40:33 -0800
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <C9306408-D304-49C5-AD19-65EF5D1BAE5B@twistedmatrix.com>
References: <201003081412.51255.victor.stinner@haypocalc.com> 
	<4B95307B.7060202@voidspace.org.uk>
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com> 
	<201003082150.02469.victor.stinner@haypocalc.com>
	<ca471dc21003081306s7b229146mb48e283e57052b0e@mail.gmail.com> 
	<C9306408-D304-49C5-AD19-65EF5D1BAE5B@twistedmatrix.com>
Message-ID: <ca471dc21003081340s23bf58ceu6a59f1f66d980ca8@mail.gmail.com>

On Mon, Mar 8, 2010 at 1:18 PM, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
>
> On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote:
>
> I am trying to remember why I made site.py failures non-fatal in the
> first place. I don't have any specific recollection but it must've
> been either from before the separation between site.py (part of the
> stdlib) and sitecustomize.py (site-specific) or out of a worry that if
> some external cause broke site.py (which does a lot of I/O) it would
> be a fatal breakdown of all Python execution.
>
> The thing that occurs to me is that one might want to write an
> administrative tool in Python to manipulate site.py, or even just some data
> that something in site.py would load.

This would be a more likely theory if we didn't have sitecustomize.py
to be manipulated.

> If exceptions from site.py were
> fatal, then bugs in such a tool would be completely unrecoverable; in trying
> to run it to un-do the buggy operation, it would crash immediately.
> On the other hand, such a tool should *really* be invoked with the -S option
> anyway, so... maybe not that pressing of a concern.

Right. I'm leaning towards the theory that treating site.py failures
as non-fatal is older than the separation between site.py and
sitecustomize.py.

-- 
--Guido van Rossum (python.org/~guido)

From victor.stinner at haypocalc.com  Mon Mar  8 22:43:45 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 8 Mar 2010 22:43:45 +0100
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <C9306408-D304-49C5-AD19-65EF5D1BAE5B@twistedmatrix.com>
References: <201003081412.51255.victor.stinner@haypocalc.com>
	<ca471dc21003081306s7b229146mb48e283e57052b0e@mail.gmail.com>
	<C9306408-D304-49C5-AD19-65EF5D1BAE5B@twistedmatrix.com>
Message-ID: <201003082243.45999.victor.stinner@haypocalc.com>

Le lundi 08 mars 2010 22:18:34, Glyph Lefkowitz a ?crit :
> On Mar 8, 2010, at 4:06 PM, Guido van Rossum wrote:
> > I am trying to remember why I made site.py failures non-fatal in the
> > first place. I don't have any specific recollection but it must've
> > been either from before the separation between site.py (part of the
> > stdlib) and sitecustomize.py (site-specific) or out of a worry that if
> > some external cause broke site.py (which does a lot of I/O) it would
> > be a fatal breakdown of all Python execution.
> 
> The thing that occurs to me is that one might want to write an
>  administrative tool in Python to manipulate site.py, or even just some
>  data that something in site.py would load.  If exceptions from site.py
>  were fatal, then bugs in such a tool would be completely unrecoverable; in
>  trying to run it to un-do the buggy operation, it would crash immediately.

I don't think that modifying the site.py file is a good idea. Why not 
editing/generating the sitecustomize or usercustomize module?

Anyway, if you write such tool: use a different language or use a failsafe 
Python interpreter dedicated to this tool ;-)

-- 
Victor Stinner
http://www.haypocalc.com/

From martin at v.loewis.de  Mon Mar  8 22:52:50 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 08 Mar 2010 22:52:50 +0100
Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1
In-Reply-To: <1268078767.5175.4.camel@kratos>
References: <1268078767.5175.4.camel@kratos>
Message-ID: <4B9571B2.2040601@v.loewis.de>

> thanks for any tips.

Set sqlite_setup_debug to True in setup.py

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Mon Mar  8 23:03:47 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 09 Mar 2010 11:03:47 +1300
Subject: [Python-Dev] [PEP 3148] futures - execute computations
 asynchronously
In-Reply-To: <e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
	<e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
Message-ID: <4B957443.7060904@canterbury.ac.nz>

Dj Gilcrease wrote:

> executor = executors.create(NAME, *args, **kwargs) # NAME is 'process'
> or 'thread' by default
> 
> from concurrent.futures import executors, ExecutorBase
> class MyExecutor(ExecutorBase): ...
> executors.register(NAME, MyExecutor)

I don't understand the reason for using a registration system
rather than just importing names from a module.

You mentioned wanting to globally change the executor class
being used by a program without having to make changes throughout.
Registering a different class under the same name would be one
way to do that, but you could achieve the same thing just by
assigning to a name in a module.

In other words, instead of inventing your own mechanism for
managing a namespace, just use a module as your namespace.

-- 
Greg

From ben+python at benfinney.id.au  Mon Mar  8 22:55:50 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 09 Mar 2010 08:55:50 +1100
Subject: [Python-Dev] argparse ugliness
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
	<de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>
Message-ID: <87lje2zdeh.fsf@benfinney.id.au>

David Stanek <dstanek at dstanek.com> writes:

> On Mon, Mar 8, 2010 at 10:40 AM, Steven Bethard
> <steven.bethard at gmail.com> wrote:
> > ?parser.add_argument ('--plot', action='store_true')
[?]

+1.

> Any reason not to do something like:
>
>   from argparse import actions
>   ...
>   parser.add_argument('--plot', actions.store_true)
>
> Basically a small namespace for the constants.

-1.

That ignores the value to the programmer in specifying keyword arguments
(there are a lot of arguments to the function, making keyword arguments
a better mean-what-you-say choice). To be equivalent, it would have to
be:

    from argparse import actions
    # ...

    parser.add_argument('--plot', action=actions.store_true)

I prefer the string values.

-- 
 \           ?Value your freedom or you will lose it, teaches history. |
  `\     ?Don't bother us with politics,? respond those who don't want |
_o__)                               to learn.? ?Richard Stallman, 2002 |
Ben Finney


From victor.stinner at haypocalc.com  Mon Mar  8 23:01:34 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 8 Mar 2010 23:01:34 +0100
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <ca471dc21003081306s7b229146mb48e283e57052b0e@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com>
	<201003082150.02469.victor.stinner@haypocalc.com>
	<ca471dc21003081306s7b229146mb48e283e57052b0e@mail.gmail.com>
Message-ID: <201003082301.34649.victor.stinner@haypocalc.com>

Le lundi 08 mars 2010 22:06:36, vous avez ?crit :
> OTOH I think the try/except in site.py around the execution of
> sitecustomize.py might be changed so that it prints a full traceback
> whenever it raises an exception other than ImportError or, again,
> exceptions inheriting from BaseException but not from Exception. IOW I
> think that exceptions coming out of sitecustomize.py should continued
> to be treated non-fatally, apart from ^C and friends, but I think that
> exceptions coming out of site.py *might* be considered more fatally
> once the change to exceptions coming out of sitecustomize.py is made.

Do you mean something like the following code?
----------------------
def execsitecustomize():
    """Run custom site specific code, if available."""
    try:
        import sitecustomize
    except ImportError:
        pass
    except Exception:
        if sys.flags.verbose:
            sys.excepthook(*sys.exc_info())
        else:
            print >>sys.stderr, "'import sitecustomize' failed; use -v for 
traceback"

def execusercustomize():
    """Run custom user specific code, if available."""
    try:
        import usercustomize
    except ImportError:
        pass
    except Exception:
        if sys.flags.verbose:
            sys.excepthook(*sys.exc_info())
        else:
            print >>sys.stderr, "'import usercustomize' failed; use -v for 
traceback"
----------------------

Using these functions, even if sitecustomize fails, usercustomize is imported. 
Can it be a problem? Does usercustomize requires that sitecustomize has been 
initialized correctly?

(The code can be changed to only import usercustomize if sitecutomize succeed 
or failed with an ImportError, but not if it fails with a different error)

-- 
Victor Stinner
http://www.haypocalc.com/

From guido at python.org  Mon Mar  8 23:14:51 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Mar 2010 14:14:51 -0800
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <201003082301.34649.victor.stinner@haypocalc.com>
References: <201003081412.51255.victor.stinner@haypocalc.com> 
	<201003082150.02469.victor.stinner@haypocalc.com>
	<ca471dc21003081306s7b229146mb48e283e57052b0e@mail.gmail.com> 
	<201003082301.34649.victor.stinner@haypocalc.com>
Message-ID: <ca471dc21003081414m47517fb6sa239f84511bde1fd@mail.gmail.com>

On Mon, Mar 8, 2010 at 2:01 PM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Le lundi 08 mars 2010 22:06:36, vous avez ?crit :
>> OTOH I think the try/except in site.py around the execution of
>> sitecustomize.py might be changed so that it prints a full traceback
>> whenever it raises an exception other than ImportError or, again,
>> exceptions inheriting from BaseException but not from Exception. IOW I
>> think that exceptions coming out of sitecustomize.py should continued
>> to be treated non-fatally, apart from ^C and friends, but I think that
>> exceptions coming out of site.py *might* be considered more fatally
>> once the change to exceptions coming out of sitecustomize.py is made.
>
> Do you mean something like the following code?
> ----------------------
> def execsitecustomize():
> ? ?"""Run custom site specific code, if available."""
> ? ?try:
> ? ? ? ?import sitecustomize
> ? ?except ImportError:
> ? ? ? ?pass
> ? ?except Exception:
> ? ? ? ?if sys.flags.verbose:
> ? ? ? ? ? ?sys.excepthook(*sys.exc_info())
> ? ? ? ?else:
> ? ? ? ? ? ?print >>sys.stderr, "'import sitecustomize' failed; use -v for
> traceback"
>
> def execusercustomize():
> ? ?"""Run custom user specific code, if available."""
> ? ?try:
> ? ? ? ?import usercustomize
> ? ?except ImportError:
> ? ? ? ?pass
> ? ?except Exception:
> ? ? ? ?if sys.flags.verbose:
> ? ? ? ? ? ?sys.excepthook(*sys.exc_info())
> ? ? ? ?else:
> ? ? ? ? ? ?print >>sys.stderr, "'import usercustomize' failed; use -v for
> traceback"
> ----------------------

Yes, roughly.

> Using these functions, even if sitecustomize fails, usercustomize is imported.

Until 5 minutes ago I didn't even know we had usercustomize. :-)

> Can it be a problem? Does usercustomize requires that sitecustomize has been
> initialized correctly?

I don't know, but if it is, execsitecustomize() could return a flag to
be checked by the code that calls execusercustomize().

> (The code can be changed to only import usercustomize if sitecutomize succeed
> or failed with an ImportError, but not if it fails with a different error)

Right. It doesn't strike me as a big deal either way.

-- 
--Guido van Rossum (python.org/~guido)

From darren at ontrenet.com  Mon Mar  8 23:28:10 2010
From: darren at ontrenet.com (Darren Govoni)
Date: Mon, 08 Mar 2010 17:28:10 -0500
Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1
In-Reply-To: <4B9571B2.2040601@v.loewis.de>
References: <1268078767.5175.4.camel@kratos>  <4B9571B2.2040601@v.loewis.de>
Message-ID: <1268087290.5175.9.camel@kratos>

Thanks. It says.

sqlite: found /usr/local/include/sqlite3.h
/usr/local/include/sqlite3.h: version 3.6.22

Which is correct. Yet when I run my progam using the newly built python
executable and attempt to use sqlite3
functionality across threads it still says:

ProgrammingError: SQLite objects created in a thread can only be used in
that same thread.The object was created in thread id -1217128768 and
this is thread id -1218753680

I set the -DSQLITE_THREADSAFE=1 flag on sqlite3 when I configured, built
and installed the lib.

Darren

On Mon, 2010-03-08 at 22:52 +0100, "Martin v. L?wis" wrote:

> > thanks for any tips.
> 
> Set sqlite_setup_debug to True in setup.py
> 
> Regards,
> Martin


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100308/6ce2ca77/attachment.html>

From martin at v.loewis.de  Mon Mar  8 23:39:19 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 08 Mar 2010 23:39:19 +0100
Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1
In-Reply-To: <1268087290.5175.9.camel@kratos>
References: <1268078767.5175.4.camel@kratos> <4B9571B2.2040601@v.loewis.de>
	<1268087290.5175.9.camel@kratos>
Message-ID: <4B957C97.2080508@v.loewis.de>

> Which is correct. Yet when I run my progam using the newly built python
> executable and attempt to use sqlite3
> functionality across threads it still says:
> 
> ProgrammingError: SQLite objects created in a thread can only be used in
> that same thread.The object was created in thread id -1217128768 and
> this is thread id -1218753680
> 
> I set the -DSQLITE_THREADSAFE=1 flag on sqlite3 when I configured, built
> and installed the lib.

So you'll need to run Python in a debugger, trying to find out under
what conditions the exception is raised.

This kind of question is off-topic for python-dev (which is about
development of Python, not about using it).

Regards,
Martin

From phd at phd.pp.ru  Mon Mar  8 23:52:41 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Tue, 9 Mar 2010 01:52:41 +0300
Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1
In-Reply-To: <1268087290.5175.9.camel@kratos>
References: <1268078767.5175.4.camel@kratos> <4B9571B2.2040601@v.loewis.de>
	<1268087290.5175.9.camel@kratos>
Message-ID: <20100308225241.GA14115@phd.pp.ru>

On Mon, Mar 08, 2010 at 05:28:10PM -0500, Darren Govoni wrote:
> ProgrammingError: SQLite objects created in a thread can only be used in
> that same thread.The object was created in thread id -1217128768 and
> this is thread id -1218753680

   Darren, try to pass check_same_thread=False when creating a connection.

Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From barry at python.org  Tue Mar  9 13:56:36 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 9 Mar 2010 07:56:36 -0500
Subject: [Python-Dev] Python 2.6.5 rc 2
Message-ID: <20100309075636.09f52b27@heresy.wooz.org>

Hi Python hackateers!

It looks like we finally have no more release blockers for 2.6.5rc2.  I would
like to tag the tree tonight for rc2 so that Martin can build the Windows
installer for a release tomorrow.  I am also moving the final release back to
Friday March 19.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100309/c2cb702a/attachment.pgp>

From darren at ontrenet.com  Tue Mar  9 14:05:52 2010
From: darren at ontrenet.com (Darren Govoni)
Date: Tue, 09 Mar 2010 08:05:52 -0500
Subject: [Python-Dev] Building thread-safe sqlite3 with Python 2.6.5rc1
In-Reply-To: <20100308225241.GA14115@phd.pp.ru>
References: <1268078767.5175.4.camel@kratos> <4B9571B2.2040601@v.loewis.de>
	<1268087290.5175.9.camel@kratos>  <20100308225241.GA14115@phd.pp.ru>
Message-ID: <1268139952.18649.0.camel@kratos>

That worked great Oleg! Thank you!

On Tue, 2010-03-09 at 01:52 +0300, Oleg Broytman wrote:

> On Mon, Mar 08, 2010 at 05:28:10PM -0500, Darren Govoni wrote:
> > ProgrammingError: SQLite objects created in a thread can only be used in
> > that same thread.The object was created in thread id -1217128768 and
> > this is thread id -1218753680
> 
>    Darren, try to pass check_same_thread=False when creating a connection.
> 
> Oleg.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100309/1a577e0d/attachment.html>

From jeremy at alum.mit.edu  Tue Mar  9 14:42:28 2010
From: jeremy at alum.mit.edu (Jeremy Hylton)
Date: Tue, 9 Mar 2010 08:42:28 -0500
Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts
In-Reply-To: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com>
References: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com>
Message-ID: <e8bf7a531003090542o2c145f58td7ecd7f74a7e86f4@mail.gmail.com>

On Sat, Mar 6, 2010 at 11:27 AM, Gregg Lind <gregg.lind at gmail.com> wrote:
> Python-devs,
>
> I'm writing to you for some help in understanding the Python grammar.? As an
> excuse to deep dive into Python's tokenizer / grammar, I decided (as a
> hideous, hideous joke) to want to allow braces where colons are allowed (as
> flow control).
>
> Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer
>
> As a first example:
>
> funcdef: ('def' NAME parameters ['->' test] ':' suite |
> ????????? 'def' NAME parameters ['->' test] '{' suite '}' )
>
> I reran Parser/pgen and the dfa changes, but python (3.1) when recompiled,
> throws errors on things like:
>
> def a() { None }
>
> Strangely enough:
>
> lambdef: ( 'lambda' [varargslist] ':' test? |
> ?????????????? 'lambda' [varargslist] '{' test '}' )
>
> works fine!? I this simplely some difference between "test" and "suite".
>
> I have tried tackling this with gdb, looking at err_input clearly isn't
> enough.
>
> ??? (gdb) break err_input
> ??? (gdb) break PyParser_ASTFromString
> ??? import sys
> ??? b = compile("def a() {pass}","sys.stdout","single")
> ??? # yet a simple grammar fix is enough for this!
> ??? c = compile("lambda x {None}","sys.stdout","single")
>
> I'm in over my head!

You don't say what errors occur when you try to compile strings in
your new language.  You may have changed the Grammar, which allows you
to tokenize the input.  That isn't enough to get the input to compile.
 You also need to change the compiler to understand the new tokens.

Jeremy

> Any insights / help would be appreciated.? Full-on flaming is also
> appropriate, but would be less appreciated.
>
> Specific questions
>
> 1.)? I assume the Grammar/grammar is read top to bottom.? Confirm?
> 2.)? More help figuring out how to debug what python *thinks* it's seeing
> when it see "def a() {pass}".? It's not getting to the ast construction
> stage, as near as I can tell.? What additional breakpoints can I set to see
> where it's failing.
>
> Gregg L.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
>
>

From aahz at pythoncraft.com  Tue Mar  9 17:08:29 2010
From: aahz at pythoncraft.com (Aahz)
Date: Tue, 9 Mar 2010 08:08:29 -0800
Subject: [Python-Dev] FWD: http://www.python.org/dev/patches/
Message-ID: <20100309160828.GA22698@panix.com>

----- Forwarded message from Bob Vadnais <bob at boblicious.com> -----

> Date: Tue, 09 Mar 2010 00:37:33 -0500
> From: Bob Vadnais <bob at boblicious.com>
> To: webmaster at python.org
> Subject: http://www.python.org/dev/patches/
> 
> > Submit documentation patches the same way. When adding the patch, be > 
> sure to set the "Category" field to "Documentation".
>
> There doesn't appear to be a field named "Category".  Perhaps there used  
> to be?  The correct field name now appears to be "Components".

----- End forwarded message -----

-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer

From solipsis at pitrou.net  Tue Mar  9 00:23:23 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 8 Mar 2010 18:23:23 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
	<e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
	<20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>
Message-ID: <20100308182323.1f7b991e@msiwind>

Le Mon, 08 Mar 2010 21:11:45 -0000,
exarkun at twistedmatrix.com a ?crit :
> 
> Getting rid of the process-global state like this simplifies testing 
> (both testing of the executors themselves and of application code
> which uses them).  It also eliminates the unpleasant interpreter 
> shutdown/module globals interactions that have plagued a number of 
> stdlib systems that keep global state.

+1.



From tjreedy at udel.edu  Tue Mar  9 06:14:28 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 09 Mar 2010 00:14:28 -0500
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B956E82.4080707@canterbury.ac.nz>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>	<20100307035657.111563A4077@sparrow.telecommunity.com>	<4B93303E.9050301@gmail.com>	<20100307154809.742E73A4062@sparrow.telecommunity.com>	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>	<20100307175732.307513A4062@sparrow.telecommunity.com>	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>	<20100307195614.BBD693A4062@sparrow.telecommunity.com>	<4B94DC30.2030904@gmail.com>
	<hn37su$tq9$1@dough.gmane.org> <4B956E82.4080707@canterbury.ac.nz>
Message-ID: <hn4lfg$9im$1@dough.gmane.org>

On 3/8/2010 4:39 PM, Greg Ewing wrote:
> Terry Reedy wrote:
>> Looking more close, I gather that the prime results will be printed
>> 'in order' (waiting on each even if others are done) while the url
>> results will be printed 'as available'.
>
> Seems to me that if you care about the order of the results,
> you should be able to just wait for each result separately
> in the order you want them. Something like
>
> task1 = start_task(proc1)
> task2 = start_task(proc2)
> task3 = start_task(proc3)
> result1 = task1.wait_for_result()
> result2 = task2.wait_for_result()
> result3 = task3.wait_for_result()

*If* I understand the first example correctly, this is effectively what 
the first example does with two loops. But I was hoping for 
clarification and amplification in the PEP.
>
> This would also be a natural way to write things even if
> you don't care about the order, but you need all the results
> before proceeding. You're going to be held up until the
> longest-running task completes anyway, so it doesn't matter
> if some of them finish earlier and have to sit around
> waiting for you to collect the result.
>



From rdmurray at bitdance.com  Tue Mar  9 18:31:23 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 09 Mar 2010 12:31:23 -0500
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <hn3qji$ba6$1@dough.gmane.org>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>
	<67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk>
	<4B956A26.6010204@canterbury.ac.nz> <hn3qji$ba6$1@dough.gmane.org>
Message-ID: <20100309173123.8268E1AB57F@kimball.webabinitio.net>

On Mon, 08 Mar 2010 15:35:46 -0600, Robert Kern <robert.kern at gmail.com> wrote:
> On 2010-03-08 15:20 PM, Greg Ewing wrote:
> > Mark Russell wrote:
> >> Boolean flags are a common enough case that I'd be inclined to add a
> >> wrapper method,
> >>
> >> parser.add_bool_argument('--plot')
> >
> > +1, this looks good.
> 
> I've added it to the argparse bugtracker, along with my suggested spelling 
> add_flag():
> 
>    http://code.google.com/p/argparse/issues/detail?id=62

Shouldn't argparse bugs/enhancement tickets be going to bugs.python.org
now?

--
R. David Murray                                      www.bitdance.com

From robert.kern at gmail.com  Tue Mar  9 18:46:17 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 9 Mar 2010 11:46:17 -0600
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <20100309173123.8268E1AB57F@kimball.webabinitio.net>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com> 
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com> 
	<67850396-A793-4547-9532-CDF0F65E0977@zen.co.uk>
	<4B956A26.6010204@canterbury.ac.nz> <hn3qji$ba6$1@dough.gmane.org>
	<20100309173123.8268E1AB57F@kimball.webabinitio.net>
Message-ID: <3d375d731003090946p4e3e3d90x70649b4b25c43843@mail.gmail.com>

On Tue, Mar 9, 2010 at 11:31, R. David Murray <rdmurray at bitdance.com> wrote:
> On Mon, 08 Mar 2010 15:35:46 -0600, Robert Kern <robert.kern at gmail.com> wrote:
>> On 2010-03-08 15:20 PM, Greg Ewing wrote:
>> > Mark Russell wrote:
>> >> Boolean flags are a common enough case that I'd be inclined to add a
>> >> wrapper method,
>> >>
>> >> parser.add_bool_argument('--plot')
>> >
>> > +1, this looks good.
>>
>> I've added it to the argparse bugtracker, along with my suggested spelling
>> add_flag():
>>
>> ? ?http://code.google.com/p/argparse/issues/detail?id=62
>
> Shouldn't argparse bugs/enhancement tickets be going to bugs.python.org
> now?

Probably. Having used the the argparse tracker before, though, it was
the first place I thought to file it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

From digitalxero at gmail.com  Tue Mar  9 22:32:53 2010
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Tue, 9 Mar 2010 14:32:53 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com> 
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com> 
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com> 
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com> 
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com> 
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com> 
	<e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com> 
	<20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>
Message-ID: <e9764b731003091332y701e167s4e04ab521b4510de@mail.gmail.com>

On Mon, Mar 8, 2010 at 2:11 PM,  <exarkun at twistedmatrix.com> wrote:
> Getting rid of the process-global state like this simplifies testing (both
> testing of the executors themselves and of application code which uses
> them). ?It also eliminates the unpleasant interpreter shutdown/module
> globals interactions that have plagued a number of stdlib systems that keep
> global state.


Ok the new patch is submitted @
http://code.google.com/p/pythonfutures/issues/detail?id=1

*note there are 2 tests that fail and 1 test that dead locks on
windows even without this patch, the deadlock test I am skipping in
the patch and the two that fail do so for a reason that does not make
sense to me.

From brian at sweetapp.com  Wed Mar 10 09:19:35 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 10 Mar 2010 19:19:35 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <e9764b731003091332y701e167s4e04ab521b4510de@mail.gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
	<e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
	<20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>
	<e9764b731003091332y701e167s4e04ab521b4510de@mail.gmail.com>
Message-ID: <40C6F203-726D-4543-AE75-D090A1DA580E@sweetapp.com>


On 10 Mar 2010, at 08:32, Dj Gilcrease wrote:

> On Mon, Mar 8, 2010 at 2:11 PM,  <exarkun at twistedmatrix.com> wrote:
>> Getting rid of the process-global state like this simplifies  
>> testing (both
>> testing of the executors themselves and of application code which  
>> uses
>> them).  It also eliminates the unpleasant interpreter shutdown/module
>> globals interactions that have plagued a number of stdlib systems  
>> that keep
>> global state.
>
>
> Ok the new patch is submitted @
> http://code.google.com/p/pythonfutures/issues/detail?id=1

Cool, thanks.

> *note there are 2 tests that fail and 1 test that dead locks on
> windows even without this patch, the deadlock test I am skipping in
> the patch and the two that fail do so for a reason that does not make
> sense to me.

I'll investigate but I don't have convenient access to a windows  
machine.

Cheers,
Brian

From brian at sweetapp.com  Wed Mar 10 09:19:38 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 10 Mar 2010 19:19:38 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B956E82.4080707@canterbury.ac.nz>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
	<4B94DC30.2030904@gmail.com> <hn37su$tq9$1@dough.gmane.org>
	<4B956E82.4080707@canterbury.ac.nz>
Message-ID: <9DDE3DD4-29F2-4062-B28A-C9394BB565B6@sweetapp.com>


On 9 Mar 2010, at 08:39, Greg Ewing wrote:

> Terry Reedy wrote:
>> Looking more close, I gather that the prime results will be printed  
>> 'in order' (waiting on each even if others are done) while the url  
>> results will be printed 'as available'.
>
> Seems to me that if you care about the order of the results,
> you should be able to just wait for each result separately
> in the order you want them. Something like
>
>  task1 = start_task(proc1)
>  task2 = start_task(proc2)
>  task3 = start_task(proc3)
>  result1 = task1.wait_for_result()
>  result2 = task2.wait_for_result()
>  result3 = task3.wait_for_result()

You can write this as:

executor = ...
future1 = executor.submit(proc1)
future2 = executor.submit(proc2)
future3 = executor.submit(proc3)
result1 = task1.result()
result2 = task2.result()
result3 = task3.result()

> This would also be a natural way to write things even if
> you don't care about the order, but you need all the results
> before proceeding. You're going to be held up until the
> longest-running task completes anyway, so it doesn't matter
> if some of them finish earlier and have to sit around
> waiting for you to collect the result.

Often you don't want to continue if there is a failure.

In the example that you gave, if "proc3" raises an exception  
immediately, you still wait for "proc1" and "proc2" to complete even  
though you will end up discarding their results.

Cheers,
Brian

From brian at sweetapp.com  Wed Mar 10 09:19:42 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 10 Mar 2010 19:19:42 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
	<e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
	<20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>
Message-ID: <D4E9C22C-5066-47E8-8353-BA8ACEFD9F5A@sweetapp.com>


On 9 Mar 2010, at 08:11, exarkun at twistedmatrix.com wrote:

> On 08:56 pm, digitalxero at gmail.com wrote:
>> On Mon, Mar 8, 2010 at 12:04 PM, Dj Gilcrease  
>> <digitalxero at gmail.com> wrote:
>>> A style I have used in my own code in the past is a Singleton class
>>> with register and create methods, where the register takes a
>>> name(string) and the class and the create method takes the name and
>>> *args, **kwargs and acts as a factory.
>>
>>
>> So I decided to play with this design a little and since I made it a
>> singleton I decided to place all the thread/process tracking and exit
>> handle code in it instead of having the odd semi-global scoped
>> _shutdown, _thread_references, _remove_dead_thread_references and
>> _python_exit objects floating around in each executor file, seems to
>> work well. The API would be
>>
>> from concurrent.futures import executors
>>
>> executor = executors.create(NAME, *args, **kwargs) # NAME is  
>> 'process'
>> or 'thread' by default
>>
>>
>> To create your own executor you create your executor class and add  
>> the
>> following at the end
>
> Getting rid of the process-global state like this simplifies testing  
> (both testing of the executors themselves and of application code  
> which uses them).  It also eliminates the unpleasant interpreter  
> shutdown/module globals interactions that have plagued a number of  
> stdlib systems that keep global state.

I'm not sure what you mean, could you clarify?

Cheers,
Brian

From brian at sweetapp.com  Wed Mar 10 09:19:53 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Wed, 10 Mar 2010 19:19:53 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <hn386j$uuk$1@dough.gmane.org>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>	<eae285401003050821q29aed45drefc86162cbb01a9e@mail.gmail.com>
	<7EFBC33B-06AD-4BAD-8173-005D6F08240E@sweetapp.com>
	<hn386j$uuk$1@dough.gmane.org>
Message-ID: <8AAB0679-897B-4D08-8C42-0DE1732055DC@sweetapp.com>


On 9 Mar 2010, at 03:21, Terry Reedy wrote:

> On 3/6/2010 4:20 AM, Brian Quinlan wrote:
>>
>> On 6 Mar 2010, at 03:21, Daniel Stutzbach wrote:
>>
>>> On Fri, Mar 5, 2010 at 12:03 AM, Brian Quinlan <brian at sweetapp.com
>>> <mailto:brian at sweetapp.com>> wrote:
>>>
>>>    import futures
>>>
>>>
>>> +1 on the idea, -1 on the name. It's too similar to "from __future__
>>> import ...".
>>>
>>> Also, the PEP should probably link to the discussions on stdlib-sig?
>>
>> I thoug ht about that but this discussion is spread over many threads
>> and many months.
>
> This is pretty typical. I would say just that, and link to the first.
> "This PEP was discussed over many months in many threads in the  
> stdlib-sig list. The first was .... . Python-dev discussion occured  
> in <this thread>.

I'll add that.

Cheers,
Brian

From ncoghlan at gmail.com  Wed Mar 10 13:32:15 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 Mar 2010 22:32:15 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <D4E9C22C-5066-47E8-8353-BA8ACEFD9F5A@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
	<e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
	<20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>
	<D4E9C22C-5066-47E8-8353-BA8ACEFD9F5A@sweetapp.com>
Message-ID: <4B97914F.50402@gmail.com>

Brian Quinlan wrote:
>> Getting rid of the process-global state like this simplifies testing
>> (both testing of the executors themselves and of application code
>> which uses them).  It also eliminates the unpleasant interpreter
>> shutdown/module globals interactions that have plagued a number of
>> stdlib systems that keep global state.
> 
> I'm not sure what you mean, could you clarify?

Assuming your question refers to the second sentence, Jean-Paul is
referring to a trick of the CPython interpreter when it terminates. To
maximise the chances of objects being deleted properly rather than just
dumped from memory when the process exits, module dictionaries are
filled with None values before the interpreter shuts down.

This can cause weirdness (usually intermittent name errors during
shutdown) when __del__ methods directly or indirectly reference module
globals.

One of the easiest ways to avoid that is to put the state on a singleton
object, then give the affected classes a reference to that object.

Cheers,
Nick.

P.S. This problem is actually the reason we don't have a context manager
for temporary directories yet. Something that should have been simple
became a twisty journey down the rabbit hole:
http://bugs.python.org/issue5178

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed Mar 10 13:37:38 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 Mar 2010 22:37:38 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <19349.25574.880099.668647@montanaro.dyndns.org>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
	<4B94DC30.2030904@gmail.com>
	<19349.25574.880099.668647@montanaro.dyndns.org>
Message-ID: <4B979292.4080906@gmail.com>

skip at pobox.com wrote:
>     >> I'm +1 on adding a nice task queuing system, -1 on calling it by any
>     >> other name.  ;-)
> 
>     Nick> As Guido said, let's call the nice task queuing system "futures"
>     Nick> and point people wanting a full-power asynchronous process model
>     Nick> to Twisted
> 
> Can this module at least be pushed down into a package?  I think
> "concurrent" or "concurrency" were both suggested at one point.

Yep, I believe "concurrent.futures" was picked as the name elsewhere in
the thread.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed Mar 10 13:46:14 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 Mar 2010 22:46:14 +1000
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <4B956CA4.4050108@canterbury.ac.nz>
References: <hmrjr4$kkd$1@dough.gmane.org>	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>	<hn0676$br0$1@dough.gmane.org>	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>
	<4B956CA4.4050108@canterbury.ac.nz>
Message-ID: <4B979496.1080705@gmail.com>

Greg Ewing wrote:
> Steven Bethard wrote:
> 
>> Because the names are so long and you'd have to import them, I've left
>> them as private attributes of the module, but if there's really
>> demand, we could rename them to argparse.StoreTrueAction, etc.
> 
> What's wrong with just StoreTrue?

All of this discussion about the class names is ignoring the main
benefit of using the string names:

- with Python variables, you just get a generic name error at the
reference site (which may or may not be useful, depending on the program
structure)
- with a string, the parser *tells* you that the problem is with the
requested action for a particular argument

The second, explicit error is going to be more informative in most cases.

Using strings also reduces the verbosity of the code, avoiding either an
"argparse." buried in the middle of the function call, or else a "from
argparse import <whatever>".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed Mar 10 13:48:23 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 Mar 2010 22:48:23 +1000
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <4B957050.90808@canterbury.ac.nz>
References: <hmrjr4$kkd$1@dough.gmane.org>	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>	<hn0676$br0$1@dough.gmane.org>	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>	<de32cc031003080753o2cd6bccfw8c1e79996e72dc61@mail.gmail.com>	<A2F055BA-ECBC-474E-A076-2AF571C4B582@masklinn.net>
	<4B957050.90808@canterbury.ac.nz>
Message-ID: <4B979517.10406@gmail.com>

Greg Ewing wrote:
> Xavier Morel wrote:
>> So you'd have to write add_argument('--plot',
>> action=actions.store_true) which is straight from the department of
>> redundant redundancies.
> 
> This could easily be fixed with
> 
> from argparse.actions import store_true

Converting argparse from a module to a package (which would be necessary
to make the above work) is above the bar I set for "easy".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From eric at trueblade.com  Wed Mar 10 14:13:17 2010
From: eric at trueblade.com (Eric Smith)
Date: Wed, 10 Mar 2010 08:13:17 -0500
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <4B979496.1080705@gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com>	<hn0676$br0$1@dough.gmane.org>	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com>	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com>	<4B956CA4.4050108@canterbury.ac.nz>
	<4B979496.1080705@gmail.com>
Message-ID: <4B979AED.8010300@trueblade.com>

Nick Coghlan wrote:
> Greg Ewing wrote:
>> Steven Bethard wrote:
>>
>>> Because the names are so long and you'd have to import them, I've left
>>> them as private attributes of the module, but if there's really
>>> demand, we could rename them to argparse.StoreTrueAction, etc.
>> What's wrong with just StoreTrue?
> 
> All of this discussion about the class names is ignoring the main
> benefit of using the string names:
> 
> - with Python variables, you just get a generic name error at the
> reference site (which may or may not be useful, depending on the program
> structure)
> - with a string, the parser *tells* you that the problem is with the
> requested action for a particular argument
> 
> The second, explicit error is going to be more informative in most cases.
> 
> Using strings also reduces the verbosity of the code, avoiding either an
> "argparse." buried in the middle of the function call, or else a "from
> argparse import <whatever>".

I agree that it should be left as a string. About the only reason I 
would suggest changing it from a string to an object is that it makes it 
obvious that argparse does accept an object that conforms to the Action 
API. But this is already well documented, so I don't see the need. Just 
leave it as a string.

-- 
Eric.

From fdrake at acm.org  Wed Mar 10 14:22:44 2010
From: fdrake at acm.org (Fred Drake)
Date: Wed, 10 Mar 2010 08:22:44 -0500
Subject: [Python-Dev] argparse ugliness
In-Reply-To: <4B979496.1080705@gmail.com>
References: <hmrjr4$kkd$1@dough.gmane.org>
	<cf9f31f21003051115u2682b22dk4d2c3f55e0b00e27@mail.gmail.com> 
	<hn0676$br0$1@dough.gmane.org>
	<ca471dc21003071149g639a25f3wba466b1d74ea1d7a@mail.gmail.com> 
	<d11dcfba1003080740t7bd48792k403a9d7bb38f5d2d@mail.gmail.com> 
	<4B956CA4.4050108@canterbury.ac.nz> <4B979496.1080705@gmail.com>
Message-ID: <9cee7ab81003100522h1a6bcac6ib4061a48327750e@mail.gmail.com>

On Wed, Mar 10, 2010 at 7:46 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> All of this discussion about the class names is ignoring the main
> benefit of using the string names:

Another benefit of strings is that data-driven argparse configuration
will usually be slightly simpler.

Some of us find things like that useful.  :-)


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller

From barry at python.org  Wed Mar 10 18:50:10 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 10 Mar 2010 12:50:10 -0500
Subject: [Python-Dev] Python 2.6.5 release candidate 2 now available
Message-ID: <20100310125010.415f838e@heresy.wooz.org>

Hi everyone,

The source tarballs and Windows installer for Python 2.6.5 release candidate 2
are now available:

http://www.python.org/download/releases/2.6.5/

As usual, please download, install, and test them with your favorite projects
and environments.  A number of regressions and build problems on OS X have
been fixed since rc1, and I'm really hoping we will not have to do an rc3.
I'm currently planning on releasing 2.6.5 final on March 19, 2010.

Enjoy,
-Barry

P.S. The Mac installer will hopefully be available soon.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100310/9f88da04/attachment.pgp>

From fuzzyman at voidspace.org.uk  Thu Mar 11 01:45:00 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 11 Mar 2010 00:45:00 +0000
Subject: [Python-Dev] FWD: http://www.python.org/dev/patches/
In-Reply-To: <20100309160828.GA22698@panix.com>
References: <20100309160828.GA22698@panix.com>
Message-ID: <4B983D0C.1030309@voidspace.org.uk>

On 09/03/2010 16:08, Aahz wrote:
> ----- Forwarded message from Bob Vadnais<bob at boblicious.com
> >  -----
>
>    
>> Date: Tue, 09 Mar 2010 00:37:33 -0500
>> From: Bob Vadnais<bob at boblicious.com>
>> To: webmaster at python.org
>> Subject: http://www.python.org/dev/patches/
>>
>>      
>>> Submit documentation patches the same way. When adding the patch, be>
>>>        
>> sure to set the "Category" field to "Documentation".
>>
>> There doesn't appear to be a field named "Category".  Perhaps there used
>> to be?  The correct field name now appears to be "Components".
>>      

Yep, 'Components' is correct. I'm amending that page appropriately.

Thanks

Michael Foord

> ----- End forwarded message -----
>
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From victor.stinner at haypocalc.com  Thu Mar 11 02:21:42 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Thu, 11 Mar 2010 02:21:42 +0100
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com>
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com>
	<693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com>
Message-ID: <201003110221.42480.victor.stinner@haypocalc.com>

Hi,

Some news about my patch.

Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a ?crit :
> On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido at python.org> wrote:
> > Actually it sounds like there's some overly general except clause
> > somewhere that should be adjusted to catch just "Exception" instead of
> > "*".
> 
> There is at least one that prints "import 'site' failed" and continues
> to run your program.

I commited to patch and broke all buildbots \o/ There were errors in 
test_sysconfig and test_subprocess related to issues #7774 and #7880 (problems 
with sys.executable and modified sys.argv[0]).

I consider that it's a good point to detect the errors earlier, but it breaks 
code based on the errors :-) I will no backport the fix to 2.6 (or at least, 
not the fix about importing the site module)

Should I backport the fix to 3.1? (The backport to py3k is not done yet)

-- 
Victor Stinner
http://www.haypocalc.com/

From brian at sweetapp.com  Thu Mar 11 12:00:38 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Thu, 11 Mar 2010 22:00:38 +1100
Subject: [Python-Dev] [PEP 3148] futures - execute
	computations	asynchronously
In-Reply-To: <4B97914F.50402@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<e9764b731003061909q2a4db98dh388f74a1d047563a@mail.gmail.com>
	<4222a8491003070550t3b4de932qb2ce00a57c0347ef@mail.gmail.com>
	<e9764b731003071344w54c94690u3c94752ce2464b78@mail.gmail.com>
	<4B94DE96.8080100@gmail.com>
	<e9764b731003081104w217a3db3k69bc93275b66a62@mail.gmail.com>
	<e9764b731003081256j7680cb6bvb1456420f77f7648@mail.gmail.com>
	<20100308211145.2792.1191903189.divmod.xquotient.1412@localhost.localdomain>
	<D4E9C22C-5066-47E8-8353-BA8ACEFD9F5A@sweetapp.com>
	<4B97914F.50402@gmail.com>
Message-ID: <50A261E0-C876-4FA3-91ED-9BDE55572839@sweetapp.com>


On 10 Mar 2010, at 23:32, Nick Coghlan wrote:

> Brian Quinlan wrote:
>>> Getting rid of the process-global state like this simplifies testing
>>> (both testing of the executors themselves and of application code
>>> which uses them).  It also eliminates the unpleasant interpreter
>>> shutdown/module globals interactions that have plagued a number of
>>> stdlib systems that keep global state.
>>
>> I'm not sure what you mean, could you clarify?
>
> Assuming your question refers to the second sentence, Jean-Paul is
> referring to a trick of the CPython interpreter when it terminates. To
> maximise the chances of objects being deleted properly rather than  
> just
> dumped from memory when the process exits, module dictionaries are
> filled with None values before the interpreter shuts down.
>
> This can cause weirdness (usually intermittent name errors during
> shutdown) when __del__ methods directly or indirectly reference module
> globals.


Ah. I'm familiar with this problem. My approach was to install an exit  
handler that ensures that all pending futures are complete and all  
threads and processes exit before allowing the interpreter to exit.

Cheers,
Brian

> One of the easiest ways to avoid that is to put the state on a  
> singleton
> object, then give the affected classes a reference to that object.
>
> Cheers,
> Nick.
>
> P.S. This problem is actually the reason we don't have a context  
> manager
> for temporary directories yet. Something that should have been simple
> became a twisty journey down the rabbit hole:
> http://bugs.python.org/issue5178
>
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com


From victor.stinner at haypocalc.com  Thu Mar 11 14:56:39 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Thu, 11 Mar 2010 14:56:39 +0100
Subject: [Python-Dev] Bug tracker: no confirmation before deleting a message
Message-ID: <201003111456.39941.victor.stinner@haypocalc.com>

Hi,

I removed two messages by error in the bug tracker :-(

The first message is msg100432 (first message of #8063) which has been 
restored by "admin". Thank you admin!

The second message is msg98284, part of #7774. How can I restore it? Or can 
"admin" restore it please? :-)

There is an open issue on Roundup tracker to add a button to restore a message 
or a deleted file:

  http://psf.upfronthosting.co.za/roundup/meta/issue267

I tried to following URL to restore the message, but it didn't work:

  http://bugs.python.org/issue8063?@action=edit&@add at messages=100432

Could you at least add a Javascript confirmation dialog please? :-)

-- 
Victor Stinner
http://www.haypocalc.com/

From martin at v.loewis.de  Thu Mar 11 15:59:01 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 11 Mar 2010 15:59:01 +0100
Subject: [Python-Dev] Bug tracker: no confirmation before deleting a
	message
In-Reply-To: <201003111456.39941.victor.stinner@haypocalc.com>
References: <201003111456.39941.victor.stinner@haypocalc.com>
Message-ID: <4B990535.6070700@v.loewis.de>

> I removed two messages by error in the bug tracker :-(

Please discuss tracker issues on tracker-discuss or, better yet, file a
report on the meta tracker.

> The second message is msg98284, part of #7774. How can I restore it? Or can 
> "admin" restore it please? :-)

Done.

> There is an open issue on Roundup tracker to add a button to restore a message 
> or a deleted file:
> 
>   http://psf.upfronthosting.co.za/roundup/meta/issue267

Unfortunately, I didn't have time to look into this yet (and may not for
many more weeks to come).

> I tried to following URL to restore the message, but it didn't work:
> 
>   http://bugs.python.org/issue8063?@action=edit&@add at messages=100432

Roundup has stopped accepting GET requests that edit the database, for
XSS concerns.

> Could you at least add a Javascript confirmation dialog please? :-)

That would be even more effort than looking into the existing issue,
unless code is provided.

Regards,
Martin

From brett at python.org  Thu Mar 11 21:58:11 2010
From: brett at python.org (Brett Cannon)
Date: Thu, 11 Mar 2010 12:58:11 -0800
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <201003110221.42480.victor.stinner@haypocalc.com>
References: <201003081412.51255.victor.stinner@haypocalc.com> 
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com> 
	<693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> 
	<201003110221.42480.victor.stinner@haypocalc.com>
Message-ID: <bbaeab101003111258o5b87d97dk69a098aaabbb5e9c@mail.gmail.com>

On Wed, Mar 10, 2010 at 17:21, Victor Stinner
<victor.stinner at haypocalc.com>wrote:

> Hi,
>
> Some news about my patch.
>
> Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a ?crit :
> > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido at python.org>
> wrote:
> > > Actually it sounds like there's some overly general except clause
> > > somewhere that should be adjusted to catch just "Exception" instead of
> > > "*".
> >
> > There is at least one that prints "import 'site' failed" and continues
> > to run your program.
>
> I commited to patch and broke all buildbots \o/ There were errors in
> test_sysconfig and test_subprocess related to issues #7774 and #7880
> (problems
> with sys.executable and modified sys.argv[0]).
>
> I consider that it's a good point to detect the errors earlier, but it
> breaks
> code based on the errors :-) I will no backport the fix to 2.6 (or at
> least,
> not the fix about importing the site module)
>
> Should I backport the fix to 3.1? (The backport to py3k is not done yet)
>
>
Once you are happy with the stability then please port it to 3.1.

-Brett




>  --
> Victor Stinner
> http://www.haypocalc.com/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100311/757eddd4/attachment.html>

From benjamin at python.org  Thu Mar 11 22:33:43 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 11 Mar 2010 15:33:43 -0600
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <bbaeab101003111258o5b87d97dk69a098aaabbb5e9c@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com>
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com>
	<693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com>
	<201003110221.42480.victor.stinner@haypocalc.com>
	<bbaeab101003111258o5b87d97dk69a098aaabbb5e9c@mail.gmail.com>
Message-ID: <1afaf6161003111333k791e21afy8c1f47d30338ca76@mail.gmail.com>

2010/3/11 Brett Cannon <brett at python.org>:
>
>
> On Wed, Mar 10, 2010 at 17:21, Victor Stinner <victor.stinner at haypocalc.com>
> wrote:
>>
>> Hi,
>>
>> Some news about my patch.
>>
>> Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a ?crit :
>> > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido at python.org>
>> > wrote:
>> > > Actually it sounds like there's some overly general except clause
>> > > somewhere that should be adjusted to catch just "Exception" instead of
>> > > "*".
>> >
>> > There is at least one that prints "import 'site' failed" and continues
>> > to run your program.
>>
>> I commited to patch and broke all buildbots \o/ There were errors in
>> test_sysconfig and test_subprocess related to issues #7774 and #7880
>> (problems
>> with sys.executable and modified sys.argv[0]).
>>
>> I consider that it's a good point to detect the errors earlier, but it
>> breaks
>> code based on the errors :-) I will no backport the fix to 2.6 (or at
>> least,
>> not the fix about importing the site module)
>>
>> Should I backport the fix to 3.1? (The backport to py3k is not done yet)
>>
>
> Once you are happy with the stability then please port it to 3.1.
> -Brett

But after 3.1.2 please!



-- 
Regards,
Benjamin

From brett at python.org  Thu Mar 11 22:35:48 2010
From: brett at python.org (Brett Cannon)
Date: Thu, 11 Mar 2010 13:35:48 -0800
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <1afaf6161003111333k791e21afy8c1f47d30338ca76@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com> 
	<ca471dc21003080947g325657ceg91ee1078ec39947e@mail.gmail.com> 
	<693bc9ab1003081021icc69731i719df4b4f5d81430@mail.gmail.com> 
	<201003110221.42480.victor.stinner@haypocalc.com>
	<bbaeab101003111258o5b87d97dk69a098aaabbb5e9c@mail.gmail.com> 
	<1afaf6161003111333k791e21afy8c1f47d30338ca76@mail.gmail.com>
Message-ID: <bbaeab101003111335s672fd800lb9d3392d90e29655@mail.gmail.com>

On Thu, Mar 11, 2010 at 13:33, Benjamin Peterson <benjamin at python.org>wrote:

> 2010/3/11 Brett Cannon <brett at python.org>:
> >
> >
> > On Wed, Mar 10, 2010 at 17:21, Victor Stinner <
> victor.stinner at haypocalc.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> Some news about my patch.
> >>
> >> Le lundi 08 mars 2010 19:21:20, Maciej Fijalkowski a ?crit :
> >> > On Mon, Mar 8, 2010 at 10:47 AM, Guido van Rossum <guido at python.org>
> >> > wrote:
> >> > > Actually it sounds like there's some overly general except clause
> >> > > somewhere that should be adjusted to catch just "Exception" instead
> of
> >> > > "*".
> >> >
> >> > There is at least one that prints "import 'site' failed" and continues
> >> > to run your program.
> >>
> >> I commited to patch and broke all buildbots \o/ There were errors in
> >> test_sysconfig and test_subprocess related to issues #7774 and #7880
> >> (problems
> >> with sys.executable and modified sys.argv[0]).
> >>
> >> I consider that it's a good point to detect the errors earlier, but it
> >> breaks
> >> code based on the errors :-) I will no backport the fix to 2.6 (or at
> >> least,
> >> not the fix about importing the site module)
> >>
> >> Should I backport the fix to 3.1? (The backport to py3k is not done yet)
> >>
> >
> > Once you are happy with the stability then please port it to 3.1.
> > -Brett
>
> But after 3.1.2 please!
>

Sorry, I meant py3k (aka 3.2) and not 3.1. Since this is a behavior change I
don't think it should be in 2.6/3.1.

-Brett



>
>
>
> --
> Regards,
> Benjamin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100311/8a56cc8e/attachment-0001.html>

From victor.stinner at haypocalc.com  Fri Mar 12 01:05:39 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Fri, 12 Mar 2010 01:05:39 +0100
Subject: [Python-Dev] Catch SIGINT at Python startup
In-Reply-To: <bbaeab101003111258o5b87d97dk69a098aaabbb5e9c@mail.gmail.com>
References: <201003081412.51255.victor.stinner@haypocalc.com>
	<201003110221.42480.victor.stinner@haypocalc.com>
	<bbaeab101003111258o5b87d97dk69a098aaabbb5e9c@mail.gmail.com>
Message-ID: <201003120105.39900.victor.stinner@haypocalc.com>

Le jeudi 11 mars 2010 21:58:11, Brett Cannon a ?crit :
> > (...)
> > Should I backport the fix to 3.1? (The backport to py3k is not done yet)
> 
> Once you are happy with the stability then please port it to 3.1.

My commit broke test_sysconfig, test_platform and test_subprocess. Some 
commits later, #7880 is fixed and part of #7774 is also fixed. So yes, it will 
be possible to port the fix to py3k.

-- 
Victor Stinner
http://www.haypocalc.com/

From ndbecker2 at gmail.com  Fri Mar 12 02:52:57 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 11 Mar 2010 20:52:57 -0500
Subject: [Python-Dev] interesting article on regex performance
Message-ID: <hnc6pp$sr2$1@dough.gmane.org>

http://swtch.com/~rsc/regexp/regexp1.html


From peter.a.portante at gmail.com  Fri Mar 12 03:12:12 2010
From: peter.a.portante at gmail.com (Peter Portante)
Date: Thu, 11 Mar 2010 21:12:12 -0500
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <hnc6pp$sr2$1@dough.gmane.org>
Message-ID: <C7BF0D2C.2C422%peter.a.portante@gmail.com>

http://code.google.com/p/re2/


On 3/11/10 8:52 PM, "Neal Becker" <ndbecker2 at gmail.com> wrote:

> http://swtch.com/~rsc/regexp/regexp1.html
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com



From g.brandl at gmx.net  Fri Mar 12 11:13:10 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 12 Mar 2010 11:13:10 +0100
Subject: [Python-Dev] Modifying Grammar/grammar and other foul acts
In-Reply-To: <e8bf7a531003090542o2c145f58td7ecd7f74a7e86f4@mail.gmail.com>
References: <ff30f7581003060827h58d3fa25pb4e28201997fb628@mail.gmail.com>
	<e8bf7a531003090542o2c145f58td7ecd7f74a7e86f4@mail.gmail.com>
Message-ID: <hnd468$80u$1@dough.gmane.org>

Am 09.03.2010 14:42, schrieb Jeremy Hylton:
> On Sat, Mar 6, 2010 at 11:27 AM, Gregg Lind <gregg.lind at gmail.com> wrote:
>> Python-devs,
>>
>> I'm writing to you for some help in understanding the Python grammar.  As an
>> excuse to deep dive into Python's tokenizer / grammar, I decided (as a
>> hideous, hideous joke) to want to allow braces where colons are allowed (as
>> flow control).
>>
>> Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer
>>
>> As a first example:
>>
>> funcdef: ('def' NAME parameters ['->' test] ':' suite |
>>           'def' NAME parameters ['->' test] '{' suite '}' )
>>
>> I reran Parser/pgen and the dfa changes, but python (3.1) when recompiled,
>> throws errors on things like:
>>
>> def a() { None }
>>
>> Strangely enough:
>>
>> lambdef: ( 'lambda' [varargslist] ':' test  |
>>                'lambda' [varargslist] '{' test '}' )
>>
>> works fine!  I this simplely some difference between "test" and "suite".
>>
>> I have tried tackling this with gdb, looking at err_input clearly isn't
>> enough.
>>
>>     (gdb) break err_input
>>     (gdb) break PyParser_ASTFromString
>>     import sys
>>     b = compile("def a() {pass}","sys.stdout","single")
>>     # yet a simple grammar fix is enough for this!
>>     c = compile("lambda x {None}","sys.stdout","single")
>>
>> I'm in over my head!
> 
> You don't say what errors occur when you try to compile strings in
> your new language.  You may have changed the Grammar, which allows you
> to tokenize the input.  That isn't enough to get the input to compile.
>  You also need to change the compiler to understand the new tokens.

In particular, many AST creation functions check for specific counts of
children on many nodes.   I haven't checked, but in the case of the
"funcdef" rule, it may check for either 7 or 5 children to determine
whether the optional return annotation ['->' test] is present.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ncoghlan at gmail.com  Fri Mar 12 17:12:23 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 13 Mar 2010 02:12:23 +1000
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
Message-ID: <4B9A67E7.30801@gmail.com>

Peter Portante wrote:
> http://code.google.com/p/re2/
> 
> On 3/11/10 8:52 PM, "Neal Becker" <ndbecker2 at gmail.com> wrote:
> 
>> http://swtch.com/~rsc/regexp/regexp1.html

Both interesting links. I'll add another one to the list:
http://google-opensource.blogspot.com/2010/03/re2-principled-approach-to-regular.html

To bring this on-topic for python-dev by considering how it could apply
to Python's default re engine, I think the key issue is that any updates
to the default engine would need to remain backwards compatible with all
of the tricks that re2 doesn't support.

There are major practical problems associated with making such a leap
directly (Google's re2 engine is in C++ rather than C and we'd have to
keep the existing implementation around regardless to handle the
features that re2 doesn't support).

I would say it is better to let re2 bake for a while and see if anyone
is motivated to come up with Python bindings for it and release them on
PyPI. Once that happens (and assuming the bindings earn a good
reputation), the first step towards integration would be to include a
"See Also" in the re module documentation to point people towards the
more limited (but more robust) regex engine implementation.

The next step would probably be a hybrid third party library that
exploits the NFA approach when it can, but resorts to backtracking when
it has to in order to handle full regex functionality. (Although
developers would need to be able to disable the backtracking support in
order to restore re2's guarantees of linear time execution)

Only once such a hybrid library had been proven in practice could the
default re module be considered for possible update.

Basically, I see some interesting ideas there, but I don't see anything
that will impact the Python core or standard library in the short term.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From tseaver at palladion.com  Fri Mar 12 18:02:46 2010
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 12 Mar 2010 12:02:46 -0500
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <4B9A67E7.30801@gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
Message-ID: <hnds3n$tea$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Nick Coghlan wrote:
> Peter Portante wrote:
>> http://code.google.com/p/re2/
>>
>> On 3/11/10 8:52 PM, "Neal Becker" <ndbecker2 at gmail.com> wrote:
>>
>>> http://swtch.com/~rsc/regexp/regexp1.html
> 
> Both interesting links. I'll add another one to the list:
> http://google-opensource.blogspot.com/2010/03/re2-principled-approach-to-regular.html
> 
> To bring this on-topic for python-dev by considering how it could apply
> to Python's default re engine, I think the key issue is that any updates
> to the default engine would need to remain backwards compatible with all
> of the tricks that re2 doesn't support.
> 
> There are major practical problems associated with making such a leap
> directly (Google's re2 engine is in C++ rather than C and we'd have to
> keep the existing implementation around regardless to handle the
> features that re2 doesn't support).
> 
> I would say it is better to let re2 bake for a while and see if anyone
> is motivated to come up with Python bindings for it and release them on
> PyPI. Once that happens (and assuming the bindings earn a good
> reputation), the first step towards integration would be to include a
> "See Also" in the re module documentation to point people towards the
> more limited (but more robust) regex engine implementation.
> 
> The next step would probably be a hybrid third party library that
> exploits the NFA approach when it can, but resorts to backtracking when
> it has to in order to handle full regex functionality. (Although
> developers would need to be able to disable the backtracking support in
> order to restore re2's guarantees of linear time execution)
> 
> Only once such a hybrid library had been proven in practice could the
> default re module be considered for possible update.
> 
> Basically, I see some interesting ideas there, but I don't see anything
> that will impact the Python core or standard library in the short term.

Agreed on the whole.  However, it might be interesting to explore the
"hybrid mode" using some of the older, C-based libraries referenced by
the original (Russ Cox) article, e.g. Rob Pike's Unicode-aware 1992
implmentation, as ported for Unix:

  http://swtch.com/plan9port/unix/


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuac7AACgkQ+gerLs4ltQ6t3wCgxQPsnpbyE1N9BEGg7nHqcBVP
q04AnA6RMJw83+uc5u6Oyud89SXEoP5C
=+0aL
-----END PGP SIGNATURE-----


From brent.longborough at gmail.com  Fri Mar 12 18:42:13 2010
From: brent.longborough at gmail.com (Brent Longborough)
Date: Fri, 12 Mar 2010 17:42:13 +0000 (UTC)
Subject: [Python-Dev] interesting article on regex performance
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
Message-ID: <loom.20100312T183815-52@post.gmane.org>

I have some regex-intensive Python that might benefit from this, but I don't
think I have enough skill to do a set of Python bindings.

An ideal first cut would be to enable this:

      import re2 as re

I live in hope...
Brent L 


From arigo at tunes.org  Fri Mar 12 19:31:21 2010
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 12 Mar 2010 19:31:21 +0100
Subject: [Python-Dev] PyPy 1.2, JIT included
Message-ID: <20100312183121.GA17630@code0.codespeak.net>

==================================
PyPy 1.2: Just-in-Time Compilation
==================================

PyPy 1.2 has been released.  The highlight of this release is
to be the first that ships with a Just-in-Time compiler that is
known to be faster than CPython (and unladen swallow) on some
real-world applications (or the best benchmarks we could get for
them).  The main theme for the 1.2 release is speed.

Main site:

    http://pypy.org/

The JIT is stable and we don't observe crashes. Nevertheless we
would recommend you to treat it as beta software and as a way to try
out the JIT to see how it works for you.


Highlights of This Release
==========================

* The JIT compiler.

* Various interpreter optimizations that improve performance
  as well as help save memory.

* Introducing a new PyPy website at http://pypy.org/ , made by
  tav and improved by the PyPy team.

* Introducing http://speed.pypy.org/ , a new service that
  monitors our performance nightly, made by Miquel Torres.

* There will be ubuntu packages on "PyPy's PPA" made by
  Bartosz Skowron; however various troubles prevented us from
  having them as of now.


Known JIT problems (or why you should consider this beta software):

* The only supported platform is 32bit x86 for now, we're
  looking for help with other platforms.

* It is still memory-hungry.  There is no limit on the amount
  of RAM that the assembler can consume; it is thus possible
  (although unlikely) that the assembler ends up using
  unreasonable amounts of memory.


If you want to try PyPy, go to the "download page" on our excellent
new site at http://pypy.org/download.html and find the binary for
your platform. If the binary does not work (e.g. on Linux, because
of different versions of external .so dependencies), or if your
platform is not supported, you can try building from the source.


What is PyPy?
=============

Technically, PyPy is both a Python interpreter implementation and an
advanced compiler, or more precisely a framework for implementing
dynamic languages and generating virtual machines for them.  The
focus of this release is the introduction of a new transformation,
the JIT Compiler Generator, and its application to the Python
interpreter.

Socially, PyPy is a collaborative effort of many individuals working
together in a distributed and sprint-driven way since 2003.  PyPy
would not have gotten as far as it has without the coding, feedback
and general support from numerous people.


The PyPy release team,
    Armin Rigo, Maciej Fijalkowski and Amaury Forgeot d'Arc

Together with
    Antonio Cuni, Carl Friedrich Bolz, Holger Krekel and
    Samuele Pedroni

and many others:
    http://codespeak.net/pypy/dist/pypy/doc/contributor.html

From collinwinter at google.com  Fri Mar 12 20:08:44 2010
From: collinwinter at google.com (Collin Winter)
Date: Fri, 12 Mar 2010 11:08:44 -0800
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <4B9A67E7.30801@gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
Message-ID: <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>

On Fri, Mar 12, 2010 at 8:12 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
[snip]
> To bring this on-topic for python-dev by considering how it could apply
> to Python's default re engine, I think the key issue is that any updates
> to the default engine would need to remain backwards compatible with all
> of the tricks that re2 doesn't support.
>
> There are major practical problems associated with making such a leap
> directly (Google's re2 engine is in C++ rather than C and we'd have to
> keep the existing implementation around regardless to handle the
> features that re2 doesn't support).

I don't see why C++ would be a deal-breaker in this case, since it
would be restricted to an extension module.

> I would say it is better to let re2 bake for a while and see if anyone
> is motivated to come up with Python bindings for it and release them on
> PyPI.

FWIW, re2 is heavily, heavily used in production at Google.
Stabilizing any proposed Python bindings would be a good idea, but I'm
not sure how much more "baking" re2's core functionality needs.

> Once that happens (and assuming the bindings earn a good
> reputation), the first step towards integration would be to include a
> "See Also" in the re module documentation to point people towards the
> more limited (but more robust) regex engine implementation.
>
> The next step would probably be a hybrid third party library that
> exploits the NFA approach when it can, but resorts to backtracking when
> it has to in order to handle full regex functionality. (Although
> developers would need to be able to disable the backtracking support in
> order to restore re2's guarantees of linear time execution)

We considered such a hybrid approach for Unladen Swallow, but rejected
it on the advice of the V8 team [1]: you end up maintaining two
totally separate, incompatible regex engines; the hybrid system comes
with interesting, possibly unintuitive performance/correctness issues
when bailing from one implementation to another; performance is
unstable as small changes are made to the regexes; and it's not
obvious that enough Python regexes would benefit from re2's
performance/restrictions tradeoff to make such a hybrid system
worthwhile in the long term. (There is no representative corpus of
real-world Python regexes weighted for dynamic execution frequency to
use in assessing such tradeoffs empirically like there is for
JavaScript.)

re2 is very useful when you want to run user-provided regexes and want
to protect your backends against pathological/malicious regex input,
but I'm not sure how applicable it is to Python. I think there are
more promising strategies to improve regex performance, such as
reusing the new JIT infrastructure to JIT-compile regular expressions
to machine code (along the lines of V8's irregexp). Some work has
already been done in this direction, and I'd be thrilled to mentor any
GSoC students interested in working on such a project this summer.

Lastly, anyone interested in working on Python regex performance
should take a look at the regex benchmarks in the standard benchmark
suite [2].

Thanks,
Collin Winter

[1] - http://blog.chromium.org/2009/02/irregexp-google-chromes-new-regexp.html#c4843826268005492354
[2] - http://hg.python.org/benchmarks/file/5b8fe389710b/performance

From skip at pobox.com  Fri Mar 12 20:29:09 2010
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 12 Mar 2010 13:29:09 -0600
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
Message-ID: <19354.38405.67977.218983@montanaro.dyndns.org>


    >> There are major practical problems associated with making such a leap
    >> directly (Google's re2 engine is in C++ rather than C and we'd have
    >> to keep the existing implementation around regardless to handle the
    >> features that re2 doesn't support).

    Collin> I don't see why C++ would be a deal-breaker in this case, since
    Collin> it would be restricted to an extension module.

Traditionally Python has run on some (minority) platforms where C++ was
unavailable.  While the re module is a dynamically linked extension module
and thus could be considered "optional", I doubt anybody thinks of it as
optional nowadays.  It's used in the regression test suite anyway.  It would
be tough to run unit tests on such minority platforms without it.  You'd
have to maintain both the current sre implementation and the new re2
implementation for a long while into the future.

As I was reading the code I thought, "Great! This stuff is so simple.  It's
even all written in C."  Then I looked at the re2 page.  :-(

Skip

From barry at python.org  Fri Mar 12 20:53:12 2010
From: barry at python.org (Barry Warsaw)
Date: Fri, 12 Mar 2010 14:53:12 -0500
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B8FB6D2.4040501@gmail.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
Message-ID: <20100312145312.1f0a0aa0@heresy.wooz.org>

On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote:

>The remaining open question to my mind is whether or not there should be
>a -X option to control the bytecode generation. E.g.:
>
>-Xcache_bytecode=no (don't write bytecode files at all)

-B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use
case.

>-Xcache_bytecode=file (write a classic "foo.pyc" file)
>-Xcache_bytecode=dir (write to the "__pycache__" directory)
>
>With cache_bytecode=dir being the default for future releases.

I believe we've decided /not/ to support creation of bytecode-only
distributions out of the box.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100312/837d9b43/attachment.pgp>

From collinwinter at google.com  Fri Mar 12 22:40:50 2010
From: collinwinter at google.com (Collin Winter)
Date: Fri, 12 Mar 2010 13:40:50 -0800
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <19354.38405.67977.218983@montanaro.dyndns.org>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com> 
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> 
	<19354.38405.67977.218983@montanaro.dyndns.org>
Message-ID: <3c8293b61003121340i70e5cca7p602aa03d76d69855@mail.gmail.com>

On Fri, Mar 12, 2010 at 11:29 AM,  <skip at pobox.com> wrote:
>
> ? ?>> There are major practical problems associated with making such a leap
> ? ?>> directly (Google's re2 engine is in C++ rather than C and we'd have
> ? ?>> to keep the existing implementation around regardless to handle the
> ? ?>> features that re2 doesn't support).
>
> ? ?Collin> I don't see why C++ would be a deal-breaker in this case, since
> ? ?Collin> it would be restricted to an extension module.
>
> Traditionally Python has run on some (minority) platforms where C++ was
> unavailable. ?While the re module is a dynamically linked extension module
> and thus could be considered "optional", I doubt anybody thinks of it as
> optional nowadays. ?It's used in the regression test suite anyway. ?It would
> be tough to run unit tests on such minority platforms without it. ?You'd
> have to maintain both the current sre implementation and the new re2
> implementation for a long while into the future.

re2 is not a full replacement for Python's current regex semantics: it
would only serve as an accelerator for a subset of the current regex
language. Given that, it makes perfect sense that it would be optional
on such minority platforms (much like the incoming JIT).

Collin

From solipsis at pitrou.net  Fri Mar 12 23:03:08 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 12 Mar 2010 17:03:08 -0500
Subject: [Python-Dev] C++
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
Message-ID: <20100312170308.111b90a5@msiwind>

Le Fri, 12 Mar 2010 13:29:09 -0600,
skip at pobox.com a ?crit :
> 
> Traditionally Python has run on some (minority) platforms where C++
> was unavailable.

Is this concern still valid? We are in the 2010s now.
I'm not saying I want us to put some C++ in the core interpreter, but
the portability argument sounds a little old...

Regards

Antoine.



From barry at python.org  Fri Mar 12 23:15:20 2010
From: barry at python.org (Barry Warsaw)
Date: Fri, 12 Mar 2010 17:15:20 -0500
Subject: [Python-Dev] C++
In-Reply-To: <20100312170308.111b90a5@msiwind>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind>
Message-ID: <20100312171520.0409e36d@heresy.wooz.org>

On Mar 12, 2010, at 05:03 PM, Antoine Pitrou wrote:

>Le Fri, 12 Mar 2010 13:29:09 -0600,
>skip at pobox.com a ?crit :
>> 
>> Traditionally Python has run on some (minority) platforms where C++
>> was unavailable.
>
>Is this concern still valid?

Certainly not if Unladen Swallow blazes the trail.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100312/7c09af1e/attachment.pgp>

From nyamatongwe at gmail.com  Fri Mar 12 23:18:41 2010
From: nyamatongwe at gmail.com (Neil Hodgson)
Date: Sat, 13 Mar 2010 09:18:41 +1100
Subject: [Python-Dev] C++
In-Reply-To: <20100312170308.111b90a5@msiwind>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind>
Message-ID: <50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com>

Antoine Pitrou:

> Is this concern still valid? We are in the 2010s now.
> I'm not saying I want us to put some C++ in the core interpreter, but
> the portability argument sounds a little old...

   There are still viable platforms which only support subsets of C++.
IIRC, Android does not support exceptions in C++.

   Neil

From fuzzyman at voidspace.org.uk  Fri Mar 12 23:48:42 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 12 Mar 2010 22:48:42 +0000
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100312145312.1f0a0aa0@heresy.wooz.org>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>	<20100303204923.6d597f03@heresy.wooz.org>	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
Message-ID: <4B9AC4CA.3070302@voidspace.org.uk>

On 12/03/2010 19:53, Barry Warsaw wrote:
> On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote:
>
>    
>> The remaining open question to my mind is whether or not there should be
>> a -X option to control the bytecode generation. E.g.:
>>
>> -Xcache_bytecode=no (don't write bytecode files at all)
>>      
> -B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use
> case.
>
>    
>> -Xcache_bytecode=file (write a classic "foo.pyc" file)
>> -Xcache_bytecode=dir (write to the "__pycache__" directory)
>>
>> With cache_bytecode=dir being the default for future releases.
>>      
> I believe we've decided /not/ to support creation of bytecode-only
> distributions out of the box.
>    

I thought Guido said on this topic [1]:

" FWIW, I started at -1 and am still -1. I think the PEP is overreaching
in this aspect; it does not serve the stated purpose of the PEP to
make life easier for distros that share code between Python versions."

Has something changed since then?

Michael

[1] http://mail.python.org/pipermail/python-dev/2010-March/098098.html
> -Barry
>    
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100312/224656b3/attachment-0001.html>

From fuzzyman at voidspace.org.uk  Fri Mar 12 23:54:19 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 12 Mar 2010 22:54:19 +0000
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B9AC4CA.3070302@voidspace.org.uk>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>	<20100303204923.6d597f03@heresy.wooz.org>	<4B8FB6D2.4040501@gmail.com>	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
Message-ID: <4B9AC61B.6010007@voidspace.org.uk>

On 12/03/2010 22:48, Michael Foord wrote:
> On 12/03/2010 19:53, Barry Warsaw wrote:
>> On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote:
>>
>>    
>>> The remaining open question to my mind is whether or not there should be
>>> a -X option to control the bytecode generation. E.g.:
>>>
>>> -Xcache_bytecode=no (don't write bytecode files at all)
>>>      
>> -B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use
>> case.
>>
>>    
>>> -Xcache_bytecode=file (write a classic "foo.pyc" file)
>>> -Xcache_bytecode=dir (write to the "__pycache__" directory)
>>>
>>> With cache_bytecode=dir being the default for future releases.
>>>      
>> I believe we've decided /not/ to support creation of bytecode-only
>> distributions out of the box.
>>    
>
> I thought Guido said on this topic [1]:
>

Ok - so oops, I misunderstood. I'll go back into my corner now. :-)

All the best,

Michael

> " FWIW, I started at -1 and am still -1. I think the PEP is overreaching
> in this aspect; it does not serve the stated purpose of the PEP to
> make life easier for distros that share code between Python versions."
>
> Has something changed since then?
>
> Michael
>
> [1] http://mail.python.org/pipermail/python-dev/2010-March/098098.html
>> -Barry
>>    
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>>    
>
>
> -- 
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
> READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
>
>    
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100312/8ae4d111/attachment.html>

From barry at python.org  Fri Mar 12 23:56:57 2010
From: barry at python.org (Barry Warsaw)
Date: Fri, 12 Mar 2010 17:56:57 -0500
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B9AC4CA.3070302@voidspace.org.uk>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
Message-ID: <20100312175657.287bb456@heresy.wooz.org>

On Mar 12, 2010, at 10:48 PM, Michael Foord wrote:

>On 12/03/2010 19:53, Barry Warsaw wrote:

>> I believe we've decided /not/ to support creation of bytecode-only
>> distributions out of the box.
>>    
>
>I thought Guido said on this topic [1]:
>
>" FWIW, I started at -1 and am still -1. I think the PEP is overreaching
>in this aspect; it does not serve the stated purpose of the PEP to
>make life easier for distros that share code between Python versions."
>
>Has something changed since then?

Nope, sorry I should have been clearer.

*creation* is the key here.  As per BDFL pronouncement, we'll support reading
pyc-only modules just like we do today.  This is in PEP 3147.  We won't
support creating them though.

BTW, I'm actually starting to work on the implementation now, so if you want
to play along:

% bzr branch lp:~barry/python/pep3147

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100312/48ce9082/attachment.pgp>

From jyasskin at gmail.com  Fri Mar 12 23:59:16 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Fri, 12 Mar 2010 14:59:16 -0800
Subject: [Python-Dev] C++
In-Reply-To: <50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind>
	<50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com>
Message-ID: <5d44f72f1003121459u6b91f1dbwe0bc33c2011b3f6b@mail.gmail.com>

On Fri, Mar 12, 2010 at 2:18 PM, Neil Hodgson <nyamatongwe at gmail.com> wrote:
> Antoine Pitrou:
>
>> Is this concern still valid? We are in the 2010s now.
>> I'm not saying I want us to put some C++ in the core interpreter, but
>> the portability argument sounds a little old...
>
> ? There are still viable platforms which only support subsets of C++.
> IIRC, Android does not support exceptions in C++.

Looks like they'll be getting exceptions "soon":
http://groups.google.com/group/android-ndk/browse_thread/thread/89db67ed1fbf6450/4de3dd6105eb26ce?#4de3dd6105eb26ce

But yeah, thanks for the concrete example, and I'd agree that Python
should compile with -fno-exceptions, for a couple more years at least.

From g.brandl at gmx.net  Sat Mar 13 00:23:11 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 13 Mar 2010 00:23:11 +0100
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <19354.38405.67977.218983@montanaro.dyndns.org>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>	<4B9A67E7.30801@gmail.com>	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
Message-ID: <hneifi$qcp$1@dough.gmane.org>

Am 12.03.2010 20:29, schrieb skip at pobox.com:
> 
>     >> There are major practical problems associated with making such a leap
>     >> directly (Google's re2 engine is in C++ rather than C and we'd have
>     >> to keep the existing implementation around regardless to handle the
>     >> features that re2 doesn't support).
> 
>     Collin> I don't see why C++ would be a deal-breaker in this case, since
>     Collin> it would be restricted to an extension module.
> 
> Traditionally Python has run on some (minority) platforms where C++ was
> unavailable.  While the re module is a dynamically linked extension module
> and thus could be considered "optional", I doubt anybody thinks of it as
> optional nowadays.

Actually, _sre is a builtin module by default since at least Python 2.5.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From skip at pobox.com  Sat Mar 13 00:22:13 2010
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 12 Mar 2010 17:22:13 -0600
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <3c8293b61003121340i70e5cca7p602aa03d76d69855@mail.gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<3c8293b61003121340i70e5cca7p602aa03d76d69855@mail.gmail.com>
Message-ID: <19354.52389.642682.376209@montanaro.dyndns.org>


    Collin> re2 is not a full replacement for Python's current regex
    Collin> semantics: it would only serve as an accelerator for a subset of
    Collin> the current regex language. Given that, it makes perfect sense
    Collin> that it would be optional on such minority platforms (much like
    Collin> the incoming JIT).

Sure, but over the years Python has supported at least four different
regular expression modules that I'm aware of (regex, regexp, and the current
re module with different extension modules underneath it, perhaps there were
others).  During some of that time more than one module was distributed with
Python proper.  I think the desire today would be that only one regular
expression module be distributed with Python (that would be my vote anyway).
Getting people to move off the older libraries was difficult.  If re2 can't
replace sre under the covers than I think it belongs in PyPI, not the Python
distribution.  That said, that suggests to me that a different NFA or DFA
implementation written in C would replace sre, one not written in C++.

Hopefully that provides some context for my earlier response.

Skip

From jared.grubb at gmail.com  Sat Mar 13 02:59:12 2010
From: jared.grubb at gmail.com (Jared Grubb)
Date: Fri, 12 Mar 2010 17:59:12 -0800
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <19354.52389.642682.376209@montanaro.dyndns.org>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<3c8293b61003121340i70e5cca7p602aa03d76d69855@mail.gmail.com>
	<19354.52389.642682.376209@montanaro.dyndns.org>
Message-ID: <AA1292D1-97ED-4ED4-8484-2E9498B9AB31@gmail.com>

On 12 Mar 2010, at 15:22, skip at pobox.com wrote:
> 
>  Collin> re2 is not a full replacement for Python's current regex
>  Collin> semantics: it would only serve as an accelerator for a subset of
>  Collin> the current regex language. Given that, it makes perfect sense
>  Collin> that it would be optional on such minority platforms (much like
>  Collin> the incoming JIT).
> 
> Sure, but over the years Python has supported at least four different
> regular expression modules that I'm aware of (regex, regexp, and the current
> re module with different extension modules underneath it, perhaps there were
> others).  During some of that time more than one module was distributed with
> Python proper.  I think the desire today would be that only one regular
> expression module be distributed with Python (that would be my vote anyway).
> Getting people to move off the older libraries was difficult.  If re2 can't
> replace sre under the covers than I think it belongs in PyPI, not the Python
> distribution.  That said, that suggests to me that a different NFA or DFA
> implementation written in C would replace sre, one not written in C++.

re2 would be a supplement to re -- it is not a replacement, and Python would run fine if it's not present on some platforms. 

It's like a floating-point processor: you can do all math you need with just an integer processor. But if you have an FPU present, then it makes sense to use it for the FP operations. 

Jared

From meadori at gmail.com  Sat Mar 13 04:17:05 2010
From: meadori at gmail.com (Meador Inge)
Date: Fri, 12 Mar 2010 21:17:05 -0600
Subject: [Python-Dev] C++
In-Reply-To: <20100312170308.111b90a5@msiwind>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind>
Message-ID: <4095897c1003121917m36e9cd12je0965c7c3a99c615@mail.gmail.com>

On Fri, Mar 12, 2010 at 4:03 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Fri, 12 Mar 2010 13:29:09 -0600,
> skip at pobox.com a ?crit :
> >
> > Traditionally Python has run on some (minority) platforms where C++
> > was unavailable.
>
> Is this concern still valid? We are in the 2010s now.
> I'm not saying I want us to put some C++ in the core interpreter, but
> the portability argument sounds a little old...
>

One area where this _may_ be a problem is with embedded systems.  I believe
there are some instances where folks have built Python into an embedded
system (with an RTOS say VxWorks, Symbian, QNX Neutrino, Nucleus, etc...)
where C++ is not always the easiest to develop in.  Admittedly, though,
these types of systems are by far a minority with respect to Python.  Just
thought I would mention it anyhow.

-- Meador
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100312/ba090f23/attachment.html>

From skip at pobox.com  Sat Mar 13 04:54:35 2010
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 12 Mar 2010 21:54:35 -0600
Subject: [Python-Dev] C++
In-Reply-To: <20100312170308.111b90a5@msiwind>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind>
Message-ID: <19355.3195.511985.327292@montanaro.dyndns.org>

    Antoine> skip at pobox.com a ??crit :
    >> 
    >> Traditionally Python has run on some (minority) platforms where C++
    >> was unavailable.

    Antoine> Is this concern still valid? We are in the 2010s now.

Like I said, *minority* platforms.  Here are some which come to mind as
quite possibly not supporting C++ well, if at all, yet all have some dialect
of Python available:

    Windows CE: http://sourceforge.net/projects/pythonce/
    Palm: http://pippy.sourceforge.net/
    iPod: http://ciberjacobo.com/python-ipod
    OS/2: http://www.andymac.org/
    QNX: http://sourceforge.net/projects/pyqnx/

If all you care about are mainstream Windows, Unix/Linux and Mac OSX
environments then C++ is no problem.

Skip


From jyasskin at gmail.com  Sat Mar 13 07:54:59 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Fri, 12 Mar 2010 22:54:59 -0800
Subject: [Python-Dev] C++
In-Reply-To: <19355.3195.511985.327292@montanaro.dyndns.org>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind>
	<19355.3195.511985.327292@montanaro.dyndns.org>
Message-ID: <5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com>

On Fri, Mar 12, 2010 at 7:54 PM,  <skip at pobox.com> wrote:
> ? ?Antoine> skip at pobox.com a ??crit :
> ? ?>>
> ? ?>> Traditionally Python has run on some (minority) platforms where C++
> ? ?>> was unavailable.
>
> ? ?Antoine> Is this concern still valid? We are in the 2010s now.
>
> Like I said, *minority* platforms. ?Here are some which come to mind as
> quite possibly not supporting C++ well, if at all, yet all have some dialect
> of Python available:
>
> ? ?Windows CE: http://sourceforge.net/projects/pythonce/

http://www.wirelessdevnet.com/channels/pda/training/vcce.html

> ? ?Palm: http://pippy.sourceforge.net/

http://prc-tools.sourceforge.net/

> ? ?iPod: http://ciberjacobo.com/python-ipod

http://ipodlinux.org/wiki/Toolchain

> ? ?OS/2: http://www.andymac.org/

I can't find a modern c++ compiler for OS/2. Then again, your link
only provides python 2.4.

> ? ?QNX: http://sourceforge.net/projects/pyqnx/

http://www.qnx.com/products/tools/

> If all you care about are mainstream Windows, Unix/Linux and Mac OSX
> environments then C++ is no problem.

If you care about any of these minority platforms except OS/2, C++ is
also available, as a google search for "<platform name> C++" quickly
finds. OS/2 might be available too; I just didn't look for very long.
If you know of platforms that don't support particular features of
C++, please link to documentation of that like Neil Hodgson did. If
not, please stop spreading FUD.

From g.brandl at gmx.net  Sat Mar 13 14:04:08 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 13 Mar 2010 14:04:08 +0100
Subject: [Python-Dev] Desired changes to Hg emails to python-checkins
In-Reply-To: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
References: <bbaeab101003041220v5db3dd32o8bad6705fda91ae@mail.gmail.com>
Message-ID: <hng2ir$ff8$1@dough.gmane.org>

Am 04.03.2010 21:20, schrieb Brett Cannon:
> 1) I miss not having the affected files listed in the subject line.
> 
> 2) The To field is set to hg at python.org <mailto:hg at python.org> which
> gets rejected as an invalid email address if you reply. Would be better
> to set it to python-checkins so that the habitual reply to a checkin
> won't get rejected.

Before I forget:  for CPython, if all branches are to be in one hg repo
with named branches, the branch name also needs to be in the subject line.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ncoghlan at gmail.com  Sat Mar 13 14:59:57 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 13 Mar 2010 23:59:57 +1000
Subject: [Python-Dev] interesting article on regex performance
In-Reply-To: <3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
Message-ID: <4B9B9A5D.5090800@gmail.com>

Collin Winter wrote:
> On Fri, Mar 12, 2010 at 8:12 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> There are major practical problems associated with making such a leap
>> directly (Google's re2 engine is in C++ rather than C and we'd have to
>> keep the existing implementation around regardless to handle the
>> features that re2 doesn't support).
> 
> I don't see why C++ would be a deal-breaker in this case, since it
> would be restricted to an extension module.

It wouldn't be a dealbreaker for an optional regex acceleration module,
I agree. It's still a practical problem in the sense that C++ code is
very much in the minority in the CPython code base.

>> I would say it is better to let re2 bake for a while and see if anyone
>> is motivated to come up with Python bindings for it and release them on
>> PyPI.
> 
> FWIW, re2 is heavily, heavily used in production at Google.
> Stabilizing any proposed Python bindings would be a good idea, but I'm
> not sure how much more "baking" re2's core functionality needs.

It was more the bindings side of things I was referring to - the
algorithm's performance is no doubt well established, but how easy it is
for non-Google users to pick up and integrate with other software is
still to be determined.

> We considered such a hybrid approach for Unladen Swallow, but rejected
> it on the advice of the V8 team [1]: you end up maintaining two
> totally separate, incompatible regex engines; the hybrid system comes
> with interesting, possibly unintuitive performance/correctness issues
> when bailing from one implementation to another; performance is
> unstable as small changes are made to the regexes; and it's not
> obvious that enough Python regexes would benefit from re2's
> performance/restrictions tradeoff to make such a hybrid system
> worthwhile in the long term. (There is no representative corpus of
> real-world Python regexes weighted for dynamic execution frequency to
> use in assessing such tradeoffs empirically like there is for
> JavaScript.)
> 
> re2 is very useful when you want to run user-provided regexes and want
> to protect your backends against pathological/malicious regex input,
> but I'm not sure how applicable it is to Python. I think there are
> more promising strategies to improve regex performance, such as
> reusing the new JIT infrastructure to JIT-compile regular expressions
> to machine code (along the lines of V8's irregexp). Some work has
> already been done in this direction, and I'd be thrilled to mentor any
> GSoC students interested in working on such a project this summer.
> 
> Lastly, anyone interested in working on Python regex performance
> should take a look at the regex benchmarks in the standard benchmark
> suite [2].

This is the kind of baking that I was really talking about - there's an
awful lot of investigation to be done before moving Python in the
direction of an NFA based regex engine could be seriously considered.

Good to hear the perspective of someone that has already looked into
this in some detail though.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ndbecker2 at gmail.com  Sat Mar 13 15:00:33 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sat, 13 Mar 2010 09:00:33 -0500
Subject: [Python-Dev] C++
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind>
	<50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com>
Message-ID: <hng5q2$npo$3@dough.gmane.org>

Neil Hodgson wrote:

> Antoine Pitrou:
> 
>> Is this concern still valid? We are in the 2010s now.
>> I'm not saying I want us to put some C++ in the core interpreter, but
>> the portability argument sounds a little old...
> 
>    There are still viable platforms which only support subsets of C++.
> IIRC, Android does not support exceptions in C++.
> 
>    Neil

Does re2 require exceptions?


From ncoghlan at gmail.com  Sat Mar 13 15:17:09 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Mar 2010 00:17:09 +1000
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100312175657.287bb456@heresy.wooz.org>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>	<20100303204923.6d597f03@heresy.wooz.org>	<4B8FB6D2.4040501@gmail.com>	<20100312145312.1f0a0aa0@heresy.wooz.org>	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
Message-ID: <4B9B9E65.4080902@gmail.com>

Barry Warsaw wrote:
> *creation* is the key here.  As per BDFL pronouncement, we'll support reading
> pyc-only modules just like we do today.  This is in PEP 3147.  We won't
> support creating them though.

The creation side could be made a little more explicit in the PEP. We
could also do something via the compileall module.

(Pause while Nick goes and reads the source code for compileall for the
first time ever...)

Hmm - methinks the PEP actually needs to talk explicitly about the
py_compile and compileall modules. These compile the files directly
rather than using the import system's side-effect, so they'll need to
understand the intricacies of the new system.

While it's probably OK if the import side-effects only create files
using the new scheme, the standard library modules will likely need to
support both schemes (although I'm not sure if "same as import system"
or "same as Python 3.1" make more sense as the default semantics -
probably the former).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From skip at pobox.com  Sat Mar 13 18:07:49 2010
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 13 Mar 2010 11:07:49 -0600
Subject: [Python-Dev] C++
In-Reply-To: <5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com>
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind>
	<19355.3195.511985.327292@montanaro.dyndns.org>
	<5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com>
Message-ID: <19355.50789.388095.93289@montanaro.dyndns.org>


    Jeffrey> If you know of platforms that don't support particular features
    Jeffrey> of C++, please link to documentation of that like Neil Hodgson
    Jeffrey> did. If not, please stop spreading FUD.

No need to get your knickers in a twist.  Every couple years someone asks,
"why isn't Python written in C++?"  The lack of good C++ support for many
(minority) platforms is often the primary explanation.  If that's changing,
I have no problem with it.

I spent a fair amount of time with Google trying to find representative
threads here or in comp.lang.python but failed to find any needles in the
haystack.  Perhaps you will have better luck.

Skip

From dudek at cim.mcgill.ca  Sat Mar 13 18:24:27 2010
From: dudek at cim.mcgill.ca (gregory dudek)
Date: Sat, 13 Mar 2010 12:24:27 -0500
Subject: [Python-Dev] Patch to telnetlib.py
Message-ID: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>

The Telnet module telnetlib.py can be
very slow -- unusably slow -- for large automated data transfers.  There are typically done in raw mode.

The attached patch greatly increased the speed of telnet interactions in raw mode.  I submitted this a couple of year ago, but it was for an older branch of python.

There are 2 key things being done:
 1) concatenations string with string.join instead of '+'  (which is probably a minor issue)
 2) wholesale appending the raw and processed buffers when the IAC character is not found.  The should be examined
     carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup.


--- installed/telnetlib.py	2010-02-02 22:57:58.000000000 -0500
+++ telnetlib.py	2010-03-13 12:17:02.000000000 -0500
@@ -30,6 +30,7 @@
 - timeout should be intrinsic to the connection object instead of an
   option on one of the read calls only
 
+Modified by G. Dudek for greater efficiency.
 """
 
 
@@ -420,6 +421,14 @@
         """
         buf = ['', '']
         try:
+            if self.rawq:
+                if not IAC in self.rawq:
+                    # speed hack, no IAC just grab whole queue. --Dudek
+                    buf[self.sb] = "".join((buf[self.sb] , self.rawq))
+                    self.cookedq = "".join((self.cookedq , buf[0] ))
+                    self.sbdataq = "".join((self.sbdataq , buf[1] ))
+                    self.rawq_flush()
+                    return
             while self.rawq:
                 c = self.rawq_getchar()
                 if not self.iacseq:
@@ -428,7 +437,7 @@
                     if c == "\021":
                         continue
                     if c != IAC:
-                        buf[self.sb] = buf[self.sb] + c
+                        buf[self.sb] = "".join((buf[self.sb] , c))
                         continue
                     else:
                         self.iacseq += c
@@ -480,8 +489,14 @@
             self.iacseq = '' # Reset on EOF
             self.sb = 0
             pass
-        self.cookedq = self.cookedq + buf[0]
-        self.sbdataq = self.sbdataq + buf[1]
+        self.cookedq = "".join((self.cookedq , buf[0] ))
+        self.sbdataq = "".join((self.sbdataq , buf[1] ))
+
+
+    def rawq_flush(self):
+        """ Set the queue to empty status """
+        self.rawq = ''
+        self.irawq = 0
 
     def rawq_getchar(self):
         """Get next char from raw queue.
@@ -516,7 +531,7 @@
         buf = self.sock.recv(50)
         self.msg("recv %r", buf)
         self.eof = (not buf)
-        self.rawq = self.rawq + buf
+        self.rawq = "".join((self.rawq,buf))
 
     def sock_avail(self):
         """Test whether data is available on the socket."""


From tjreedy at udel.edu  Sat Mar 13 19:22:56 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 13 Mar 2010 13:22:56 -0500
Subject: [Python-Dev] Patch to telnetlib.py
In-Reply-To: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>
References: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>
Message-ID: <hngl5v$53e$1@dough.gmane.org>

On 3/13/2010 12:24 PM, gregory dudek wrote:
> The Telnet module telnetlib.py can be
> very slow -- unusably slow -- for large automated data transfers.  There are typically done in raw mode.
>
> The attached patch

Please submit to the tracker. If there is an existing issue, attach it 
to that.


From eric at trueblade.com  Sat Mar 13 19:23:16 2010
From: eric at trueblade.com (Eric Smith)
Date: Sat, 13 Mar 2010 13:23:16 -0500
Subject: [Python-Dev] Patch to telnetlib.py
In-Reply-To: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>
References: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>
Message-ID: <4B9BD814.50806@trueblade.com>

Can you create an issue on the bug tracker? Otherwise this will get lost.

Eric.

On 3/13/2010 12:24 PM, gregory dudek wrote:
> The Telnet module telnetlib.py can be
> very slow -- unusably slow -- for large automated data transfers.  There are typically done in raw mode.
>
> The attached patch greatly increased the speed of telnet interactions in raw mode.  I submitted this a couple of year ago, but it was for an older branch of python.
>
> There are 2 key things being done:
>   1) concatenations string with string.join instead of '+'  (which is probably a minor issue)
>   2) wholesale appending the raw and processed buffers when the IAC character is not found.  The should be examined
>       carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup.
>
>
> --- installed/telnetlib.py	2010-02-02 22:57:58.000000000 -0500
> +++ telnetlib.py	2010-03-13 12:17:02.000000000 -0500
> @@ -30,6 +30,7 @@
>   - timeout should be intrinsic to the connection object instead of an
>     option on one of the read calls only
>
> +Modified by G. Dudek for greater efficiency.
>   """
>
>
> @@ -420,6 +421,14 @@
>           """
>           buf = ['', '']
>           try:
> +            if self.rawq:
> +                if not IAC in self.rawq:
> +                    # speed hack, no IAC just grab whole queue. --Dudek
> +                    buf[self.sb] = "".join((buf[self.sb] , self.rawq))
> +                    self.cookedq = "".join((self.cookedq , buf[0] ))
> +                    self.sbdataq = "".join((self.sbdataq , buf[1] ))
> +                    self.rawq_flush()
> +                    return
>               while self.rawq:
>                   c = self.rawq_getchar()
>                   if not self.iacseq:
> @@ -428,7 +437,7 @@
>                       if c == "\021":
>                           continue
>                       if c != IAC:
> -                        buf[self.sb] = buf[self.sb] + c
> +                        buf[self.sb] = "".join((buf[self.sb] , c))
>                           continue
>                       else:
>                           self.iacseq += c
> @@ -480,8 +489,14 @@
>               self.iacseq = '' # Reset on EOF
>               self.sb = 0
>               pass
> -        self.cookedq = self.cookedq + buf[0]
> -        self.sbdataq = self.sbdataq + buf[1]
> +        self.cookedq = "".join((self.cookedq , buf[0] ))
> +        self.sbdataq = "".join((self.sbdataq , buf[1] ))
> +
> +
> +    def rawq_flush(self):
> +        """ Set the queue to empty status """
> +        self.rawq = ''
> +        self.irawq = 0
>
>       def rawq_getchar(self):
>           """Get next char from raw queue.
> @@ -516,7 +531,7 @@
>           buf = self.sock.recv(50)
>           self.msg("recv %r", buf)
>           self.eof = (not buf)
> -        self.rawq = self.rawq + buf
> +        self.rawq = "".join((self.rawq,buf))
>
>       def sock_avail(self):
>           """Test whether data is available on the socket."""
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com
>


From guido at python.org  Sat Mar 13 19:22:59 2010
From: guido at python.org (Guido van Rossum)
Date: Sat, 13 Mar 2010 10:22:59 -0800
Subject: [Python-Dev] Patch to telnetlib.py
In-Reply-To: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>
References: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>
Message-ID: <ca471dc21003131022x2b7b39bcx2dfc8eebb8df85a5@mail.gmail.com>

On Sat, Mar 13, 2010 at 9:24 AM, gregory dudek <dudek at cim.mcgill.ca> wrote:
> The Telnet module telnetlib.py can be
> very slow -- unusably slow -- for large automated data transfers. ?There are typically done in raw mode.
>
> The attached patch greatly increased the speed of telnet interactions in raw mode. ?I submitted this a couple of year ago, but it was for an older branch of python.

For which Python version is your new patch?

> There are 2 key things being done:
> ?1) concatenations string with string.join instead of '+' ?(which is probably a minor issue)

This looks like superstition in the patch below -- there is no reason
why "".join((a, b)) should be any faster than a+b. (The join trick is
important when joining *many* strings, but for two it doesn't make a
difference, both strings have to be copied at least once no matter
what you try.)

> ?2) wholesale appending the raw and processed buffers when the IAC character is not found. ?The should be examined
> ? ? carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup.

This makes sense and is likely the reason your patch is faster.

You should really submit this to the bug tracker at bugs.python.org.

> --- installed/telnetlib.py ? ? ?2010-02-02 22:57:58.000000000 -0500
> +++ telnetlib.py ? ? ? ?2010-03-13 12:17:02.000000000 -0500
> @@ -30,6 +30,7 @@
> ?- timeout should be intrinsic to the connection object instead of an
> ? option on one of the read calls only
>
> +Modified by G. Dudek for greater efficiency.

You're gonna need to submit a Python contributor agreement
(http://www.python.org/psf/contrib/contrib-form/) if/when your patch
gets accepted. Welcome to the club!

> ?"""
>
>
> @@ -420,6 +421,14 @@
> ? ? ? ? """
> ? ? ? ? buf = ['', '']
> ? ? ? ? try:
> + ? ? ? ? ? ?if self.rawq:
> + ? ? ? ? ? ? ? ?if not IAC in self.rawq:
> + ? ? ? ? ? ? ? ? ? ?# speed hack, no IAC just grab whole queue. --Dudek
> + ? ? ? ? ? ? ? ? ? ?buf[self.sb] = "".join((buf[self.sb] , self.rawq))
> + ? ? ? ? ? ? ? ? ? ?self.cookedq = "".join((self.cookedq , buf[0] ))
> + ? ? ? ? ? ? ? ? ? ?self.sbdataq = "".join((self.sbdataq , buf[1] ))
> + ? ? ? ? ? ? ? ? ? ?self.rawq_flush()
> + ? ? ? ? ? ? ? ? ? ?return
> ? ? ? ? ? ? while self.rawq:
> ? ? ? ? ? ? ? ? c = self.rawq_getchar()
> ? ? ? ? ? ? ? ? if not self.iacseq:
> @@ -428,7 +437,7 @@
> ? ? ? ? ? ? ? ? ? ? if c == "\021":
> ? ? ? ? ? ? ? ? ? ? ? ? continue
> ? ? ? ? ? ? ? ? ? ? if c != IAC:
> - ? ? ? ? ? ? ? ? ? ? ? ?buf[self.sb] = buf[self.sb] + c
> + ? ? ? ? ? ? ? ? ? ? ? ?buf[self.sb] = "".join((buf[self.sb] , c))
> ? ? ? ? ? ? ? ? ? ? ? ? continue
> ? ? ? ? ? ? ? ? ? ? else:
> ? ? ? ? ? ? ? ? ? ? ? ? self.iacseq += c
> @@ -480,8 +489,14 @@
> ? ? ? ? ? ? self.iacseq = '' # Reset on EOF
> ? ? ? ? ? ? self.sb = 0
> ? ? ? ? ? ? pass
> - ? ? ? ?self.cookedq = self.cookedq + buf[0]
> - ? ? ? ?self.sbdataq = self.sbdataq + buf[1]
> + ? ? ? ?self.cookedq = "".join((self.cookedq , buf[0] ))
> + ? ? ? ?self.sbdataq = "".join((self.sbdataq , buf[1] ))
> +
> +
> + ? ?def rawq_flush(self):
> + ? ? ? ?""" Set the queue to empty status """
> + ? ? ? ?self.rawq = ''
> + ? ? ? ?self.irawq = 0
>
> ? ? def rawq_getchar(self):
> ? ? ? ? """Get next char from raw queue.
> @@ -516,7 +531,7 @@
> ? ? ? ? buf = self.sock.recv(50)
> ? ? ? ? self.msg("recv %r", buf)
> ? ? ? ? self.eof = (not buf)
> - ? ? ? ?self.rawq = self.rawq + buf
> + ? ? ? ?self.rawq = "".join((self.rawq,buf))
>
> ? ? def sock_avail(self):
> ? ? ? ? """Test whether data is available on the socket."""
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From greg at krypto.org  Sat Mar 13 19:42:22 2010
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 13 Mar 2010 10:42:22 -0800
Subject: [Python-Dev] C++
In-Reply-To: <hng5q2$npo$3@dough.gmane.org>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>
	<4B9A67E7.30801@gmail.com> 
	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com> 
	<19354.38405.67977.218983@montanaro.dyndns.org>
	<20100312170308.111b90a5@msiwind> 
	<50862ebd1003121418s5e7dd608o31185b42d3fdb468@mail.gmail.com> 
	<hng5q2$npo$3@dough.gmane.org>
Message-ID: <52dc1c821003131042i4e3ec141g84155a1c4a85a855@mail.gmail.com>

On Sat, Mar 13, 2010 at 6:00 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> Neil Hodgson wrote:
>
>> Antoine Pitrou:
>>
>>> Is this concern still valid? We are in the 2010s now.
>>> I'm not saying I want us to put some C++ in the core interpreter, but
>>> the portability argument sounds a little old...
>>
>> ? ?There are still viable platforms which only support subsets of C++.
>> IIRC, Android does not support exceptions in C++.
>>
>> ? ?Neil
>
> Does re2 require exceptions?

Nope.  No exceptions.

It was written following this style guide:
http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Exceptions

From jackdied at gmail.com  Sat Mar 13 19:47:58 2010
From: jackdied at gmail.com (Jack Diederich)
Date: Sat, 13 Mar 2010 13:47:58 -0500
Subject: [Python-Dev] Patch to telnetlib.py
In-Reply-To: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>
References: <637D2792-9FD7-478A-935C-1BF6A392F7AD@cim.mcgill.ca>
Message-ID: <b8e622741003131047p15414a75v93f9b596a2ed9845@mail.gmail.com>

On Sat, Mar 13, 2010 at 12:24 PM, gregory dudek <dudek at cim.mcgill.ca> wrote:
> The Telnet module telnetlib.py can be
> very slow -- unusably slow -- for large automated data transfers. ?There are typically done in raw mode.
>
> The attached patch greatly increased the speed of telnet interactions in raw mode. ?I submitted this a couple of year ago, but it was for an older branch of python.
>
> There are 2 key things being done:
> ?1) concatenations string with string.join instead of '+' ?(which is probably a minor issue)
> ?2) wholesale appending the raw and processed buffers when the IAC character is not found. ?The should be examined
> ? ? carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup.

As others mentioned, please post the bug to the tracker.  Also, please
assign the patch to me ("jackdied") and mention the previous bug
number - I thought I had reviewed all the telnetlib bugs including
those that were closed WONTFIX.

Thanks for the kick in the pants, I have a whole new inner loop for
data processing but I haven't applied it.  I've been adding unit tests
to the module so I could be sure I wouldn't break anything but never
finished the job.

-Jack

From solipsis at pitrou.net  Sat Mar 13 22:46:51 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 13 Mar 2010 16:46:51 -0500
Subject: [Python-Dev] "Fixing" the new GIL
Message-ID: <20100313164651.7eb322b1@msiwind>


Hello,

As some of you may know, Dave Beazley recently exhibited a situation
where the new GIL shows quite a poor behaviour (the old GIL isn't very
good either, but still a little better). This issue is followed in
http://bugs.python.org/issue7946

This situation is when an IO-bound thread wants to process a lot of
incoming packets, while one (or several) CPU-bound thread is also
running. Each time the IO-bound thread releases the GIL, the CPU-bound
thread gets it and keeps holding it for at least 5 milliseconds
(default setting), which limits the number of individual packets which
can be recv()'ed and processed per second.

I have proposed two mechanisms, based on the same idea: IO-bound
threads should be able to steal the GIL very quickly, rather than
having to wait for the whole "thread switching interval" (again, 5 ms
by default). They differ in how they detect an "IO-bound threads":

- the first mechanism is actually the same mechanism which was
  embodied in the original new GIL patch before being removed. In this
  approach, IO methods (such as socket.read() in socketmodule.c)
  releasing the GIL must use a separate C macro when trying to get the
  GIL back again.

- the second mechanism dynamically computes the "interactiveness" of a
  thread and allows interactive threads to steal the GIL quickly. In
  this approach, IO methods don't have to be modified at all.

Both approaches show similar benchmark results (for the benchmarks
that I know of) and basically fix the issue put forward by Dave Beazley.

Any thoughts?

Regards

Antoine.



From benjamin at python.org  Sat Mar 13 22:52:03 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 13 Mar 2010 15:52:03 -0600
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100313164651.7eb322b1@msiwind>
References: <20100313164651.7eb322b1@msiwind>
Message-ID: <1afaf6161003131352n3a7bd423w75a8a36ce9ba7a70@mail.gmail.com>

2010/3/13 Antoine Pitrou <solipsis at pitrou.net>:
> Any thoughts?
\
The latter solution seems best to me as it would help any 3rd party IO
libraries and require less code modification.



-- 
Regards,
Benjamin

From martin at v.loewis.de  Sat Mar 13 23:03:07 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 13 Mar 2010 23:03:07 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100313164651.7eb322b1@msiwind>
References: <20100313164651.7eb322b1@msiwind>
Message-ID: <4B9C0B9B.3080805@v.loewis.de>

> Any thoughts?

My initial reaction to this report, and still my current opinion is:

This issue is not a problem.

It's a boundary case, so it's not clear whether Python should be able to
deal with it gracefully. I doubt it's a realistic case.

Regards,
Martin

From daniel at stutzbachenterprises.com  Sat Mar 13 23:18:23 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Sat, 13 Mar 2010 16:18:23 -0600
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100313164651.7eb322b1@msiwind>
References: <20100313164651.7eb322b1@msiwind>
Message-ID: <eae285401003131418t4eb5cc0ao57e390ed37991e61@mail.gmail.com>

On Sat, Mar 13, 2010 at 3:46 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> - the second mechanism dynamically computes the "interactiveness" of a
>  thread and allows interactive threads to steal the GIL quickly. In
>  this approach, IO methods don't have to be modified at all.
>

I like the second approach as well, assuming "interactiveness" can be
computed cheaply.
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100313/b57f2779/attachment.html>

From skip at pobox.com  Sun Mar 14 00:11:41 2010
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 13 Mar 2010 17:11:41 -0600
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100313164651.7eb322b1@msiwind>
References: <20100313164651.7eb322b1@msiwind>
Message-ID: <19356.7085.340102.469770@montanaro.dyndns.org>


If the 5ms interval is too long would it be possible to adaptively reduce it
in this situation?  How would you detect it?  I assume this would be akin to
your interactiveness computation.

Skip


From solipsis at pitrou.net  Sun Mar 14 00:23:00 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 13 Mar 2010 18:23:00 -0500
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <19356.7085.340102.469770@montanaro.dyndns.org>
References: <20100313164651.7eb322b1@msiwind>
	<19356.7085.340102.469770@montanaro.dyndns.org>
Message-ID: <20100313182300.4f5376a0@msiwind>

Le Sat, 13 Mar 2010 17:11:41 -0600,
skip at pobox.com a ?crit :
> 
> If the 5ms interval is too long would it be possible to adaptively
> reduce it in this situation?  How would you detect it?  I assume this
> would be akin to your interactiveness computation.

I think modulating the interval is what Dave Beazley's own patch
proposal does. It has a full-blown priority system. It is more
complicated than either of the two approaches I am more proposing,
though.
(also, per Dave's own admission, his patch is a proof-of-concept rather
than a finished piece of work)

From brett at python.org  Sun Mar 14 01:53:00 2010
From: brett at python.org (Brett Cannon)
Date: Sat, 13 Mar 2010 16:53:00 -0800
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <1afaf6161003131352n3a7bd423w75a8a36ce9ba7a70@mail.gmail.com>
References: <20100313164651.7eb322b1@msiwind>
	<1afaf6161003131352n3a7bd423w75a8a36ce9ba7a70@mail.gmail.com>
Message-ID: <bbaeab101003131653s3bd641b8qe959a9c7840f4b0c@mail.gmail.com>

On Sat, Mar 13, 2010 at 13:52, Benjamin Peterson <benjamin at python.org>wrote:

> 2010/3/13 Antoine Pitrou <solipsis at pitrou.net>:
> > Any thoughts?
> \
> The latter solution seems best to me as it would help any 3rd party IO
> libraries and require less code modification.
>

Plus the interactiveness approach has been tested by OS thread schedulers
for years and is shown to work. So I vote for the latter approach as well.

-Brett



>
>
>
> --
> Regards,
> Benjamin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100313/d23c0bd3/attachment.html>

From ncoghlan at gmail.com  Sun Mar 14 04:32:51 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Mar 2010 13:32:51 +1000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <bbaeab101003131653s3bd641b8qe959a9c7840f4b0c@mail.gmail.com>
References: <20100313164651.7eb322b1@msiwind>	<1afaf6161003131352n3a7bd423w75a8a36ce9ba7a70@mail.gmail.com>
	<bbaeab101003131653s3bd641b8qe959a9c7840f4b0c@mail.gmail.com>
Message-ID: <4B9C58E3.6000305@gmail.com>

Brett Cannon wrote:
> 
> 
> On Sat, Mar 13, 2010 at 13:52, Benjamin Peterson <benjamin at python.org
> <mailto:benjamin at python.org>> wrote:
> 
>     2010/3/13 Antoine Pitrou <solipsis at pitrou.net
>     <mailto:solipsis at pitrou.net>>:
>     > Any thoughts?
>     \
>     The latter solution seems best to me as it would help any 3rd party IO
>     libraries and require less code modification.
> 
> 
> Plus the interactiveness approach has been tested by OS thread
> schedulers for years and is shown to work. So I vote for the latter
> approach as well.

+1

Cheap to calculate and means developer don't need to guess if they
should be using the "I am CPU bound" or the "I am IO bound" GIL
acquisition macro.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From cs at zip.com.au  Sun Mar 14 06:16:34 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Sun, 14 Mar 2010 16:16:34 +1100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4B9C0B9B.3080805@v.loewis.de>
References: <4B9C0B9B.3080805@v.loewis.de>
Message-ID: <20100314051634.GA9503@cskk.homeip.net>

On 13Mar2010 23:03, Martin v. L?wis <martin at v.loewis.de> wrote:
| > Any thoughts?
| 
| My initial reaction to this report, and still my current opinion is:
| This issue is not a problem.
| It's a boundary case, so it's not clear whether Python should be able to
| deal with it gracefully. I doubt it's a realistic case.

How was it tripped over?

Speaking for myself, I have an app with a daemon mode which I expect
will sometimes behave as described; it answers requests and thus has I/O
bound threads waiting for requests and dispatching replies, and threads
doing data handling, which make constant use of the zlib library. On the
client side the same app is often throughput bound by a data examination
process that is compute bound; I can easily see it having compute bound
threads and I/O bound threads talking to daemon instances.

Currently all the above is somewhat "batchy"; the client end looks like
an archiving command line tool, but it's intended to have a FUSE mode to
present the archive as a filesystem; that I can imagine tripping over
this issue as a user uses the filesystem for "stuff".

Still, it is unusual and I suspect it will self limit to an extent
anyway; I can't fairly claim to have had it cause me trouble yet.
(Indeed, I've not used the new GIL at all - it's mostly using python
2.6).

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

I very strongly suggest that you periodically place ice packs over the abused
areas.  - Steve Garnier

From nir at winpdb.org  Sun Mar 14 12:31:51 2010
From: nir at winpdb.org (Nir Aides)
Date: Sun, 14 Mar 2010 13:31:51 +0200
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100313164651.7eb322b1@msiwind>
References: <20100313164651.7eb322b1@msiwind>
Message-ID: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>

There are two possible problems with Dave's benchmark:

1) On my system setting TCP_NODELAY option on the accepted server socket
changes results dramatically.
2) What category of socket servers is dave's spin() function intended to
simulate?

In a server which involves CPU intensive work in response to a socket
request the behavior may be significantly different.
In such a system, high CPU load will significantly reduce socket
responsiveness which in turn will reduce CPU load and increase socket
responsiveness.

Testing with a modified server that reflects the above indicates the new GIL
behaves just fine in terms of throughput.
So a change to the GIL may not be required at all.

There is still the question of latency - a single request which takes long
time to process will affect the latency of other "small" requests. However,
it can be argued if such a scenario is practical, or if modifying the GIL is
the solution.

If a change is still required, then I vote for the simpler approach - that
of having a special macro for socket code.
I remember there was reluctance in the past to repeat the OS scheduling
functionality and for a good reason.

Nir


On Sat, Mar 13, 2010 at 11:46 PM, Antoine Pitrou <solipsis at pitrou.net>wrote:

>
> Hello,
>
> As some of you may know, Dave Beazley recently exhibited a situation
> where the new GIL shows quite a poor behaviour (the old GIL isn't very
> good either, but still a little better). This issue is followed in
> http://bugs.python.org/issue7946
>
> This situation is when an IO-bound thread wants to process a lot of
> incoming packets, while one (or several) CPU-bound thread is also
> running. Each time the IO-bound thread releases the GIL, the CPU-bound
> thread gets it and keeps holding it for at least 5 milliseconds
> (default setting), which limits the number of individual packets which
> can be recv()'ed and processed per second.
>
> I have proposed two mechanisms, based on the same idea: IO-bound
> threads should be able to steal the GIL very quickly, rather than
> having to wait for the whole "thread switching interval" (again, 5 ms
> by default). They differ in how they detect an "IO-bound threads":
>
> - the first mechanism is actually the same mechanism which was
>  embodied in the original new GIL patch before being removed. In this
>  approach, IO methods (such as socket.read() in socketmodule.c)
>  releasing the GIL must use a separate C macro when trying to get the
>  GIL back again.
>
> - the second mechanism dynamically computes the "interactiveness" of a
>  thread and allows interactive threads to steal the GIL quickly. In
>  this approach, IO methods don't have to be modified at all.
>
> Both approaches show similar benchmark results (for the benchmarks
> that I know of) and basically fix the issue put forward by Dave Beazley.
>
> Any thoughts?
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100314/777a1227/attachment.html>

From andymac at bullseye.apana.org.au  Sun Mar 14 12:28:46 2010
From: andymac at bullseye.apana.org.au (Andrew MacIntyre)
Date: Sun, 14 Mar 2010 22:28:46 +1100
Subject: [Python-Dev] C++
In-Reply-To: <5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com>
References: <C7BF0D2C.2C422%peter.a.portante@gmail.com>	<4B9A67E7.30801@gmail.com>	<3c8293b61003121108n4772951el30927dbe0885b5cb@mail.gmail.com>	<19354.38405.67977.218983@montanaro.dyndns.org>	<20100312170308.111b90a5@msiwind>	<19355.3195.511985.327292@montanaro.dyndns.org>
	<5d44f72f1003122254s4c015780ia7c72a1cab48f5ba@mail.gmail.com>
Message-ID: <4B9CC86E.7000505@bullseye.andymac.org>

Jeffrey Yasskin wrote:
> On Fri, Mar 12, 2010 at 7:54 PM,  <skip at pobox.com> wrote:

>>    OS/2: http://www.andymac.org/
> 
> I can't find a modern c++ compiler for OS/2. Then again, your link
> only provides python 2.4.

While the gcc used as part of the EMX toolchain is 2.8.1, there are
ports of gcc 3.3.5 and 4.4.x using a somewhat compatible runtime (known
as klibc).  I have an unreleased 2.6 binary package which I'm holding
up pending 2.6.5.

OpenWatcom is also supported on OS/2, though I don't know whether the
C++ support is advanced enough...  while I like this compiler, the
runtime support (or lack thereof) for Posixish features makes platform
support a lot of work.

Andrew.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia

From peter.a.portante at gmail.com  Sun Mar 14 14:54:13 2010
From: peter.a.portante at gmail.com (Peter Portante)
Date: Sun, 14 Mar 2010 08:54:13 -0500
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>
Message-ID: <C7C254B5.2C6D8%peter.a.portante@gmail.com>

On 3/14/10 7:31 AM, "Nir Aides" <nir at winpdb.org> wrote:

> There are two possible problems with Dave's benchmark:
> 
> 1) On my system setting TCP_NODELAY option on the accepted server socket
> changes results dramatically.
Could you document what you saw and explain how you think TCP_NODELAY makes
a difference, including what kind of system you ran your tests and what the
application was that demonstrates those dramatic results?

> 2) What category of socket servers is dave's spin() function intended to
> simulate?
What is the problem you are trying to get at with this question?

Does Dave? spin() function have to have a category? Or perhaps the question
is, for these solutions, what category of application do they hurt? Perhaps
we can look at the solutions as general, but consider their impact in
various categories.

> In a server which involves CPU intensive work in response to a socket request
> the behavior may be significantly different. In such a system, high CPU load
> will significantly reduce socket responsiveness which in turn will reduce CPU
> load and increase socket responsiveness.
Not sure I follow how high CPU load will in turn reduce CPU load. :) Can you
explain more about what you are trying to say here?

> Testing with a modified server that reflects the above indicates the new GIL
> behaves just fine in terms of throughput. So a change to the GIL may not be
> required at all.
Are you saying that a change to the new GIL may not be required at all?

Did your modified server run any worse with any of the proposed changes?
Could you document what you saw?

> There is still the question of latency - a single request which takes long
> time to process will affect the latency of other "small" requests. However, it
> can be argued if such a scenario is practical, or if modifying the GIL is the
> solution.
Perhaps Dave already documented this effect in his visualizations, no?

> If a change is still required, then I vote for the simpler approach - that of
> having a special macro for socket code.
What is that simpler approach? How would that special macro work?

> I remember there was reluctance in the past to repeat the OS?scheduling
> functionality and for a good reason.

Folks,
 
Do we believe that every other application that has contention for one
resource leaves the solution of that problem to the OS scheduler?

In what ways do we consider the CPython interpreter to be different than
another application that has multiple threads and contention for one
resource? Perhaps we have a unique problem among all other user space
applications. Perhaps we don?t.

Do we see Antoine?s or Dave?s proposed solutions as being solutions to a
niche problem? Or are they algorithms for handling one or more of the
behaviors the CPython interpreter will encounter when running the many
python applications across OS/CPU environments?

As for the behavior of the GIL, how are the proposed solutions repeating OS
scheduling functionality? Could we instead look at these solutions as
repeating what other applications have done that have contention for one
resource?

Can anyone show that Antoine?s proposed solution hurts performance?

-peter

> Nir
> 
> 
> On Sat, Mar 13, 2010 at 11:46 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> 
>> Hello,
>> 
>> As some of you may know, Dave Beazley recently exhibited a situation
>> where the new GIL shows quite a poor behaviour (the old GIL isn't very
>> good either, but still a little better). This issue is followed in
>> http://bugs.python.org/issue7946
>> 
>> This situation is when an IO-bound thread wants to process a lot of
>> incoming packets, while one (or several) CPU-bound thread is also
>> running. Each time the IO-bound thread releases the GIL, the CPU-bound
>> thread gets it and keeps holding it for at least 5 milliseconds
>> (default setting), which limits the number of individual packets which
>> can be recv()'ed and processed per second.
>> 
>> I have proposed two mechanisms, based on the same idea: IO-bound
>> threads should be able to steal the GIL very quickly, rather than
>> having to wait for the whole "thread switching interval" (again, 5 ms
>> by default). They differ in how they detect an "IO-bound threads":
>> 
>> - the first mechanism is actually the same mechanism which was
>>  ?embodied in the original new GIL patch before being removed. In this
>>  ?approach, IO methods (such as socket.read() in socketmodule.c)
>>  ?releasing the GIL must use a separate C macro when trying to get the
>>  ?GIL back again.
>> 
>> - the second mechanism dynamically computes the "interactiveness" of a
>>  ?thread and allows interactive threads to steal the GIL quickly. In
>>  ?this approach, IO methods don't have to be modified at all.
>> 
>> Both approaches show similar benchmark results (for the benchmarks
>> that I know of) and basically fix the issue put forward by Dave Beazley.
>> 
>> Any thoughts?
>> 
>> Regards
>> 
>> Antoine.
>> 
>> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100314/9d5527c4/attachment-0001.html>

From jon+python-dev at unequivocal.co.uk  Sun Mar 14 13:59:50 2010
From: jon+python-dev at unequivocal.co.uk (Jon Ribbens)
Date: Sun, 14 Mar 2010 12:59:50 +0000
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100312175657.287bb456@heresy.wooz.org>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
Message-ID: <20100314125950.GH19240@snowy.squish.net>

On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote:
> Nope, sorry I should have been clearer.
> 
> *creation* is the key here.  As per BDFL pronouncement, we'll support reading
> pyc-only modules just like we do today.  This is in PEP 3147.  We won't
> support creating them though.
> 
> BTW, I'm actually starting to work on the implementation now, so if you want
> to play along:
> 
> % bzr branch lp:~barry/python/pep3147

Sorry if I missed it, but why on earth is the bytecode directory
__pycache__ and not .pycache? (Or indeed anything else that starts
with a '.') Surely this is a classic ideal use case for a "hidden"
directory?

From steve at pearwood.info  Sun Mar 14 16:07:23 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 15 Mar 2010 02:07:23 +1100
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100314125950.GH19240@snowy.squish.net>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100312175657.287bb456@heresy.wooz.org>
	<20100314125950.GH19240@snowy.squish.net>
Message-ID: <201003150207.24396.steve@pearwood.info>

On Sun, 14 Mar 2010 11:59:50 pm Jon Ribbens wrote:

> Sorry if I missed it, but why on earth is the bytecode directory
> __pycache__ and not .pycache? (Or indeed anything else that starts
> with a '.') Surely this is a classic ideal use case for a "hidden"
> directory?

I disagree with your assumption that there is *any* use-case for a 
hidden directory, let alone an ideal one.

I despise hidden directories and dot files. I know it is the "Unix way", 
and I suppose it made sense back in 1975 when users had two or three 
dot files in their home directory, but I count 215 dot files in my home 
directory compared to only 77 visible files, and I have no idea how 
most of them got there or what they do. Programs that litter the file 
system with dot files are bad enough when they do it in $HOME, but 
sprinkling dot files everywhere they can is inexcusable.

This is not the place for me to rant over the evil that is dot files, so 
I'll just say this: Python works on platforms other than Unix/Linux, 
and some of those platforms don't treat dot files as anything more than 
a file with a leading dot in the file name.



-- 
Steven D'Aprano

From solipsis at pitrou.net  Sun Mar 14 16:24:11 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 14 Mar 2010 11:24:11 -0400
Subject: [Python-Dev] __file__ and bytecode-only
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
	<20100314125950.GH19240@snowy.squish.net>
Message-ID: <20100314112411.2e5e1604@msiwind>

Le Sun, 14 Mar 2010 12:59:50 +0000,
Jon Ribbens <jon+python-dev at unequivocal.co.uk> a ?crit :
> 
> Sorry if I missed it, but why on earth is the bytecode directory
> __pycache__ and not .pycache? (Or indeed anything else that starts
> with a '.') Surely this is a classic ideal use case for a "hidden"
> directory?

I suppose the same reason pyc files are not hidden nowadays. We want
people to be aware that these files exist.



From p.f.moore at gmail.com  Sun Mar 14 17:37:57 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 14 Mar 2010 16:37:57 +0000
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B9B9E65.4080902@gmail.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org> <4B9B9E65.4080902@gmail.com>
Message-ID: <79990c6b1003140937x783bfa02pc5c0a949a5f42890@mail.gmail.com>

On 13 March 2010 14:17, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The creation side could be made a little more explicit in the PEP. We
> could also do something via the compileall module.
>
> (Pause while Nick goes and reads the source code for compileall for the
> first time ever...)
>
> Hmm - methinks the PEP actually needs to talk explicitly about the
> py_compile and compileall modules. These compile the files directly
> rather than using the import system's side-effect, so they'll need to
> understand the intricacies of the new system.

The bdist_wininst installer also compiles modules explicitly on
install (as does the python.org Windows MSI installer). I've always
assumed that this worked via compileall, but haven't checked.
Regardless, these should probably also be covered in the PEP.

Paul.

From rdmurray at bitdance.com  Sun Mar 14 17:46:11 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 14 Mar 2010 12:46:11 -0400
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100314125950.GH19240@snowy.squish.net>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
	<20100314125950.GH19240@snowy.squish.net>
Message-ID: <20100314164611.0B5051AB5B9@kimball.webabinitio.net>

On Sun, 14 Mar 2010 12:59:50 -0000, Jon Ribbens <jon+python-dev at unequivocal.co.uk> wrote:
> On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote:
> > Nope, sorry I should have been clearer.
> > 
> > *creation* is the key here.  As per BDFL pronouncement, we'll support reading
> > pyc-only modules just like we do today.  This is in PEP 3147.  We won't
> > support creating them though.
> > 
> > BTW, I'm actually starting to work on the implementation now, so if you want
> > to play along:
> > 
> > % bzr branch lp:~barry/python/pep3147
> 
> Sorry if I missed it, but why on earth is the bytecode directory
> __pycache__ and not .pycache? (Or indeed anything else that starts
> with a '.') Surely this is a classic ideal use case for a "hidden"
> directory?

You did miss it.  Perhaps a link to the discussion should be added to
the PEP, or a summary of it.  If I remember correctly the BDFL even
weighed in on this point.

--
R. David Murray                                      www.bitdance.com

From martin at v.loewis.de  Sun Mar 14 19:32:52 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 14 Mar 2010 19:32:52 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100314051634.GA9503@cskk.homeip.net>
References: <4B9C0B9B.3080805@v.loewis.de>
	<20100314051634.GA9503@cskk.homeip.net>
Message-ID: <4B9D2BD4.8010307@v.loewis.de>

> Speaking for myself, I have an app with a daemon mode which I expect
> will sometimes behave as described; it answers requests and thus has I/O
> bound threads waiting for requests and dispatching replies, and threads
> doing data handling, which make constant use of the zlib library.

This is then already different from the scenario described. Every call
into zlib will release the GIL, for the period of the zlib computation,
allowing other threads to run.

> On the
> client side the same app is often throughput bound by a data examination
> process that is compute bound; I can easily see it having compute bound
> threads and I/O bound threads talking to daemon instances.

I can't see that. I would expect that typically (and specifically
including your application), the compute bound threads will synchronize
with the IO bound ones, asking for more requests to perform.

That's the whole point of using the "bound" adjective (?): execution
time is *bound* by the computation time. It can't get faster than what
the computation can process.

Regards,
Martni

From janssen at parc.com  Sun Mar 14 20:37:48 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 14 Mar 2010 12:37:48 PDT
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100314051634.GA9503@cskk.homeip.net>
References: <4B9C0B9B.3080805@v.loewis.de>
	<20100314051634.GA9503@cskk.homeip.net>
Message-ID: <70987.1268595468@parc.com>

Cameron Simpson <cs at zip.com.au> wrote:

> Currently all the above is somewhat "batchy"; the client end looks like
> an archiving command line tool, but it's intended to have a FUSE mode to
> present the archive as a filesystem; that I can imagine tripping over
> this issue as a user uses the filesystem for "stuff".

Me, too :-).  Though I also do a lot of processing in pure Python...

> Still, it is unusual and I suspect it will self limit to an extent
> anyway; I can't fairly claim to have had it cause me trouble yet.
> (Indeed, I've not used the new GIL at all - it's mostly using python
> 2.6).

The old GIL is causing me a great deal of grief.  My server is heavily
threaded, and when I get a number of Python 2.6 threads doing layout
analysis, my 4 and 8 core machines turn into slow thrashers.

I'm dismayed that the backport of the new GIL work to the 2.x line has
been stopped (see issue 7753); I see a future of OS X 10.7 with
presumably 64-bit Python 2.7 on, say, 32-core machines running even more
slowly than it currently does.  That's a pretty hard-core backwards
compatibility decision.

Maybe the Mac-SIG should adopt that patch...

Bill

From nir at winpdb.org  Sun Mar 14 22:11:44 2010
From: nir at winpdb.org (Nir Aides)
Date: Sun, 14 Mar 2010 23:11:44 +0200
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <C7C254B5.2C6D8%peter.a.portante@gmail.com>
References: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>
	<C7C254B5.2C6D8%peter.a.portante@gmail.com>
Message-ID: <5d2328511003141411j2bfb8c12s72e2ed54744fe0d4@mail.gmail.com>

inline:

On Sun, Mar 14, 2010 at 3:54 PM, Peter Portante
<peter.a.portante at gmail.com>wrote:

>  On 3/14/10 7:31 AM, "Nir Aides" <nir at winpdb.org> wrote:
>
> There are two possible problems with Dave's benchmark:
>
> 1) On my system setting TCP_NODELAY option on the accepted server socket
> changes results dramatically.
>
> Could you document what you saw and explain how you think TCP_NODELAY makes
> a difference, including what kind of system you ran your tests and what the
> application was that demonstrates those dramatic results?
>

I first disabled the call to spin() but client running time remained around
30 seconds.
I then added TCP_NODELAY and running time dropped to a few dozen
milliseconds for the entire no-spin run.
The system is Ubuntu Karmic 64bit with latest revision of python 3.2.

2) What category of socket servers is dave's spin() function intended to
> simulate?
>
> What is the problem you are trying to get at with this question?
>
> Does Dave? spin() function have to have a category? Or perhaps the question
> is, for these solutions, what category of application do they hurt? Perhaps
> we can look at the solutions as general, but consider their impact in
> various categories.
>

In Dave's code sample, spin() is loading the CPU regardless of
requests. This may demonstrate how short IO bound requests will behave while
the server is processing a long Python-algorithmic CPU intensive request, or
an ongoing CPU intensive task unrelated to incoming requests. However is
this typical for socket servers?

If you change the sample server code to start a CPU bound thread with work X
for each incoming request you will see different behavior.

There is still the question of latency - a single request which takes long
> time to process will affect the latency of other "small" requests. However,
> it can be argued if such a scenario is practical, or if modifying the GIL is
> the solution.
>
> Perhaps Dave already documented this effect in his visualizations, no?
>

Naturally. I did not imply otherwise. His analysis is brilliant.


> If a change is still required, then I vote for the simpler approach - that
> of having a special macro for socket code.
>
> What is that simpler approach? How would that special macro work?
>

The special macro for socket code is one of the alternatives proposed by
Antoine above.

However, thinking about it again, with this approach as soon as the new
incoming request tries to read a file, query the DB, decompress some data or
do anything which releases the GIL, it goes back to square one. no?

I remember there was reluctance in the past to repeat the OS scheduling
> functionality and for a good reason.
>
>
> In what ways do we consider the CPython interpreter to be different than
> another application that has multiple threads and contention for one
> resource? Perhaps we have a unique problem among all other user space
> applications. Perhaps we don?t.
>

I think a rule of thumb is to protect a resource (typically a data
structure), as tightly as possible, avoiding for example locking across
function calls, etc, if possible. In contrast CPython is "locked" the entire
time.

As for the behavior of the GIL, how are the proposed solutions repeating OS
> scheduling functionality?
>

Dave discussed this in his analysis.

Nir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100314/200248df/attachment-0001.html>

From ncoghlan at gmail.com  Sun Mar 14 22:43:12 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Mar 2010 07:43:12 +1000
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100314164611.0B5051AB5B9@kimball.webabinitio.net>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>	<20100303204923.6d597f03@heresy.wooz.org>	<4B8FB6D2.4040501@gmail.com>	<20100312145312.1f0a0aa0@heresy.wooz.org>	<4B9AC4CA.3070302@voidspace.org.uk>	<20100312175657.287bb456@heresy.wooz.org>	<20100314125950.GH19240@snowy.squish.net>
	<20100314164611.0B5051AB5B9@kimball.webabinitio.net>
Message-ID: <4B9D5870.8080507@gmail.com>

R. David Murray wrote:
> On Sun, 14 Mar 2010 12:59:50 -0000, Jon Ribbens <jon+python-dev at unequivocal.co.uk> wrote:
>> Sorry if I missed it, but why on earth is the bytecode directory
>> __pycache__ and not .pycache? (Or indeed anything else that starts
>> with a '.') Surely this is a classic ideal use case for a "hidden"
>> directory?
> 
> You did miss it.  Perhaps a link to the discussion should be added to
> the PEP, or a summary of it.  If I remember correctly the BDFL even
> weighed in on this point.

He did (in favour of keeping the directory visible).

Here's one reference (although not the main one):
http://osdir.com/ml/python-dev/2010-02/msg00114.html

My Google-fu is failing me, so I don't have a link to the other part of
the thread that I refer to in that message.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Sun Mar 14 23:09:29 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 14 Mar 2010 18:09:29 -0400
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <70987.1268595468@parc.com>
References: <4B9C0B9B.3080805@v.loewis.de>
	<20100314051634.GA9503@cskk.homeip.net> <70987.1268595468@parc.com>
Message-ID: <20100314180929.4053d683@msiwind>

Le Sun, 14 Mar 2010 12:37:48 PDT,
Bill Janssen <janssen at parc.com> a ?crit :
> 
> The old GIL is causing me a great deal of grief.  My server is heavily
> threaded, and when I get a number of Python 2.6 threads doing layout
> analysis, my 4 and 8 core machines turn into slow thrashers.

Have you checked whether the 2.x patch improved your use case?
It would be nice to get real-world feedback of which situations the
new GIL helps (or doesn't help) improve.

Thanks

Antoine.

From solipsis at pitrou.net  Sun Mar 14 23:17:03 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 14 Mar 2010 18:17:03 -0400
Subject: [Python-Dev] "Fixing" the new GIL
References: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>
	<C7C254B5.2C6D8%peter.a.portante@gmail.com>
	<5d2328511003141411j2bfb8c12s72e2ed54744fe0d4@mail.gmail.com>
Message-ID: <20100314181703.0f65fdcf@msiwind>

Le Sun, 14 Mar 2010 23:11:44 +0200,
Nir Aides <nir at winpdb.org> a ?crit :
> 
> I first disabled the call to spin() but client running time remained
> around 30 seconds.
> I then added TCP_NODELAY and running time dropped to a few dozen
> milliseconds for the entire no-spin run.

You don't want the benchmark to be dependent on the TCP stack's
implementation specifics. Therefore, you should use the UDP version
of Dave Beazley's benchmark.
I have posted it on http://bugs.python.org/issue7946, and included a
variant of it in ccbench.

> The special macro for socket code is one of the alternatives proposed
> by Antoine above.
> 
> However, thinking about it again, with this approach as soon as the
> new incoming request tries to read a file, query the DB, decompress
> some data or do anything which releases the GIL, it goes back to
> square one. no?

Indeed. This approach involves using the macro in every place where
releasing the GIL is meant to cover OS- or IO-dependent latencies.
Even then, though, there may be cases (such as doing a very small
amount of zlib compression) where the background CPU-bound thread will
get "too much time" compared to the "mostly IO" thread. But the other
approach isn't perfect either. These are heuristics anyway.

> I remember there was reluctance in the past to repeat the OS
> scheduling
> functionality and for a good reason.

I agree with this. Hence my posting on this list, about whether we
should go with an additional amount of complexity to solve what
may be considered an annoying performance problem.

Regards

Antoine.



From martin at v.loewis.de  Sun Mar 14 23:23:53 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 14 Mar 2010 23:23:53 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>
References: <20100313164651.7eb322b1@msiwind>
	<5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>
Message-ID: <4B9D61F9.7070106@v.loewis.de>

> Testing with a modified server that reflects the above indicates the new
> GIL behaves just fine in terms of throughput.
> So a change to the GIL may not be required at all.

Thanks - I think you put it better than I did.

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Mon Mar 15 00:10:52 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 15 Mar 2010 12:10:52 +1300
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100314125950.GH19240@snowy.squish.net>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org> <4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
	<20100314125950.GH19240@snowy.squish.net>
Message-ID: <4B9D6CFC.7020709@canterbury.ac.nz>

Jon Ribbens wrote:

> Sorry if I missed it, but why on earth is the bytecode directory
> __pycache__ and not .pycache? (Or indeed anything else that starts
> with a '.') Surely this is a classic ideal use case for a "hidden"
> directory?

Please don't try to hide it. On MacOSX, it makes it impossible
to look inside using the Finder if you want to, and it won't
have any effect on Windows anyway.

-- 
Greg

From greg at krypto.org  Mon Mar 15 00:29:48 2010
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 14 Mar 2010 16:29:48 -0700
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>
References: <20100313164651.7eb322b1@msiwind>
	<5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>
Message-ID: <52dc1c821003141629o1a48c7fp83847c342ef2d9e7@mail.gmail.com>

On Sun, Mar 14, 2010 at 4:31 AM, Nir Aides <nir at winpdb.org> wrote:
> There are two possible problems with Dave's benchmark:
> 1) On my system setting TCP_NODELAY option on the accepted server socket
> changes results dramatically.
> 2) What category of socket servers is dave's spin() function intended to
> simulate?
> In a server which involves CPU intensive work in response to a socket
> request the behavior may be significantly different.
> In such a system, high CPU load will significantly reduce socket
> responsiveness which in turn will reduce CPU load and increase socket
> responsiveness.
> Testing with a modified server that reflects the above indicates the new GIL
> behaves just fine in terms of throughput.
> So a change to the GIL may not be required at all.
> There is still the question of latency - a single request which takes long
> time to process will affect the latency of other "small" requests. However,
> it can be argued if such a scenario is practical, or if modifying the GIL is
> the solution.

Such a scenario is practical and is an area that we really should not
fall flat on our face in.  Python should not regress to performing
worse when an application adds a cpu intensive thread that isn't
tempered by the activity of its IO threads.

As for the argument that an application with cpu intensive work being
driven by the IO itself will work itself out...  No it won't, it can
get into beat patterns where it is handling requests quite rapidly up
until one that causes a long computation to start comes in.  At that
point it'll stop performing well on other requests for as long (it
could be a significant amount of time) as the cpu intensive request
threads are running.  That is not a graceful degration in serving
capacity / latency as one would normally expect.  It is a sudden drop
off.  (followed by a sudden ramp up, and a sudden drop off and a
sudden ramp up, and drop off... etc.  not pretty.  not predictable.
not easy to deploy and manage as a service)

If we don't fix this issue, we need to document as an official part of
the Python threading module docs that people should not mix
computation with IO in threaded applications when using CPython if
they care about IO performance.  This will blow people's mind.

> If a change is still required, then I vote for the simpler approach - that
> of having a special macro for socket code.
> I remember there was reluctance in the past to repeat the OS?scheduling
> functionality and for a good reason.

I don't consider needing to modify code specifically for it to
indicate if it is IO bound or CPU bound to be the simpler approach.

So +1 on the interactiveness calculation for me.

+0.5 on annotating the IO lock release/acquisitions.

Basically: I want to see this fixed, even if its not my preferred
approach I still think it is important.

-gps

From janssen at parc.com  Mon Mar 15 02:46:25 2010
From: janssen at parc.com (Bill Janssen)
Date: Sun, 14 Mar 2010 18:46:25 PDT
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100314180929.4053d683@msiwind>
References: <4B9C0B9B.3080805@v.loewis.de>
	<20100314051634.GA9503@cskk.homeip.net>
	<70987.1268595468@parc.com> <20100314180929.4053d683@msiwind>
Message-ID: <74196.1268617585@parc.com>

Antoine Pitrou <solipsis at pitrou.net> wrote:

> Le Sun, 14 Mar 2010 12:37:48 PDT,
> Bill Janssen <janssen at parc.com> a ?crit :
> > 
> > The old GIL is causing me a great deal of grief.  My server is heavily
> > threaded, and when I get a number of Python 2.6 threads doing layout
> > analysis, my 4 and 8 core machines turn into slow thrashers.
> 
> Have you checked whether the 2.x patch improved your use case?
> It would be nice to get real-world feedback of which situations the
> new GIL helps (or doesn't help) improve.

Yes, good thought.  I'll try it and see if I can get some numbers.  I do
have a mix of I/O and CPU bound threads, and we are seeing core usage by
Python similar to that David originally discovered.

Bill

From dave at dabeaz.com  Mon Mar 15 04:07:05 2010
From: dave at dabeaz.com (David Beazley)
Date: Sun, 14 Mar 2010 22:07:05 -0500
Subject: [Python-Dev] "Fixing" the new GIL
Message-ID: <49653.1268622425@dabeaz.com>

I have been following this discussion about fixing the GIL and just wanted to make a few comments about it.

To the doubters who don't think this is a real problem worth fixing, I must respectfully disagree.   Multicore systems and parallel programming issues are not 
going away any time in the foreseeable future.  Moreover, there are plenty of programmers who are using Python to build applications involving concurrency, 
distributed computing, and compute intensive processing.    While everyone agrees that it would be great if the GIL simply disappeared, the next best option is 
for the GIL to have well-defined and reasonably predictable behavior (i.e., behavior that can be easily explained and justified).   Specifically, if multiple threads 
happen to be performing CPU intensive work at the same time, it would be nice if they didn't thrash on multiple cores (the problem with the old GIL) and if I/O is 
mixed with CPU-bound processing it would be nice if I/O requests didn't have their responsiveness and priority penalized  (the problem with the new GIL).

So, just to be clear about the my bug report, it is directly related to the problem of overlapping I/O requests with CPU-bound processing.  This kind of scenario 
comes up in the context of many applications--especially those based on cooperating processes, multiprocessing, and message passing.  Ironically, these are 
exactly the kinds of parallel programming techniques that people are going to use because of the GIL in the first place.   So, why would you want the new GIL to 
behave poorly here?

In any case, if the GIL can be improved in a way that is simple and which either improves or doesn't negatively impact the performance of existing applications, 
why wouldn't you want to do it?  Seems like a no-brainer.

Cheers,
Dave








From martin at v.loewis.de  Mon Mar 15 09:28:40 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 15 Mar 2010 09:28:40 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <52dc1c821003141629o1a48c7fp83847c342ef2d9e7@mail.gmail.com>
References: <20100313164651.7eb322b1@msiwind>	<5d2328511003140431w458f034eo1dfe4ad1843a2a87@mail.gmail.com>
	<52dc1c821003141629o1a48c7fp83847c342ef2d9e7@mail.gmail.com>
Message-ID: <4B9DEFB8.3010107@v.loewis.de>

> As for the argument that an application with cpu intensive work being
> driven by the IO itself will work itself out...  No it won't, it can
> get into beat patterns where it is handling requests quite rapidly up
> until one that causes a long computation to start comes in.  At that
> point it'll stop performing well on other requests for as long (it
> could be a significant amount of time) as the cpu intensive request
> threads are running.  That is not a graceful degration in serving
> capacity / latency as one would normally expect.  It is a sudden drop
> off.

Why do you say that? The other threads continue to be served - and
Python couldn't use more than one CPU, anyway. Can you demonstrate that
in an example?

Regards,
Martin


From martin at v.loewis.de  Mon Mar 15 09:34:03 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 15 Mar 2010 09:34:03 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <49653.1268622425@dabeaz.com>
References: <49653.1268622425@dabeaz.com>
Message-ID: <4B9DF0FB.8000002@v.loewis.de>

> So, just to be clear about the my bug report, it is directly related
> to the problem of overlapping I/O requests with CPU-bound processing.
> This kind of scenario comes up in the context of many
> applications--especially those based on cooperating processes,
> multiprocessing, and message passing.

How so? if you have cooperating processes, multiprocessing, and message
passing, you *don't* have CPU bound threads along with IO "bound"
threads in the same process - you may not have threads at all!!!

> In any case, if the GIL can be improved in a way that is simple and
> which either improves or doesn't negatively impact the performance of
> existing applications, why wouldn't you want to do it?  Seems like a
> no-brainer.

Unfortunately, a simple improvement doesn't really seem to exist.

Regards,
Martin

From dave at dabeaz.com  Mon Mar 15 13:21:29 2010
From: dave at dabeaz.com (David Beazley)
Date: Mon, 15 Mar 2010 07:21:29 -0500
Subject: [Python-Dev] (no subject)
Message-ID: <50994.1268655689@dabeaz.com>



On Mon 15/03/10  4:34 AM , "Martin v. L?wis" martin at v.loewis.de sent:
> > So, just to be clear about the my bug report, it
> is directly related> to the problem of overlapping I/O requests with
> CPU-bound processing.> This kind of scenario comes up in the context of
> many> applications--especially those based on
> cooperating processes,> multiprocessing, and message passing.
> 
> How so? if you have cooperating processes, multiprocessing, and message
> passing, you *don't* have CPU bound threads along with IO
> "bound"threads in the same process - you may not have threads at all!!!
> 

You're right in that end user will probably not be using threads in this situation.  However, threads are still often used inside the associated programming 
libraries and frameworks that provide communication.   For example, threads might be used to implement queuing, background monitoring, event notification, 
routing, and other similar low-level features.   Just as an example, the multiprocessing module currently uses background threads as part of its implementation 
of queues.   In a similar vein, threads are sometimes used in asynchronous I/O frameworks (e.g., Twisted) to handle certain kinds of deferred operations.  
Bottom line: just because a user isn't directly programming with threads doesn't mean that threads aren't there.

> > In any case, if the GIL can be improved in a way
> that is simple and> which either improves or doesn't negatively
> impact the performance of> existing applications, why wouldn't you want to
> do it?  Seems like a> no-brainer.
> 
> Unfortunately, a simple improvement doesn't really seem to exist.
> 

Well, I think this problem can be fixed in a manner that is pretty well isolated to just one specific part of Python (the GIL)--especially since several prototypes, 
including my own, have already demonstrated that it's possible.    In any case, as it stands now, the current "new GIL" ignores about 40 years of work concerning 
operating system thread/process scheduling and the efficient handling of I/O requests.   I suspect that I'm not the only one who be disappointed if the Python 
community simply blew it off and said it's not an issue.    Trying to fix it is a worthwhile goal.

Cheers,
Dave




From barry at python.org  Mon Mar 15 16:16:54 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Mar 2010 11:16:54 -0400
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B9B9E65.4080902@gmail.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
	<4B9B9E65.4080902@gmail.com>
Message-ID: <20100315111654.1aa7acf5@heresy.wooz.org>

On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote:

>Hmm - methinks the PEP actually needs to talk explicitly about the
>py_compile and compileall modules. These compile the files directly
>rather than using the import system's side-effect, so they'll need to
>understand the intricacies of the new system.

Good point.  I'll add this to the PEP.

>While it's probably OK if the import side-effects only create files
>using the new scheme, the standard library modules will likely need to
>support both schemes (although I'm not sure if "same as import system"
>or "same as Python 3.1" make more sense as the default semantics -
>probably the former).

I don't understand this point.

compileall probably /could/ be extended to understand bytecode-only
(i.e. legacy or <3.2) layout.  I've added that to the PEP too.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100315/514c798e/attachment.pgp>

From barry at python.org  Mon Mar 15 16:18:34 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Mar 2010 11:18:34 -0400
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <79990c6b1003140937x783bfa02pc5c0a949a5f42890@mail.gmail.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
	<4B9B9E65.4080902@gmail.com>
	<79990c6b1003140937x783bfa02pc5c0a949a5f42890@mail.gmail.com>
Message-ID: <20100315111834.66190720@heresy.wooz.org>

On Mar 14, 2010, at 04:37 PM, Paul Moore wrote:

>The bdist_wininst installer also compiles modules explicitly on
>install (as does the python.org Windows MSI installer). I've always
>assumed that this worked via compileall, but haven't checked.
>Regardless, these should probably also be covered in the PEP.

Thanks; added to PEP.
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100315/c681e965/attachment.pgp>

From barry at python.org  Mon Mar 15 16:28:29 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 15 Mar 2010 11:28:29 -0400
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B9D5870.8080507@gmail.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
	<20100314125950.GH19240@snowy.squish.net>
	<20100314164611.0B5051AB5B9@kimball.webabinitio.net>
	<4B9D5870.8080507@gmail.com>
Message-ID: <20100315112829.4b36f160@heresy.wooz.org>

On Mar 15, 2010, at 07:43 AM, Nick Coghlan wrote:
>He did (in favour of keeping the directory visible).

http://www.mail-archive.com/python-dev at python.org/msg45203.html

(added to pep)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100315/89672313/attachment.pgp>

From cs at zip.com.au  Tue Mar 16 06:34:45 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 16 Mar 2010 16:34:45 +1100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4B9DEFB8.3010107@v.loewis.de>
References: <4B9DEFB8.3010107@v.loewis.de>
Message-ID: <20100316053445.GA11229@cskk.homeip.net>

On 15Mar2010 09:28, Martin v. L?wis <martin at v.loewis.de> wrote:
| > As for the argument that an application with cpu intensive work being
| > driven by the IO itself will work itself out...  No it won't, it can
| > get into beat patterns where it is handling requests quite rapidly up
| > until one that causes a long computation to start comes in.  At that
| > point it'll stop performing well on other requests for as long (it
| > could be a significant amount of time) as the cpu intensive request
| > threads are running.  That is not a graceful degration in serving
| > capacity / latency as one would normally expect.  It is a sudden drop
| > off.
| 
| Why do you say that? The other threads continue to be served - and
| Python couldn't use more than one CPU, anyway. Can you demonstrate that
| in an example?

Real example:

I have a FuncMultiQueue class which manages a pool of worker threads.
Other tasks can make requests of it by passing a callable (usually
obtained via partial()) to its .call(), .bgcall() or .qbgcall() methods
depending on how they want to collect the result of the callable.

The idea here is that one has a few threads receiving requests (eg a
daemon watching a socket or monitoring a db queue table) which then use
the FuncMultiQueue to manage how many actual requests are processed
in parallel (yes, a semaphore can cover a lot of this, but not the
asynchronous call modes).

So, suppose a couple of CPU-intensive callables get queued which work for a
substantial time, and meanwhile a bunch of tiny tiny cheap requests arrive.
Their timely response will be impacted by this issue.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

hybrid rather than pure; compromising rather than clean;
distorted rather than straightforward; ambiguous rather than
articulated; both-and rather than either-or; the difficult
unity of inclusion rather than the easy unity of exclusion.
        - Paul Barton-Davis <pauld at cs.washington.edu>

From cs at zip.com.au  Tue Mar 16 06:38:47 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 16 Mar 2010 16:38:47 +1100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4B9D2BD4.8010307@v.loewis.de>
References: <4B9D2BD4.8010307@v.loewis.de>
Message-ID: <20100316053847.GA12552@cskk.homeip.net>

On 14Mar2010 19:32, "Martin v. L?wis" <martin at v.loewis.de> wrote:
| > Speaking for myself, I have an app with a daemon mode which I expect
| > will sometimes behave as described; it answers requests and thus has I/O
| > bound threads waiting for requests and dispatching replies, and threads
| > doing data handling, which make constant use of the zlib library.
| 
| This is then already different from the scenario described. Every call
| into zlib will release the GIL, for the period of the zlib computation,
| allowing other threads to run.
| 
| > On the
| > client side the same app is often throughput bound by a data examination
| > process that is compute bound; I can easily see it having compute bound
| > threads and I/O bound threads talking to daemon instances.
| 
| I can't see that. I would expect that typically (and specifically
| including your application), the compute bound threads will synchronize
| with the IO bound ones, asking for more requests to perform.

In the single threaded case, sure. The usual command line "archive this"
mode fit this. But...

| That's the whole point of using the "bound" adjective (?): execution
| time is *bound* by the computation time. It can't get faster than what
| the computation can process.

If it's lurking behind a filesystem interface or in its daemon mode
(remote archive store), multiple client processes can be using it at once,
and it will be processing multiple tasks somewhat in parallel. Here one
can get a compute bound thread answering one request, impacting quick
response to other parallel-and-cheap requests.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Here was a man who not only had a ready mind and a quick wit,
but could also sing.    - _Rope_

From martin at v.loewis.de  Tue Mar 16 08:59:16 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 16 Mar 2010 08:59:16 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100316053445.GA11229@cskk.homeip.net>
References: <4B9DEFB8.3010107@v.loewis.de>
	<20100316053445.GA11229@cskk.homeip.net>
Message-ID: <4B9F3A54.3040907@v.loewis.de>

Cameron Simpson wrote:
> On 15Mar2010 09:28, Martin v. L?wis <martin at v.loewis.de> wrote:
> | > As for the argument that an application with cpu intensive work being
> | > driven by the IO itself will work itself out...  No it won't, it can
> | > get into beat patterns where it is handling requests quite rapidly up
> | > until one that causes a long computation to start comes in.  At that
> | > point it'll stop performing well on other requests for as long (it
> | > could be a significant amount of time) as the cpu intensive request
> | > threads are running.  That is not a graceful degration in serving
> | > capacity / latency as one would normally expect.  It is a sudden drop
> | > off.
> | 
> | Why do you say that? The other threads continue to be served - and
> | Python couldn't use more than one CPU, anyway. Can you demonstrate that
> | in an example?
> 
> Real example:

... unfortunately without a demonstration. What's the throughput under
the old GIL? What's the throughput of this application under the new
GIL? How can I observe the "beat pattern"?

> The idea here is that one has a few threads receiving requests (eg a
> daemon watching a socket or monitoring a db queue table) which then use
> the FuncMultiQueue to manage how many actual requests are processed
> in parallel (yes, a semaphore can cover a lot of this, but not the
> asynchronous call modes).

Why do you say processing is in parallel? In Python, processing is
normally never in parallel, but always sequential (potentially
interleaving). Are you releasing the GIL for processing?

> So, suppose a couple of CPU-intensive callables get queued which work for a
> substantial time, and meanwhile a bunch of tiny tiny cheap requests arrive.
> Their timely response will be impacted by this issue.

By how much exactly? What operating system?

Regards,
Martin

From martin at v.loewis.de  Tue Mar 16 09:02:41 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 16 Mar 2010 09:02:41 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100316053847.GA12552@cskk.homeip.net>
References: <4B9D2BD4.8010307@v.loewis.de>
	<20100316053847.GA12552@cskk.homeip.net>
Message-ID: <4B9F3B21.1090106@v.loewis.de>

> If it's lurking behind a filesystem interface or in its daemon mode
> (remote archive store), multiple client processes can be using it at once,
> and it will be processing multiple tasks somewhat in parallel. Here one
> can get a compute bound thread answering one request, impacting quick
> response to other parallel-and-cheap requests.

However, "impacting" will always be the case. *Of course* any thread
that performs computation impacts everything else on the same processor.
There is nothing to prevent that, unless you schedule the long-running
activity at a point in time where you know the system will be idle for
the entire run of that task.

Regards,
Martin

From cs at zip.com.au  Tue Mar 16 10:15:44 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 16 Mar 2010 20:15:44 +1100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4B9F3A54.3040907@v.loewis.de>
References: <4B9F3A54.3040907@v.loewis.de>
Message-ID: <20100316091544.GA3799@cskk.homeip.net>

On 16Mar2010 08:59, "Martin v. L?wis" <martin at v.loewis.de> wrote:
| Cameron Simpson wrote:
| > On 15Mar2010 09:28, Martin v. L?wis <martin at v.loewis.de> wrote:
| > | > As for the argument that an application with cpu intensive work being
| > | > driven by the IO itself will work itself out...  No it won't, it can
| > | > get into beat patterns where it is handling requests quite rapidly up
| > | > until one that causes a long computation to start comes in.  At that
| > | > point it'll stop performing well on other requests for as long (it
| > | > could be a significant amount of time) as the cpu intensive request
| > | > threads are running.  That is not a graceful degration in serving
| > | > capacity / latency as one would normally expect.  It is a sudden drop
| > | > off.
| > | 
| > | Why do you say that? The other threads continue to be served - and
| > | Python couldn't use more than one CPU, anyway. Can you demonstrate that
| > | in an example?
| > 
| > Real example:
| 
| ... unfortunately without a demonstration. What's the throughput under
| the old GIL? What's the throughput of this application under the new
| GIL? How can I observe the "beat pattern"?

You can't. This isn't an example where I am personally bitten by the GIL.
I may be, but there's plenty of other stuff in terms of tuning or
optimisation in my particular app before I get anywhere near the GIL.

My point is not that it's biting me right now but that earlier you
said the scenario was probably unrealistic, but I have an app which can
probably exhibit exactly the issue under discussion.

| > The idea here is that one has a few threads receiving requests (eg a
| > daemon watching a socket or monitoring a db queue table) which then use
| > the FuncMultiQueue to manage how many actual requests are processed
| > in parallel (yes, a semaphore can cover a lot of this, but not the
| > asynchronous call modes).
| 
| Why do you say processing is in parallel?
| In Python, processing is
| normally never in parallel, but always sequential (potentially
| interleaving). Are you releasing the GIL for processing?


I mean that I can have multiple actual requests being served, and they
are _not_ handled sequentially - the multi queue is to permit more than
one to be in progress at once so that an expensive request need not
inherently delay a following cheap request.

And in my it's-a-filesystem mode there will often be a bottleneck in the
pure-python data scanning part. And thus the CPU bound python active
while still accepting requests in I/O blocked threads.

| > So, suppose a couple of CPU-intensive callables get queued which work for a
| > substantial time, and meanwhile a bunch of tiny tiny cheap requests arrive.
| > Their timely response will be impacted by this issue.
| 
| By how much exactly? What operating system?

Can't tell you - I'm not claiming the current GIL behaviour is biting me
right now; I'm saying your claim that the scenario is unrealistic is a
bit unfair; my app wil probably be just the kind of app to be affected,
even only subtly. And it can hardly be alone in the "daemon" world.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

There's two kinds of climbers...smart ones, and dead ones.      - Don Whillans

From cs at zip.com.au  Tue Mar 16 10:21:20 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 16 Mar 2010 20:21:20 +1100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4B9F3B21.1090106@v.loewis.de>
References: <4B9F3B21.1090106@v.loewis.de>
Message-ID: <20100316092120.GA6286@cskk.homeip.net>

On 16Mar2010 09:02, "Martin v. L?wis" <martin at v.loewis.de> wrote:
| > If it's lurking behind a filesystem interface or in its daemon mode
| > (remote archive store), multiple client processes can be using it at once,
| > and it will be processing multiple tasks somewhat in parallel. Here one
| > can get a compute bound thread answering one request, impacting quick
| > response to other parallel-and-cheap requests.
| 
| However, "impacting" will always be the case. *Of course* any thread
| that performs computation impacts everything else on the same processor.
| There is nothing to prevent that, unless you schedule the long-running
| activity at a point in time where you know the system will be idle for
| the entire run of that task.

Sure, but one has only to dip one's toe in the long running Linux
interactivity scheduling wars to know that there's a _lot_ of attraction
in a scheduling approach that favours the usually-blocked task over the
CPU bound task, and various approaches like the early low-latency patches
to the later in-the-scheduler changes were greeted with much rejoicing.

Certainly there's only so much CPU to go around, but if the tasks blocked
waiting for something to happen get to run more immediately when an event
does occur, you tend to get better "responsiveness".

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Be smart, be safe, be paranoid.
        - Ryan Cousineau, courier at compdyn.com DoD#863, KotRB, KotKWaWCRH

From ncoghlan at gmail.com  Tue Mar 16 10:44:27 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Mar 2010 19:44:27 +1000
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <20100315111654.1aa7acf5@heresy.wooz.org>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>	<20100303204923.6d597f03@heresy.wooz.org>	<4B8FB6D2.4040501@gmail.com>	<20100312145312.1f0a0aa0@heresy.wooz.org>	<4B9AC4CA.3070302@voidspace.org.uk>	<20100312175657.287bb456@heresy.wooz.org>	<4B9B9E65.4080902@gmail.com>
	<20100315111654.1aa7acf5@heresy.wooz.org>
Message-ID: <4B9F52FB.2090701@gmail.com>

Barry Warsaw wrote:
> On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote:
>> While it's probably OK if the import side-effects only create files
>> using the new scheme, the standard library modules will likely need to
>> support both schemes (although I'm not sure if "same as import system"
>> or "same as Python 3.1" make more sense as the default semantics -
>> probably the former).
> 
> I don't understand this point.
> 
> compileall probably /could/ be extended to understand bytecode-only
> (i.e. legacy or <3.2) layout.  I've added that to the PEP too.

In Python 3.1, *invoking* py_compile.compile() will create 2.x style
bytecode. Similarly, when force==False, compileall.compile_dir() and
compileall.compile_path() will check for 2.x style bytecode in order to
decide whether or not to compile the module.

The question for 3.2 is what bytecode layout py_compile.compile() should
generate. For the precompile-a-system-library use case it should clearly
generate a PEP 3147 layout and this probably makes sense as the default
behaviour in 3.2.

However, for production of bytecode-only packages, it would be
convenient to be able to explicitly invoke the 2.x style behaviour
without having to specify the target filename explicitly using the
'cfile' parameter (which isn't exposed at the compileall layer anyway).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Tue Mar 16 11:37:52 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Mar 2010 20:37:52 +1000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4B9F3A54.3040907@v.loewis.de>
References: <4B9DEFB8.3010107@v.loewis.de>	<20100316053445.GA11229@cskk.homeip.net>
	<4B9F3A54.3040907@v.loewis.de>
Message-ID: <4B9F5F80.7030102@gmail.com>

Martin v. L?wis wrote:
> Cameron Simpson wrote:
>> The idea here is that one has a few threads receiving requests (eg a
>> daemon watching a socket or monitoring a db queue table) which then use
>> the FuncMultiQueue to manage how many actual requests are processed
>> in parallel (yes, a semaphore can cover a lot of this, but not the
>> asynchronous call modes).
> 
> Why do you say processing is in parallel? In Python, processing is
> normally never in parallel, but always sequential (potentially
> interleaving). Are you releasing the GIL for processing?

We know the GIL is timeslicing - that's the whole point. The real
question is *how often* is it time slicing and *which threads* are
getting to run.

The key issue is that the current new GIL implementation means that, in
the presence of a CPU bound GIL-holding thread, the *minimum* duration
of *any* C call that releases the GIL becomes sys.getcheckinterval().
The CPU bound thread *always* wants the GIL and once it gets it, it
won't give it back until sys.getcheckinterval() expires. This is in
contrast to the I/O bound (or non-GIL holding CPU bound) threads that
nearly always yield the GIL early, since they're waiting for something
else to happen (be it I/O or a native calculation).

Antoine's interactiveness patch resolves this by rewarding threads that
regularly release the GIL early: the interpreter quickly recognises them
as doing so and they are subsequently given priority over the threads
that only release the GIL when the check interval expires. When threads
that are flagged as interactive ask for the GIL back, the interpreter
gives it to them immediately instead of forcing them to wait until the
check interval expires (because it trusts those threads not to hang onto
the GIL for too long).

I like this significantly better than the explicit priority patch,
because it means threads are given priority based on their behaviour
(i.e. frequently holding the GIL for less than the check interval)
rather than relying on developers to correctly decide between using a
normal GIL request and a priority request.

Handling of thread pools that mix I/O bound and CPU bound GIL-holding
tasks isn't quite optimal with this approach (since the nature of any
given thread will change over time), but it isn't terrible either - the
interpreter doesn't need much time to reclassify a thread as interactive
or non-interactive (and there are a couple of parameters that can be
tuned to control how quickly these changes in status can happen).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From kristjan at ccpgames.com  Tue Mar 16 11:52:28 2010
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Tue, 16 Mar 2010 10:52:28 +0000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <49653.1268622425@dabeaz.com>
References: <49653.1268622425@dabeaz.com>
Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>

How about attacking the original problem, then?

The reason they thrash on pthreads implementation is that a pthreads mutex is assumed to be a short-held resource.  Therefore it will be optimized in the following ways for multicore machines:
1) There is a certain amount of spinning done, to try to acquire it before blocking
2) It will employ un-fair tactics to avoid lock-convoying, meaning that a thread coming in to acquire the mutex may get in before others that are queued.  This is why "ticking" the GIL works so badly:  The thread that releases the lock is usually the one that reaquires it even though others may be waiting.  See e.g. http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspx for a discussion of this (albeit on windows.)

On Windows, this isn't a problem.  The reason is, that the GIL on windows is implemented using Event objects that don't cut these corners.  The Event provides you with a strict FIFO queue of objects waiting for the event.

If pthreads doesn't provide a synchronization primitive similar to that, someone that doesn't thrash and has a true FIFO queue, it is possible to construct such a thing using condition variables and critical sections.  Perhaps the posix semaphore api is more appropriate in this case.

By the way, this also shows another problem with (old) python.  There is only one core locking primitive, the PyThread_type_lock.  It is being used both as a critical section in the traditional sense, and also as this sort-of-inverse lock that the GIL is.  In the modern world, where the intended behaviour of these is quite different, there is no one-size-fits all.  On windows in particular, the use of the Event object based lock is not ideal for other uses than the GIL.


In the new GIL, there appear to be several problems:
1) There is no FIFO queue of threads wanting the queue, thus thread scheduling becomes non-deterministic
2) The "ticking" of the GIL is now controled by a condition variable timeout.  There appears to be no way to prevent many such timeouts to be in progress at the same time, thus you may have an unnecessarily high rate of ticking going on.
3) There isn't an immediate gil request made when an IO thread requests the gil back, only after an initial timeout.

What we are trying to write here is a thread scheduler, and that is complex business.
K



> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
> Of David Beazley
> Sent: 15. mars 2010 03:07
> To: python-dev at python.org
> Subject: Re: [Python-Dev] "Fixing" the new GIL
> 
> happen to be performing CPU intensive work at the same time, it would
> be nice if they didn't thrash on multiple cores (the problem with the
> old GIL) and if I/O is


From dave at dabeaz.com  Tue Mar 16 12:02:32 2010
From: dave at dabeaz.com (David Beazley)
Date: Tue, 16 Mar 2010 06:02:32 -0500
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
Message-ID: <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com>

Python doesn't use a pthreads mutex for the GIL.    It has always used a binary semaphore implemented with condition variables (or just a pthreads semaphore if available).    The reason the performance is so bad is precisely due to the fact that it is using this implementation and the fact that there *IS* a FIFO queue of threads (associated with the condition variable).   The I/O performance problem with the new GIL is gets much worse with many CPU-bound threads precisely because there is a FIFO queue involved.   This has been covered in my past GIL presentations.

-Dave



On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote:

> How about attacking the original problem, then?
> 
> The reason they thrash on pthreads implementation is that a pthreads mutex is assumed to be a short-held resource.  Therefore it will be optimized in the following ways for multicore machines:
> 1) There is a certain amount of spinning done, to try to acquire it before blocking
> 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a thread coming in to acquire the mutex may get in before others that are queued.  This is why "ticking" the GIL works so badly:  The thread that releases the lock is usually the one that reaquires it even though others may be waiting.  See e.g. http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspx for a discussion of this (albeit on windows.)
> 
> On Windows, this isn't a problem.  The reason is, that the GIL on windows is implemented using Event objects that don't cut these corners.  The Event provides you with a strict FIFO queue of objects waiting for the event.
> 
> If pthreads doesn't provide a synchronization primitive similar to that, someone that doesn't thrash and has a true FIFO queue, it is possible to construct such a thing using condition variables and critical sections.  Perhaps the posix semaphore api is more appropriate in this case.
> 
> By the way, this also shows another problem with (old) python.  There is only one core locking primitive, the PyThread_type_lock.  It is being used both as a critical section in the traditional sense, and also as this sort-of-inverse lock that the GIL is.  In the modern world, where the intended behaviour of these is quite different, there is no one-size-fits all.  On windows in particular, the use of the Event object based lock is not ideal for other uses than the GIL.
> 
> 
> In the new GIL, there appear to be several problems:
> 1) There is no FIFO queue of threads wanting the queue, thus thread scheduling becomes non-deterministic
> 2) The "ticking" of the GIL is now controled by a condition variable timeout.  There appears to be no way to prevent many such timeouts to be in progress at the same time, thus you may have an unnecessarily high rate of ticking going on.
> 3) There isn't an immediate gil request made when an IO thread requests the gil back, only after an initial timeout.
> 
> What we are trying to write here is a thread scheduler, and that is complex business.
> K
> 
> 
> 
>> -----Original Message-----
>> From: python-dev-bounces+kristjan=ccpgames.com at python.org
>> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
>> Of David Beazley
>> Sent: 15. mars 2010 03:07
>> To: python-dev at python.org
>> Subject: Re: [Python-Dev] "Fixing" the new GIL
>> 
>> happen to be performing CPU intensive work at the same time, it would
>> be nice if they didn't thrash on multiple cores (the problem with the
>> old GIL) and if I/O is
> 


From solipsis at pitrou.net  Tue Mar 16 13:00:11 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 16 Mar 2010 08:00:11 -0400
Subject: [Python-Dev] "Fixing" the new GIL
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
Message-ID: <20100316080011.34589b00@msiwind>


Hi Kristj?n,

> In the new GIL, there appear to be several problems:
> 1) There is no FIFO queue of threads wanting the queue, thus thread
> scheduling becomes non-deterministic

Thread scheduling has always been non-deterministic.

> 2) The "ticking" of the GIL is
> now controled by a condition variable timeout.  There appears to be
> no way to prevent many such timeouts to be in progress at the same
> time, thus you may have an unnecessarily high rate of ticking going
> on.

Unless there's a bug, no, there isn't.

> 3) There isn't an immediate gil request made when an IO thread
> requests the gil back, only after an initial timeout.

This is what we are looking to fix (perhaps).

> What we are trying to write here is a thread scheduler, and that is
> complex business. K

I would have rephrased:
"What we are trying to avoid here is a thread scheduler".

regards

Antoine.



From kristjan at ccpgames.com  Tue Mar 16 14:46:08 2010
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Tue, 16 Mar 2010 13:46:08 +0000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com>
Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A1821825F@exchis.ccp.ad.local>

You are right, I was assuming a mutex. (why not then, use a semaphore?)
But the implementation isn't a simple binary semaphore.
It suffers from case 2) that I explained before:  Greedy aquisition of the "semaphore" even if someone is waiting.
Also, I should add, a condition variable doesn't specify that it employs a FIFO.  pythread_cond_signals() wakes up _a_ thread.  If you want particular behaviour then you have to add code to deal with that.  But it tries to be fair, and we can probably assum e FIFO behavioud, so let's ignore that part.  So, in the following, let's assume that the condition variable is fair, and that it queues up the waiting threads in a FIFO manner:

Fixing 2) in this case is easy: Make sure you wait if others are waiting.  In python pseudocode:
def Aquire(gil):
  with gil.condition:
    if gil.locked or gil.n_waiting:
      gil.n_waiting+=1
        while gil.locked:
          gil.condition.wait()
      gil.n_waiting-=1
    gil.locked = True

def Release(gil):
  with gil.condition:
    gil.locked=False
    if gil.n_waiting:
      gil.condition.notify()

This ought to fix problem 2) below.  Multicore CPUs ought to not behave worse than single core anymore.
(Btw, the C code is erroneous, one should not call pthread_cond_signal without having the lock held, but that has been brought up before.)

Now, if you want to improve IO performance, by ensuring that IO threads get woken up before (yielding) cpu threads, you can do something like this:
class GIL():
  def __init__(self, npri):
    self.lock=RLock()
    self.queues = [[Condition(self.lock), 0] for i in range(npri)]
    self.locked = False

  def Acquire(self, pri):
    """Acquire the lock with priority "pri", where 0 is highest priority"""
    with self.lock:
    if self.locked or sum([q[1] for q in self.queues[:pri+1]]) #locked or someone with same or higher priority waiting
      self.queues[pri][1]+=1
      while self.locked:
        self.queues[pri][0].wait()
      self.queues[pri][1]-=1
    self.locked = True

  def Release(self):
    with self.lock:
      self.locked = False
      for q in self.queues:
        if q[1]:
          q[0].notify()
          break

  def Blip(self):
    """volountailily give up the lock and allow higher priority requests to run"""
    self.Release()
    self.Acquire(len(self.queues)-1)

Typically you would have only two priority classes. IO would then be done with:
gil.Release()
r = socket.recv(l)
gil.Acquire(0)

But here we are on dangerous grounds. Priority scheduling is full of pitfalls.
I should add here maybe, that I am a strong proponent of strict FIFO scheduling with perhaps few special cases.  We have been using FIFO tasklet scheduling for all tasklets, including those that are returing with IO, in EVE Online for three years now.  Before, we had somewhat chaotic scheduling (immediate switching to tasklets) and behavior was very erratic.  Switching to FIFO cured that and was very wholesome for the cluster in general..  I have been toying with the idea of adding a special IO tasklet priority class to stackless for a while, but haven't done it yet.  I am doubtful that it will on the whole change matters much.
        
Cheers,

Kristj?n


> -----Original Message-----
> From: David Beazley [mailto:dave at dabeaz.com]
> Sent: 16. mars 2010 11:03
> To: Kristj?n Valur J?nsson
> Cc: David Beazley; python-dev at python.org
> Subject: Re: [Python-Dev] "Fixing" the new GIL
> 
> Python doesn't use a pthreads mutex for the GIL.    It has always used
> a binary semaphore implemented with condition variables (or just a
> pthreads semaphore if available).    The reason the performance is so
> bad is precisely due to the fact that it is using this implementation
> and the fact that there *IS* a FIFO queue of threads (associated with
> the condition variable).   The I/O performance problem with the new GIL
> is gets much worse with many CPU-bound threads precisely because there
> is a FIFO queue involved.   This has been covered in my past GIL
> presentations.
> 
> -Dave
> 
> 
> 
> On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote:
> 
> > How about attacking the original problem, then?
> >
> > The reason they thrash on pthreads implementation is that a pthreads
> mutex is assumed to be a short-held resource.  Therefore it will be
> optimized in the following ways for multicore machines:
> > 1) There is a certain amount of spinning done, to try to acquire it
> before blocking
> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning
> that a thread coming in to acquire the mutex may get in before others
> that are queued.  This is why "ticking" the GIL works so badly:  The
> thread that releases the lock is usually the one that reaquires it even
> though others may be waiting.  See e.g.
> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-
> 8f85-616ef7b031aa.aspx for a discussion of this (albeit on windows.)
> >
> > On Windows, this isn't a problem.  The reason is, that the GIL on
> windows is implemented using Event objects that don't cut these
> corners.  The Event provides you with a strict FIFO queue of objects
> waiting for the event.
> >
> > If pthreads doesn't provide a synchronization primitive similar to
> that, someone that doesn't thrash and has a true FIFO queue, it is
> possible to construct such a thing using condition variables and
> critical sections.  Perhaps the posix semaphore api is more appropriate
> in this case.
> >
> > By the way, this also shows another problem with (old) python.  There
> is only one core locking primitive, the PyThread_type_lock.  It is
> being used both as a critical section in the traditional sense, and
> also as this sort-of-inverse lock that the GIL is.  In the modern
> world, where the intended behaviour of these is quite different, there
> is no one-size-fits all.  On windows in particular, the use of the
> Event object based lock is not ideal for other uses than the GIL.
> >
> >
> > In the new GIL, there appear to be several problems:
> > 1) There is no FIFO queue of threads wanting the queue, thus thread
> scheduling becomes non-deterministic
> > 2) The "ticking" of the GIL is now controled by a condition variable
> timeout.  There appears to be no way to prevent many such timeouts to
> be in progress at the same time, thus you may have an unnecessarily
> high rate of ticking going on.
> > 3) There isn't an immediate gil request made when an IO thread
> requests the gil back, only after an initial timeout.
> >
> > What we are trying to write here is a thread scheduler, and that is
> complex business.
> > K
> >
> >
> >
> >> -----Original Message-----
> >> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> >> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On
> Behalf
> >> Of David Beazley
> >> Sent: 15. mars 2010 03:07
> >> To: python-dev at python.org
> >> Subject: Re: [Python-Dev] "Fixing" the new GIL
> >>
> >> happen to be performing CPU intensive work at the same time, it
> would
> >> be nice if they didn't thrash on multiple cores (the problem with
> the
> >> old GIL) and if I/O is
> >
> 


From kristjan at ccpgames.com  Tue Mar 16 15:10:33 2010
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Tue, 16 Mar 2010 14:10:33 +0000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100316080011.34589b00@msiwind>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<20100316080011.34589b00@msiwind>
Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local>



> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
> Of Antoine Pitrou
> Sent: 16. mars 2010 12:00
> To: python-dev at python.org
> Subject: Re: [Python-Dev] "Fixing" the new GIL
> 
> 
> Hi Kristj?n,
> 
> > In the new GIL, there appear to be several problems:
> > 1) There is no FIFO queue of threads wanting the queue, thus thread
> > scheduling becomes non-deterministic
> 
> Thread scheduling has always been non-deterministic.
Yes, but not the scheduling of "which thread has the GIL" which is what we are talking about.  With a gil, you have neutered the non-deterministic OS scheduler and forced the system to _effectively_ schedule threads as you with them to be scheduled.  The key to this is implementing your GIL in such a way that you (and not the system) chooses which threads runs next.  On Windows it works in a nice, determinitstic FIFO order becoause that's how the underlying Event object is supposed to work.  under pthreads, we rely on the behaviour of the condition variable which _probably_ is well behaved in this manner too.

> 
> > 2) The "ticking" of the GIL is
> > now controled by a condition variable timeout.  There appears to be
> > no way to prevent many such timeouts to be in progress at the same
> > time, thus you may have an unnecessarily high rate of ticking going
> > on.
> 
> Unless there's a bug, no, there isn't.
You're right, I had forgotten about "gil_switch_number".  Never mind.

> 
> > 3) There isn't an immediate gil request made when an IO thread
> > requests the gil back, only after an initial timeout.
> 
> This is what we are looking to fix (perhaps).
> 
> > What we are trying to write here is a thread scheduler, and that is
> > complex business. K
> 
> I would have rephrased:
> "What we are trying to avoid here is a thread scheduler".

I can't argue with that, but I'm afraid that in the end, you have to.  Python threads do behave very much in the same way as stackless python tasklets do.  They own the cpu until they volounteeringly give it up.  You want to decide, then, which of the threads next gets its turn.  You are working on a completely different level than what the OS usually uses for its scheduling (timeslices, IO) so _you_ have to be in control.  By a happy coincidence, in the past on a single CPU, the old GIL did give us this, without us being explicit about it:  FIFO execution of tasklets.  This fell out as a byproduct of the very simple way that the GIL was implemented, using a single Condition variable.  Now that we have a different mechanism, _where threads time out_ on the condition, this behaviour no longer falls out of the implementation on its own.  It may turn out, in this new, more complicated world, that you have to explicitly schedule the threads, by chaining them and giving each their own condition to wait on.

Btw, if you want to take up a private converstaion with me, I'd like to point out to you some problems with the Windows version of the Condition variable :)

Cheers,

Kristj?n


From solipsis at pitrou.net  Tue Mar 16 15:16:23 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 16 Mar 2010 15:16:23 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<20100316080011.34589b00@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local>
Message-ID: <20100316151623.05e8e570@msiwind>

Le Tue, 16 Mar 2010 14:10:33 +0000,
Kristj?n Valur J?nsson <kristjan at ccpgames.com> a ?crit :
> 
> The key to this
> is implementing your GIL in such a way that you (and not the system)
> chooses which threads runs next.  On Windows it works in a nice,
> determinitstic FIFO order becoause that's how the underlying Event
> object is supposed to work.

Well, I don't think this has ever been by design, and it's not obvious
this is desirable either (see Dave Beazley's benchmark).

> Btw, if you want to take up a private converstaion with me, I'd like
> to point out to you some problems with the Windows version of the
> Condition variable :)

You can report a bug.

cheers

Antoine.

From dickinsm at gmail.com  Tue Mar 16 15:41:26 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 16 Mar 2010 14:41:26 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
Message-ID: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>

Hello all,

Currently in Python 2.x, Decimal-to-float comparisons behave as follows:

>>> Decimal(1) < float(4)
False
>>> Decimal(4) < float(1)
False

That is, any Decimal sorts before any float (though this is arbitrary:
 it's possible that on some implementations any float sorts before any
Decimal).  This causes (a) confusion, and (b) bugs, especially when
floats and Decimals are accidentally combined in a program.  There
probably aren't too many legitimate reasons for deliberately mixing
floats and Decimals in the same calculation, so preventing accidents
is the main concern here.

In Python 3.x, however, such comparisons raise a TypeError
("unorderable types: Decimal() < float()").

http://bugs.python.org/issue2531 ('float compared to decimal is
silently incorrect') was opened for this a while ago.

I'm planning to commit a change to trunk that changes the behaviour so
that the comparison result is based on the respective values of the
arguments (so the results above would be True and False respectively).

Question for python-dev people (and the point of this email): should
this change be forward ported to py3k?

On the one hand there's something to be said for maintaining a clean
separation between the float and Decimal types, allowing only explicit
conversions from one to the other;  mixed-type arithmetic between
floats and Decimals was very deliberately not permitted in the
original PEP, and that's unlikely to change in a hurry.  On the other
hand, there's value in keeping 2.x and 3.x aligned where possible for
the sake of 2-to-3 porters, and the new behaviour may even be useful.
Even with the TypeError above, there are still some py3k surprises
arising from the ability to compare ints and Decimals, and ints and
floats, but not floats and Decimals.

A quick tour of some of these surprises, in trunk:

>>> from decimal import Decimal
>>> Decimal(1) < 2 < float(3) < Decimal(1)  # < is non-transitive
True
>>> Decimal(1) == 1 == float(1)   # so is equality
True
>>> Decimal(1) == float(1)
False
>>> d1, i1, f1 = Decimal(1), float(1), 1
>>> set([d1, i1, f1]) == set([f1, i1, d1])  # sets with the same elements are different
False
>>> sorted([d1, i1, f1]) == sorted([f1, i1, d1])
False

and in py3k:

>>> from decimal import Decimal
>>> Decimal(1) < 2 < float(3) < Decimal(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: float() < Decimal()
>>> Decimal(1) == 1 == float(1)
True
>>> Decimal(1) == float(1)
False
>>> d1, i1, f1 = Decimal(1), float(1), 1
>>> set([d1, i1, f1]) == set([f1, i1, d1])
False
>>> sorted([Decimal(1), 2, float(3)])
[Decimal('1'), 2, 3.0]
>>> sorted([2, Decimal(1), float(3)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: float() < Decimal()
>>> sorted([float(3), 2, Decimal(1)])
[Decimal('1'), 2, 3.0]

By the way, even with the patch there are still problems with other
numeric types: comparisons or set operations involving both Fraction
and Decimal instances are going to cause similar difficulties to those
above.  In practice I think this is much less of an issue than the
float/Decimal problem, since the chance of accidentally combining
Fraction and Decimal types in a calculation seems significantly
smaller than the chance of accidentally combining float and Decimal
types.

--
Mark

From facundobatista at gmail.com  Tue Mar 16 15:58:45 2010
From: facundobatista at gmail.com (Facundo Batista)
Date: Tue, 16 Mar 2010 09:58:45 -0500
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
Message-ID: <e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>

On Tue, Mar 16, 2010 at 9:41 AM, Mark Dickinson <dickinsm at gmail.com> wrote:

> On the one hand there's something to be said for maintaining a clean
> separation between the float and Decimal types, allowing only explicit
> conversions from one to the other; ?mixed-type arithmetic between
> floats and Decimals was very deliberately not permitted in the
> original PEP, and that's unlikely to change in a hurry. ?On the other

But, to be fair, we didn't have "true value of the float at that time".

I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
"Decimal(1) < Decimal.from_float(.3)"

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From janssen at parc.com  Tue Mar 16 16:11:39 2010
From: janssen at parc.com (Bill Janssen)
Date: Tue, 16 Mar 2010 08:11:39 PDT
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4B9F3B21.1090106@v.loewis.de>
References: <4B9D2BD4.8010307@v.loewis.de>
	<20100316053847.GA12552@cskk.homeip.net>
	<4B9F3B21.1090106@v.loewis.de>
Message-ID: <82842.1268752299@parc.com>

Martin v. L?wis <martin at v.loewis.de> wrote:

> > If it's lurking behind a filesystem interface or in its daemon mode
> > (remote archive store), multiple client processes can be using it at once,
> > and it will be processing multiple tasks somewhat in parallel. Here one
> > can get a compute bound thread answering one request, impacting quick
> > response to other parallel-and-cheap requests.
> 
> However, "impacting" will always be the case. *Of course* any thread
> that performs computation impacts everything else on the same processor.

That's not the problem, Martin.  The problem is that it also impacts
other threads on other cores, because of the interlock between the OS
thread scheduler and the internal Pythonic struggle to obtain the GIL.
And it shouldn't.

Bill

From dickinsm at gmail.com  Tue Mar 16 16:17:47 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 16 Mar 2010 15:17:47 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
Message-ID: <5c6f2a5d1003160817jdfea407q4af516f1770bda19@mail.gmail.com>

On Tue, Mar 16, 2010 at 2:58 PM, Facundo Batista
<facundobatista at gmail.com> wrote:
> On Tue, Mar 16, 2010 at 9:41 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>
>> On the one hand there's something to be said for maintaining a clean
>> separation between the float and Decimal types, allowing only explicit
>> conversions from one to the other; ?mixed-type arithmetic between
>> floats and Decimals was very deliberately not permitted in the
>> original PEP, and that's unlikely to change in a hurry. ?On the other
>
> But, to be fair, we didn't have "true value of the float at that time".

That's true.  I'd still be reluctant to start supporting operations
like Decimal('1.2') + 0.71, though.  At least comparisons have the
nice feature that the return type is unambiguous.

>
> I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
> "Decimal(1) < Decimal.from_float(.3)"

Yes, I should have clarified that those are exactly the semantics I'm proposing.

Thanks!

Mark

From pje at telecommunity.com  Tue Mar 16 16:58:22 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 16 Mar 2010 11:58:22 -0400
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.co
 m>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
Message-ID: <20100316155815.DAE983A4097@sparrow.telecommunity.com>

At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote:
>I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
>"Decimal(1) < Decimal.from_float(.3)"

Does Decimal.from_float() use the "shortest decimal representation" approach?

If not, it might be confusing if a number that prints as '.1' 
compares unequal to Decimal('.1').


From dickinsm at gmail.com  Tue Mar 16 17:11:38 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 16 Mar 2010 16:11:38 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <20100316155815.DAE983A4097@sparrow.telecommunity.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
	<20100316155815.DAE983A4097@sparrow.telecommunity.com>
Message-ID: <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>

On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote:
>>
>> I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
>> "Decimal(1) < Decimal.from_float(.3)"
>
> Does Decimal.from_float() use the "shortest decimal representation"
> approach?

No.  It does exact conversions:

>>> Decimal.from_float(1.1)
Decimal('1.100000000000000088817841970012523233890533447265625')
>>> Decimal.from_float(1.1) == 1.1
False
>>> Decimal('1.1') == float('1.1')  # returns False both pre- and post-patch
False

> If not, it might be confusing if a number that prints as '.1' compares
> unequal to Decimal('.1').

Agreed, but this is just your everyday floating-point confusion, to be
dealt with by social means (e.g., educating the programmer).  Any
technical solution that made "Decimal('1.1') == float('1.1')" evaluate
to True would, I suspect, be a cure worse than the original disease.

Mark

From victor.stinner at haypocalc.com  Tue Mar 16 17:18:50 2010
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 16 Mar 2010 17:18:50 +0100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <20100316155815.DAE983A4097@sparrow.telecommunity.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
	<20100316155815.DAE983A4097@sparrow.telecommunity.com>
Message-ID: <201003161718.50634.victor.stinner@haypocalc.com>

Le mardi 16 mars 2010 16:58:22, P.J. Eby a ?crit :
> At 09:58 AM 3/16/2010 -0500, Facundo Batista wrote:
> >I'm +0 to allow these comparisons, being "Decimal(1) < .3" the same as
> >"Decimal(1) < Decimal.from_float(.3)"
> 
> Does Decimal.from_float() use the "shortest decimal representation"
>  approach?
> 
> If not, it might be confusing if a number that prints as '.1'
> compares unequal to Decimal('.1').

In py3k, comparing bytes and str raise a TypeError("unorderable types: bytes() 
< str()"). I like this behaviour :-)

If comparaison of Decimal and float can have "unpredictable" result, I would 
suggest the same behaviour (raise an error).

-- 
Victor Stinner
http://www.haypocalc.com/

From dickinsm at gmail.com  Tue Mar 16 17:23:30 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 16 Mar 2010 16:23:30 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
	<20100316155815.DAE983A4097@sparrow.telecommunity.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
Message-ID: <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com>

On Tue, Mar 16, 2010 at 4:11 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
[...]

>>>> Decimal.from_float(1.1) == 1.1
> False

Whoops.  To clarify, this is the pre-patch behaviour;  post-patch,
this gives True.

Mark

From dickinsm at gmail.com  Tue Mar 16 17:25:32 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 16 Mar 2010 16:25:32 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003161718.50634.victor.stinner@haypocalc.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
	<20100316155815.DAE983A4097@sparrow.telecommunity.com>
	<201003161718.50634.victor.stinner@haypocalc.com>
Message-ID: <5c6f2a5d1003160925n637b9987t9af2c0bd02330bd3@mail.gmail.com>

On Tue, Mar 16, 2010 at 4:18 PM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> If comparaison of Decimal and float can have "unpredictable" result, I would
> suggest the same behaviour (raise an error).

Well, it's not really `unpredictable':  the new behaviour is perfectly
predictable and sane, provided only that you remember the basic fact
that float("0.1") is not exactly 0.1.

Mark

From guido at python.org  Tue Mar 16 17:41:48 2010
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Mar 2010 08:41:48 -0800
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
Message-ID: <ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>

I'd say if you're not going to forward-port this to Python 3, it
shouldn't go into Python 2 -- in that case it would make more sense to
me to back-port the exception-raising behavior.

Also supporting comparisons but not other mixed operations is going to
be confusing. If you are sticking to that behavior I think mixed
comparisons should also be ruled out.

--Guido

On Tue, Mar 16, 2010 at 7:41 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> Hello all,
>
> Currently in Python 2.x, Decimal-to-float comparisons behave as follows:
>
>>>> Decimal(1) < float(4)
> False
>>>> Decimal(4) < float(1)
> False
>
> That is, any Decimal sorts before any float (though this is arbitrary:
> ?it's possible that on some implementations any float sorts before any
> Decimal). ?This causes (a) confusion, and (b) bugs, especially when
> floats and Decimals are accidentally combined in a program. ?There
> probably aren't too many legitimate reasons for deliberately mixing
> floats and Decimals in the same calculation, so preventing accidents
> is the main concern here.
>
> In Python 3.x, however, such comparisons raise a TypeError
> ("unorderable types: Decimal() < float()").
>
> http://bugs.python.org/issue2531 ('float compared to decimal is
> silently incorrect') was opened for this a while ago.
>
> I'm planning to commit a change to trunk that changes the behaviour so
> that the comparison result is based on the respective values of the
> arguments (so the results above would be True and False respectively).
>
> Question for python-dev people (and the point of this email): should
> this change be forward ported to py3k?
>
> On the one hand there's something to be said for maintaining a clean
> separation between the float and Decimal types, allowing only explicit
> conversions from one to the other; ?mixed-type arithmetic between
> floats and Decimals was very deliberately not permitted in the
> original PEP, and that's unlikely to change in a hurry. ?On the other
> hand, there's value in keeping 2.x and 3.x aligned where possible for
> the sake of 2-to-3 porters, and the new behaviour may even be useful.
> Even with the TypeError above, there are still some py3k surprises
> arising from the ability to compare ints and Decimals, and ints and
> floats, but not floats and Decimals.
>
> A quick tour of some of these surprises, in trunk:
>
>>>> from decimal import Decimal
>>>> Decimal(1) < 2 < float(3) < Decimal(1) ?# < is non-transitive
> True
>>>> Decimal(1) == 1 == float(1) ? # so is equality
> True
>>>> Decimal(1) == float(1)
> False
>>>> d1, i1, f1 = Decimal(1), float(1), 1
>>>> set([d1, i1, f1]) == set([f1, i1, d1]) ?# sets with the same elements are different
> False
>>>> sorted([d1, i1, f1]) == sorted([f1, i1, d1])
> False
>
> and in py3k:
>
>>>> from decimal import Decimal
>>>> Decimal(1) < 2 < float(3) < Decimal(1)
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> TypeError: unorderable types: float() < Decimal()
>>>> Decimal(1) == 1 == float(1)
> True
>>>> Decimal(1) == float(1)
> False
>>>> d1, i1, f1 = Decimal(1), float(1), 1
>>>> set([d1, i1, f1]) == set([f1, i1, d1])
> False
>>>> sorted([Decimal(1), 2, float(3)])
> [Decimal('1'), 2, 3.0]
>>>> sorted([2, Decimal(1), float(3)])
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> TypeError: unorderable types: float() < Decimal()
>>>> sorted([float(3), 2, Decimal(1)])
> [Decimal('1'), 2, 3.0]
>
> By the way, even with the patch there are still problems with other
> numeric types: comparisons or set operations involving both Fraction
> and Decimal instances are going to cause similar difficulties to those
> above. ?In practice I think this is much less of an issue than the
> float/Decimal problem, since the chance of accidentally combining
> Fraction and Decimal types in a calculation seems significantly
> smaller than the chance of accidentally combining float and Decimal
> types.
>
> --
> Mark
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From kristjan at ccpgames.com  Tue Mar 16 18:02:13 2010
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Tue, 16 Mar 2010 17:02:13 +0000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100316151623.05e8e570@msiwind>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<20100316080011.34589b00@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local>
	<20100316151623.05e8e570@msiwind>
Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local>



> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
> Of Antoine Pitrou
> > The key to this
> > is implementing your GIL in such a way that you (and not the system)
> > chooses which threads runs next.  On Windows it works in a nice,
> > determinitstic FIFO order becoause that's how the underlying Event
> > object is supposed to work.
> 
> Well, I don't think this has ever been by design, and it's not obvious
> this is desirable either (see Dave Beazley's benchmark).
>
Did Dave benchmark a straight FIFO system?
I can tell you out of experience, engineering a similar system (stacklessIO) that you _do_ want a deterministic scheduling algorithm.  The most straightforward is round-robin (or FIFO), and the only sane option really if you know nothing about your threads.  Upon this, you can start to build, by for example adding another priority layer, but you have to have a solid foundation.  We have never had high priority for IO threads in python (and its not-by-design round robin scheduler on single core cpus) and it is unclear why that should be required now as a fix.

 
> > Btw, if you want to take up a private converstaion with me, I'd like
> > to point out to you some problems with the Windows version of the
> > Condition variable :)
> 
> You can report a bug.
Or I could commit a fix.  I just thought You might be interested, is all.  I'll submit a patch.

K

From dickinsm at gmail.com  Tue Mar 16 18:07:48 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 16 Mar 2010 17:07:48 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
Message-ID: <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com>

On Tue, Mar 16, 2010 at 4:41 PM, Guido van Rossum <guido at python.org> wrote:
> I'd say if you're not going to forward-port this to Python 3, it
> shouldn't go into Python 2 -- in that case it would make more sense to
> me to back-port the exception-raising behavior.

That's also a possible solution, and the one that I'd personally be
happiest with.  The main problem is that this has the potential to
break code:  lists containing both floats and Decimals are sortable in
2.6, but would no longer be sortable in 2.7.  If such breakage is
deemed acceptable then I'd happily backport the exception;  I really
don't have a good feeling for how much real-world code could break, if
any.

> Also supporting comparisons but not other mixed operations is going to
> be confusing. If you are sticking to that behavior I think mixed
> comparisons should also be ruled out.

Confusing, yes, but at least not bug-prone.  The current 2.x behaviour
has provoked complaints from a number of different people in various
different fora (I recently saw this come up on StackOverflow), and
after initially being skeptical I'm now convinced that it would be a
good idea to change it if at all possible.

Mark

From guido at python.org  Tue Mar 16 18:15:28 2010
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Mar 2010 09:15:28 -0800
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> 
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com> 
	<5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com>
Message-ID: <ca471dc21003161015k5eede772ucf9286bddd270eda@mail.gmail.com>

On Tue, Mar 16, 2010 at 9:07 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Tue, Mar 16, 2010 at 4:41 PM, Guido van Rossum <guido at python.org> wrote:
>> I'd say if you're not going to forward-port this to Python 3, it
>> shouldn't go into Python 2 -- in that case it would make more sense to
>> me to back-port the exception-raising behavior.
>
> That's also a possible solution, and the one that I'd personally be
> happiest with. ?The main problem is that this has the potential to
> break code: ?lists containing both floats and Decimals are sortable in
> 2.6, but would no longer be sortable in 2.7. ?If such breakage is
> deemed acceptable then I'd happily backport the exception; ?I really
> don't have a good feeling for how much real-world code could break, if
> any.

Definitely some. Stricter comparison rules are a frequent cause of
problems when code is first ported to 3.x. While you'd think that code
comparing a float and a Decimal is *already* broken, there's a
surprising number of situations where that's not necessary the case,
e.g. when an arbitrary but stable ordering is needed.

>> Also supporting comparisons but not other mixed operations is going to
>> be confusing. If you are sticking to that behavior I think mixed
>> comparisons should also be ruled out.
>
> Confusing, yes, but at least not bug-prone. ?The current 2.x behaviour
> has provoked complaints from a number of different people in various
> different fora (I recently saw this come up on StackOverflow), and
> after initially being skeptical I'm now convinced that it would be a
> good idea to change it if at all possible.

Yeah, it should have raised an exception all along. But it's too late
for that now. I wonder if it should just become a py3k warning?

-- 
--Guido van Rossum (python.org/~guido)

From nir at winpdb.org  Tue Mar 16 19:11:00 2010
From: nir at winpdb.org (Nir Aides)
Date: Tue, 16 Mar 2010 20:11:00 +0200
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com>
Message-ID: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com>

Hello Dave,

The following documentation suggests ordering in Linux is not FIFO:
http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html#tag_03_518_08_06
"Threads waiting on mutexes and condition variables are selected to proceed
in an order dependent upon the scheduling policy rather than in some fixed
order (for example, FIFO or priority). Thus, the scheduling policy
determines which thread(s) are awakened and allowed to proceed."

Here is the code:
http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz

If this is so then it should affect the proposed fixes.
For example waiting with timeout should be avoided, no?

Nir

2010/3/16 David Beazley <dave at dabeaz.com>

> Python doesn't use a pthreads mutex for the GIL.    It has always used a
> binary semaphore implemented with condition variables (or just a pthreads
> semaphore if available).    The reason the performance is so bad is
> precisely due to the fact that it is using this implementation and the fact
> that there *IS* a FIFO queue of threads (associated with the condition
> variable).   The I/O performance problem with the new GIL is gets much worse
> with many CPU-bound threads precisely because there is a FIFO queue
> involved.   This has been covered in my past GIL presentations.
>
> -Dave
>
>
>
> On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote:
>
> > How about attacking the original problem, then?
> >
> > The reason they thrash on pthreads implementation is that a pthreads
> mutex is assumed to be a short-held resource.  Therefore it will be
> optimized in the following ways for multicore machines:
> > 1) There is a certain amount of spinning done, to try to acquire it
> before blocking
> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a
> thread coming in to acquire the mutex may get in before others that are
> queued.  This is why "ticking" the GIL works so badly:  The thread that
> releases the lock is usually the one that reaquires it even though others
> may be waiting.  See e.g.
> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor a discussion of this (albeit on windows.)
> >
> > On Windows, this isn't a problem.  The reason is, that the GIL on windows
> is implemented using Event objects that don't cut these corners.  The Event
> provides you with a strict FIFO queue of objects waiting for the event.
> >
> > If pthreads doesn't provide a synchronization primitive similar to that,
> someone that doesn't thrash and has a true FIFO queue, it is possible to
> construct such a thing using condition variables and critical sections.
>  Perhaps the posix semaphore api is more appropriate in this case.
> >
> > By the way, this also shows another problem with (old) python.  There is
> only one core locking primitive, the PyThread_type_lock.  It is being used
> both as a critical section in the traditional sense, and also as this
> sort-of-inverse lock that the GIL is.  In the modern world, where the
> intended behaviour of these is quite different, there is no one-size-fits
> all.  On windows in particular, the use of the Event object based lock is
> not ideal for other uses than the GIL.
> >
> >
> > In the new GIL, there appear to be several problems:
> > 1) There is no FIFO queue of threads wanting the queue, thus thread
> scheduling becomes non-deterministic
> > 2) The "ticking" of the GIL is now controled by a condition variable
> timeout.  There appears to be no way to prevent many such timeouts to be in
> progress at the same time, thus you may have an unnecessarily high rate of
> ticking going on.
> > 3) There isn't an immediate gil request made when an IO thread requests
> the gil back, only after an initial timeout.
> >
> > What we are trying to write here is a thread scheduler, and that is
> complex business.
> > K
> >
> >
> >
> >> -----Original Message-----
> >> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> >> [mailto:python-dev-bounces+kristjan <python-dev-bounces%2Bkristjan>=
> ccpgames.com at python.org] On Behalf
> >> Of David Beazley
> >> Sent: 15. mars 2010 03:07
> >> To: python-dev at python.org
> >> Subject: Re: [Python-Dev] "Fixing" the new GIL
> >>
> >> happen to be performing CPU intensive work at the same time, it would
> >> be nice if they didn't thrash on multiple cores (the problem with the
> >> old GIL) and if I/O is
> >
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100316/b1ef99e5/attachment.html>

From peter.a.portante at gmail.com  Tue Mar 16 20:22:34 2010
From: peter.a.portante at gmail.com (Peter Portante)
Date: Tue, 16 Mar 2010 14:22:34 -0500
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com>
Message-ID: <C7C544AA.2D05F%peter.a.portante@gmail.com>

Yes, having another thread wait with a timeout is not good. CPU cycles
waisted doing things that don?t help the thread holding the GIL release it.
They just note that they are present and then the GIL-holding-thread should
be responsible for handling the rest. -peter


On 3/16/10 1:11 PM, "Nir Aides" <nir at winpdb.org> wrote:

> Hello Dave,?
> 
> The following documentation suggests ordering in Linux is not FIFO:
> http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait
> .html#tag_03_518_08_06
> "Threads waiting on mutexes and condition variables are selected to proceed in
> an order dependent upon the scheduling policy rather than in some fixed order
> (for example, FIFO or priority). Thus, the scheduling policy determines which
> thread(s) are awakened and allowed to proceed."
> 
> Here is the code:
> http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthre
> ads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&e
> xact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz
> 
> If this is so then it should affect the proposed fixes.?
> For example?waiting with timeout should be avoided, no?
> 
> Nir
> 
> 2010/3/16 David Beazley <dave at dabeaz.com>
>> Python doesn't use a pthreads mutex for the GIL. ? ?It has always used a
>> binary semaphore implemented with condition variables (or just a pthreads
>> semaphore if available). ? ?The reason the performance is so bad is precisely
>> due to the fact that it is using this implementation and the fact that there
>> *IS* a FIFO queue of threads (associated with the condition variable). ? The
>> I/O performance problem with the new GIL is gets much worse with many
>> CPU-bound threads precisely because there is a FIFO queue involved. ? This
>> has been covered in my past GIL presentations.
>> 
>> -Dave
>> 
>> 
>> 
>> On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote:
>> 
>>> > How about attacking the original problem, then?
>>> >
>>> > The reason they thrash on pthreads implementation is that a pthreads mutex
>>> is assumed to be a short-held resource. ?Therefore it will be optimized in
>>> the following ways for multicore machines:
>>> > 1) There is a certain amount of spinning done, to try to acquire it before
>>> blocking
>>> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a
>>> thread coming in to acquire the mutex may get in before others that are
>>> queued. ?This is why "ticking" the GIL works so badly: ?The thread that
>>> releases the lock is usually the one that reaquires it even though others
>>> may be waiting. ?See e.g.
>>> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-
>>> 616ef7b031aa.aspx for a discussion of this (albeit on windows.)
>>> >
>>> > On Windows, this isn't a problem. ?The reason is, that the GIL on windows
>>> is implemented using Event objects that don't cut these corners. ?The Event
>>> provides you with a strict FIFO queue of objects waiting for the event.
>>> >
>>> > If pthreads doesn't provide a synchronization primitive similar to that,
>>> someone that doesn't thrash and has a true FIFO queue, it is possible to
>>> construct such a thing using condition variables and critical sections.
>>> ?Perhaps the posix semaphore api is more appropriate in this case.
>>> >
>>> > By the way, this also shows another problem with (old) python. ?There is
>>> only one core locking primitive, the PyThread_type_lock. ?It is being used
>>> both as a critical section in the traditional sense, and also as this
>>> sort-of-inverse lock that the GIL is. ?In the modern world, where the
>>> intended behaviour of these is quite different, there is no one-size-fits
>>> all. ?On windows in particular, the use of the Event object based lock is
>>> not ideal for other uses than the GIL.
>>> >
>>> >
>>> > In the new GIL, there appear to be several problems:
>>> > 1) There is no FIFO queue of threads wanting the queue, thus thread
>>> scheduling becomes non-deterministic
>>> > 2) The "ticking" of the GIL is now controled by a condition variable
>>> timeout. ?There appears to be no way to prevent many such timeouts to be in
>>> progress at the same time, thus you may have an unnecessarily high rate of
>>> ticking going on.
>>> > 3) There isn't an immediate gil request made when an IO thread requests
>>> the gil back, only after an initial timeout.
>>> >
>>> > What we are trying to write here is a thread scheduler, and that is
>>> complex business.
>>> > K
>>> >
>>> >
>>> >
>>>> >> -----Original Message-----
>>>> >> From: python-dev-bounces+kristjan=ccpgames.com <http://ccpgames.com>
>>>> @python.org <http://python.org>
>>>> >> [mailto:python-dev-bounces+kristjan
>>>> <mailto:python-dev-bounces%2Bkristjan> =ccpgames.com <http://ccpgames.com>
>>>> @python.org <http://python.org> ] On Behalf
>>>> >> Of David Beazley
>>>> >> Sent: 15. mars 2010 03:07
>>>> >> To: python-dev at python.org
>>>> >> Subject: Re: [Python-Dev] "Fixing" the new GIL
>>>> >>
>>>> >> happen to be performing CPU intensive work at the same time, it would
>>>> >> be nice if they didn't thrash on multiple cores (the problem with the
>>>> >> old GIL) and if I/O is
>>> >
>> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100316/238d826d/attachment-0001.html>

From hancock.robert at gmail.com  Tue Mar 16 21:09:39 2010
From: hancock.robert at gmail.com (Robert Hancock)
Date: Tue, 16 Mar 2010 16:09:39 -0400
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <C7C544AA.2D05F%peter.a.portante@gmail.com>
References: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com> 
	<C7C544AA.2D05F%peter.a.portante@gmail.com>
Message-ID: <613d56581003161309p147c45f5ub0f142f1cd160776@mail.gmail.com>

The Linux kernel scheduler deals with two types of ratings and has a
heuristic algorithm that rewards and penalizes on a rapid basis.  To
determine the next thread it inspects the priority array to find the highest
priority task that is runnable and then selects the first task in that
priority.  This is a gross simplification of an extremely complex system,
but it is not simply a simple queue.

To duplicate this exact type of scheduling in the interpreter would be a
daunting task, but Dave's example illustrates that some system of reward and
penalty can be beneficial.

Peter has brought up the point, both here and at the Open Space at Pycon,
that before attempting a downsized version of the a priority thread
scheduler that we should look at other programs that have had to deal with
similar problems.

Peter, since you have dealt with the nitty-gritty of concurrency before, can
you suggest some applications that could serve as models for research?

On Tue, Mar 16, 2010 at 3:22 PM, Peter Portante
<peter.a.portante at gmail.com>wrote:

>  Yes, having another thread wait with a timeout is not good. CPU cycles
> waisted doing things that don?t help the thread holding the GIL release it.
> They just note that they are present and then the GIL-holding-thread should
> be responsible for handling the rest. -peter
>
>
>
> On 3/16/10 1:11 PM, "Nir Aides" <nir at winpdb.org> wrote:
>
> Hello Dave,
>
> The following documentation suggests ordering in Linux is not FIFO:
>
> http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html#tag_03_518_08_06
> "Threads waiting on mutexes and condition variables are selected to
> proceed in an order dependent upon the scheduling policy rather than in some
> fixed order (for example, FIFO or priority). Thus, the scheduling policy
> determines which thread(s) are awakened and allowed to proceed."
>
> Here is the code:
>
> http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz
>
> If this is so then it should affect the proposed fixes.
> For example waiting with timeout should be avoided, no?
>
> Nir
>
> 2010/3/16 David Beazley <dave at dabeaz.com>
>
> Python doesn't use a pthreads mutex for the GIL.    It has always used a
> binary semaphore implemented with condition variables (or just a pthreads
> semaphore if available).    The reason the performance is so bad is
> precisely due to the fact that it is using this implementation and the fact
> that there *IS* a FIFO queue of threads (associated with the condition
> variable).   The I/O performance problem with the new GIL is gets much worse
> with many CPU-bound threads precisely because there is a FIFO queue
> involved.   This has been covered in my past GIL presentations.
>
> -Dave
>
>
>
> On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote:
>
> > How about attacking the original problem, then?
> >
> > The reason they thrash on pthreads implementation is that a pthreads
> mutex is assumed to be a short-held resource.  Therefore it will be
> optimized in the following ways for multicore machines:
> > 1) There is a certain amount of spinning done, to try to acquire it
> before blocking
> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that a
> thread coming in to acquire the mutex may get in before others that are
> queued.  This is why "ticking" the GIL works so badly:  The thread that
> releases the lock is usually the one that reaquires it even though others
> may be waiting.  See e.g.
> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor a discussion of this (albeit on windows.)
> >
> > On Windows, this isn't a problem.  The reason is, that the GIL on windows
> is implemented using Event objects that don't cut these corners.  The Event
> provides you with a strict FIFO queue of objects waiting for the event.
> >
> > If pthreads doesn't provide a synchronization primitive similar to that,
> someone that doesn't thrash and has a true FIFO queue, it is possible to
> construct such a thing using condition variables and critical sections.
>  Perhaps the posix semaphore api is more appropriate in this case.
> >
> > By the way, this also shows another problem with (old) python.  There is
> only one core locking primitive, the PyThread_type_lock.  It is being used
> both as a critical section in the traditional sense, and also as this
> sort-of-inverse lock that the GIL is.  In the modern world, where the
> intended behaviour of these is quite different, there is no one-size-fits
> all.  On windows in particular, the use of the Event object based lock is
> not ideal for other uses than the GIL.
> >
> >
> > In the new GIL, there appear to be several problems:
> > 1) There is no FIFO queue of threads wanting the queue, thus thread
> scheduling becomes non-deterministic
> > 2) The "ticking" of the GIL is now controled by a condition variable
> timeout.  There appears to be no way to prevent many such timeouts to be in
> progress at the same time, thus you may have an unnecessarily high rate of
> ticking going on.
> > 3) There isn't an immediate gil request made when an IO thread requests
> the gil back, only after an initial timeout.
> >
> > What we are trying to write here is a thread scheduler, and that is
> complex business.
> > K
> >
> >
> >
> >> -----Original Message-----
> >> From: python-dev-bounces+kristjan=ccpgames.com <http://ccpgames.com> @
> python.org <http://python.org>
> >> [mailto:python-dev-bounces+kristjan <python-dev-bounces+kristjan> <
> mailto:python-dev-bounces%2Bkristjan <python-dev-bounces%2Bkristjan>> =
> ccpgames.com <http://ccpgames.com> @python.org <http://python.org> ] On
> Behalf
> >> Of David Beazley
> >> Sent: 15. mars 2010 03:07
> >> To: python-dev at python.org
> >> Subject: Re: [Python-Dev] "Fixing" the new GIL
> >>
> >> happen to be performing CPU intensive work at the same time, it would
> >> be nice if they didn't thrash on multiple cores (the problem with the
> >> old GIL) and if I/O is
> >
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org
>
>
>
> ------------------------------
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/hancock.robert%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100316/8432901c/attachment.html>

From sridharr at activestate.com  Tue Mar 16 21:31:54 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Tue, 16 Mar 2010 13:31:54 -0700
Subject: [Python-Dev] Python 2.6.5 rc 2
In-Reply-To: <20100309075636.09f52b27@heresy.wooz.org>
References: <20100309075636.09f52b27@heresy.wooz.org>
Message-ID: <314E71ED-C3BA-47B2-9BD3-0D6BFE6B6AEA@activestate.com>

I just verified with our ActivePython build that 2.6.4rc2 builds fine on Linux, Windows, Mac, HP-UX, AIX and Solaris. 3.1.2rc1 builds fine except on AIX[1] and HP-UX[2] but those issues existed in 3.1.1 too, I believe.

-srid

[1] http://bugs.python.org/issue6645
[2] http://bugs.python.org/issue5999

On 2010-03-09, at 4:56 AM, Barry Warsaw wrote:

> Hi Python hackateers!
> 
> It looks like we finally have no more release blockers for 2.6.5rc2.  I would
> like to tag the tree tonight for rc2 so that Martin can build the Windows
> installer for a release tomorrow.  I am also moving the final release back to
> Friday March 19.
> 
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com


From benjamin at python.org  Tue Mar 16 21:51:10 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 16 Mar 2010 15:51:10 -0500
Subject: [Python-Dev] tagging 3.1.2
Message-ID: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com>

My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built
for the final release on Saturday. Agreeable?

-- 
Regards,
Benjamin

From brian at sweetapp.com  Tue Mar 16 21:52:22 2010
From: brian at sweetapp.com (Brian Quinlan)
Date: Tue, 16 Mar 2010 13:52:22 -0700
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <4B979292.4080906@gmail.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
	<4B94DC30.2030904@gmail.com>
	<19349.25574.880099.668647@montanaro.dyndns.org>
	<4B979292.4080906@gmail.com>
Message-ID: <E40B5636-E58C-4BDB-92B9-A7712DB2B531@sweetapp.com>

Nick Coghlan wrote:
 > You may want to consider providing global thread and process  
executors
 > in the futures module itself. Code which just wants to say "do this  
in
 > the background" without having to manage the lifecycle of its own
 > executor instance is then free to do so. I've had a lot of experience
 > with a framework that provides this and it is *very* convenient (it's
 > also a good way to avoid deadlocks due to synchronous notification  
APIs).

This seems like a reasonable idea to me.

I take it that the thread/process pool should be unlimited in size.  
Should every thread/process exit when it finishes its job or should  
there be a smarter collection strategy?

Cheers,
Brian

From barry at python.org  Tue Mar 16 22:06:36 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 16 Mar 2010 17:06:36 -0400
Subject: [Python-Dev] tagging 2.6.5 (Re:  tagging 3.1.2)
In-Reply-To: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com>
References: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com>
Message-ID: <20100316170636.69c6a141@heresy>

On Mar 16, 2010, at 03:51 PM, Benjamin Peterson wrote:

>My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built
>for the final release on Saturday. Agreeable?

I might as well do the same for 2.6.5, though I plan on releasing it Friday.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100316/6e13f198/attachment.pgp>

From greg.ewing at canterbury.ac.nz  Tue Mar 16 23:10:41 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 17 Mar 2010 11:10:41 +1300
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
Message-ID: <4BA001E1.5050106@canterbury.ac.nz>

Mark Dickinson wrote:

> On the one hand there's something to be said for maintaining a clean
> separation between the float and Decimal types, allowing only explicit
> conversions from one to the other;  mixed-type arithmetic between
> floats and Decimals was very deliberately not permitted in the
> original PEP, and that's unlikely to change in a hurry.

I think that as long as you're disallowing arithmetic
between float and decimal, comparison should be disallowed
as well.

So if you're going to change anything, it would be better
to make such comparisons raise a TypeError in 2.7 to match
the current 3.x behaviour.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Mar 16 23:16:11 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 17 Mar 2010 11:16:11 +1300
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
	<20100316155815.DAE983A4097@sparrow.telecommunity.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
Message-ID: <4BA0032B.4080307@canterbury.ac.nz>

Mark Dickinson wrote:
> On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby <pje at telecommunity.com> wrote:
> 
>>If not, it might be confusing if a number that prints as '.1' compares
>>unequal to Decimal('.1').
> 
> Agreed, but this is just your everyday floating-point confusion, to be
> dealt with by social means (e.g., educating the programmer).

Seems to me that this education would mostly consist of saying
"don't compare floats and decimals", which is why I think that
disallowing them in the first place would be better.

Then if a programmer truly needs to compare them for some
reason, he has to be explicit about how to do it.

-- 
Greg

From ncoghlan at gmail.com  Tue Mar 16 23:08:14 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Mar 2010 08:08:14 +1000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local>
References: <49653.1268622425@dabeaz.com>	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>	<20100316080011.34589b00@msiwind>	<930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local>	<20100316151623.05e8e570@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local>
Message-ID: <4BA0014E.9010207@gmail.com>

Kristj?n Valur J?nsson wrote:
>  We have never had high priority for IO threads in
> python (and its not-by-design round robin scheduler on single core
> cpus) and it is unclear why that should be required now as a fix.

David explained that in the issue tracker - 2.x typically doesn't do
that much work per bytecode instruction, so the interpreter releases and
reacquires the GIL far more frequently in a CPU-bound thread than it
does under the new time-based check interval.

The current settings mean we have less GIL overhead in the normal case,
but worse worst-case I/O latency.

There are two fairly straightforward ways to handle that:
- cut the check interval duration drastically (trading GIL overhead for
I/O responsiveness, as in 2.x)
- add some form of I/O thread prioritisation, such as Antoine's gilinter
patch (which automatically identifies and prioritises "interactive"
threads at the cost of a dozen or so additional lines of C code and a
small amount of overhead on GIL context switches to adjust thread
interactivity counters).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Tue Mar 16 23:10:41 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Mar 2010 08:10:41 +1000
Subject: [Python-Dev] [PEP 3148] futures - execute computations
	asynchronously
In-Reply-To: <E40B5636-E58C-4BDB-92B9-A7712DB2B531@sweetapp.com>
References: <31B654A3-E7DB-4185-A3BD-B29EE0BB5E7B@sweetapp.com>
	<20100306065016.DBDF33A40A7@sparrow.telecommunity.com>
	<AD8CC105-0F78-414F-BFC1-5992DB6B3697@sweetapp.com>
	<20100306160434.39FBD3A4077@sparrow.telecommunity.com>
	<022F6F17-95F3-45E4-8224-FAF6380C855A@sweetapp.com>
	<20100307035657.111563A4077@sparrow.telecommunity.com>
	<4B93303E.9050301@gmail.com>
	<20100307154809.742E73A4062@sparrow.telecommunity.com>
	<5d44f72f1003070839v6481b757j73e9b14cb808e9b7@mail.gmail.com>
	<20100307175732.307513A4062@sparrow.telecommunity.com>
	<5d44f72f1003071059v2608c145ra8b426cbfc17ef3e@mail.gmail.com>
	<20100307195614.BBD693A4062@sparrow.telecommunity.com>
	<4B94DC30.2030904@gmail.com>
	<19349.25574.880099.668647@montanaro.dyndns.org>
	<4B979292.4080906@gmail.com>
	<E40B5636-E58C-4BDB-92B9-A7712DB2B531@sweetapp.com>
Message-ID: <4BA001E1.8080105@gmail.com>

Brian Quinlan wrote:
> I take it that the thread/process pool should be unlimited in size.
> Should every thread/process exit when it finishes its job or should
> there be a smarter collection strategy?

I'd be inclined to do something slightly smarter, similar to what we do
with memory overallocation for mutable containers.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Tue Mar 16 23:22:25 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 16 Mar 2010 23:22:25 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4BA0014E.9010207@gmail.com>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<20100316080011.34589b00@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local>
	<20100316151623.05e8e570@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local>
	<4BA0014E.9010207@gmail.com>
Message-ID: <1268778145.32204.38.camel@localhost>


> David explained that in the issue tracker - 2.x typically doesn't do
> that much work per bytecode instruction,

Oh, but that's wrong in general.
Dave's *spinning loop* doesn't do much work per bytecode instruction,
however ;)

> The current settings mean we have less GIL overhead in the normal case,
> but worse worst-case I/O latency.

Actually, ccbench shows that worst case IO latency is much worse in 2.x
(when executing bytecodes which do a lot of work, e.g. matching a
regex).
What happens though is that best case IO latency is better in 2.x (e.g.
spinning loop, or short opcodes approaching the spinning loop case :-)).

cheers

Antoine.



From steve at pearwood.info  Tue Mar 16 23:32:08 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 17 Mar 2010 09:32:08 +1100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
	<5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com>
Message-ID: <201003170932.09201.steve@pearwood.info>

On Wed, 17 Mar 2010 03:23:30 am Mark Dickinson wrote:
> On Tue, Mar 16, 2010 at 4:11 PM, Mark Dickinson <dickinsm at gmail.com>
> wrote: [...]
>
> >>>> Decimal.from_float(1.1) == 1.1
> >
> > False
>
> Whoops.  To clarify, this is the pre-patch behaviour;  post-patch,
> this gives True.

Whew! You had me worried there for a second. Just to clarify, you are 
proposing:

Decimal.from_float(1.1) == 1.1
Decimal.('1.1') != float('1.1')

+1 on this behaviour, even in the absence of supporting mixed Decimal 
and float arithmetic operations.

Both Decimals and floats are representations of real numbers, and not 
being able to compare two numbers is just weird: refusing to compare 
(say) Decimal(1) with float(1) makes as little sense to me as refusing 
to compare int(1) with float(1).

But mixed arithmetic runs into the problem, what do you want the result 
type to be? Given (say) decimal+float, returning either a Decimal or a 
float will be the wrong thing to do some of the time, so better to 
prohibit mixed arithmetic and let folks handle their own conversions. 
So +1 on continuing to prohibit mixed arithmetic.

But no such problems arise with comparisons, which will always return a 
bool, and will avoid the current ... interesting ... behaviour. In 3.1:

>>> Decimal(1) == 1 == 1.0
True
>>> Decimal(1) == 1.0
False
>>> Decimal.from_float(1.0) == 1 == 1.0
True
>>> Decimal.from_float(1.0) == 1.0
False

Replacing False with an exception doesn't make it any less bizarre.



-- 
Steven D'Aprano

From solipsis at pitrou.net  Tue Mar 16 23:37:36 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 16 Mar 2010 22:37:36 +0000 (UTC)
Subject: [Python-Dev] "Fixing" the new GIL
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<20100316080011.34589b00@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local>
	<20100316151623.05e8e570@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local>
Message-ID: <hnp17g$efn$1@dough.gmane.org>

Le Tue, 16 Mar 2010 17:02:13 +0000, Kristj?n Valur J?nsson a ?crit?:
>> 
>> Well, I don't think this has ever been by design, and it's not obvious
>> this is desirable either (see Dave Beazley's benchmark).
>>
> Did Dave benchmark a straight FIFO system?

Well, he presented benchmark figures with two threads, which is a FIFO 
system with the new GIL (because when a thread releases the GIL, it 
ensures that another thread takes it).

> We have never had high priority for IO threads in python
> (and its not-by-design round robin scheduler on single core cpus) and it
> is unclear why that should be required now as a fix.

I agree that Dave's situation doesn't seem new to me, although he often 
presents it as a deficiency of the new GIL only.

Regards

Antoine.



From martin at v.loewis.de  Tue Mar 16 23:44:25 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 16 Mar 2010 23:44:25 +0100
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <20100316091544.GA3799@cskk.homeip.net>
References: <4B9F3A54.3040907@v.loewis.de>
	<20100316091544.GA3799@cskk.homeip.net>
Message-ID: <4BA009C9.8090000@v.loewis.de>

> You can't. This isn't an example where I am personally bitten by the GIL.
> I may be, but there's plenty of other stuff in terms of tuning or
> optimisation in my particular app before I get anywhere near the GIL.
> 
> My point is not that it's biting me right now but that earlier you
> said the scenario was probably unrealistic, but I have an app which can
> probably exhibit exactly the issue under discussion.

I don't understand. First you say that there is plenty of other stuff
.... before you get anywhere near the GIL, and then you say that your
app can probably exhibit the issue under discussion. Which one is it?

I can easily believe that there is tons of other stuff that make the GIL
contention irrelevant, I can not believe that it will exhibit the issue
under discussion.

> | > The idea here is that one has a few threads receiving requests (eg a
> | > daemon watching a socket or monitoring a db queue table) which then use
> | > the FuncMultiQueue to manage how many actual requests are processed
> | > in parallel (yes, a semaphore can cover a lot of this, but not the
> | > asynchronous call modes).
> | 
> | Why do you say processing is in parallel?
> | In Python, processing is
> | normally never in parallel, but always sequential (potentially
> | interleaving). Are you releasing the GIL for processing?
> 
> 
> I mean that I can have multiple actual requests being served, and they
> are _not_ handled sequentially - the multi queue is to permit more than
> one to be in progress at once so that an expensive request need not
> inherently delay a following cheap request

But, in Python, due to the GIL, *everything* is sequential, *nothing* is
in parallel.

> Can't tell you - I'm not claiming the current GIL behaviour is biting me
> right now; I'm saying your claim that the scenario is unrealistic is a
> bit unfair; my app wil probably be just the kind of app to be affected,
> even only subtly. And it can hardly be alone in the "daemon" world.

Unfortunately, you are not able to demonstrate this (because the effect
would only be subtle).

My claim is that the scenario where the effect is *visible* is
unrealistic, i.e. that, in realistic scenarios, tons of other effects
actually lessen the problem.

Regards,
Martin


From ncoghlan at gmail.com  Tue Mar 16 23:46:04 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Mar 2010 08:46:04 +1000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003170932.09201.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>	<5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com>
	<201003170932.09201.steve@pearwood.info>
Message-ID: <4BA00A2C.8070204@gmail.com>

Steven D'Aprano wrote:
> But no such problems arise with comparisons, which will always return a 
> bool, and will avoid the current ... interesting ... behaviour. In 3.1:
> 
>>>> Decimal(1) == 1 == 1.0
> True
>>>> Decimal(1) == 1.0
> False
>>>> Decimal.from_float(1.0) == 1 == 1.0
> True
>>>> Decimal.from_float(1.0) == 1.0
> False
> 
> Replacing False with an exception doesn't make it any less bizarre.

Allowing the comparisons also doesn't introduce the potential for large
cumulative errors which are possible when actual implicit arithmetic
conversions are allowed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From steve at pearwood.info  Tue Mar 16 23:46:39 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 17 Mar 2010 09:46:39 +1100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA0032B.4080307@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
	<4BA0032B.4080307@canterbury.ac.nz>
Message-ID: <201003170946.39457.steve@pearwood.info>

On Wed, 17 Mar 2010 09:16:11 am Greg Ewing wrote:
> Mark Dickinson wrote:
> > On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby <pje at telecommunity.com> 
wrote:
> >>If not, it might be confusing if a number that prints as '.1'
> >> compares unequal to Decimal('.1').
> >
> > Agreed, but this is just your everyday floating-point confusion, to
> > be dealt with by social means (e.g., educating the programmer).
>
> Seems to me that this education would mostly consist of saying
> "don't compare floats and decimals", which is why I think that
> disallowing them in the first place would be better.

I'm sure you don't mean to suggest that the only (or even the main) 
source of floating point confusion comes from mixed Decimal/float 
operations.


> Then if a programmer truly needs to compare them for some
> reason, he has to be explicit about how to do it.

More explicit than someDecimal == someFloat? Seems pretty explicit to 
me.



-- 
Steven D'Aprano

From guido at python.org  Wed Mar 17 00:01:12 2010
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Mar 2010 15:01:12 -0800
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003170932.09201.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> 
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com> 
	<5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com> 
	<201003170932.09201.steve@pearwood.info>
Message-ID: <ca471dc21003161601v16a18bcfg44e4e55b38d43ad4@mail.gmail.com>

On Tue, Mar 16, 2010 at 2:32 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> But mixed arithmetic runs into the problem, what do you want the result
> type to be? Given (say) decimal+float, returning either a Decimal or a
> float will be the wrong thing to do some of the time, so better to
> prohibit mixed arithmetic and let folks handle their own conversions.
> So +1 on continuing to prohibit mixed arithmetic.

I'm not disagreeing, but I really wonder, what is the value of
supporting mixed comparisons then? Just because you *can* assign a
meaning to it doesn't mean you should.

OTOH I'm sure a lot of people would like to see mixed arithmetic
supported, the PEP be damned, and they would probably be happy with
any simple rule about the return type even if it's not always ideal. I
note that there are cases where converting a long to a float also is
the wrong thing to do, and yet mixed long/float operations always
return floats. If you are amenable to this argument, I would propose
to make the result of mixed operations return a Decimal, since in some
"intuitive complexity" sense an int is a simpler type than a float and
a float is a simpler type than a Decimal -- so results return the more
complex type. But my intuition on this isn't super strong and I could
live with always returning a float as well -- there are always casts
to force the issue.

-- 
--Guido van Rossum (python.org/~guido)

From nir at winpdb.org  Wed Mar 17 00:05:45 2010
From: nir at winpdb.org (Nir Aides)
Date: Wed, 17 Mar 2010 01:05:45 +0200
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<5AD14AA1-72D3-4076-91BF-63B6C402139D@dabeaz.com>
	<5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com>
Message-ID: <5d2328511003161605j771a2feai83b981594b33b77d@mail.gmail.com>

Hi,

I posted a small patch to the GIL which demonstrates the scheduling policy
of pthreads conditions.

It seems that with pthreads a good policy is to allow (and help) the OS to
manage scheduling of threads via the condition queue without introducing
scheduling logic.
Some changes include removing the timeout from the new GIL wait, and
allowing CPU bound threads to run long enough to let the OS recognize them
as such.
The patch uses ticks countdown-to-switch but it should be changed to a time
based countdown (If taking a look at the clock is expensive maybe it can be
done once every N ticks during countdown).

Remains to explore what can be done on other platforms.

Nir



2010/3/16 Nir Aides <nir at winpdb.org>

> Hello Dave,
>
> The following documentation suggests ordering in Linux is not FIFO:
>
> http://www.opengroup.org/onlinepubs/000095399/functions/pthread_cond_timedwait.html#tag_03_518_08_06
> "Threads waiting on mutexes and condition variables are selected to
> proceed in an order dependent upon the scheduling policy rather than in some
> fixed order (for example, FIFO or priority). Thus, the scheduling policy
> determines which thread(s) are awakened and allowed to proceed."
>
> Here is the code:
>
> http://www.google.com/codesearch/p?hl=en#5ge3gHPB4K4/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz%7CaeB7Uqo7T9g/linuxthreads/queue.h&q=pthread_cond_timedwait&exact_package=http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.1.1.tar.gz
>
> If this is so then it should affect the proposed fixes.
> For example waiting with timeout should be avoided, no?
>
> Nir
>
> 2010/3/16 David Beazley <dave at dabeaz.com>
>
>> Python doesn't use a pthreads mutex for the GIL.    It has always used a
>> binary semaphore implemented with condition variables (or just a pthreads
>> semaphore if available).    The reason the performance is so bad is
>> precisely due to the fact that it is using this implementation and the fact
>> that there *IS* a FIFO queue of threads (associated with the condition
>> variable).   The I/O performance problem with the new GIL is gets much worse
>> with many CPU-bound threads precisely because there is a FIFO queue
>> involved.   This has been covered in my past GIL presentations.
>>
>>
>> -Dave
>>
>>
>>
>> On Mar 16, 2010, at 5:52 AM, Kristj?n Valur J?nsson wrote:
>>
>> > How about attacking the original problem, then?
>> >
>> > The reason they thrash on pthreads implementation is that a pthreads
>> mutex is assumed to be a short-held resource.  Therefore it will be
>> optimized in the following ways for multicore machines:
>> > 1) There is a certain amount of spinning done, to try to acquire it
>> before blocking
>> > 2) It will employ un-fair tactics to avoid lock-convoying, meaning that
>> a thread coming in to acquire the mutex may get in before others that are
>> queued.  This is why "ticking" the GIL works so badly:  The thread that
>> releases the lock is usually the one that reaquires it even though others
>> may be waiting.  See e.g.
>> http://www.bluebytesoftware.com/blog/PermaLink,guid,e40c2675-43a3-410f-8f85-616ef7b031aa.aspxfor a discussion of this (albeit on windows.)
>> >
>> > On Windows, this isn't a problem.  The reason is, that the GIL on
>> windows is implemented using Event objects that don't cut these corners.
>>  The Event provides you with a strict FIFO queue of objects waiting for the
>> event.
>> >
>> > If pthreads doesn't provide a synchronization primitive similar to that,
>> someone that doesn't thrash and has a true FIFO queue, it is possible to
>> construct such a thing using condition variables and critical sections.
>>  Perhaps the posix semaphore api is more appropriate in this case.
>> >
>> > By the way, this also shows another problem with (old) python.  There is
>> only one core locking primitive, the PyThread_type_lock.  It is being used
>> both as a critical section in the traditional sense, and also as this
>> sort-of-inverse lock that the GIL is.  In the modern world, where the
>> intended behaviour of these is quite different, there is no one-size-fits
>> all.  On windows in particular, the use of the Event object based lock is
>> not ideal for other uses than the GIL.
>> >
>> >
>> > In the new GIL, there appear to be several problems:
>> > 1) There is no FIFO queue of threads wanting the queue, thus thread
>> scheduling becomes non-deterministic
>> > 2) The "ticking" of the GIL is now controled by a condition variable
>> timeout.  There appears to be no way to prevent many such timeouts to be in
>> progress at the same time, thus you may have an unnecessarily high rate of
>> ticking going on.
>> > 3) There isn't an immediate gil request made when an IO thread requests
>> the gil back, only after an initial timeout.
>> >
>> > What we are trying to write here is a thread scheduler, and that is
>> complex business.
>> > K
>> >
>> >
>> >
>> >> -----Original Message-----
>> >> From: python-dev-bounces+kristjan=ccpgames.com at python.org
>> >> [mailto:python-dev-bounces+kristjan <python-dev-bounces%2Bkristjan>=
>> ccpgames.com at python.org] On Behalf
>> >> Of David Beazley
>> >> Sent: 15. mars 2010 03:07
>> >> To: python-dev at python.org
>> >> Subject: Re: [Python-Dev] "Fixing" the new GIL
>> >>
>> >> happen to be performing CPU intensive work at the same time, it would
>> >> be nice if they didn't thrash on multiple cores (the problem with the
>> >> old GIL) and if I/O is
>> >
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100317/07e23425/attachment-0001.html>

From martin at v.loewis.de  Wed Mar 17 00:06:55 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 17 Mar 2010 00:06:55 +0100
Subject: [Python-Dev] tagging 3.1.2
In-Reply-To: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com>
References: <1afaf6161003161351x664aa033i38cd7deb387012da@mail.gmail.com>
Message-ID: <4BA00F0F.8070504@v.loewis.de>

Benjamin Peterson wrote:
> My plan is to tag 3.1.2 sometime on Thursday, so binaries can be built
> for the final release on Saturday. Agreeable?

Fine with me.

Martin

From raymond.hettinger at gmail.com  Wed Mar 17 01:36:36 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 16 Mar 2010 17:36:36 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
Message-ID: <594D6395-4B01-4C9C-8417-36866992C338@gmail.com>




On Mar 16, 2010, at 9:41 AM, Guido van Rossum wrote:

> I'd say if you're not going to forward-port this to Python 3, it
> shouldn't go into Python 2 -- in that case it would make more sense to
> me to back-port the exception-raising behavior.

Python 3 doesn't need it because it is possible to not give a result
at all.  Python 2 does need it because we have to give *some*
result.

> 
> Also supporting comparisons but not other mixed operations is going to
> be confusing. If you are sticking to that behavior I think mixed
> comparisons should also be ruled out.

The difference is that mixed comparisons currently do give a result,
but one that is non-sensical.  The proposal is a make in give a
meaningful result, not as an extra feature, but in an effort to not
be wrong.

Since 2.x has to give a result, we should make it useful.
Since 3.x does not make the comparison, it is okay to punt.


Raymond




From guido at python.org  Wed Mar 17 02:05:29 2010
From: guido at python.org (Guido van Rossum)
Date: Tue, 16 Mar 2010 18:05:29 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <594D6395-4B01-4C9C-8417-36866992C338@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> 
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com> 
	<594D6395-4B01-4C9C-8417-36866992C338@gmail.com>
Message-ID: <ca471dc21003161805t4d7a5e33v11daa908d53a0a3b@mail.gmail.com>

On Tue, Mar 16, 2010 at 5:36 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
>
>
> On Mar 16, 2010, at 9:41 AM, Guido van Rossum wrote:
>
>> I'd say if you're not going to forward-port this to Python 3, it
>> shouldn't go into Python 2 -- in that case it would make more sense to
>> me to back-port the exception-raising behavior.
>
> Python 3 doesn't need it because it is possible to not give a result
> at all. ?Python 2 does need it because we have to give *some*
> result.
>
>>
>> Also supporting comparisons but not other mixed operations is going to
>> be confusing. If you are sticking to that behavior I think mixed
>> comparisons should also be ruled out.
>
> The difference is that mixed comparisons currently do give a result,
> but one that is non-sensical. ?The proposal is a make in give a
> meaningful result, not as an extra feature, but in an effort to not
> be wrong.
>
> Since 2.x has to give a result, we should make it useful.
> Since 3.x does not make the comparison, it is okay to punt.

No. You are talking like there is no path from 2.7 to 3.x. Since the
result in 2.x was never useful, you should not add a dead-end feature.
And there is no reason (other than backwards compatibility) why the
comparison couldn't raise an exception in 2.x as well -- e.g.
str<->unicode comparisons do this already. (It used to be required
that comparisons had to always return a result, but that was done away
with long before Decimal was introduced.)

-- 
--Guido van Rossum (python.org/~guido)

From raymond.hettinger at gmail.com  Wed Mar 17 02:27:01 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 16 Mar 2010 18:27:01 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA0032B.4080307@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
	<20100316155815.DAE983A4097@sparrow.telecommunity.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
	<4BA0032B.4080307@canterbury.ac.nz>
Message-ID: <32790831-2CF6-425D-8DA1-4D45C1A1F083@gmail.com>


On Mar 16, 2010, at 3:16 PM, Greg Ewing wrote:
> 
> Seems to me that this education would mostly consist of saying
> "don't compare floats and decimals", which is why I think that
> disallowing them in the first place would be better.

That makes sense.

I do worry that 2.x currently does make the comparison
and gives the wrong answer.  We have the ability to
make it a correct answer.  But, it seems like the mood
here is that wrong-is-better-than-right for an action
that someone shouldn't be doing in the first place.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100316/a4615a57/attachment.html>

From steve at pearwood.info  Wed Mar 17 11:45:51 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 17 Mar 2010 21:45:51 +1100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <32790831-2CF6-425D-8DA1-4D45C1A1F083@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<4BA0032B.4080307@canterbury.ac.nz>
	<32790831-2CF6-425D-8DA1-4D45C1A1F083@gmail.com>
Message-ID: <201003172145.52425.steve@pearwood.info>

On Wed, 17 Mar 2010 12:27:01 pm Raymond Hettinger wrote:
> On Mar 16, 2010, at 3:16 PM, Greg Ewing wrote:
> > Seems to me that this education would mostly consist of saying
> > "don't compare floats and decimals", which is why I think that
> > disallowing them in the first place would be better.
>
> That makes sense.
>
> I do worry that 2.x currently does make the comparison
> and gives the wrong answer.  We have the ability to
> make it a correct answer.  But, it seems like the mood
> here is that wrong-is-better-than-right for an action
> that someone shouldn't be doing in the first place.

I don't get this.

Why is it "wrong" to compare Decimals to floats, and why shouldn't I do 
so? What harm is there?

If the argument is that naive users who don't understand floats may be 
confused by the results, then the problem lies with floats, and if you 
really want to avoid confusing the float-naive, then we should prohibit 
all comparisons on floats:

>>> 1e20 + 1e4 < 1e20 + 2e4
False

I don't mean that as a serious suggestion -- it would be absurd to 
cripple floats for the sake of avoiding confusion of those who don't 
understand floats. Why are Decimals different?

I can't see why comparing Decimal(1) to float(1) is wrong in any sense. 
I can see that comparing Decimal("1.1") to float("1.1") may confuse the 
float-naive, but the float naive will be confused by this too:

>>> x = 1.0/3
>>> x + 1.0 - 1.0 == x
False

There's an awful lot about floats that is confusing to naive users, I 
don't see that the behaviour of Decimals will make it worse.



-- 
Steven D'Aprano

From kristjan at ccpgames.com  Wed Mar 17 12:01:16 2010
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Wed, 17 Mar 2010 11:01:16 +0000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <613d56581003161309p147c45f5ub0f142f1cd160776@mail.gmail.com>
References: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com>
	<C7C544AA.2D05F%peter.a.portante@gmail.com>
	<613d56581003161309p147c45f5ub0f142f1cd160776@mail.gmail.com>
Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A182184A4@exchis.ccp.ad.local>

I could also point you at EVE online, a large scale IO machine based on stackless python.  Our game proxies (front end servers) handle 30.000 client connections each using tasklets.  The problem is coarse grained scheduling of IO and other tasks, same as with threads in regular Python.  We have found that adhering to FIFO scheduling provides good all round performance, particularly the variance of IO latency is very low.
As I?v mentioned in another message, I am toying with the idea of adding another priority layer to the Stackless scheduler for IO tasks.  It?s not trivial though (requires me to do some stackless surgery) and the turnaround time to test new approaches in production for us is a bit high ?

K

From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Robert Hancock
Sent: 16. mars 2010 20:10
To: Peter Portante
Cc: David Beazley; python-dev at python.org
Subject: Re: [Python-Dev] "Fixing" the new GIL

The Linux kernel scheduler deals with two types of ratings and has a heuristic algorithm that rewards and penalizes on a rapid basis.  To determine the next thread it inspects the priority array to find the highest priority task that is runnable and then selects the first task in that priority.  This is a gross simplification of an extremely complex system, but it is not simply a simple queue.

To duplicate this exact type of scheduling in the interpreter would be a daunting task, but Dave's example illustrates that some system of reward and penalty can be beneficial.

Peter has brought up the point, both here and at the Open Space at Pycon, that before attempting a downsized version of the a priority thread scheduler that we should look at other programs that have had to deal with similar problems.

Peter, since you have dealt with the nitty-gritty of concurrency before, can you suggest some applications that could serve as models for research?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100317/75c31944/attachment.html>

From kristjan at ccpgames.com  Wed Mar 17 12:08:39 2010
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Wed, 17 Mar 2010 11:08:39 +0000
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <4BA0014E.9010207@gmail.com>
References: <49653.1268622425@dabeaz.com>
	<930F189C8A437347B80DF2C156F7EC7F0A18218176@exchis.ccp.ad.local>
	<20100316080011.34589b00@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A1821827D@exchis.ccp.ad.local>
	<20100316151623.05e8e570@msiwind>
	<930F189C8A437347B80DF2C156F7EC7F0A18218371@exchis.ccp.ad.local>
	<4BA0014E.9010207@gmail.com>
Message-ID: <930F189C8A437347B80DF2C156F7EC7F0A182184B0@exchis.ccp.ad.local>


> -----Original Message-----
> From: Nick Coghlan [mailto:ncoghlan at gmail.com]
> Sent: 16. mars 2010 22:08
> To: Kristj?n Valur J?nsson
> Cc: Antoine Pitrou; python-dev at python.org
> Subject: Re: [Python-Dev] "Fixing" the new GIL
> 
> Kristj?n Valur J?nsson wrote:
> >  We have never had high priority for IO threads in
> > python (and its not-by-design round robin scheduler on single core
> > cpus) and it is unclear why that should be required now as a fix.
> 
> David explained that in the issue tracker - 2.x typically doesn't do
> that much work per bytecode instruction, so the interpreter releases
> and
> reacquires the GIL far more frequently in a CPU-bound thread than it
> does under the new time-based check interval.
> 
> 
Okay, but my original post in this thread was "why don't we fix the original
problem with the GIL as implemented in 2.x" rather than to try to beat the new, more complicated, GIL into submission?
In fact, I probably should submit a patch to do that since 2.x isn't disappearing quite yet.

Cheers,
K


From steve at pearwood.info  Wed Mar 17 12:43:35 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 17 Mar 2010 22:43:35 +1100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <ca471dc21003161601v16a18bcfg44e4e55b38d43ad4@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003170932.09201.steve@pearwood.info>
	<ca471dc21003161601v16a18bcfg44e4e55b38d43ad4@mail.gmail.com>
Message-ID: <201003172243.35336.steve@pearwood.info>

On Wed, 17 Mar 2010 10:01:12 am Guido van Rossum wrote:
> On Tue, Mar 16, 2010 at 2:32 PM, Steven D'Aprano <steve at pearwood.info> 
wrote:
> > But mixed arithmetic runs into the problem, what do you want the
> > result type to be? Given (say) decimal+float, returning either a
> > Decimal or a float will be the wrong thing to do some of the time,
> > so better to prohibit mixed arithmetic and let folks handle their
> > own conversions. So +1 on continuing to prohibit mixed arithmetic.
>
> I'm not disagreeing, but I really wonder, what is the value of
> supporting mixed comparisons then? Just because you *can* assign a
> meaning to it doesn't mean you should.

Any mixed comparison with floats is already risky:

>>> 1e20 + 1e4 == 10**20 + 10**4
False

but we allow them anyway because they're too useful and too obvious not 
to. The problems with (say) floating point equality are far too well 
known to need me give examples, but I will anyway *wink*:

>>> 1.3 == 3.7 - 2.4
False

but we allow it anyway, partly from tradition ("floats have always been 
like that") and partly because, frankly, laziness is a virtue. It would 
be painful not to be able to say (e.g.):

>>> 1.25 == 3.75 - 2.5
True
>>> 10.0 > 1
True

My defence of mixed comparisons is the same. If I'm doing serious maths 
work, then I'm going to be comparing numbers carefully, and not just 
tossing a comparison operator between them. But for simple 
calculations, or using the interactive interpreter as a calculator, or 
if I need to sort a list of mixed numbers without caring whether they 
are ints or floats or Decimals, laziness is a virtue. If your needs for 
accuracy aren't high, you can go far by comparing floats with ==. Why 
shouldn't the same apply to mixed floats and Decimals?

Having said all that, I've just re-read the PEP, and spotted a fly in 
the ointment... hash.

If we allow Decimals to compare equal to floats, that presumably implies 
that they need to hash equal. That may be simple enough for integer 
valued floats, but what non-integer values?




-- 
Steven D'Aprano

From dave at dabeaz.com  Wed Mar 17 12:53:24 2010
From: dave at dabeaz.com (David Beazley)
Date: Wed, 17 Mar 2010 06:53:24 -0500
Subject: [Python-Dev] "Fixing" the new GIL
In-Reply-To: <930F189C8A437347B80DF2C156F7EC7F0A182184A4@exchis.ccp.ad.local>
References: <5d2328511003161111vbf198d1y36e974d474e38a06@mail.gmail.com>
	<C7C544AA.2D05F%peter.a.portante@gmail.com>
	<613d56581003161309p147c45f5ub0f142f1cd160776@mail.gmail.com>
	<930F189C8A437347B80DF2C156F7EC7F0A182184A4@exchis.ccp.ad.local>
Message-ID: <66A064A8-E328-49C7-9E5E-F5E8DC6D3D64@dabeaz.com>

I'm not sure I see any inherent problem in putting all I/O bound threads in some kind of FIFO queue.  However, there still should be some mechanism that gives those threads priority over a CPU-bound thread that never wants to give up the GIL.   A common OS solution is to simply have multiple FIFO queues, each with varying priorities.  Threads with low priority don't get to run unless the higher priority queues are empty.    One could probably try it with Python by simply having two queues (I/O and CPU) and separating threads into two categories depending on whether they are forcefully timed out or not.    An OS would tend to have many more levels, but maybe just two queues would be enough to solve the thread scheduling problems being discussed here.

Cheers,
Dave


On Mar 17, 2010, at 6:01 AM, Kristj?n Valur J?nsson wrote:

> I could also point you at EVE online, a large scale IO machine based on stackless python.  Our game proxies (front end servers) handle 30.000 client connections each using tasklets.  The problem is coarse grained scheduling of IO and other tasks, same as with threads in regular Python.  We have found that adhering to FIFO scheduling provides good all round performance, particularly the variance of IO latency is very low.
> As I?v mentioned in another message, I am toying with the idea of adding another priority layer to the Stackless scheduler for IO tasks.  It?s not trivial though (requires me to do some stackless surgery) and the turnaround time to test new approaches in production for us is a bit high J
>  
> K
>  
> From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Robert Hancock
> Sent: 16. mars 2010 20:10
> To: Peter Portante
> Cc: David Beazley; python-dev at python.org
> Subject: Re: [Python-Dev] "Fixing" the new GIL
>  
> The Linux kernel scheduler deals with two types of ratings and has a heuristic algorithm that rewards and penalizes on a rapid basis.  To determine the next thread it inspects the priority array to find the highest priority task that is runnable and then selects the first task in that priority.  This is a gross simplification of an extremely complex system, but it is not simply a simple queue.
>  
> To duplicate this exact type of scheduling in the interpreter would be a daunting task, but Dave's example illustrates that some system of reward and penalty can be beneficial.
>  
> Peter has brought up the point, both here and at the Open Space at Pycon, that before attempting a downsized version of the a priority thread scheduler that we should look at other programs that have had to deal with similar problems.
>  
> Peter, since you have dealt with the nitty-gritty of concurrency before, can you suggest some applications that could serve as models for research?
> 
>  


From dickinsm at gmail.com  Wed Mar 17 14:39:14 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 17 Mar 2010 13:39:14 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA0032B.4080307@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<e04bdf311003160758j652dc64cib3d700b9fba8432c@mail.gmail.com>
	<20100316155815.DAE983A4097@sparrow.telecommunity.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
	<4BA0032B.4080307@canterbury.ac.nz>
Message-ID: <5c6f2a5d1003170639r73ed8811ydad84fcd0faa3d3a@mail.gmail.com>

On Tue, Mar 16, 2010 at 10:16 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Mark Dickinson wrote:
>>
>> On Tue, Mar 16, 2010 at 3:58 PM, P.J. Eby <pje at telecommunity.com> wrote:
>>
>>> If not, it might be confusing if a number that prints as '.1' compares
>>> unequal to Decimal('.1').
>>
>> Agreed, but this is just your everyday floating-point confusion, to be
>> dealt with by social means (e.g., educating the programmer).
>
> Seems to me that this education would mostly consist of saying
> "don't compare floats and decimals", which is why I think that
> disallowing them in the first place would be better.

I was thinking of something more along the lines of:  "Sure, go ahead
and compare floats and decimals, but be aware that float('1.1') is not
exactly 1.1, so don't complain when 1.1 == Decimal('1.1') returns
False."  For me, this actually a plus of allowing these comparisons:
it makes the education easier.  "Look, the binary float stored for 1.1
is actually larger than 1.1, and here's the proof:  >>> 1.1 >
Decimal('1.1')   -> True."

--
Mark

From dickinsm at gmail.com  Wed Mar 17 14:40:41 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 17 Mar 2010 13:40:41 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003170932.09201.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
	<5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com>
	<201003170932.09201.steve@pearwood.info>
Message-ID: <5c6f2a5d1003170640i957d1d6yfca64e26e3b4e7b4@mail.gmail.com>

On Tue, Mar 16, 2010 at 10:32 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Wed, 17 Mar 2010 03:23:30 am Mark Dickinson wrote:
>> On Tue, Mar 16, 2010 at 4:11 PM, Mark Dickinson <dickinsm at gmail.com>
>> wrote: [...]
>>
>> >>>> Decimal.from_float(1.1) == 1.1
>> >
>> > False
>>
>> Whoops. ?To clarify, this is the pre-patch behaviour; ?post-patch,
>> this gives True.
>
> Whew! You had me worried there for a second. Just to clarify, you are
> proposing:
>
> Decimal.from_float(1.1) == 1.1
> Decimal.('1.1') != float('1.1')

Exactly, yes.

--
Mark

From dickinsm at gmail.com  Wed Mar 17 15:19:51 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 17 Mar 2010 14:19:51 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <594D6395-4B01-4C9C-8417-36866992C338@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
	<594D6395-4B01-4C9C-8417-36866992C338@gmail.com>
Message-ID: <5c6f2a5d1003170719m4e5abe27t1d74371da53b4914@mail.gmail.com>

On Wed, Mar 17, 2010 at 12:36 AM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 16, 2010, at 9:41 AM, Guido van Rossum wrote:
>>
>> Also supporting comparisons but not other mixed operations is going to
>> be confusing. If you are sticking to that behavior I think mixed
>> comparisons should also be ruled out.
>
> The difference is that mixed comparisons currently do give a result,
> but one that is non-sensical. ?The proposal is a make in give a
> meaningful result, not as an extra feature, but in an effort to not
> be wrong.

Yes, exactly.

A number of people (including me, early in the issue tracker
discussion) have suggested that it would be
confusing/inconsistent/strange to allow mixed-type comparisons but not
mixed-type arithmetic.  But it's not clear to me exactly why this
would be a bad thing.

It seems likely that for serious applications there's little benefit
in allowing float<->Decimal comparisons if mixed-type arithmetic isn't
allowed:  if you're checking that b > a, it's probably because you're
about to subtract b from a, or divide b by a, or do some other piece
of arithmetic that depends on that condition being true.  But for
simply playing around with numbers on the command line, and for
helping free people from their floating-point misconceptions, I think
having these float<->Decimal comparisons available is potentially
useful.

I understand the argument about not adding this feature to 2.x if it's
not going to go into 3.x;  but I'm convinced enough that the 2.x
behaviour should change that I'd prefer to add it to both 2.x and 3.x
than to neither.

The other option would be to leave 3.x alone and just add a py3k
warning to 2.x, so that at least there's some indication of where
strange results are coming from.

--
Mark

From dickinsm at gmail.com  Wed Mar 17 15:41:49 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 17 Mar 2010 14:41:49 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003172243.35336.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003170932.09201.steve@pearwood.info>
	<ca471dc21003161601v16a18bcfg44e4e55b38d43ad4@mail.gmail.com>
	<201003172243.35336.steve@pearwood.info>
Message-ID: <5c6f2a5d1003170741v20c64f29t87aa1bffa82af3df@mail.gmail.com>

On Wed, Mar 17, 2010 at 11:43 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Having said all that, I've just re-read the PEP, and spotted a fly in
> the ointment... hash.
>
> If we allow Decimals to compare equal to floats, that presumably implies
> that they need to hash equal. That may be simple enough for integer
> valued floats, but what non-integer values?

That is indeed an issue.  It can be taken care by checking whether the
Decimal instance is exactly representable as a float, and returning
the hash of the corresponding float if so.  From the patch
(http://bugs.python.org/file16544/issue2531.patch):

+        self_as_float = float(self)
+        if Decimal.from_float(self_as_float) == self:
+            return hash(self_as_float)

Strictly speaking this only works if the Decimal -> float conversion
(which relies on the platform strtod) is accurate to within <1ulp, but
that's going to be true in 2.7/3.x on all platforms using David Gay's
str<->float conversion code, and is *probably* true for the system
strtod on any sane platform, so I don't think it's a real issue.

This also slows down the (already slow) hash computation for Decimal a
touch.  Until people start complaining, or show evidence of
applications that require making large dictionaries of Decimal
instances, I'm not going to worry about that either.  It would be easy
to cache the hash value of a Decimal instance if it became necessary.

A less messy (but more intrusive) alternative would be to come up with
a single sane hash function defined for all rational numbers (plus
infinities and nans), and use this function for int, long, float,
Decimal and Fraction types.  I have a candidate function in mind...

Mark

From dickinsm at gmail.com  Wed Mar 17 16:04:10 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 17 Mar 2010 15:04:10 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <ca471dc21003161015k5eede772ucf9286bddd270eda@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
	<5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com>
	<ca471dc21003161015k5eede772ucf9286bddd270eda@mail.gmail.com>
Message-ID: <5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com>

On Tue, Mar 16, 2010 at 5:15 PM, Guido van Rossum <guido at python.org> wrote:
> Definitely some. Stricter comparison rules are a frequent cause of
> problems when code is first ported to 3.x. While you'd think that code
> comparing a float and a Decimal is *already* broken, there's a
> surprising number of situations where that's not necessary the case,
> e.g. when an arbitrary but stable ordering is needed.

Hmm.  Okay.  It seems like backporting the exception isn't a real option, then.

A nitpick:  the current 2.x behaviour fails to give an arbitrary but
stable ordering;  it merely gives an arbitrary ordering:

>>> sorted([Decimal(1), 2, 3.0])
[Decimal('1'), 2, 3.0]
>>> sorted([2, Decimal(1), 3.0])
[3.0, Decimal('1'), 2]

So long as your list contains only floats and Decimals you're fine,
but for a list containing floats, integers and Decimals the ordering
is no longer stable:  the problem, of course, being that int <-> float
and int <-> Decimal comparisons use a rule (compare by numeric value)
that's not compatible with the way that floats and Decimals are
compared.  This seems like yet another possible cause of subtle bugs,
and again would be fixed by the proposed change in behaviour.  On the
other hand, I've not seen any reports of anyone encountering this in
real life.

Mark

Mark

From guido at python.org  Wed Mar 17 18:09:27 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 17 Mar 2010 10:09:27 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> 
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com> 
	<5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> 
	<ca471dc21003161015k5eede772ucf9286bddd270eda@mail.gmail.com> 
	<5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com>
Message-ID: <ca471dc21003171009k36a11d7rea19aa5fbad067bd@mail.gmail.com>

On Wed, Mar 17, 2010 at 8:04 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Tue, Mar 16, 2010 at 5:15 PM, Guido van Rossum <guido at python.org> wrote:
>> Definitely some. Stricter comparison rules are a frequent cause of
>> problems when code is first ported to 3.x. While you'd think that code
>> comparing a float and a Decimal is *already* broken, there's a
>> surprising number of situations where that's not necessary the case,
>> e.g. when an arbitrary but stable ordering is needed.
>
> Hmm. ?Okay. ?It seems like backporting the exception isn't a real option, then.
>
> A nitpick: ?the current 2.x behaviour fails to give an arbitrary but
> stable ordering; ?it merely gives an arbitrary ordering:
>
>>>> sorted([Decimal(1), 2, 3.0])
> [Decimal('1'), 2, 3.0]
>>>> sorted([2, Decimal(1), 3.0])
> [3.0, Decimal('1'), 2]
>
> So long as your list contains only floats and Decimals you're fine,
> but for a list containing floats, integers and Decimals the ordering
> is no longer stable: ?the problem, of course, being that int <-> float
> and int <-> Decimal comparisons use a rule (compare by numeric value)
> that's not compatible with the way that floats and Decimals are
> compared. ?This seems like yet another possible cause of subtle bugs,
> and again would be fixed by the proposed change in behaviour. ?On the
> other hand, I've not seen any reports of anyone encountering this in
> real life.

Ok, I'll try to stay out of the discussion of which solution is best
of our users, and if the outcome is that mixed operations in general
are bad but mixed comparisons are good, I'll trust you. However I want
to reiterate that you really shouldn't improve the situation for 2.7
unless you also forward-port the solution to 3.x.

-- 
--Guido van Rossum (python.org/~guido)

From solipsis at pitrou.net  Wed Mar 17 19:54:33 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 17 Mar 2010 18:54:33 +0000 (UTC)
Subject: [Python-Dev] test_bigmem and test_bigaddrspace broken
Message-ID: <hnr8h8$hbo$1@dough.gmane.org>


Hello,

The title says it all. test_bigmem and test_bigaddrspace are partly 
broken under py3k, and no buildbot ever tests that they run successfully.
As an example, here is a test_bigmem run under a 16GB machine. It gets 
"killed" at some point, probably by the Linux kernel's OOM killer, 
although I instruct the test suite to only use a fraction of the 
available RAM. There are also a couple of failures:


$ ./python -m test.regrtest -v -M 8Gb test_bigmem
== CPython 3.2a0 (py3k, Mar 17 2010, 18:38:10) [GCC 4.3.2]
==   Linux-a.b.c.d-grsec-xxxx-grs-ipv4-64-x86_64-with-debian-lenny-sid
==   /home/antoine/py3k/py3k/build/test_python_10071
test_bigmem
test_capitalize (test.test_bigmem.StrTest) ... Skipping test_capitalize 
because of memory constraint
ok
test_center (test.test_bigmem.StrTest) ... ok
test_compare (test.test_bigmem.StrTest) ... Skipping test_compare because 
of memory constraint
ok
test_concat (test.test_bigmem.StrTest) ... ok
test_contains (test.test_bigmem.StrTest) ... Skipping test_contains 
because of memory constraint
ok
test_count (test.test_bigmem.StrTest) ... Skipping test_count because of 
memory constraint
ok
test_encode (test.test_bigmem.StrTest) ... ERROR
test_encode_ascii (test.test_bigmem.StrTest) ... ok
test_encode_raw_unicode_escape (test.test_bigmem.StrTest) ... ok
test_encode_utf32 (test.test_bigmem.StrTest) ... ok
test_encode_utf7 (test.test_bigmem.StrTest) ... ok
test_endswith (test.test_bigmem.StrTest) ... Skipping test_endswith 
because of memory constraint
ok
test_expandtabs (test.test_bigmem.StrTest) ... Skipping test_expandtabs 
because of memory constraint
ok
test_find (test.test_bigmem.StrTest) ... Skipping test_find because of 
memory constraint
ok
test_format (test.test_bigmem.StrTest) ... Skipping test_format because 
of memory constraint
ok
test_hash (test.test_bigmem.StrTest) ... ok
test_index (test.test_bigmem.StrTest) ... Skipping test_index because of 
memory constraint
ok
test_isalnum (test.test_bigmem.StrTest) ... Skipping test_isalnum because 
of memory constraint
ok
test_isalpha (test.test_bigmem.StrTest) ... Skipping test_isalpha because 
of memory constraint
ok
test_isdigit (test.test_bigmem.StrTest) ... Skipping test_isdigit because 
of memory constraint
ok
test_islower (test.test_bigmem.StrTest) ... Skipping test_islower because 
of memory constraint
ok
test_isspace (test.test_bigmem.StrTest) ... Skipping test_isspace because 
of memory constraint
ok
test_istitle (test.test_bigmem.StrTest) ... Skipping test_istitle because 
of memory constraint
ok
test_isupper (test.test_bigmem.StrTest) ... Skipping test_isupper because 
of memory constraint
ok
test_join (test.test_bigmem.StrTest) ... Skipping test_join because of 
memory constraint
ok
test_ljust (test.test_bigmem.StrTest) ... ok
test_lower (test.test_bigmem.StrTest) ... Skipping test_lower because of 
memory constraint
ok
test_lstrip (test.test_bigmem.StrTest) ... ok
test_repeat (test.test_bigmem.StrTest) ... ok
test_replace (test.test_bigmem.StrTest) ... Skipping test_replace because 
of memory constraint
ok
test_repr_large (test.test_bigmem.StrTest) ... Skipping test_repr_large 
because of memory constraint
ok
test_repr_small (test.test_bigmem.StrTest) ... Skipping test_repr_small 
because of memory constraint
ok
test_rfind (test.test_bigmem.StrTest) ... Skipping test_rfind because of 
memory constraint
ok
test_rindex (test.test_bigmem.StrTest) ... Skipping test_rindex because 
of memory constraint
ok
test_rjust (test.test_bigmem.StrTest) ... ok
test_rstrip (test.test_bigmem.StrTest) ... ok
test_slice_and_getitem (test.test_bigmem.StrTest) ... Skipping 
test_slice_and_getitem because of memory constraint
ok
test_split_large (test.test_bigmem.StrTest) ... Skipping test_split_large 
because of memory constraint
ok
test_split_small (test.test_bigmem.StrTest) ... Skipping test_split_small 
because of memory constraint
ok
test_splitlines (test.test_bigmem.StrTest) ... Skipping test_splitlines 
because of memory constraint
ok
test_startswith (test.test_bigmem.StrTest) ... Skipping test_startswith 
because of memory constraint
ok
test_strip (test.test_bigmem.StrTest) ... ok
test_swapcase (test.test_bigmem.StrTest) ... Skipping test_swapcase 
because of memory constraint
ok
test_title (test.test_bigmem.StrTest) ... Skipping test_title because of 
memory constraint
ok
test_translate (test.test_bigmem.StrTest) ... Skipping test_translate 
because of memory constraint
ok
test_unicode_repr (test.test_bigmem.StrTest) ... Skipping 
test_unicode_repr because of memory constraint
ok
test_unicode_repr_overflow (test.test_bigmem.StrTest) ... Skipping 
test_unicode_repr_overflow because of memory constraint
ok
test_unicode_repr_wide (test.test_bigmem.StrTest) ... Skipping 
test_unicode_repr_wide because of memory constraint
ok
test_upper (test.test_bigmem.StrTest) ... Skipping test_upper because of 
memory constraint
ok
test_zfill (test.test_bigmem.StrTest) ... ok
test_capitalize (test.test_bigmem.BytesTest) ... ok
test_center (test.test_bigmem.BytesTest) ... ok
test_compare (test.test_bigmem.BytesTest) ... ok
test_concat (test.test_bigmem.BytesTest) ... ok
test_contains (test.test_bigmem.BytesTest) ... FAIL
test_count (test.test_bigmem.BytesTest) ... ok
test_decode (test.test_bigmem.BytesTest) ... ok
test_endswith (test.test_bigmem.BytesTest) ... ok
test_expandtabs (test.test_bigmem.BytesTest) ... ok
test_find (test.test_bigmem.BytesTest) ... ok
test_hash (test.test_bigmem.BytesTest) ... ok
test_index (test.test_bigmem.BytesTest) ... ok
test_isalnum (test.test_bigmem.BytesTest) ... ok
test_isalpha (test.test_bigmem.BytesTest) ... ok
test_isdigit (test.test_bigmem.BytesTest) ... ok
test_islower (test.test_bigmem.BytesTest) ... ok
test_isspace (test.test_bigmem.BytesTest) ... ok
test_istitle (test.test_bigmem.BytesTest) ... ok
test_isupper (test.test_bigmem.BytesTest) ... ok
test_join (test.test_bigmem.BytesTest) ... ok
test_ljust (test.test_bigmem.BytesTest) ... ok
test_lower (test.test_bigmem.BytesTest) ... ok
test_lstrip (test.test_bigmem.BytesTest) ... ok
test_repeat (test.test_bigmem.BytesTest) ... ok
test_replace (test.test_bigmem.BytesTest) ... ok
test_rfind (test.test_bigmem.BytesTest) ... ok
test_rindex (test.test_bigmem.BytesTest) ... ok
test_rjust (test.test_bigmem.BytesTest) ... ok
test_rstrip (test.test_bigmem.BytesTest) ... ok
test_slice_and_getitem (test.test_bigmem.BytesTest) ... ok
test_split_large (test.test_bigmem.BytesTest) ... Skipping 
test_split_large because of memory constraint
ok
test_split_small (test.test_bigmem.BytesTest) ... ok
test_splitlines (test.test_bigmem.BytesTest) ... ok
test_startswith (test.test_bigmem.BytesTest) ... ok
test_strip (test.test_bigmem.BytesTest) ... ok
test_swapcase (test.test_bigmem.BytesTest) ... ok
test_title (test.test_bigmem.BytesTest) ... ok
test_translate (test.test_bigmem.BytesTest) ... ok
test_upper (test.test_bigmem.BytesTest) ... ok
test_zfill (test.test_bigmem.BytesTest) ... ok
test_capitalize (test.test_bigmem.BytearrayTest) ... ok
test_center (test.test_bigmem.BytearrayTest) ... ok
test_compare (test.test_bigmem.BytearrayTest) ... ok
test_concat (test.test_bigmem.BytearrayTest) ... ok
test_contains (test.test_bigmem.BytearrayTest) ... FAIL
test_count (test.test_bigmem.BytearrayTest) ... ok
test_decode (test.test_bigmem.BytearrayTest) ... ok
test_endswith (test.test_bigmem.BytearrayTest) ... ok
test_expandtabs (test.test_bigmem.BytearrayTest) ... ok
test_find (test.test_bigmem.BytearrayTest) ... ok
test_index (test.test_bigmem.BytearrayTest) ... ok
test_isalnum (test.test_bigmem.BytearrayTest) ... ok
test_isalpha (test.test_bigmem.BytearrayTest) ... ok
test_isdigit (test.test_bigmem.BytearrayTest) ... ok
test_islower (test.test_bigmem.BytearrayTest) ... ok
test_isspace (test.test_bigmem.BytearrayTest) ... ok
test_istitle (test.test_bigmem.BytearrayTest) ... ok
test_isupper (test.test_bigmem.BytearrayTest) ... ok
test_join (test.test_bigmem.BytearrayTest) ... ok
test_ljust (test.test_bigmem.BytearrayTest) ... ok
test_lower (test.test_bigmem.BytearrayTest) ... ok
test_lstrip (test.test_bigmem.BytearrayTest) ... ok
test_repeat (test.test_bigmem.BytearrayTest) ... ok
test_replace (test.test_bigmem.BytearrayTest) ... ok
test_rfind (test.test_bigmem.BytearrayTest) ... ok
test_rindex (test.test_bigmem.BytearrayTest) ... ok
test_rjust (test.test_bigmem.BytearrayTest) ... ok
test_rstrip (test.test_bigmem.BytearrayTest) ... ok
test_slice_and_getitem (test.test_bigmem.BytearrayTest) ... ok
test_split_small (test.test_bigmem.BytearrayTest) ... ok
test_splitlines (test.test_bigmem.BytearrayTest) ... ok
test_startswith (test.test_bigmem.BytearrayTest) ... ok
test_strip (test.test_bigmem.BytearrayTest) ... ok
test_swapcase (test.test_bigmem.BytearrayTest) ... ok
test_title (test.test_bigmem.BytearrayTest) ... ok
test_translate (test.test_bigmem.BytearrayTest) ... ok
test_upper (test.test_bigmem.BytearrayTest) ... ok
test_zfill (test.test_bigmem.BytearrayTest) ... ok
test_compare (test.test_bigmem.TupleTest) ... Skipping test_compare 
because of memory constraint
ok
test_concat_large (test.test_bigmem.TupleTest) ... Skipping 
test_concat_large because of memory constraint
ok
test_concat_small (test.test_bigmem.TupleTest) ... Skipping 
test_concat_small because of memory constraint
ok
test_contains (test.test_bigmem.TupleTest) ... Skipping test_contains 
because of memory constraint
ok
test_from_2G_generator (test.test_bigmem.TupleTest) ... Skipping 
test_from_2G_generator because of memory constraint
ok
test_from_almost_2G_generator (test.test_bigmem.TupleTest) ... Skipping 
test_from_almost_2G_generator because of memory constraint
ok
test_hash (test.test_bigmem.TupleTest) ... Skipping test_hash because of 
memory constraint
ok
test_index_and_slice (test.test_bigmem.TupleTest) ... Skipping 
test_index_and_slice because of memory constraint
ok
test_repeat_large (test.test_bigmem.TupleTest) ... Skipping 
test_repeat_large because of memory constraint
ok
test_repeat_large_2 (test.test_bigmem.TupleTest) ... Skipping 
test_repeat_large_2 because of memory constraint
ok
test_repeat_small (test.test_bigmem.TupleTest) ... Skipping 
test_repeat_small because of memory constraint
ok
test_repr_large (test.test_bigmem.TupleTest) ... Skipping test_repr_large 
because of memory constraint
ok
test_repr_small (test.test_bigmem.TupleTest) ... ERROR
test_append (test.test_bigmem.ListTest) ... Skipping test_append because 
of memory constraint
ok
test_compare (test.test_bigmem.ListTest) ... Skipping test_compare 
because of memory constraint
ok
test_concat_large (test.test_bigmem.ListTest) ... Skipping 
test_concat_large because of memory constraint
ok
test_concat_small (test.test_bigmem.ListTest) ... Skipping 
test_concat_small because of memory constraint
ok
test_contains (test.test_bigmem.ListTest) ... Skipping test_contains 
because of memory constraint
ok
test_count (test.test_bigmem.ListTest) ... Skipping test_count because of 
memory constraint
ok
test_extend_large (test.test_bigmem.ListTest) ... Skipping 
test_extend_large because of memory constraint
ok
test_extend_small (test.test_bigmem.ListTest) ... Skipping 
test_extend_small because of memory constraint
ok
test_hash (test.test_bigmem.ListTest) ... Skipping test_hash because of 
memory constraint
ok
test_index (test.test_bigmem.ListTest) ... Skipping test_index because of 
memory constraint
ok
test_index_and_slice (test.test_bigmem.ListTest) ... Skipping 
test_index_and_slice because of memory constraint
ok
test_inplace_concat_large (test.test_bigmem.ListTest) ... Skipping 
test_inplace_concat_large because of memory constraint
ok
test_inplace_concat_small (test.test_bigmem.ListTest) ... Skipping 
test_inplace_concat_small because of memory constraint
ok
test_inplace_repeat_large (test.test_bigmem.ListTest) ... Skipping 
test_inplace_repeat_large because of memory constraint
ok
test_inplace_repeat_small (test.test_bigmem.ListTest) ... Skipping 
test_inplace_repeat_small because of memory constraint
ok
test_insert (test.test_bigmem.ListTest) ... Skipping test_insert because 
of memory constraint
ok
test_pop (test.test_bigmem.ListTest) ... Skipping test_pop because of 
memory constraint
ok
test_remove (test.test_bigmem.ListTest) ... Skipping test_remove because 
of memory constraint
ok
test_repeat_large (test.test_bigmem.ListTest) ... Skipping 
test_repeat_large because of memory constraint
ok
test_repeat_small (test.test_bigmem.ListTest) ... Skipping 
test_repeat_small because of memory constraint
ok
test_repr_large (test.test_bigmem.ListTest) ... Skipping test_repr_large 
because of memory constraint
ok
test_repr_small (test.test_bigmem.ListTest) ... Killed



From solipsis at pitrou.net  Wed Mar 17 20:08:33 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 17 Mar 2010 20:08:33 +0100
Subject: [Python-Dev] test_bigmem and test_bigaddrspace broken
In-Reply-To: <20100317185831.GA7296@idyll.org>
References: <hnr8h8$hbo$1@dough.gmane.org> <20100317185831.GA7296@idyll.org>
Message-ID: <1268852913.3507.15.camel@localhost>


Hello,

Le mercredi 17 mars 2010 ? 11:58 -0700, C. Titus Brown a ?crit :
> On Wed, Mar 17, 2010 at 06:54:33PM +0000, Antoine Pitrou wrote:
> > The title says it all. test_bigmem and test_bigaddrspace are partly 
> > broken under py3k, and no buildbot ever tests that they run successfully.
> > As an example, here is a test_bigmem run under a 16GB machine. It gets 
> > "killed" at some point, probably by the Linux kernel's OOM killer, 
> > although I instruct the test suite to only use a fraction of the 
> > available RAM. There are also a couple of failures:
> 
> I can provide login access to a 24gb machine (Linux), if you or others are
> interested.  Sometimes it's running other stuff, but it's got lots of swap :)

Thanks for the proposal.
That wasn't really the point of my message, though :)

As mentioned I have access to a 16GB machine which is able to run the
tests. The problem is that nobody seems interested in maintaining these
tests.
As far as I can say, the initial committer is Thomas Wouters, and not
much happened since then. It seems to me that big companies such as
Google and Apple would be the primarily concerned ones by such
scenarios, and willing to ensure they run fine.

(by the way, even without swapping and in non-debug mode, the tests are
quite long to run)

Regards

Antoine.



From tjreedy at udel.edu  Wed Mar 17 20:34:19 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 17 Mar 2010 15:34:19 -0400
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <ca471dc21003171009k36a11d7rea19aa5fbad067bd@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
	<5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com>
	<ca471dc21003161015k5eede772ucf9286bddd270eda@mail.gmail.com>
	<5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com>
	<ca471dc21003171009k36a11d7rea19aa5fbad067bd@mail.gmail.com>
Message-ID: <hnrars$2cg$1@dough.gmane.org>

On 3/17/2010 1:09 PM, Guido van Rossum wrote:

> Ok, I'll try to stay out of the discussion of which solution is best
> of our users, and if the outcome is that mixed operations in general
> are bad but mixed comparisons are good, I'll trust you. However I want
> to reiterate that you really shouldn't improve the situation for 2.7
> unless you also forward-port the solution to 3.x.

I agree. Improving 2.7 and not 3.2+ would give people a reason to not 
move to 3.x.

tjr




From tlesher at gmail.com  Wed Mar 17 20:53:45 2010
From: tlesher at gmail.com (Tim Lesher)
Date: Wed, 17 Mar 2010 15:53:45 -0400
Subject: [Python-Dev] Removing Windows 95-related code in posixmodule.c
Message-ID: <9613db601003171253y389a17aam68823b225c97a34@mail.gmail.com>

Now that Windows 9x is no longer supported (per PEP 11), would it be
appropriate to replace the ANSI ("A") versions of Windows OS calls in
posixmodule.c with their Unicode ("W") equivalents and remove the
unicode_file_names() function that determines which version to call?

There are a number of places in that module where we try to decide whether
to call ANSI or UNICODE versions (because only the ANSI versions were fully
supported on Windows 9x).  This is still fine on Win32, where the ANSI
versions are still supported for backwards compatibility, but it causes
issues with Windows CE, which doesn't have the ANSI versions at all.

I've been maintaining some fairly ugly patches to cope with this in my
company's Python 2.5.x for CE build, and I'd love to drop them as we move to
a post-2.6 version.

Thanks!
-- 
Tim Lesher <tlesher at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100317/ef78c021/attachment-0001.html>

From raymond.hettinger at gmail.com  Wed Mar 17 21:44:21 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 17 Mar 2010 13:44:21 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <hnrars$2cg$1@dough.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
	<5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com>
	<ca471dc21003161015k5eede772ucf9286bddd270eda@mail.gmail.com>
	<5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com>
	<ca471dc21003171009k36a11d7rea19aa5fbad067bd@mail.gmail.com>
	<hnrars$2cg$1@dough.gmane.org>
Message-ID: <826E9179-6933-4316-91AB-044F5453A97A@gmail.com>


On Mar 17, 2010, at 12:34 PM, Terry Reedy wrote:
> 
> I agree. Improving 2.7 and not 3.2+ would give people a reason to not move to 3.x.
> 

FWIW, I think this is mischaracterizing the proposal.

The spectrum of options from worst to best is
1) compare but give the wrong answer
2) compare but give the right answer
3) refuse to compare.

Py3.x is already in the best position, it refuses to compare.
IOW, is already is as improved as it can get.

P2.6 is in the worst position.  The proposal is to make
it better, but not as good as 3.x.


Raymond



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100317/d7bdabe9/attachment.html>

From steve at pearwood.info  Wed Mar 17 21:59:24 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 18 Mar 2010 07:59:24 +1100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <826E9179-6933-4316-91AB-044F5453A97A@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<hnrars$2cg$1@dough.gmane.org>
	<826E9179-6933-4316-91AB-044F5453A97A@gmail.com>
Message-ID: <201003180759.25229.steve@pearwood.info>

On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote:
> The spectrum of options from worst to best is
> 1) compare but give the wrong answer
> 2) compare but give the right answer
> 3) refuse to compare.

Why is 3 the best? If there is a right answer to give, surely giving the 
right answer it is better than not?


-- 
Steven D'Aprano

From martin at v.loewis.de  Wed Mar 17 22:16:45 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 17 Mar 2010 22:16:45 +0100
Subject: [Python-Dev] test_bigmem and test_bigaddrspace broken
In-Reply-To: <1268852913.3507.15.camel@localhost>
References: <hnr8h8$hbo$1@dough.gmane.org> <20100317185831.GA7296@idyll.org>
	<1268852913.3507.15.camel@localhost>
Message-ID: <4BA146BD.2010909@v.loewis.de>

> As mentioned I have access to a 16GB machine which is able to run the
> tests. The problem is that nobody seems interested in maintaining these
> tests.

So what do you propose to do?

I personally don't see that as a problem. Many rarely-used features get
infrequent testing - that's the nature of rarely-used features. In turn,
they tend to break over time, until somebody comes along and fixes them.
That's open source: scratch your own itches.

> As far as I can say, the initial committer is Thomas Wouters, and not
> much happened since then. It seems to me that big companies such as
> Google and Apple would be the primarily concerned ones by such
> scenarios, and willing to ensure they run fine.

I'm not sure Apple has really Python applications running that use a lot
of memory - being a big company doesn't necessarily mean you operate
large machines. I'm sure Google has applications that use that much
memory, and I wouldn't be surprised if Google has also bug fixes for
various bugs in large objects that they haven't been contributing (yet).

In any case, I think it's a good thing that these tests are there for
people to run if they have the hardware and the time. If somebody wanted
to contribute such hardware as a build slave - that could certainly be
arranged.

Regards,
Martin

From martin at v.loewis.de  Wed Mar 17 22:22:43 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 17 Mar 2010 22:22:43 +0100
Subject: [Python-Dev] Removing Windows 95-related code in posixmodule.c
In-Reply-To: <9613db601003171253y389a17aam68823b225c97a34@mail.gmail.com>
References: <9613db601003171253y389a17aam68823b225c97a34@mail.gmail.com>
Message-ID: <4BA14823.10603@v.loewis.de>

Tim Lesher wrote:
> Now that Windows 9x is no longer supported (per PEP 11), would it be
> appropriate to replace the ANSI ("A") versions of Windows OS calls in
> posixmodule.c with their Unicode ("W") equivalents and remove the
> unicode_file_names() function that determines which version to call?

Not directly. The A versions are still used if you pass byte strings as
file names.

It would be possible to convert byte strings to Unicode strings (using
CP_ACP), but I would only want to consider that if the code complexity
does not increase under that approach (which I doubt).

FWIW, the unicode_file_names function *is* removed in Python 2.7.

Regards,
Martin


From solipsis at pitrou.net  Wed Mar 17 22:26:06 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 17 Mar 2010 21:26:06 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?test=5Fbigmem_and_test=5Fbigaddrspace_brok?=
	=?utf-8?q?en?=
References: <hnr8h8$hbo$1@dough.gmane.org> <20100317185831.GA7296@idyll.org>
	<1268852913.3507.15.camel@localhost> <4BA146BD.2010909@v.loewis.de>
Message-ID: <loom.20100317T222120-910@post.gmane.org>

Martin v. L?wis <martin <at> v.loewis.de> writes:
> 
> > As mentioned I have access to a 16GB machine which is able to run the
> > tests. The problem is that nobody seems interested in maintaining these
> > tests.
> 
> So what do you propose to do?

Nothing. I was pointing out the issue here because it is not obvious that
interested people would follow tracker updates.

Of course, there is also the problem of having tests which never get run and
whose failures don't get noticed (test_zipfile64 is another example, although
the last time I tried it ran ok). And the additional problem of buggy tests
(test_bigmem shouldn't OOM if I ask it to use 8GB on a machine with around 14GB
of free mem).

Regards

Antoine.



From ncoghlan at gmail.com  Wed Mar 17 22:33:19 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Mar 2010 07:33:19 +1000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003180759.25229.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<hnrars$2cg$1@dough.gmane.org>	<826E9179-6933-4316-91AB-044F5453A97A@gmail.com>
	<201003180759.25229.steve@pearwood.info>
Message-ID: <4BA14A9F.10902@gmail.com>

Steven D'Aprano wrote:
> On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote:
>> The spectrum of options from worst to best is
>> 1) compare but give the wrong answer
>> 2) compare but give the right answer
>> 3) refuse to compare.
> 
> Why is 3 the best? If there is a right answer to give, surely giving the 
> right answer it is better than not?

I agree with Steven here - for mixed *arithmetic* refusing to get
involved is a reasonable choice, because the whole point of Decimals is
to get the answers according to base 10 expectations. Allowing implicit
conversion of base 2 floats puts that foundation at risk.

In what way do comparisons carry the same risk? Decimal.from_float shows
that a perfectly adequate mapping from float into the Decimal space can
be made, and the comparisons have a clear well-defined answer. It may be
slightly confusing to those not familiar with the subtleties of limited
precision base 2 vs larger precision base 10, but the boolean result
places them in a clearly different category to my mind.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From raymond.hettinger at gmail.com  Wed Mar 17 22:58:25 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 17 Mar 2010 14:58:25 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003180759.25229.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<hnrars$2cg$1@dough.gmane.org>
	<826E9179-6933-4316-91AB-044F5453A97A@gmail.com>
	<201003180759.25229.steve@pearwood.info>
Message-ID: <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>


On Mar 17, 2010, at 1:59 PM, Steven D'Aprano wrote:

> On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote:
>> The spectrum of options from worst to best is
>> 1) compare but give the wrong answer
>> 2) compare but give the right answer
>> 3) refuse to compare.
> 
> Why is 3 the best? If there is a right answer to give, surely giving the 
> right answer it is better than not?

From the early days of the decimal module,
we've thought that mixed float-decimal operations
are 1) a bit perilous and 2) have few, if any good
use cases.     

Accordingly, any mixed operations should be explicit 
rather than implicit:

     Decimal('1.1') + Decimal.from_float(2.2)

is better than:

     Decimal('1.1') + 2.2

To help the user avoid confusion, we flag the latter with a TypeError:
unsupported operand type(s) for +: 'Decimal' and 'float'.

Unfortunately, in Py2.x, implicit mixed comparisons do not
raise an exception, and instead will silently fail by giving 
an incorrect answer: 

    >>> Decimal('1.1') < 2.2
    False

IMO, raising the exception is the right thing to do.
Short of that though, if we're going to give a result,
it should at least be a correct one.


Raymond


New zen:

* right answers are better wrong
* but ill-formed questions are best not answered at all


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100317/48095c0b/attachment-0001.html>

From guido at python.org  Thu Mar 18 00:07:44 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 17 Mar 2010 16:07:44 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <826E9179-6933-4316-91AB-044F5453A97A@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> 
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com> 
	<5c6f2a5d1003161007q2503f7acy3bc131fc14b9ae10@mail.gmail.com> 
	<ca471dc21003161015k5eede772ucf9286bddd270eda@mail.gmail.com> 
	<5c6f2a5d1003170804s35eed086j14c1e48d81a53193@mail.gmail.com> 
	<ca471dc21003171009k36a11d7rea19aa5fbad067bd@mail.gmail.com> 
	<hnrars$2cg$1@dough.gmane.org>
	<826E9179-6933-4316-91AB-044F5453A97A@gmail.com>
Message-ID: <ca471dc21003171607p47c43b1cxac73e3a5ec0c78a8@mail.gmail.com>

On Wed, Mar 17, 2010 at 1:44 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 17, 2010, at 12:34 PM, Terry Reedy wrote:
>
> I agree. Improving 2.7 and not 3.2+ would give people a reason to not move
> to 3.x.
>
>
> FWIW, I think this is mischaracterizing the proposal.

> The spectrum of options from worst to best is
> 1) compare but give the wrong answer
> 2) compare but give the right answer
> 3) refuse to compare.

> Py3.x is already in the best position, it refuses to compare.
> IOW, is already is as improved as it can get.

> P2.6 is in the worst position. ?The proposal is to make
> it better, but not as good as 3.x.

Some people have argued that (2) is better than (3). At the very
least, there is a marked discontinuity in this spectrum between (2)
and (3). With (1), users are less likely to rely on this feature in
2.x and then have no recourse when converting to 3.x.

-- 
--Guido van Rossum (python.org/~guido)

From greg.ewing at canterbury.ac.nz  Thu Mar 18 00:49:24 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 18 Mar 2010 12:49:24 +1300
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003170946.39457.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
	<4BA0032B.4080307@canterbury.ac.nz>
	<201003170946.39457.steve@pearwood.info>
Message-ID: <4BA16A84.9010307@canterbury.ac.nz>

Steven D'Aprano wrote:

> More explicit than someDecimal == someFloat? Seems pretty explicit to 
> me.

Yes. I mean at least specifying either float(someDecimal) == someFloat 
or someDecimal == Decimal(someFloat). Preferably also whether the
conversion is to be as exact as possible or on a minimum-digits basis.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Mar 18 00:58:40 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 18 Mar 2010 12:58:40 +1300
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <ca471dc21003161601v16a18bcfg44e4e55b38d43ad4@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<5c6f2a5d1003160911x44f04f28s763483530142298f@mail.gmail.com>
	<5c6f2a5d1003160923v1519a3advf95df39cb7f884af@mail.gmail.com>
	<201003170932.09201.steve@pearwood.info>
	<ca471dc21003161601v16a18bcfg44e4e55b38d43ad4@mail.gmail.com>
Message-ID: <4BA16CB0.4040908@canterbury.ac.nz>

Guido van Rossum wrote:
> in some
> "intuitive complexity" sense an int is a simpler type than a float and
> a float is a simpler type than a Decimal

I don't think this analogy holds. In a mathematical sense,
ints are a subset of reals, but binary and decimal floats
are just alternative approximate representations of reals,
neither one being inherently preferable over the other.

One could argue that since all binary floats are exactly
representable in decimal but not vice versa, decimal should
be regarded as the wider type. But even this doesn't hold
when you have a limited number of decimal digits available,
which you always do at any given moment with the Decimal
type.

And even if there are enough digits, an exact conversion
mightn't be what you really want. This problem doesn't arise
with int->float conversion -- there is only one obvious way
of chopping it to fit.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Mar 18 01:02:31 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 18 Mar 2010 13:02:31 +1300
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <594D6395-4B01-4C9C-8417-36866992C338@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
	<594D6395-4B01-4C9C-8417-36866992C338@gmail.com>
Message-ID: <4BA16D97.2010502@canterbury.ac.nz>

Raymond Hettinger wrote:

> Python 3 doesn't need it because it is possible to not give a result
> at all.  Python 2 does need it because we have to give *some*
> result.

That's not true -- it's possible for comparisons to raise
an exception in 2.x, and they sometimes do already:

Python 2.5.4 (r254:67916, May 15 2009, 15:21:20)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> 1+2j < 3+4j
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: no ordering relation is defined for complex numbers

-- 
Greg

From raymond.hettinger at gmail.com  Thu Mar 18 02:03:36 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 17 Mar 2010 18:03:36 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA16D97.2010502@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<ca471dc21003160941s28b8097cxdbe9a6d3c33ecb1f@mail.gmail.com>
	<594D6395-4B01-4C9C-8417-36866992C338@gmail.com>
	<4BA16D97.2010502@canterbury.ac.nz>
Message-ID: <C845C90F-33D5-48A5-8CF4-9E53D58D83E4@gmail.com>


On Mar 17, 2010, at 5:02 PM, Greg Ewing wrote:

> Raymond Hettinger wrote:
> 
>> Python 3 doesn't need it because it is possible to not give a result
>> at all.  Python 2 does need it because we have to give *some*
>> result.
> 
> That's not true -- it's possible for comparisons to raise
> an exception in 2.x, and they sometimes do already:

Complex objects do not support __float__.  Decimal objects do.
If an object supports __float__, then a float comparison coerces 
its other argument via __float__ and the other argument
never gets a chance to raise an exception.

>>> class D:
	def __float__(self):
		return 3.14

	
>>> float(D())
3.1400000000000001
>>> float(complex(3.14))

Traceback (most recent call last):
  File "<pyshell#14>", line 1, in <module>
    float(complex(3.14))
TypeError: can't convert complex to float

>>> D() < 10.0
True
>>> complex(3.14) < 10.0

Traceback (most recent call last):
  File "<pyshell#16>", line 1, in <module>
    complex(3.14) < 10.0
TypeError: no ordering relation is defined for complex numbers


Raymond



From jafo at tummy.com  Thu Mar 18 12:58:44 2010
From: jafo at tummy.com (Sean Reifschneider)
Date: Thu, 18 Mar 2010 05:58:44 -0600
Subject: [Python-Dev] Bug? syslog.openlog using ident "python" by default.
Message-ID: <20100318115844.GA7374@tummy.com>

I'd appreciate some opinions on this.  Personally, I'm in the "the current
code is buggy" camp.  :-)  I can code up the changes to the syslog module
if we decide that's the right way to go.  Looks like Raymond, Guido, and I
are the last ones to do syslog-specific changes to this module in the last
15 years.

If you call:

   from syslog import syslog, openlog
   syslog('My error message')

Something like the following gets logged:

   Mar 18 05:20:22 guin python: My error message
                        ^^^^^^

Where I'm annoyed by the "python" in the above.  This is pulled from
the C argv[0].  IMHO, what we really want is the Python sys.argv[0].
Because of this feature, I always have to do the following when I'm using
syslog:

   openlog(os.path.basename(sys.argv[0]))

I would propose changing the Python syslog() call to do the C equivalent of:

   if openlog_hasnt_been_called_before_now:
      if sys.argv[0]:
         syslog.openlog(os.path.basename(sys.argv[0]))

In other words, if there's a script name and openlog hasn't already been
called, call openlog with the basename of the script name.

This is effectively what happens in normal C code that calls syslog.

Note that the Python syslog.openlog says that the default for ident is
"(usually) 'syslog'".

The benefit of this change is that you get a more identifiable ident string
in Python programs, so when you look at the logs you can tell what script
it came from, not just that it came from some Python program.  One way of
looking at it would be that the syslog module logging "python" as the
program name is a bug.

The downside I see is that there might be some users doing log scraping
depending on this (IMHO, buggy) log ident.

Something else just occurred to me though, a nice enhancement would be for
openlog() to have a "None" default for ident, something like:

   orig_openlog = openlog
   def openlog(ident = None, logopt = 0, facility = LOG_USER)
      if ident == None:
         if sys.argv[0]:
            ident = os.path.basename(sys.argv[0])
         else:
            ident = 'python'
      orig_openlog(ident, logopt, facility)

So you could call openlog and rely on the default ident (sys.argv[0]),
and set the logopt and facility.

The gory details of why this is occurring are: If you don't call
"openlog()" before "syslog()", the system syslog() function calls
something similar to: "openlog(basename(argv[0]))", which causes the
"ident" of the syslog messages to be "python" rather than the specific
program.

Thoughts?

Thanks,
Sean
-- 
 "Every increased possession loads us with new weariness."
                 -- John Ruskin
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability


From steve at pearwood.info  Thu Mar 18 13:23:18 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 18 Mar 2010 23:23:18 +1100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
Message-ID: <201003182323.19458.steve@pearwood.info>

On Thu, 18 Mar 2010 08:58:25 am Raymond Hettinger wrote:
> On Mar 17, 2010, at 1:59 PM, Steven D'Aprano wrote:
> > On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote:
> >> The spectrum of options from worst to best is
> >> 1) compare but give the wrong answer
> >> 2) compare but give the right answer
> >> 3) refuse to compare.
> >
> > Why is 3 the best? If there is a right answer to give, surely
> > giving the right answer it is better than not?
>
> From the early days of the decimal module,
> we've thought that mixed float-decimal operations
> are 1) a bit perilous and 2) have few, if any good
> use cases.

When it comes to *arithmetic* operations, I agree. Is there anyone on 
python-dev willing to argue the case for allowing implicit mixed 
float/Decimal arithmetic operations? The arguments in the PEP seem 
pretty convincing to me, and I'm not suggesting we change that.

But comparison operations are different. For starters, you don't need to 
worry about whether to return a float or a Decimal, because you always 
get a bool. In theory, both Decimals and floats are representations of 
the same underlying thing, namely real numbers, and it seems strange to 
me that I can't ask whether two such real numbers are equal just 
because their storage implementation is different.

I can see three reasonable reasons for avoiding mixed comparisons:

(1) To avoid confusing float-naive users (but they're confused by pure 
float comparisons too).

(2) To avoid mixed arithmetic operations (but comparisons aren't 
arithmetic).

(3) If Decimals and floats compare equal, they must hash equal, and 
currently they don't (but Mark Dickinson thinks he has a solution for 
that).



> Accordingly, any mixed operations should be explicit
> rather than implicit:
>
>      Decimal('1.1') + Decimal.from_float(2.2)
>
> is better than:
>
>      Decimal('1.1') + 2.2

Agreed. The user should explicitly choose whether they want a float 
answer or a Decimal answer.


> To help the user avoid confusion, we flag the latter with a
> TypeError: unsupported operand type(s) for +: 'Decimal' and 'float'.
>
> Unfortunately, in Py2.x, implicit mixed comparisons do not
> raise an exception, and instead will silently fail by giving
>
> an incorrect answer:
>     >>> Decimal('1.1') < 2.2
>     False

That is clearly the wrong thing to do. 

Do you envisage any problems from allowing this instead?

>>> Decimal('1.1') < 2.2
True


> IMO, raising the exception is the right thing to do.
> Short of that though, if we're going to give a result,
> it should at least be a correct one.

+1




-- 
Steven D'Aprano

From raymond.hettinger at gmail.com  Thu Mar 18 18:55:41 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Thu, 18 Mar 2010 10:55:41 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003182323.19458.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
Message-ID: <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>


On Mar 18, 2010, at 5:23 AM, Steven D'Aprano wrote:

> On Thu, 18 Mar 2010 08:58:25 am Raymond Hettinger wrote:
>> On Mar 17, 2010, at 1:59 PM, Steven D'Aprano wrote:
>>> On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote:
>>>> The spectrum of options from worst to best is
>>>> 1) compare but give the wrong answer
>>>> 2) compare but give the right answer
>>>> 3) refuse to compare.
>>> 
>>> Why is 3 the best? If there is a right answer to give, surely
>>> giving the right answer it is better than not?
>> 
>> From the early days of the decimal module,
>> we've thought that mixed float-decimal operations
>> are 1) a bit perilous and 2) have few, if any good
>> use cases.
> 
> When it comes to *arithmetic* operations, I agree. Is there anyone on 
> python-dev willing to argue the case for allowing implicit mixed 
> float/Decimal arithmetic operations? The arguments in the PEP seem 
> pretty convincing to me, and I'm not suggesting we change that.
> 
> But comparison operations are different. For starters, you don't need to 
> worry about whether to return a float or a Decimal, because you always 
> get a bool. In theory, both Decimals and floats are representations of 
> the same underlying thing, namely real numbers, and it seems strange to 
> me that I can't ask whether two such real numbers are equal just 
> because their storage implementation is different.
> 
> I can see three reasonable reasons for avoiding mixed comparisons:
> 
> (1) To avoid confusing float-naive users (but they're confused by pure 
> float comparisons too).
> 
> (2) To avoid mixed arithmetic operations (but comparisons aren't 
> arithmetic).
> 
> (3) If Decimals and floats compare equal, they must hash equal, and 
> currently they don't (but Mark Dickinson thinks he has a solution for 
> that).

Thanks for the thoughtful post.

The question is which behavior is most useful, most of the time.
If a person truly wants to do mixed comparisons, it's trivially
easy to do so in a way that is explicit:

   somedecimal < Decimal.from_float(somefloat)

My thought is that intentional mixed compares of float and decimal 
are very rare relative to unintentional cases.  IOW, most of the
time that x<y makes a float/decimal comparison, it is actually an error
(or the user simply doesn't understand what his or her code is
actually doing).  That user is best served by refusing the temptation
to guess that they really wanted to go down this path.

The zen of this particular problem:

    explicit is already easy
    implicit is likely to let errors pass silently
    therefore, explicit is better than implicit

   the special and rare case of really wanting a float/decimal comparison
   is rare enough that it isn't worth making the 
   operation implicit, especially when the explicit option is so easy.

Do you want protection from probable errors or do you have
compelling use cases for implicit conversion behavior
where an explicit conversion would have been too much of 
a burden?


Raymond






From v+python at g.nevcal.com  Thu Mar 18 19:27:06 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 18 Mar 2010 11:27:06 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003182323.19458.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
Message-ID: <4BA2707A.80801@g.nevcal.com>

On 3/18/2010 5:23 AM, Steven D'Aprano wrote:
> On Thu, 18 Mar 2010 08:58:25 am Raymond Hettinger wrote:
>    
>> On Mar 17, 2010, at 1:59 PM, Steven D'Aprano wrote:
>>      
>>> On Thu, 18 Mar 2010 07:44:21 am Raymond Hettinger wrote:
>>>        
>>>> The spectrum of options from worst to best is
>>>> 1) compare but give the wrong answer
>>>> 2) compare but give the right answer
>>>> 3) refuse to compare.
>>>>          
>>> Why is 3 the best? If there is a right answer to give, surely
>>> giving the right answer it is better than not?
>>>        
>>  From the early days of the decimal module,
>> we've thought that mixed float-decimal operations
>> are 1) a bit perilous and 2) have few, if any good
>> use cases.
>>      
> When it comes to *arithmetic* operations, I agree. Is there anyone on
> python-dev willing to argue the case for allowing implicit mixed
> float/Decimal arithmetic operations? The arguments in the PEP seem
> pretty convincing to me, and I'm not suggesting we change that.
>
> But comparison operations are different. For starters, you don't need to
> worry about whether to return a float or a Decimal, because you always
> get a bool. In theory, both Decimals and floats are representations of
> the same underlying thing, namely real numbers, and it seems strange to
> me that I can't ask whether two such real numbers are equal just
> because their storage implementation is different.
>
> I can see three reasonable reasons for avoiding mixed comparisons:
>
> (1) To avoid confusing float-naive users (but they're confused by pure
> float comparisons too).
>
> (2) To avoid mixed arithmetic operations (but comparisons aren't
> arithmetic).
>
> (3) If Decimals and floats compare equal, they must hash equal, and
> currently they don't (but Mark Dickinson thinks he has a solution for
> that).
>
>
>
>    
>> Accordingly, any mixed operations should be explicit
>> rather than implicit:
>>
>>       Decimal('1.1') + Decimal.from_float(2.2)
>>
>> is better than:
>>
>>       Decimal('1.1') + 2.2
>>      
> Agreed. The user should explicitly choose whether they want a float
> answer or a Decimal answer.
>
>
>    
>> To help the user avoid confusion, we flag the latter with a
>> TypeError: unsupported operand type(s) for +: 'Decimal' and 'float'.
>>
>> Unfortunately, in Py2.x, implicit mixed comparisons do not
>> raise an exception, and instead will silently fail by giving
>>
>> an incorrect answer:
>>      >>>  Decimal('1.1')<  2.2
>>      False
>>      
> That is clearly the wrong thing to do.
>
> Do you envisage any problems from allowing this instead?
>
>    
>>>> Decimal('1.1')<  2.2
>>>>          
> True
>    

Yes.

As any non-na?ve float user is aware, the proper form of float 
comparisons is not to use < or > or == or !=, but rather, instead of 
using < (to follow along with your example), one should use:

Decimal('1.1') - 2.2 < epsilon

However, while even this is only useful in certain circumstances as a 
gross simplification [1], it immediately shows the need to do mixed 
arithmetic to produce (sometimes) correct results.  More correct 
comparisons require much more code (even the 20-line C code in [1], 
which understands the float format to some extent, admits to being 
deficient in some circumstances).

For all the reasons that mixed decimal and float arithmetic is bad, 
mixed decimal and float comparisons are also bad.  To do proper 
comparisons, you need to know the number of significant digits of both 
numbers, and the precision and numeric ranges being dealt with by the 
application.

For the single purpose of sorting, one could make an argument that not 
knowing the significant digits, precision, and numeric ranges, that the 
sort would probably produce results where floats and decimals that 
should compare equal would be clustered similarly as they would if the 
significant digits, precision, and numeric ranges were known, and that 
would probably be close to truth, but only if the decimal vs float key 
were the last in the composite sort key.  I don't think Python informs 
its comparison operations that it is being used as part of a sort, nor 
would there be a way for user-written sorts to inform the comparison 
operations of that fact.

Seems like it would be better to raise an exception, and in the 
documentation for the exception point out that turning off the exception 
(if it should be decided that that should be possible, which could be 
good for compatibility), would regress to the current behavior, which 
doesn't sort numerically, but by type.

[1]
http://www.cprogramming.com/tutorial/floating_point/understanding_floating_point_representation.html


Glenn

From dickinsm at gmail.com  Thu Mar 18 19:41:08 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 18 Mar 2010 18:41:08 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
Message-ID: <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>

On Thu, Mar 18, 2010 at 5:55 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> My thought is that intentional mixed compares of float and decimal
> are very rare relative to unintentional cases. ?IOW, most of the
> time that x<y makes a float/decimal comparison, it is actually an error
> (or the user simply doesn't understand what his or her code is
> actually doing). ?That user is best served by refusing the temptation
> to guess that they really wanted to go down this path.

In this case, could we consider again the idea of making these
comparisons produce TypeErrors in 2.7?  That's what's been requested
or proposed multiple times on the tracker issue (by Imri Goldberg[1],
Jeremy Dunck[2], Bert Hughes[3] and Stefan Krah[4]), and also by
participants in this discussion (Greg Ewing).

I'm only seeing two arguments against this at the moment: (1) it has
the potential to break code that relies on being able to sort
heterogeneous lists.  But given that heterogeneous lists can't be
sorted stably anyway (see my earlier post about lists containing ints,
floats and Decimals), perhaps this is an acceptable risk. (2) A few of
the posters here (Steven, Nick, and me) feel that it's slightly more
natural to allow these comparisons;  but I think the argument's fairly
evenly balanced at the moment between those who'd prefer an exception
and those who'd prefer to allow the comparisons.

I'd really like to get this sorted for 2.7:  as far as I'm concerned,
either of the proposed behaviours (raise an exception, or allow
comparisons) would be an improvement on the current 2.7 behaviour.

Could everyone live with making float<->Decimal comparisons raise an
exception in 2.7?

Mark

[1] http://bugs.python.org/issue2531#msg83691
[2] http://bugs.python.org/issue2531#msg83818
[3] http://bugs.python.org/issue2531#msg97891
[4] http://bugs.python.org/issue2531#msg98217

From solipsis at pitrou.net  Thu Mar 18 19:44:52 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 18 Mar 2010 18:44:52 +0000 (UTC)
Subject: [Python-Dev] Decimal &lt;-&gt; float comparisons in py3k.
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<4BA2707A.80801@g.nevcal.com>
Message-ID: <loom.20100318T193940-109@post.gmane.org>

Glenn Linderman <v+python <at> g.nevcal.com> writes:
> 
> For all the reasons that mixed decimal and float arithmetic is bad, 
> mixed decimal and float comparisons are also bad.  To do proper 
> comparisons, you need to know the number of significant digits of both 
> numbers, and the precision and numeric ranges being dealt with by the 
> application.

What is a "proper comparison"?
A comparison is a lossy operation by construction (it returns only 1 bit of
output, or 2 bits for rich comparison), so the loss in precision that results
from mixing float and decimal operands should not be a concern here.

Unless there are situations where the comparison algorithm might return wrong
results (are there?), I don't see why we should forbid such comparisons.

> Seems like it would be better to raise an exception, and in the 
> documentation for the exception point out that turning off the exception 
> (if it should be decided that that should be possible, which could be 
> good for compatibility), would regress to the current behavior, which 
> doesn't sort numerically, but by type.

Regressing to an useless behaviour sounds like a deliberate annoyance. I don't
think this proposal should be implemented.

Regards

Antoine.



From fuzzyman at voidspace.org.uk  Thu Mar 18 19:48:48 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 18 Mar 2010 18:48:48 +0000
Subject: [Python-Dev] Decimal &lt;-&gt; float comparisons in py3k.
In-Reply-To: <loom.20100318T193940-109@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<4BA2707A.80801@g.nevcal.com>
	<loom.20100318T193940-109@post.gmane.org>
Message-ID: <4BA27590.9020304@voidspace.org.uk>

On 18/03/2010 18:44, Antoine Pitrou wrote:
> [snip...]
>> Seems like it would be better to raise an exception, and in the
>> documentation for the exception point out that turning off the exception
>> (if it should be decided that that should be possible, which could be
>> good for compatibility), would regress to the current behavior, which
>> doesn't sort numerically, but by type.
>>      
> Regressing to an useless behaviour sounds like a deliberate annoyance. I don't
> think this proposal should be implemented.
>    

I agree, comparisons here have completely defined semantics - it sounds 
crazy not to allow it. (The argument 'because some programmers might do 
it without realising' doesn't hold much water with me.)

Michael

> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From steve at pearwood.info  Thu Mar 18 20:20:41 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 19 Mar 2010 06:20:41 +1100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
Message-ID: <201003190620.42295.steve@pearwood.info>

On Fri, 19 Mar 2010 05:41:08 am Mark Dickinson wrote:

> I'd really like to get this sorted for 2.7:  as far as I'm concerned,
> either of the proposed behaviours (raise an exception, or allow
> comparisons) would be an improvement on the current 2.7 behaviour.
>
> Could everyone live with making float<->Decimal comparisons raise an
> exception in 2.7?

Yes. I would far prefer an exception than the current incorrect results 
in 2.6.



-- 
Steven D'Aprano

From steve at pearwood.info  Thu Mar 18 20:34:19 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 19 Mar 2010 06:34:19 +1100
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA2707A.80801@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<4BA2707A.80801@g.nevcal.com>
Message-ID: <201003190634.19250.steve@pearwood.info>

On Fri, 19 Mar 2010 05:27:06 am Glenn Linderman wrote:

> > Do you envisage any problems from allowing this instead?
> >
> >>>> Decimal('1.1')<  2.2
> >
> > True
>
> Yes.
>
> As any non-na?ve float user is aware, the proper form of float
> comparisons is not to use < or > or == or !=, but rather, instead of
> using < (to follow along with your example), one should use:
>
> Decimal('1.1') - 2.2 < epsilon

And yet we allow 

1.1 < 2.2

instead of forcing users to do the "proper form". One can only wonder 
why the various standards (actual and de-facto) for floating point 
allows comparisons at all.

But they do, and so does Python, and frankly even if the only reason is 
to satisfy lazy coders who don't have a requirement for high accuracy, 
then that's a good reason in my book, and one equally applicable to 
Decimal and float.



-- 
Steven D'Aprano

From robert.kern at gmail.com  Thu Mar 18 20:45:50 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 18 Mar 2010 14:45:50 -0500
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA2707A.80801@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>
	<4BA2707A.80801@g.nevcal.com>
Message-ID: <hntvtf$593$1@dough.gmane.org>

On 2010-03-18 13:27 PM, Glenn Linderman wrote:

> As any non-na?ve float user is aware, the proper form of float
> comparisons is not to use < or > or == or !=, but rather, instead of
> using < (to follow along with your example), one should use:
>
> Decimal('1.1') - 2.2 < epsilon

Not at all. This is quite incorrect for most use cases. Fuzzy comparisons are 
sometimes useful for equality testing, but almost never for ordering.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From v+python at g.nevcal.com  Thu Mar 18 21:35:55 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 18 Mar 2010 13:35:55 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <hntvtf$593$1@dough.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<4BA2707A.80801@g.nevcal.com>
	<hntvtf$593$1@dough.gmane.org>
Message-ID: <4BA28EAB.3000707@g.nevcal.com>

On 3/18/2010 12:45 PM, Robert Kern wrote:
> On 2010-03-18 13:27 PM, Glenn Linderman wrote:
>
>> As any non-na?ve float user is aware, the proper form of float
>> comparisons is not to use < or > or == or !=, but rather, instead of
>> using < (to follow along with your example), one should use:
>>
>> Decimal('1.1') - 2.2 < epsilon
>
> Not at all. This is quite incorrect for most use cases. Fuzzy 
> comparisons are sometimes useful for equality testing, but almost 
> never for ordering.
>

I wondered if anyone would catch me on that!  I did it mostly to keep 
the same example going, but partly to see if there were any real 
floating point experts in the discussion!  You'll have noted my 
reference was to using the technique for equality.

From omega_force2003 at yahoo.com  Thu Mar 18 21:54:20 2010
From: omega_force2003 at yahoo.com (OMEGA RED)
Date: Thu, 18 Mar 2010 13:54:20 -0700 (PDT)
Subject: [Python-Dev] Is Scala better than Python ???
Message-ID: <405517.58108.qm@web34408.mail.mud.yahoo.com>

To All,
?
I have done some more review of python and Scala and I am very impressed with both languages.? However, it seems as though that Scala is far more able to support parallel processing than Python ??? Is this true ??
?
O.R. 


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100318/adbc4b6e/attachment.html>

From phd at phd.pp.ru  Thu Mar 18 22:04:40 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Fri, 19 Mar 2010 00:04:40 +0300
Subject: [Python-Dev] Is Scala better than Python ???
In-Reply-To: <405517.58108.qm@web34408.mail.mud.yahoo.com>
References: <405517.58108.qm@web34408.mail.mud.yahoo.com>
Message-ID: <20100318210440.GA4142@phd.pp.ru>

Hello.

   We'are sorry but we cannot help you. This mailing list is for working on
developing Python (fixing bugs and adding new features to Python itself); if
you're having problems using Python, please find another forum. Probably
python-list (comp.lang.python) news group/mailing list is the best place. See
http://www.python.org/community/lists/ for other lists/news groups/forums.
Thank you for understanding.

On Thu, Mar 18, 2010 at 01:54:20PM -0700, OMEGA RED wrote:
> To All,
> ?
> I have done some more review of python and Scala and I am very impressed with both languages.? However, it seems as though that Scala is far more able to support parallel processing than Python ??? Is this true ??
> ?
> O.R. 

Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From tjreedy at udel.edu  Thu Mar 18 22:38:06 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 18 Mar 2010 17:38:06 -0400
Subject: [Python-Dev] Joel Spolsky on Mercurial
Message-ID: <hnu6fu$ufa$1@dough.gmane.org>

In what might be his last Joel on Software post (for awhile, at least)
http://www.joelonsoftware.com/items/2010/03/17.html

Summary:
Early 2009: Joel disses DVCSes. His programmers switch from subversion 
to hg. Joel grumbles. His programmers develop an hg-related product. 
Joel takes a better look.

Joel sees the light: Subversion controls versions, hg controls changes; 
this make merging much easier.
Joel flips (his opinion): Switch now!

Joel write hg tutorial: http://hginit.com/
It starts with "Subversion Re-education" for existing version-oriented 
system users and continue with "Ground up Mercurial" where new vcs users 
can start. It has several simple examples.

Having gotten that far, I think this might be worth referencing in new 
dev docs.

Terry Jan Reedy



From ncoghlan at gmail.com  Thu Mar 18 22:48:33 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Mar 2010 07:48:33 +1000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
Message-ID: <4BA29FB1.9020909@gmail.com>

Mark Dickinson wrote:
> Could everyone live with making float<->Decimal comparisons raise an
> exception in 2.7?

I could, with the caveat that *if* this causes problems for real world
code, then changing it to produce the correct answer (as per your patch)
should be applied as a bug fix in both 2.7 and 3.2.

Note that even in Py3k there are some fairly serious weirdnesses kicking
around due to the intransitive nature of numeric equality though:

>>> from decimal import Decimal as dec
>>> set((1, 1.0, dec("1.0")))
{1}
>>> set((1.0, dec("1.0")))
{1.0, Decimal('1.0')}

>>> d = {}
>>> from decimal import Decimal as dec
>>> d[1] = d[1.0] = d[dec("1.0")] = 42
>>> d
{1: 42}
>>> d[1.0] = d[dec("1.0")] = 42
>>> d
{1: 42}
>>> del d[1]
>>> d[1.0] = d[dec("1.0")] = 42
>>> d
{1.0: 42, Decimal('1.0'): 42}

When there is a clear, correct way (based on Decimal.from_float) to make
numeric comparison behave in accordance with the rules of mathematics,
do we really want to preserve strange, unintuitive behaviour like the above?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From glenn at nevcal.com  Thu Mar 18 21:31:33 2010
From: glenn at nevcal.com (Glenn Linderman)
Date: Thu, 18 Mar 2010 13:31:33 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <201003190634.19250.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003182323.19458.steve@pearwood.info>	<4BA2707A.80801@g.nevcal.com>
	<201003190634.19250.steve@pearwood.info>
Message-ID: <4BA28DA5.8050608@nevcal.com>

On 3/18/2010 12:34 PM, Steven D'Aprano wrote:
> On Fri, 19 Mar 2010 05:27:06 am Glenn Linderman wrote:
>
>    
>>> Do you envisage any problems from allowing this instead?
>>>
>>>        
>>>>>> Decimal('1.1')<   2.2
>>>>>>              
>>> True
>>>        
>> Yes.
>>
>> As any non-na?ve float user is aware, the proper form of float
>> comparisons is not to use<  or>  or == or !=, but rather, instead of
>> using<  (to follow along with your example), one should use:
>>
>> Decimal('1.1') - 2.2<  epsilon
>>      
> And yet we allow
>
> 1.1<  2.2
>
> instead of forcing users to do the "proper form". One can only wonder
> why the various standards (actual and de-facto) for floating point
> allows comparisons at all.
>    

Hard to tell

1.1 < 2.2

from the second line of

diff = 1.1 - 2.2
diff < epsilon

Hard to enforce the "proper form", without being omniscient, and the 
"proper form" isn't always as simple as my example, to be truly proper 
often requires a lot more code (and a lot more omniscience).

I'm +1 on adding an exception for Decimal/float comparisons, -0 on 
allowing it to be turned off to achieve compatible behavior, since 
Raymond pointed out that the compatible sorting behavior isn't stable in 
the presence of int, Decimal, and float.


> But they do, and so does Python, and frankly even if the only reason is
> to satisfy lazy coders who don't have a requirement for high accuracy,
> then that's a good reason in my book, and one equally applicable to
> Decimal and float.
>    

A point well understood, but I personally would rather force 
Decimal/float comparisons to be explicitly converted to be lazily 
compared, if the arithmetic operations are forced to be explicit.


From v+python at g.nevcal.com  Fri Mar 19 00:25:53 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 18 Mar 2010 16:25:53 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA29FB1.9020909@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
Message-ID: <4BA2B681.6090709@g.nevcal.com>

On 3/18/2010 2:48 PM, Nick Coghlan wrote:
> When there is a clear, correct way (based on Decimal.from_float) to make
> numeric comparison behave in accordance with the rules of mathematics,
> do we really want to preserve strange, unintuitive behaviour like the above?
>
> Cheers,
> Nick.
>    

I'm aware of nothing that prevents the lazy coder from having a class 
unifiedNumber in his toolbox that implements his favorite type of 
conversions from various numeric types to whatever he thinks is the 
"best" one for his application, and then using it places where sources 
might be of various other numeric types.  I'm aware of nothing that 
would prevent the lazy coder from implement comparison operators and 
even arithmetic operators on such a class.

I believe, but haven't proven, and haven't used Python long enough to 
"just know", that such class could even implement what would appear to 
be operators that would seem to provide implicit conversion from the 
various other numeric classes, as long as one item was of the class.

So I think it would be possible, with such a class, to have one's cake 
(nothing implicit), and eat it too (providing a way do comparisons and 
numerical operations, mostly implicitly).

Glenn

From tjreedy at udel.edu  Fri Mar 19 01:27:25 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 18 Mar 2010 20:27:25 -0400
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA29FB1.9020909@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
Message-ID: <hnugdd$vht$1@dough.gmane.org>

On 3/18/2010 5:48 PM, Nick Coghlan wrote:
> Mark Dickinson wrote:
>> Could everyone live with making float<->Decimal comparisons raise an
>> exception in 2.7?
>
> I could, with the caveat that *if* this causes problems for real world
> code, then changing it to produce the correct answer (as per your patch)
> should be applied as a bug fix in both 2.7 and 3.2.
>
> Note that even in Py3k there are some fairly serious weirdnesses kicking
> around due to the intransitive nature of numeric equality though:
>
>>>> from decimal import Decimal as dec
>>>> set((1, 1.0, dec("1.0")))
> {1}
>>>> set((1.0, dec("1.0")))
> {1.0, Decimal('1.0')}
>
>>>> d = {}
>>>> from decimal import Decimal as dec
>>>> d[1] = d[1.0] = d[dec("1.0")] = 42
>>>> d
> {1: 42}
>>>> d[1.0] = d[dec("1.0")] = 42
>>>> d
> {1: 42}
>>>> del d[1]
>>>> d[1.0] = d[dec("1.0")] = 42
>>>> d
> {1.0: 42, Decimal('1.0'): 42}
>
> When there is a clear, correct way (based on Decimal.from_float) to make
> numeric comparison behave in accordance with the rules of mathematics,
> do we really want to preserve strange, unintuitive behaviour like the above?

I would still strongly prefer that equality intransitivity be fixed one 
way or another (I suggested two on the tracker). The many ways it makes 
sets misbehave was discussed in both of
http://mail.python.org/pipermail/python-list/2008-September/508859.html
http://bugs.python.org/issue4087

Terry Jan Reedy


From solipsis at pitrou.net  Fri Mar 19 02:18:27 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Mar 2010 01:18:27 +0000 (UTC)
Subject: [Python-Dev] Decimal &lt;-&gt; float comparisons in py3k.
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
Message-ID: <loom.20100319T021726-50@post.gmane.org>

Glenn Linderman <v+python <at> g.nevcal.com> writes:
> 
> On 3/18/2010 2:48 PM, Nick Coghlan wrote:
> > When there is a clear, correct way (based on Decimal.from_float) to make
> > numeric comparison behave in accordance with the rules of mathematics,
> > do we really want to preserve strange, unintuitive behaviour like the above?
> 
> I'm aware of nothing that prevents the lazy coder from having a class 
> unifiedNumber in his toolbox [snip]

Please stick to the topic. We are talking about Python's default behaviour here.

Antoine.



From glenn at nevcal.com  Fri Mar 19 03:08:56 2010
From: glenn at nevcal.com (Glenn Linderman)
Date: Thu, 18 Mar 2010 19:08:56 -0700
Subject: [Python-Dev] Decimal &lt;-&gt; float comparisons in py3k.
In-Reply-To: <loom.20100319T021726-50@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
Message-ID: <4BA2DCB8.2040406@nevcal.com>

On 3/18/2010 6:18 PM, Antoine Pitrou wrote:
> Glenn Linderman<v+python<at>  g.nevcal.com>  writes:
>    
>> On 3/18/2010 2:48 PM, Nick Coghlan wrote:
>>      
>>> When there is a clear, correct way (based on Decimal.from_float) to make
>>> numeric comparison behave in accordance with the rules of mathematics,
>>> do we really want to preserve strange, unintuitive behaviour like the above?
>>>        
>> I'm aware of nothing that prevents the lazy coder from having a class
>> unifiedNumber in his toolbox [snip]
>>      
> Please stick to the topic. We are talking about Python's default behaviour here.
>    

Yes, I consider my comment relevant, and think that you should apologize 
for claiming it is off topic.

There are two choices on the table -- doing comparisons implicitly 
between Decimal and float, and raising an exception.  It seems the 
current behavior, sorting by type, is universally disliked, but doing 
nothing is a third choice.

So if the default behavior is to raise an exception, my comment pointed 
out the way comparisons could be provided, for those that need them.  
This allows both behaviors to exist concurrently.  Python developers 
could even consider including such a library in the standard library, 
although my suggestion didn't include that.

On the other hand, if the default behavior is to do an implicit 
conversion, I don't know of any way that that could be turned into an 
exception for those coders that don't want or don't like the particular 
type of implicit conversion chosen.

Glenn

From ctb at msu.edu  Fri Mar 19 03:36:30 2010
From: ctb at msu.edu (C. Titus Brown)
Date: Thu, 18 Mar 2010 19:36:30 -0700
Subject: [Python-Dev] GSoC 2010 is on -- projects?
Message-ID: <20100319023630.GC1617@idyll.org>

Hi all,

once again, the PSF has been accepted as a mentoring foundation for the Google
Summer of Code!  This year, we're going to emphasize python 3 porting, so
please think of projects you'd like to see tackled.

Please submit ideas for projects as soon as possible, as students will be able
to start applying soon.  You can add them to this page:

   http://wiki.python.org/moin/SummerOfCode/2010

You can subscribe to the mentors list here,

   http://mail.python.org/mailman/listinfo/soc2010-mentors

and the general discussion list (students included) here:

   http://mail.python.org/mailman/listinfo/soc2010-general

thanks,
--titus
-- 
C. Titus Brown, ctb at msu.edu

From benjamin at python.org  Fri Mar 19 04:13:42 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 18 Mar 2010 22:13:42 -0500
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <20100319023630.GC1617@idyll.org>
References: <20100319023630.GC1617@idyll.org>
Message-ID: <1afaf6161003182013j43de82aeheae2982bc6e4dc8b@mail.gmail.com>

2010/3/18 C. Titus Brown <ctb at msu.edu>:
> Hi all,
>
> once again, the PSF has been accepted as a mentoring foundation for the Google
> Summer of Code! ?This year, we're going to emphasize python 3 porting, so
> please think of projects you'd like to see tackled.

This is not completely to the exclusion of other, projects, though,
correct? For example, I think building a 64 bit PyPy backend would be
a very worthy task.



-- 
Regards,
Benjamin

From ctb at msu.edu  Fri Mar 19 04:15:54 2010
From: ctb at msu.edu (C. Titus Brown)
Date: Thu, 18 Mar 2010 20:15:54 -0700
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <1afaf6161003182013j43de82aeheae2982bc6e4dc8b@mail.gmail.com>
References: <20100319023630.GC1617@idyll.org>
	<1afaf6161003182013j43de82aeheae2982bc6e4dc8b@mail.gmail.com>
Message-ID: <20100319031554.GB5376@idyll.org>

On Thu, Mar 18, 2010 at 10:13:42PM -0500, Benjamin Peterson wrote:
> 2010/3/18 C. Titus Brown <ctb at msu.edu>:
> > Hi all,
> >
> > once again, the PSF has been accepted as a mentoring foundation for the Google
> > Summer of Code! ??This year, we're going to emphasize python 3 porting, so
> > please think of projects you'd like to see tackled.
> 
> This is not completely to the exclusion of other, projects, though,
> correct? For example, I think building a 64 bit PyPy backend would be
> a very worthy task.

Right -- emphasize only ;).  Good projects + good students + good mentors are
always welcome (and, actually, we'd like to focus on good students -- better
that something get done than that something particularly relevant get chosen
but NOT done by a poor student).

thanks,
--titus
-- 
C. Titus Brown, ctb at msu.edu

From benjamin at python.org  Fri Mar 19 04:20:17 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 18 Mar 2010 22:20:17 -0500
Subject: [Python-Dev] 3.1.2 tagging delayed a little
Message-ID: <1afaf6161003182020l1dd73856k30942c590ebcfb8f@mail.gmail.com>

Pending the resolution of a few Mac OS issues (#8068, #8069, and
#8133), I'm not going to tag the release at the moment. Hopefully,
we'll still be able to have a release in the next week.

-- 
Regards,
Benjamin

From lgautier at gmail.com  Fri Mar 19 07:23:20 2010
From: lgautier at gmail.com (Laurent Gautier)
Date: Fri, 19 Mar 2010 07:23:20 +0100
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <20100319023630.GC1617@idyll.org>
References: <20100319023630.GC1617@idyll.org>
Message-ID: <4BA31858.3010904@gmail.com>

On 3/19/10 3:36 AM, C. Titus Brown wrote:
> Hi all,
>
> once again, the PSF has been accepted as a mentoring foundation for the Google
> Summer of Code!  This year, we're going to emphasize python 3 porting, so
> please think of projects you'd like to see tackled.

Hi,


Does this mean that any other python project could potentially see 
itself ported to Python 3 in the course of this SoC ? If so, can any 
project owner submit a request for help, or is there going to be a list 
of projects that would nice to port, or will a voting system of some 
sort be put in place ?


Best,


Laurent

From tjreedy at udel.edu  Fri Mar 19 07:45:52 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 19 Mar 2010 02:45:52 -0400
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <4BA31858.3010904@gmail.com>
References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com>
Message-ID: <hnv6j0$id6$1@dough.gmane.org>

On 3/19/2010 2:23 AM, Laurent Gautier wrote:
> On 3/19/10 3:36 AM, C. Titus Brown wrote:
>> Hi all,
>>
>> once again, the PSF has been accepted as a mentoring foundation for
>> the Google
>> Summer of Code! This year, we're going to emphasize python 3 porting, so
>> please think of projects you'd like to see tackled.
>
> Hi,
>
>
> Does this mean that any other python project could potentially see
> itself ported to Python 3 in the course of this SoC ?

The theme should include both general porting tools and specific 
projects, especially infrastructure projects like numeric, which are 
blocking the porting of other projects. It would be nice if those doing 
specific projects contributed their experience/knowledge to a central pool.

> If so, can any project owner submit a request for help,

Any project owner is *always* free to ask for help (on python-list, but 
now here in this thread). Those who can also mentor might be more likely 
to get it. If I were a student, I would consider serious interest from a 
project owner (and a promise to distribute a port, when ready), a 
prerequisite.

 > or is there going to be a list
> of projects that would nice to port, or will a voting system of some
> sort be put in place ?

Like most contributors, students choose projects, within the limits of 
what they can get mentors for, that scratch their itches. They may or 
may not otherwise be swayed by requests and opinions.

My views.

Terry Jan Reedy


From techtonik at gmail.com  Fri Mar 19 09:09:16 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 19 Mar 2010 10:09:16 +0200
Subject: [Python-Dev] Tracker reviews workflow and flags
Message-ID: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>

I want to push some of my patches before 2.7 and use 5-1 rule for
that, but I can't come up with any review workflow other than mailing
status of my comments to the issues here. I can't mark issues in any
way. How about giving users ability to set flags or keywords? Maybe
entering a separate field like "Review status"?

Real world example with issue8151. It is an issue with a trivial patch
in it. Everything what is needed is to dispatch it to stable `commit
queue` and port to trunk. It is not 'easy' - it is 'trivial', but I
have no means to mark it as 'easy' either, so even this trivial fix
lies in tracker for three days waiting for review.

About 'easy' flag:
"6	easy	This is an easy task (e.g. suitable for GHOP or bug day beginners)"
It seems that it is for the issue that requires a patch, but if the
patch is already there and nobody wants to commit it - what should I
do?


Finally, review queue proposal.
To follow 5-1 rule, I need to review 5 issues, before mine is
reviewed, so I need some help from tracker.
1. I lack expertise in certain areas (C/C++, Mac OS, etc.), so I would
like to hide some issues under "Needs review" query, so they won't
bother me (if too many people marked issues in this way - the stats
may become a good reason for highly professional bug party)
2. I need ability to set "Needs review" flag back. Some issues were
once reviewed, but since then they've got modified and need further
comments.  The need the review again. That means pushed back _into the
end_ of patch queue.
3. Setting bug dependency by users. Often nobody wants to care about
issue, because it is just too long, there is much irrelevant comments
and nobody has time to read it. We don't have personal digg-like
comment filters to cope with the amount of text to be brain-loaded.
But it is possible to refactor such issue thread into separate
digestable issue/issues.


P.S.  About personal comment filters if anybody is interested in that.
Digg-like +1, -1 are good for voting, but for filtering it would be
nice to: 1. have several filters for different aspects of the thread,
2. have JS filter by marking individual phrases in comments and adding
ranges to filter using jquery / ajax

This way reviews will be more fun and easy.
-- 
anatoly t.

From dirkjan at ochtman.nl  Fri Mar 19 10:00:07 2010
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Fri, 19 Mar 2010 10:00:07 +0100
Subject: [Python-Dev] Joel Spolsky on Mercurial
In-Reply-To: <hnu6fu$ufa$1@dough.gmane.org>
References: <hnu6fu$ufa$1@dough.gmane.org>
Message-ID: <ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>

On Thu, Mar 18, 2010 at 22:38, Terry Reedy <tjreedy at udel.edu> wrote:
> Having gotten that far, I think this might be worth referencing in new dev
> docs.

Will do. I finally read hginit yesterday, after having seen people
rave about it on twitter for a few weeks, and it's a very friendly
introduction.

Cheers,

Dirkjan

From dickinsm at gmail.com  Fri Mar 19 10:37:23 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Fri, 19 Mar 2010 09:37:23 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA29FB1.9020909@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
Message-ID: <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>

On Thu, Mar 18, 2010 at 9:48 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Note that even in Py3k there are some fairly serious weirdnesses kicking
> around due to the intransitive nature of numeric equality though:

Yep.  My personal favourite is:

>>> from decimal import Decimal as dec
>>> set((1, 1.0, dec(1))) == set((1.0, 1, dec(1)))
False

(The sets even have different sizes!)

Note that while the originally proposed change does fix problems with
sets of Decimals, ints and floats, and with sorting of lists of those
types, it's not a complete fix:  as soon as you throw Fractions into
the mix, all the same problems arise again.  Making hashes of int,
float, Decimal *and* Fraction all compatible with one another,
efficient for ints and floats, and not grossly inefficient for
Fractions and Decimals, is kinda hairy, though I have a couple of
ideas of how it could be done.  Making Decimal-to-Fraction comparisons
give correct results isn't easy either:  the conversion in one
direction (Fraction to Decimal) is lossy, while the conversion in the
other direction (Decimal to Fraction) can't be performed efficiently
if the Decimal has a large exponent (Decimal('1e123456'));  so you
can't just do the Decimal.from_float trick of whacking everything into
a single domain and comparing there.  Again, this is solvable, but not
trivially so.

See, this is what happens when a language conflates numeric equality
with the equivalence relation used for membership testing.  ;-).  (I'm
not really complaining, but this is definitely one of the downsides of
this particular design decision.)

> When there is a clear, correct way (based on Decimal.from_float) to make
> numeric comparison behave in accordance with the rules of mathematics,
> do we really want to preserve strange, unintuitive behaviour like the above?

No.  Not really, no.  :)

Mark

From dickinsm at gmail.com  Fri Mar 19 11:07:55 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Fri, 19 Mar 2010 10:07:55 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
Message-ID: <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>

On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> Making hashes of int,
> float, Decimal *and* Fraction all compatible with one another,
> efficient for ints and floats, and not grossly inefficient for
> Fractions and Decimals, is kinda hairy, though I have a couple of
> ideas of how it could be done.

To elaborate, here's a cute scheme for the above, which is actually
remarkably close to what Python already does for ints and floats, and
which doesn't require any of the numeric types to figure out whether
it's exactly equal to an instance of some other numeric type.

After throwing out infinities and nans (which can easily be dealt with
separately), everything we care about is a rational number, so it's
enough to come up with some mapping from the set of all rationals to
the set of possible hashes, subject to the requirement that the
mapping can be computed efficiently for the types we care about.

For any prime P there's a natural 'reduce modulo P' map

  reduce : {rational numbers} --> {0, 1, ..., P-1, infinity}

defined in pseudocode by:

  reduce(m/n) = ((m % P) * inv(n, P)) % P  if n % P != 0  else infinity

where inv(n, P) represents the modular inverse to n modulo P.

Now let P be the handy Mersenne prime P = 2**31-1 (on a 64-bit
machine, the almost equally handy prime 2**61-1 could be used
instead), and define a hash function from the set of rationals to the
set [-2**31, 2**31) by:

hash(0) = 0
hash(m/n) = 1 + reduce(m/n - 1) if m/n > 0   # i.e., reduce m/n modulo
P, but to [1..P] rather than [0..P-1].
hash(m/n) = -hash(-m/n) if m/n < 0.

and in the last two cases, map a result of infinity to the unused hash
value -2**31.

For ints, this hash function is almost identical to what Python
already has, except that the current int hash does a reduction modulo
2**32-1 or 2**64-1 rather than 2**31-1.  For all small ints, hash(n)
== n, as currently.  Either way, the hash can be computed
digit-by-digit in exactly the same manner.  For floats, it's also easy
to compute:  express the float as m * 2**e for some integers m and e,
compute hash(m), and rotate e bits in the appropriate direction.  And
it's straightforward to implement for the Decimal and Fraction types,
too.

(One minor detail:  as usual, some postprocessing would be required to
replace a hash of -1 with something else, since a hash value of -1 is
invalid.)

Mark

From solipsis at pitrou.net  Fri Mar 19 12:50:16 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Mar 2010 11:50:16 +0000 (UTC)
Subject: [Python-Dev] Decimal &amp; lt; -&amp; gt;
	float comparisons in py3k.
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>
Message-ID: <loom.20100319T124539-150@post.gmane.org>

Glenn Linderman <glenn <at> nevcal.com> writes:
> 
> On the other hand, if the default behavior is to do an implicit 
> conversion, I don't know of any way that that could be turned into an 
> exception for those coders that don't want or don't like the particular 
> type of implicit conversion chosen.

You still haven't given a good reason why we should raise an exception rather
than do a comparison, though.

The fact that some particular coders don't like "the particular type of implicit
conversion chosen" is a particularly weak argument. Python isn't a language
construction framework; we try to choose useful defaults rather than simply give
out a box of tools. If some people don't like the defaults (significant
indentation, limitless precision integers, etc.), there are other choices out 
there.


Antoine.



From ncoghlan at gmail.com  Fri Mar 19 12:52:57 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Mar 2010 21:52:57 +1000
Subject: [Python-Dev] Joel Spolsky on Mercurial
In-Reply-To: <ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
References: <hnu6fu$ufa$1@dough.gmane.org>
	<ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
Message-ID: <4BA36599.7000106@gmail.com>

Dirkjan Ochtman wrote:
> On Thu, Mar 18, 2010 at 22:38, Terry Reedy <tjreedy at udel.edu> wrote:
>> Having gotten that far, I think this might be worth referencing in new dev
>> docs.
> 
> Will do. I finally read hginit yesterday, after having seen people
> rave about it on twitter for a few weeks, and it's a very friendly
> introduction.

The key insight for me was the "manage versions" vs "manage change sets"
distinction between a traditional VCS and a DVCS. I kind of knew that
already, but that page expressed it really well.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From arcriley at gmail.com  Fri Mar 19 12:57:26 2010
From: arcriley at gmail.com (Arc Riley)
Date: Fri, 19 Mar 2010 07:57:26 -0400
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <4BA31858.3010904@gmail.com>
References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com>
Message-ID: <bad82a81003190457w5c0e1770hd7c0f4af3f0a8746@mail.gmail.com>

Hi Laurent

If your community project would like help porting to Python 3, and you feel
this work is enough for a student to work full time for several weeks on,
then please do add it to the GSoC ideas page on the wiki.

There will be another program running for high school students which is more
suitable for smaller tasks (2-3 days each), more on-par with the actual time
it takes to port most Python packages.


On Fri, Mar 19, 2010 at 2:23 AM, Laurent Gautier <lgautier at gmail.com> wrote:

>
> Hi,
>
>
> Does this mean that any other python project could potentially see itself
> ported to Python 3 in the course of this SoC ? If so, can any project owner
> submit a request for help, or is there going to be a list of projects that
> would nice to port, or will a voting system of some sort be put in place ?
>
>
> Best,
>
>
> Laurent
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/arcriley%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/35879244/attachment.html>

From aries.shuaib at gmail.com  Fri Mar 19 12:59:38 2010
From: aries.shuaib at gmail.com (M. Shuaib Khan)
Date: Fri, 19 Mar 2010 16:59:38 +0500
Subject: [Python-Dev] Joel Spolsky on Mercurial
In-Reply-To: <4BA36599.7000106@gmail.com>
References: <hnu6fu$ufa$1@dough.gmane.org>
	<ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
	<4BA36599.7000106@gmail.com>
Message-ID: <d6bd61d31003190459m33ff69b2gb93b3b8f88e3c29f@mail.gmail.com>

On Fri, Mar 19, 2010 at 4:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Dirkjan Ochtman wrote:
> > On Thu, Mar 18, 2010 at 22:38, Terry Reedy <tjreedy at udel.edu> wrote:
> >> Having gotten that far, I think this might be worth referencing in new
> dev
> >> docs.
> >
> > Will do. I finally read hginit yesterday, after having seen people
> > rave about it on twitter for a few weeks, and it's a very friendly
> > introduction.
>
> The key insight for me was the "manage versions" vs "manage change sets"
> distinction between a traditional VCS and a DVCS. I kind of knew that
> already, but that page expressed it really well.
>

Hi,

Is Python-dev going to consider shifting their repo to mercurial/git instead
of SVN? :)



>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/aries.shuaib%40gmail.com
>



-- 
M. Shuaib Khan
http://mshuaibkhan.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/3450a22a/attachment.html>

From ncoghlan at gmail.com  Fri Mar 19 13:03:25 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Mar 2010 22:03:25 +1000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	
	<201003180759.25229.steve@pearwood.info>	
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	
	<201003182323.19458.steve@pearwood.info>	
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	
	<4BA29FB1.9020909@gmail.com>	
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
Message-ID: <4BA3680D.8070909@gmail.com>

Mark Dickinson wrote:
> On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> For ints, this hash function is almost identical to what Python
> already has, except that the current int hash does a reduction modulo
> 2**32-1 or 2**64-1 rather than 2**31-1.  For all small ints, hash(n)
> == n, as currently.  Either way, the hash can be computed
> digit-by-digit in exactly the same manner.  For floats, it's also easy
> to compute:  express the float as m * 2**e for some integers m and e,
> compute hash(m), and rotate e bits in the appropriate direction.  And
> it's straightforward to implement for the Decimal and Fraction types,
> too.

It seems to me that given the existing conflation of numeric equivalence
and containment testing, going the whole hog and fixing the set
membership problem for all of our rational types would be the right
thing to do.

Would it be worth providing the underlying implementation of the hash
algorithm as a math.hash_rational function and then use it for Decimal
and Fraction?

That would have the virtue of making it easy for others to define
numeric types that "played well" with numeric equivalence.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From phd at phd.pp.ru  Fri Mar 19 13:09:44 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Fri, 19 Mar 2010 15:09:44 +0300
Subject: [Python-Dev] Joel Spolsky on Mercurial
In-Reply-To: <d6bd61d31003190459m33ff69b2gb93b3b8f88e3c29f@mail.gmail.com>
References: <hnu6fu$ufa$1@dough.gmane.org>
	<ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
	<4BA36599.7000106@gmail.com>
	<d6bd61d31003190459m33ff69b2gb93b3b8f88e3c29f@mail.gmail.com>
Message-ID: <20100319120944.GA12240@phd.pp.ru>

On Fri, Mar 19, 2010 at 04:59:38PM +0500, M. Shuaib Khan wrote:
> Is Python-dev going to consider shifting their repo to mercurial/git instead
> of SVN? :)

http://mail.python.org/pipermail/python-dev/2009-March/087931.html

   The decision was made about a year ago.

http://www.python.org/dev/peps/pep-0374/
http://www.python.org/dev/peps/pep-0385/

Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From alex.nanou at gmail.com  Fri Mar 19 13:46:07 2010
From: alex.nanou at gmail.com (Alex A. Naanou)
Date: Fri, 19 Mar 2010 15:46:07 +0300
Subject: [Python-Dev] binary operation heuristics -- bug or undocumented?
Message-ID: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>

Hi,

A friend of mine stumbled upon the following behavior:


---cut---

>>> class A(object): pass
...
>>> class B(object):
...     def __add__(self, other):
...         print 'B: adding B and %s objects.' % other.__class__.__name__
...
>>> class C(object):
...     def __radd__(self, other):
...         print 'C: adding C and %s objects.' % other.__class__.__name__
...
>>> a, b, c = A(), B(), C()

>>> b + c
B: adding B and C objects.

>>> a + c
C: adding C and A objects.


# so far, quite logical. now let's do this:

>>> 1 + c
C: adding C and int objects.


--uncut--

My first expectation would be to get a TypeError here, as ints indeed
have an __add__ method, and they do not know anything about C objects
(obviously :) ). On second thought, giving client code priority to
handle things has it's merits.

The problem is that I found no mention of this behavior in the docs.


P.S. tested in 2.5 through 3.0 and PyPy

Thanks.

-- 
Alex.

From dickinsm at gmail.com  Fri Mar 19 13:55:58 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Fri, 19 Mar 2010 12:55:58 +0000
Subject: [Python-Dev] binary operation heuristics -- bug or undocumented?
In-Reply-To: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>
References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>
Message-ID: <5c6f2a5d1003190555l56cd5237g9b7a410a835390f6@mail.gmail.com>

On Fri, Mar 19, 2010 at 12:46 PM, Alex A. Naanou <alex.nanou at gmail.com> wrote:
> A friend of mine stumbled upon the following behavior:
>
>
> ---cut---
>
>>>> class A(object): pass
> ...
>>>> class B(object):
> ... ? ? def __add__(self, other):
> ... ? ? ? ? print 'B: adding B and %s objects.' % other.__class__.__name__
> ...
>>>> class C(object):
> ... ? ? def __radd__(self, other):
> ... ? ? ? ? print 'C: adding C and %s objects.' % other.__class__.__name__
> ...
>>>> a, b, c = A(), B(), C()
>
>>>> b + c
> B: adding B and C objects.
>
>>>> a + c
> C: adding C and A objects.
>
>
> # so far, quite logical. now let's do this:
>
>>>> 1 + c
> C: adding C and int objects.
>
>
> --uncut--
>
> My first expectation would be to get a TypeError here, as ints indeed
> have an __add__ method, and they do not know anything about C objects
> (obviously :) ).

Yes:  the int.__add__ method is tried first.  Since it doesn't know
anything about C objects it returns NotImplemented, and then
C.__radd__ is given a chance to do the addition.  The rules are
documented here:

http://docs.python.org/reference/datamodel.html#coercion-rules

Mark

From fuzzyman at voidspace.org.uk  Fri Mar 19 14:05:20 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 19 Mar 2010 13:05:20 +0000
Subject: [Python-Dev] binary operation heuristics -- bug or undocumented?
In-Reply-To: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>
References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>
Message-ID: <4BA37690.1040601@voidspace.org.uk>

On 19/03/2010 12:46, Alex A. Naanou wrote:
> [snip...]
>>>> class C(object):
>>>>          
> ...     def __radd__(self, other):
> ...         print 'C: adding C and %s objects.' % other.__class__.__name__
> ...
>    
>>>> a, b, c = A(), B(), C()
>>>> 1 + c
>>>>          
> C: adding C and int objects.
>
>
>    

That is the whole point of the __radd__ method. ints don't know how to 
add themselves to C objects, so int.__add__ will return NotImplemented 
and then Python will try C.__radd__.

All the best,

Michael

> --uncut--
>
> My first expectation would be to get a TypeError here, as ints indeed
> have an __add__ method, and they do not know anything about C objects
> (obviously :) ). On second thought, giving client code priority to
> handle things has it's merits.
>
> The problem is that I found no mention of this behavior in the docs.
>
>
> P.S. tested in 2.5 through 3.0 and PyPy
>
> Thanks.
>
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From phd at phd.pp.ru  Fri Mar 19 14:07:34 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Fri, 19 Mar 2010 16:07:34 +0300
Subject: [Python-Dev] right-side binary operations
In-Reply-To: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>
References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>
Message-ID: <20100319130734.GA20925@phd.pp.ru>

On Fri, Mar 19, 2010 at 03:46:07PM +0300, Alex A. Naanou wrote:
> >>> class C(object):
> ...     def __radd__(self, other):
> ...         print 'C: adding C and %s objects.' % other.__class__.__name__
> ...
> >>> 1 + c
> C: adding C and int objects.
> 
> My first expectation would be to get a TypeError here, as ints indeed
> have an __add__ method, and they do not know anything about C objects
> (obviously :) ). On second thought, giving client code priority to
> handle things has it's merits.
> 
> The problem is that I found no mention of this behavior in the docs.

   It's well-known and documented behavior. It's what r-methods are for.
See http://docs.python.org/reference/datamodel.html#emulating-numeric-types

   "These methods are called to implement the binary arithmetic operations
(+, -, *, /, %, divmod(), pow(), **, <<, >>, &, ^, |) with reflected
(swapped) operands. These functions are only called if the left operand
does not support the corresponding operation and the operands are of
different types. [3] For instance, to evaluate the expression x - y, where
y is an instance of a class that has an __rsub__() method, y.__rsub__(x) is
called if x.__sub__(y) returns NotImplemented."

Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From skip at pobox.com  Fri Mar 19 14:13:18 2010
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 19 Mar 2010 08:13:18 -0500
Subject: [Python-Dev] Joel Spolsky on Mercurial
In-Reply-To: <d6bd61d31003190459m33ff69b2gb93b3b8f88e3c29f@mail.gmail.com>
References: <hnu6fu$ufa$1@dough.gmane.org>
	<ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
	<4BA36599.7000106@gmail.com>
	<d6bd61d31003190459m33ff69b2gb93b3b8f88e3c29f@mail.gmail.com>
Message-ID: <19363.30830.383690.400879@montanaro.dyndns.org>


    M> Is Python-dev going to consider shifting their repo to mercurial/git
    M> instead of SVN? :)

Yes, Dirkjan has been working on it.  My initial experience with hg as a
quick way to revision control anything anywhere on a small scale ("hmmm, I
should track changes to /etc/hosts... cd /etc ; hg init ; hg add hosts")
worked quite well, but when I tried to use it for something where I was
actually working collaboratively (my lockfile module) I managed to get my
code/branches/heads/whatever completely f**ked up.  I gave up and just went
back to Subversion.

Skip

From alex.nanou at gmail.com  Fri Mar 19 14:16:42 2010
From: alex.nanou at gmail.com (Alex A. Naanou)
Date: Fri, 19 Mar 2010 16:16:42 +0300
Subject: [Python-Dev] binary operation heuristics -- bug or undocumented?
In-Reply-To: <4BA37690.1040601@voidspace.org.uk>
References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com> 
	<4BA37690.1040601@voidspace.org.uk>
Message-ID: <36f889221003190616j25053655h11f1d9417f77821b@mail.gmail.com>

Thanks!

On Fri, Mar 19, 2010 at 16:05, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> On 19/03/2010 12:46, Alex A. Naanou wrote:
>>
>> [snip...]
>>>>>
>>>>> class C(object):
>>>>>
>>
>> ... ? ? def __radd__(self, other):
>> ... ? ? ? ? print 'C: adding C and %s objects.' % other.__class__.__name__
>> ...
>>
>>>>>
>>>>> a, b, c = A(), B(), C()
>>>>> 1 + c
>>>>>
>>
>> C: adding C and int objects.
>>
>>
>>
>
> That is the whole point of the __radd__ method. ints don't know how to add
> themselves to C objects, so int.__add__ will return NotImplemented and then
> Python will try C.__radd__.
>
> All the best,
>
> Michael
>
>> --uncut--
>>
>> My first expectation would be to get a TypeError here, as ints indeed
>> have an __add__ method, and they do not know anything about C objects
>> (obviously :) ). On second thought, giving client code priority to
>> handle things has it's merits.
>>
>> The problem is that I found no mention of this behavior in the docs.
>>
>>
>> P.S. tested in 2.5 through 3.0 and PyPy
>>
>> Thanks.
>>
>>
>
>
> --
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
> READ CAREFULLY. By accepting and reading this email you agree, on behalf of
> your employer, to release me from all obligations and waivers arising from
> any and all NON-NEGOTIATED agreements, licenses, terms-of-service,
> shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure,
> non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have
> entered into with your employer, its partners, licensors, agents and
> assigns, in perpetuity, without prejudice to my ongoing rights and
> privileges. You further represent that you have the authority to release me
> from any BOGUS AGREEMENTS on behalf of your employer.

P.S. like the licence :)

>
>
>



-- 
Alex.

From casevh at gmail.com  Fri Mar 19 14:17:53 2010
From: casevh at gmail.com (Case Vanhorsen)
Date: Fri, 19 Mar 2010 06:17:53 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
Message-ID: <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com>

On Fri, Mar 19, 2010 at 3:07 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>> Making hashes of int,
>> float, Decimal *and* Fraction all compatible with one another,
>> efficient for ints and floats, and not grossly inefficient for
>> Fractions and Decimals, is kinda hairy, though I have a couple of
>> ideas of how it could be done.
>
> To elaborate, here's a cute scheme for the above, which is actually
> remarkably close to what Python already does for ints and floats, and
> which doesn't require any of the numeric types to figure out whether
> it's exactly equal to an instance of some other numeric type.
>
> After throwing out infinities and nans (which can easily be dealt with
> separately), everything we care about is a rational number, so it's
> enough to come up with some mapping from the set of all rationals to
> the set of possible hashes, subject to the requirement that the
> mapping can be computed efficiently for the types we care about.
>
> For any prime P there's a natural 'reduce modulo P' map
>
> ?reduce : {rational numbers} --> {0, 1, ..., P-1, infinity}
>
> defined in pseudocode by:
>
> ?reduce(m/n) = ((m % P) * inv(n, P)) % P ?if n % P != 0 ?else infinity
>
> where inv(n, P) represents the modular inverse to n modulo P.
>
> Now let P be the handy Mersenne prime P = 2**31-1 (on a 64-bit
> machine, the almost equally handy prime 2**61-1 could be used
> instead), and define a hash function from the set of rationals to the
> set [-2**31, 2**31) by:
>
> hash(0) = 0
> hash(m/n) = 1 + reduce(m/n - 1) if m/n > 0 ? # i.e., reduce m/n modulo
> P, but to [1..P] rather than [0..P-1].
> hash(m/n) = -hash(-m/n) if m/n < 0.
>
> and in the last two cases, map a result of infinity to the unused hash
> value -2**31.
>
> For ints, this hash function is almost identical to what Python
> already has, except that the current int hash does a reduction modulo
> 2**32-1 or 2**64-1 rather than 2**31-1. ?For all small ints, hash(n)
> == n, as currently. ?Either way, the hash can be computed
> digit-by-digit in exactly the same manner. ?For floats, it's also easy
> to compute: ?express the float as m * 2**e for some integers m and e,
> compute hash(m), and rotate e bits in the appropriate direction. ?And
> it's straightforward to implement for the Decimal and Fraction types,
> too.
Will this change the result of hashing a long? I know that both gmpy
and SAGE use their own hash implementations for performance reasons. I
understand that CPython's hash function is an implementation detail,
but there are external modules that rely on the existing hash
behavior.

FWIW, I'd prefer 2.7 and 3.2 have the same behavior. I don't mind the
existing 3.1 behavior and I'd rather not have a difference between 3.1
and 3.2.

casevh
>
> (One minor detail: ?as usual, some postprocessing would be required to
> replace a hash of -1 with something else, since a hash value of -1 is
> invalid.)
>
> Mark
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/casevh%40gmail.com
>

From eric at trueblade.com  Fri Mar 19 14:33:15 2010
From: eric at trueblade.com (Eric Smith)
Date: Fri, 19 Mar 2010 09:33:15 -0400
Subject: [Python-Dev] Bug? syslog.openlog using ident "python" by
	default.
In-Reply-To: <20100318115844.GA7374@tummy.com>
References: <20100318115844.GA7374@tummy.com>
Message-ID: <4BA37D1B.9020306@trueblade.com>

Sean Reifschneider wrote:
> If you call:
> 
>    from syslog import syslog, openlog
>    syslog('My error message')
> 
> Something like the following gets logged:
> 
>    Mar 18 05:20:22 guin python: My error message
>                         ^^^^^^
> 
> Where I'm annoyed by the "python" in the above.  This is pulled from
> the C argv[0].  IMHO, what we really want is the Python sys.argv[0].

I agree with this. The existing behavior is very annoying. I don't have 
any particular thoughts about the implementation.

Eric.

From brian.curtin at gmail.com  Fri Mar 19 15:20:00 2010
From: brian.curtin at gmail.com (Brian Curtin)
Date: Fri, 19 Mar 2010 09:20:00 -0500
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
Message-ID: <cf9f31f21003190720v678c8368yacf90a3e9419b20c@mail.gmail.com>

On Fri, Mar 19, 2010 at 03:09, anatoly techtonik <techtonik at gmail.com>wrote:

> I want to push some of my patches before 2.7 and use 5-1 rule for
> that, but I can't come up with any review workflow other than mailing
> status of my comments to the issues here. I can't mark issues in any
> way. How about giving users ability to set flags or keywords? Maybe
> entering a separate field like "Review status"?
>

We already have "Patch Review" as a stage, and "needs review" as a flag,
which I feel is more than enough. If you don't have the privileges to modify
an issue, you can always comment on the issue with something like "can this
issue be set to the patch review stage?" Someone will probably take a look
at it and set it accordingly.

Real world example with issue8151. It is an issue with a trivial patch
> in it. Everything what is needed is to dispatch it to stable `commit
> queue` and port to trunk. It is not 'easy' - it is 'trivial', but I
> have no means to mark it as 'easy' either, so even this trivial fix
> lies in tracker for three days waiting for review.
>

Given that we have hundreds of issues with patches waiting for review, three
days isn't so bad :) As with any project, there are only so many people and
so much time to get the work done.


> About 'easy' flag:
> "6      easy    This is an easy task (e.g. suitable for GHOP or bug day
> beginners)"
> It seems that it is for the issue that requires a patch, but if the
> patch is already there and nobody wants to commit it - what should I
> do?
>

Post a comment on the issue asking what the status is. If an approved patch
has been sitting for a few weeks, make a comment. If it has been there for a
few days, I'd let it slide but keep it on your radar.


>
> Finally, review queue proposal.
> To follow 5-1 rule, I need to review 5 issues, before mine is
> reviewed, so I need some help from tracker.
> 1. I lack expertise in certain areas (C/C++, Mac OS, etc.), so I would
> like to hide some issues under "Needs review" query, so they won't
> bother me (if too many people marked issues in this way - the stats
> may become a good reason for highly professional bug party)
>

There already exists a component field which has Macintosh as an option, so
you could filter on that. Same goes for Windows.

Drilling down to the language(s) involved in the issue feels like just
another step that would get limited use. I do see what you are saying,
though, because I'm sure it's frustrating to want to look out there for a
nice pure-Python issue, then the first 10 issues you click on require C code
(or vice versa).


> 2. I need ability to set "Needs review" flag back. Some issues were
> once reviewed, but since then they've got modified and need further
> comments.  The need the review again. That means pushed back _into the
> end_ of patch queue.
>

I don't think the "needs review" flag gets unset, but if it has been unset
manually and you've made changes, you can ask for another review. If you can
get the changes made quickly, you might be able to get the previous reviewer
to look at it again while it's still fresh in their mind.


> 3. Setting bug dependency by users. Often nobody wants to care about
> issue, because it is just too long, there is much irrelevant comments
> and nobody has time to read it. We don't have personal digg-like
> comment filters to cope with the amount of text to be brain-loaded.
> But it is possible to refactor such issue thread into separate
> digestable issue/issues.
>
>
> P.S.  About personal comment filters if anybody is interested in that.
> Digg-like +1, -1 are good for voting, but for filtering it would be
> nice to: 1. have several filters for different aspects of the thread,
> 2. have JS filter by marking individual phrases in comments and adding
> ranges to filter using jquery / ajax
>
> This way reviews will be more fun and easy.
> --
> anatoly t.
>

I think that would take away from the goal of fixing the issue. Even some -1
comments could contain helpful information, but you might have to explicitly
click on that hidden comment to find out.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/3c75a976/attachment.html>

From rdmurray at bitdance.com  Fri Mar 19 15:22:05 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 19 Mar 2010 10:22:05 -0400
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
Message-ID: <20100319142205.E11FC1FDA39@kimball.webabinitio.net>

On Fri, 19 Mar 2010 10:09:16 +0200, anatoly techtonik <techtonik at gmail.com> wrote:
> I want to push some of my patches before 2.7 and use 5-1 rule for
> that, but I can't come up with any review workflow other than mailing
> status of my comments to the issues here. I can't mark issues in any
> way. How about giving users ability to set flags or keywords? Maybe
> entering a separate field like "Review status"?

With our current workflow, mailing the status of your reviews here is
pretty much the only option.  ("I've reviewed issues a, b, c, d, and e,
could someone please review y?").

> Real world example with issue8151. It is an issue with a trivial patch
> in it. Everything what is needed is to dispatch it to stable `commit
> queue` and port to trunk. It is not 'easy' - it is 'trivial', but I
> have no means to mark it as 'easy' either, so even this trivial fix
> lies in tracker for three days waiting for review.

Three days?  That's nothing :)

> About 'easy' flag:
> "6	easy	This is an easy task (e.g. suitable for GHOP or bug day beginners)"
> It seems that it is for the issue that requires a patch, but if the
> patch is already there and nobody wants to commit it - what should I
> do?

If an issue has a patch the 'easy' flag doesn't really apply any more,
but it doesn't always get deleted, either.

Issues that have a patch should get set to "patch review" stage, unless
tests are missing, in which case it should get set to "needs test".

Making changes like that (and also updating keywords) requires tracker
privileges, but we give those out more freely than we do commit privs.
You could try hanging out on #python-dev on Freenode, where there will
(usually) be people who can respond to your requests to update the status
of tracker issues.  Make enough good requests and we'll get you tracker
privs so you can do it yourself.  ("Enough" is probably a pretty small
number, by the way :)

> Finally, review queue proposal.
> To follow 5-1 rule, I need to review 5 issues, before mine is
> reviewed, so I need some help from tracker.
> 1. I lack expertise in certain areas (C/C++, Mac OS, etc.), so I would
> like to hide some issues under "Needs review" query, so they won't
> bother me (if too many people marked issues in this way - the stats
> may become a good reason for highly professional bug party)

To avoid issues involving C coding, try restricting your search to
'Library' issues.  There is C code in some library modules, but less so
than in the core ;).  It would be nice if you could search on more than
one component, but currently the interface doesn't allow that.  (You might
be able to do it by hand constructing a query string, I haven't checked.)

> 2. I need ability to set "Needs review" flag back. Some issues were
> once reviewed, but since then they've got modified and need further
> comments.  The need the review again. That means pushed back _into the
> end_ of patch queue.

I'm not sure exactly what the 'needs review' keyword is for, frankly :)
Talking about pushing things back into the end of the patch queue makes
the assumption that there *is* a queue.  There isn't, there's just the
set of open issues with patches that committers look at when they can or
when they have an itch to scratch.  Making a comment on an issue (such
as doing a review) generates an email to everyone on the nosy list and
to the python-bugs list (for those people who are subscribed), and thus
in a fashion brings the issue to the 'top of the pile' in some sense and
for a short period.  Often things happen to the issue when a comment is
posted, but not always.  As with everything else, it depends on whether
anyone has both the time and the interest to take action.

> 3. Setting bug dependency by users. Often nobody wants to care about
> issue, because it is just too long, there is much irrelevant comments
> and nobody has time to read it. We don't have personal digg-like
> comment filters to cope with the amount of text to be brain-loaded.
> But it is possible to refactor such issue thread into separate
> digestable issue/issues.
>
> P.S.  About personal comment filters if anybody is interested in that.
> Digg-like +1, -1 are good for voting, but for filtering it would be
> nice to: 1. have several filters for different aspects of the thread,
> 2. have JS filter by marking individual phrases in comments and adding
> ranges to filter using jquery / ajax

-1 on comment filtering.  The text in the issue is the history of the
issue.  Reading it is part of doing an issue review.  I can't imagine that
the effort of manually blocking certain comments so that you didn't see
them when later looking at the issue again would be worth it.  It seems
to me you'd be much more likely to accidentally block something you should
be reading than you would be to make the issue usefully more readable.

If an issue needs to be refactored, new issues can be opened with
pointers to the old issue, and pointers added to the old issue to the new.
This can be done by anyone just by saying, eg: 'see issue 1234' (roundup
turns that into a link), and you can also suggest that the bug be added
to the dependency list by someone who can, if that is appropriate.
This happens fairly regularly with complex issues, or issues where we
discover more than one bug is involved.  Of course, it doesn't always
happen when it should.

The roundup code lives in the SVN repository, and you can set up your own
instance of the tracker to experiment with changes.  Patches are welcome,
although we have a few in the meta-tracker[1] already that haven't been
reviewed.  Currently Martin is pretty much the only one doing code-level
tracker work, and while I hope to get involved in that at some point it
hasn't happened yet.  So if you want to contribute some time in that
area, that would be great.  You could join the tracker-discuss list
(though it is very low volume).

[1] The 'report tracker problems' link on the lower left, if you
haven't noticed it.

--
R. David Murray                                      www.bitdance.com

From lgautier at gmail.com  Fri Mar 19 15:23:33 2010
From: lgautier at gmail.com (Laurent Gautier)
Date: Fri, 19 Mar 2010 15:23:33 +0100
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <bad82a81003190457w5c0e1770hd7c0f4af3f0a8746@mail.gmail.com>
References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com>
	<bad82a81003190457w5c0e1770hd7c0f4af3f0a8746@mail.gmail.com>
Message-ID: <4BA388E5.1030703@gmail.com>

On 3/19/10 12:57 PM, Arc Riley wrote:
> Hi Laurent
>
> If your community project would like help porting to Python 3, and you
> feel this work is enough for a student to work full time for several
> weeks on, then please do add it to the GSoC ideas page on the wiki.

Whether this is worth weeks of work or not will depend on a given 
student's knowledge about Python 3, and I'd suspect that the GSoC would 
be an opportunity for a number of applicant to actually learn the 
intricacies of Python 3.
Developing Python 3-specific features could be used to increase the 
amount of work on the project, but I am uncertain of whether this is 
worth a full GSoC.

> There will be another program running for high school students which is
> more suitable for smaller tasks (2-3 days each), more on-par with the
> actual time it takes to port most Python packages.
>

My project is roughly 6000 lines of Python and 6000 lines of C (Mostly 
C-level Python API bindings). I tried porting to Python 3 in the past, 
and it went fast (less than a day). The hurdle is that this resulted in 
segfaults for a lot of string-related features (You know, the 
byte/string thing). Tracing issues at the C level can be time-consuming, 
so I am hesitating to claim that 2-3 days of an high-school student 
would be enough.


Could several ports be bundled in a GSoC (the target projects would be 
grouped by theme, somehow).


L.



> On Fri, Mar 19, 2010 at 2:23 AM, Laurent Gautier <lgautier at gmail.com
> <mailto:lgautier at gmail.com>> wrote:
>
>
>     Hi,
>
>
>     Does this mean that any other python project could potentially see
>     itself ported to Python 3 in the course of this SoC ? If so, can any
>     project owner submit a request for help, or is there going to be a
>     list of projects that would nice to port, or will a voting system of
>     some sort be put in place ?
>
>
>     Best,
>
>
>     Laurent
>
>     _______________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     http://mail.python.org/mailman/listinfo/python-dev
>     Unsubscribe:
>     http://mail.python.org/mailman/options/python-dev/arcriley%40gmail.com
>
>


From rhamph at gmail.com  Fri Mar 19 16:22:12 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Fri, 19 Mar 2010 09:22:12 -0600
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
Message-ID: <aac2c7cb1003190822v4af12314kb709ae3658ca778@mail.gmail.com>

On Thu, Mar 18, 2010 at 12:41, Mark Dickinson <dickinsm at gmail.com> wrote:
> I'm only seeing two arguments against this at the moment: (1) it has
> the potential to break code that relies on being able to sort
> heterogeneous lists. ?But given that heterogeneous lists can't be
> sorted stably anyway (see my earlier post about lists containing ints,
> floats and Decimals), perhaps this is an acceptable risk. (2) A few of
> the posters here (Steven, Nick, and me) feel that it's slightly more
> natural to allow these comparisons; ?but I think the argument's fairly
> evenly balanced at the moment between those who'd prefer an exception
> and those who'd prefer to allow the comparisons.

Conceptually I like the idea of them all being comparable, but are
there any real use cases involving heterogeneous lists?  All the
examples I've seen have focused on how they're broken, not on how
they'd be correct (including possible math after the comparison!) if
the language compared properly.

Without such use cases allowing comparison seems like a lot of work for nothing.


-- 
Adam Olsen, aka Rhamphoryncus

From raymond.hettinger at gmail.com  Fri Mar 19 17:38:27 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 19 Mar 2010 09:38:27 -0700
Subject: [Python-Dev] Decimal &amp; lt; -&amp; gt;
	float comparisons in py3k.
In-Reply-To: <loom.20100319T124539-150@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
Message-ID: <4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>


On Mar 19, 2010, at 4:50 AM, Antoine Pitrou wrote:

> Glenn Linderman <glenn <at> nevcal.com> writes:
>> 
>> On the other hand, if the default behavior is to do an implicit 
>> conversion, I don't know of any way that that could be turned into an 
>> exception for those coders that don't want or don't like the particular 
>> type of implicit conversion chosen.
> 
> You still haven't given a good reason why we should raise an exception rather
> than do a comparison, though.
> 
> The fact that some particular coders don't like "the particular type of implicit
> conversion chosen" is a particularly weak argument. Python isn't a language
> construction framework; we try to choose useful defaults rather than simply give
> out a box of tools. If some people don't like the defaults (significant
> indentation, limitless precision integers, etc.), there are other choices out 
> there.


The reason to prefer an exception is that decimal/float comparisons
are more likely to be a programmer error than an intended behavior.
Real use cases for decimal/float comparisons are rare and would
be more clearly expressed with an explicit conversion using Decimal.from_float().

Of course there is a precedent, I can compare "120" < 140 in AWK
and get an automatic implicit conversion ;-)  

Just because we can compare, doesn't mean we should.


Raymond

From amk at amk.ca  Fri Mar 19 17:39:58 2010
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 19 Mar 2010 12:39:58 -0400
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <20100319142205.E11FC1FDA39@kimball.webabinitio.net>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
	<20100319142205.E11FC1FDA39@kimball.webabinitio.net>
Message-ID: <20100319163958.GB8716@amk-desktop.matrixgroup.net>

On Fri, Mar 19, 2010 at 10:22:05AM -0400, R. David Murray wrote:
> > Real world example with issue8151. It is an issue with a trivial patch
> > in it. Everything what is needed is to dispatch it to stable `commit
> > queue` and port to trunk. It is not 'easy' - it is 'trivial', but I
> > have no means to mark it as 'easy' either, so even this trivial fix
> > lies in tracker for three days waiting for review.
> 
> Three days?  That's nothing :)

I think Anatoly has a point, though; there are issues where the people
discussing the fix have come to a consensus, but none of them are
committers, or the committers involved are too busy.  There's a patch
ready to apply, but no committer knows about it.

> To avoid issues involving C coding, try restricting your search to
> 'Library' issues.  There is C code in some library modules, but less so
> than in the core ;).  It would be nice if you could search on more than

Actually Library should pretty much always be pure-Python.  Issues
related to C modules such as socket are classified under the
'Extension modules' component.  Of course, sometimes the ultimate
cause will migrate (a bug in SocketServer.py is traced to a flaw in
socket, for example).

--amk

From tjreedy at udel.edu  Fri Mar 19 18:15:46 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 19 Mar 2010 13:15:46 -0400
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <20100319142205.E11FC1FDA39@kimball.webabinitio.net>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
	<20100319142205.E11FC1FDA39@kimball.webabinitio.net>
Message-ID: <ho0bg1$q6c$1@dough.gmane.org>

On 3/19/2010 10:22 AM, R. David Murray wrote:

> This can be done by anyone just by saying, eg: 'see issue 1234' (roundup
> turns that into a link),

That should be 'see issue #1234' to get the autolink.
 From http://wiki.python.org/moin/TrackerDocs/

The tracker converts some specially formatted words in messages into 
links. The list includes
       "#<number>" links to the issue <number>
       "msg<number>" links to the message <number>
       "r<number>, "rev<number>", "revision <number>" links to 
svn.python.org displaying the checked in changes.


From tjreedy at udel.edu  Fri Mar 19 18:20:33 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 19 Mar 2010 13:20:33 -0400
Subject: [Python-Dev] Joel Spolsky on Mercurial
In-Reply-To: <ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
References: <hnu6fu$ufa$1@dough.gmane.org>
	<ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
Message-ID: <ho0bov$q6c$2@dough.gmane.org>

On 3/19/2010 5:00 AM, Dirkjan Ochtman wrote:
> On Thu, Mar 18, 2010 at 22:38, Terry Reedy<tjreedy at udel.edu>  wrote:
>> Having gotten that far, I think this might be worth referencing in new dev
>> docs.
>
> Will do. I finally read hginit yesterday, after having seen people
> rave about it on twitter for a few weeks, and it's a very friendly
> introduction.

Having read further 'lessons', you might also want to note that the 
Python group workflow differs (and briefly how so) from that given as a 
possibility, assuming that it does.

Terry Jan Reedy


From tjreedy at udel.edu  Fri Mar 19 18:33:14 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 19 Mar 2010 13:33:14 -0400
Subject: [Python-Dev] binary operation heuristics -- bug or undocumented?
In-Reply-To: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>
References: <36f889221003190546p35dedaf6u8b7ecb11b58f97dd@mail.gmail.com>
Message-ID: <ho0cgo$q6c$3@dough.gmane.org>

On 3/19/2010 8:46 AM, Alex A. Naanou wrote:

> A friend of mine stumbled upon the following behavior:
[snip]
Questions like this are more appropriately posted to python-list, where 
you would have gotten the same answer.

tjr



From solipsis at pitrou.net  Fri Mar 19 19:11:18 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Mar 2010 18:11:18 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?b?RGVjaW1hbCAmYW1wO2FtcDsgbHQ7IC0mYW1wO2Ft?=
	=?utf-8?q?p=3B_gt=3B=09float_comparisons_in_py3k=2E?=
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
Message-ID: <loom.20100319T190158-232@post.gmane.org>

Raymond Hettinger <raymond.hettinger <at> gmail.com> writes:
> 
> The reason to prefer an exception is that decimal/float comparisons
> are more likely to be a programmer error than an intended behavior.

Not more so than float/int or decimal/int or bool/int comparisons, which all 
work.

We forbid comparisons when there is a real danger or ambiguity, such as unicode
vs. bytes. There is no such danger or ambiguity when comparing a decimal with a
float. I don't see the point of being so restrictive; Python is not Java, and
typing is not supposed to be a synonym for bondage.

> Of course there is a precedent, I can compare "120" < 140 in AWK
> and get an automatic implicit conversion

The proper precedent in this context, though, is this one (py3k):

>>> 1 < 2.0
True
>>> 1 < Decimal("2.0")
True
>>> 1 > Decimal("2.0")
False
>>> 1 > 2.0
False
>>> True > 0.5
True
>>> True > 1.5
False
>>> True > Decimal("0.5")
True
>>> True > Decimal("1.5")
False

Are you suggesting to change all the above comparisons to raise a TypeError?

cheers

Antoine.



From zvezdan at zope.com  Fri Mar 19 19:09:28 2010
From: zvezdan at zope.com (Zvezdan Petkovic)
Date: Fri, 19 Mar 2010 14:09:28 -0400
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <ho0bg1$q6c$1@dough.gmane.org>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
	<20100319142205.E11FC1FDA39@kimball.webabinitio.net>
	<ho0bg1$q6c$1@dough.gmane.org>
Message-ID: <46EF5EEA-CA98-44C5-92AD-0BC83E39090D@zope.com>


On Mar 19, 2010, at 1:15 PM, Terry Reedy wrote:

> On 3/19/2010 10:22 AM, R. David Murray wrote:
> 
>> This can be done by anyone just by saying, eg: 'see issue 1234' (roundup
>> turns that into a link),
> 
> That should be 'see issue #1234' to get the autolink.
> From http://wiki.python.org/moin/TrackerDocs/

The list in the docs is not complete then.
David is right.  The words 'issue 1234' will become a link in roundup automatically.  As well as 'issue1234'.

	Zvezdan


From raymond.hettinger at gmail.com  Fri Mar 19 19:42:42 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 19 Mar 2010 11:42:42 -0700
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float comparisons in py3k.
In-Reply-To: <loom.20100319T190158-232@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
Message-ID: <7A59AEF6-A4F9-4D81-BE9A-EAD89053AD7F@gmail.com>


On Mar 19, 2010, at 11:11 AM, Antoine Pitrou wrote:

> Raymond Hettinger <raymond.hettinger <at> gmail.com> writes:
>> 
>> The reason to prefer an exception is that decimal/float comparisons
>> are more likely to be a programmer error than an intended behavior.
> 
> Not more so than float/int or decimal/int or bool/int comparisons, which all 
> work.

The float/int and decimal/int comparisons have valid and common use cases.
In contrast, comparing binary floats and decimal floats is rare, and more 
likely to be an accident.

When an int is converted to a float or decimal, the result usually isn't
surprising.  The conversion of a float to a decimal is not as straight-forward
(most people don't even know that an exact conversion is possible).

Also, the float and decimal constructors also natively accept ints as inputs,
but the decimal constructor intentionally excludes a float input.


Raymond




From v+python at g.nevcal.com  Fri Mar 19 19:42:39 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 11:42:39 -0700
Subject: [Python-Dev] Decimal &amp; lt; -&amp; gt;
 float comparisons in py3k.
In-Reply-To: <loom.20100319T124539-150@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
Message-ID: <4BA3C59F.7030508@g.nevcal.com>

On 3/19/2010 4:50 AM, Antoine Pitrou wrote:
> Glenn Linderman<glenn<at>  nevcal.com>  writes:
>    
>> On the other hand, if the default behavior is to do an implicit
>> conversion, I don't know of any way that that could be turned into an
>> exception for those coders that don't want or don't like the particular
>> type of implicit conversion chosen.
>>      
> You still haven't given a good reason why we should raise an exception rather
> than do a comparison, though.
>
> The fact that some particular coders don't like "the particular type of implicit
> conversion chosen" is a particularly weak argument. Python isn't a language
> construction framework; we try to choose useful defaults rather than simply give
> out a box of tools. If some people don't like the defaults (significant
> indentation, limitless precision integers, etc.), there are other choices out
> there.
>    

The whole point of providing Decimal is for applications for which float 
is inappropriate.  I didn't think I needed to reproduce PEP 327 in my email.

So when a coder choose to use Decimal, it is because float is 
inappropriate.  Because float is inappropriate, mixing Decimal and float 
is inappropriate.  Having the language coerce implicitly, is 
inappropriate.  All this is in the PEP.  Note that the PEP actually says 
that the problem is not doing implicit arithmetic (as some have reported 
in this thread) but in doing implicit coercions.  In order to do 
implicit comparisons, one must do an implicit coercion.  Hence the PEP 
actually already prohibits implicit comparisons, as well as implicit 
arithmetic.

Now the proposal is to start down the slippery slope by allowing 
comparisons.  To start with, neither decimal nor float comparisons, are, 
in general, exact. (Although it is more common for decimal calculations 
to contain the code to do rounding and significant digit calculations 
than float code, due to its use in monetary calculations.  People that 
don't care about the precision tend to just use float, and no 
significant digit concerns are coded.)  Comparisons need to be done with 
full knowledge of the precision of the numbers.  The additional 
information necessary to do so cannot be encoded in a binary operator.

People that do care about precision would rather not have imprecise data 
introduced by accidentally forgetting to include a coercion constructor, 
and have it implicitly converted; hence, the exception is much better 
for them.

My original message pointed out that providing an exception solves the 
problem for people that care about precision, and doesn't hinder a 
solution (only a convenience) for people that somehow bothered to use 
Decimal but then suddenly lost interest in doing correct math.  For 
those people, the class I proposed can workaround the existence of the 
exception, so the language can better serve both types of people, those 
that care, and those that don't.

My personal opinion is that real applications that use Decimal are much 
more likely to care, and much more likely to appreciate the exception, 
whereas applications that don't care, aren't likely to use Decimal in 
the first place, and so won't encounter the problem anyway.  And if they 
do, for some reason, hop on the Decimal bandwagon, then it seems to be a 
simple matter to implement a class with implicit conversions using 
whatever parameters they choose to implement, for their sloppy endeavors.

Yes, if implicit coercions for comparisons exist, it is possible to 
write code that avoids using them... but much friendlier for people that 
wish to avoid them, if the exception remains in place, unless explicitly 
sidestepped through coding another class.

Implementing an exception for Decimal/float comparison attempts, rather 
than comparing their types, is a giant step forward.  Implementing an 
implicit coercion for Decimal/float comparisons is a step down a 
slippery slope, to reduce the benefits of Decimal for its users.

Glenn




From tjreedy at udel.edu  Fri Mar 19 19:43:01 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 19 Mar 2010 14:43:01 -0400
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float comparisons in py3k.
In-Reply-To: <loom.20100319T190158-232@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
Message-ID: <ho0gjk$dd4$1@dough.gmane.org>

On 3/19/2010 2:11 PM, Antoine Pitrou wrote:
> Raymond Hettinger<raymond.hettinger<at>  gmail.com>  writes:
>>
>> The reason to prefer an exception is that decimal/float comparisons
>> are more likely to be a programmer error than an intended behavior.

If you really believe that, then equality comparisons should also be 
disabled by raising NotImplemented or whatever. Clearly, someone who 
writes 'if somefloat == somedecimal:'assumes (now wrongly) that the test 
might be true. This is just as buggy as an order comparison. Raising an 
exception would consistently isolate decimals from other numbers and 
eliminate the equality intransitivity mess and its nasty effect on sets.

It still strikes me as a bit crazy for Python to say that 0.0 == 0 and 0 
== Decimal(0) but that 0.0 != Decimal(0). Who would expect such a thing?

Terry Jan Reedy


From raymond.hettinger at gmail.com  Fri Mar 19 19:47:45 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 19 Mar 2010 11:47:45 -0700
Subject: [Python-Dev] Decimal &amp; lt; -&amp; gt;
	float comparisons in py3k.
In-Reply-To: <4BA3C59F.7030508@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4BA3C59F.7030508@g.nevcal.com>
Message-ID: <3909DF13-B2FA-43E7-9B15-22326271BA86@gmail.com>


On Mar 19, 2010, at 11:42 AM, Glenn Linderman wrote:
>> 
> 
> The whole point of providing Decimal is for applications for which float is inappropriate.  I didn't think I needed to reproduce PEP 327 in my email.
> 
> So when a coder choose to use Decimal, it is because float is inappropriate.  Because float is inappropriate, mixing Decimal and float is inappropriate.  Having the language coerce implicitly, is inappropriate.  All this is in the PEP.  Note that the PEP actually says that the problem is not doing implicit arithmetic (as some have reported in this thread) but in doing implicit coercions.  In order to do implicit comparisons, one must do an implicit coercion.  Hence the PEP actually already prohibits implicit comparisons, as well as implicit arithmetic.

Well said.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/dfaa421b/attachment.html>

From solipsis at pitrou.net  Fri Mar 19 19:53:26 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Mar 2010 18:53:26 +0000 (UTC)
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float comparisons in py3k.
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4BA3C59F.7030508@g.nevcal.com>
Message-ID: <loom.20100319T194911-297@post.gmane.org>

Glenn Linderman <v+python <at> g.nevcal.com> writes:
> 
> So when a coder choose to use Decimal, it is because float is 
> inappropriate.  Because float is inappropriate, mixing Decimal and float 
> is inappropriate.  Having the language coerce implicitly, is 
> inappropriate.

I'm sorry but this is very dogmatic. What is the concrete argument against an
accurate comparison between floats and decimals?

> Comparisons need to be done with 
> full knowledge of the precision of the numbers.  The additional 
> information necessary to do so cannot be encoded in a binary operator.

This doesn't have anything to do with the mixing of floats and decimals, though,
since it also applies to unmixed comparisons. Again, is there an argument
specific to mixed comparisons?



From solipsis at pitrou.net  Fri Mar 19 20:02:35 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Mar 2010 19:02:35 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?b?RGVjaW1hbCAmYW1wO2FtcDsgYW1wOyBsdDsgLSZh?=
	=?utf-8?q?mp=3Bamp=3B_amp=3B_gt=3B=09float_comparisons_in_py3k=2E?=
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<7A59AEF6-A4F9-4D81-BE9A-EAD89053AD7F@gmail.com>
Message-ID: <loom.20100319T195334-851@post.gmane.org>

Raymond Hettinger <raymond.hettinger <at> gmail.com> writes:
> 
> The conversion of a float to a decimal is not as straight-forward
> (most people don't even know that an exact conversion is possible).

I still don't follow you. You are saying that an exact conversion is possible,
but don't want it to be done because it is "not as straight-forward"?



From v+python at g.nevcal.com  Fri Mar 19 20:16:35 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 12:16:35 -0700
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <ho0gjk$dd4$1@dough.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>
	<ho0gjk$dd4$1@dough.gmane.org>
Message-ID: <4BA3CD93.8080502@g.nevcal.com>

On 3/19/2010 11:43 AM, Terry Reedy wrote:
> On 3/19/2010 2:11 PM, Antoine Pitrou wrote:
>> Raymond Hettinger<raymond.hettinger<at>  gmail.com>  writes:
>>>
>>> The reason to prefer an exception is that decimal/float comparisons
>>> are more likely to be a programmer error than an intended behavior.
>
> If you really believe that, then equality comparisons should also be 
> disabled by raising NotImplemented or whatever. 

Totally agree.  While the example most used in this thread is a less 
than operator, the text of the thread has never (as far as I have read) 
distinguished between equality, inequality, or relative signed magnitude 
comparisons.  Sort has also been mentioned explicitly as an example.  I 
perceive that the whole thread is about _all_ comparison operators with 
one float and one decimal operand currently producing an exception (3.x) 
or a type-based ordering (2.x).

The type-based ordering has been demonstrated to produce unstable sorts 
in the presence of other types also, so is more of a problem than first 
perceived, and should be changed.


> Clearly, someone who writes 'if somefloat == somedecimal:'assumes (now 
> wrongly) that the test might be true. This is just as buggy as an 
> order comparison. Raising an exception would consistently isolate 
> decimals from other numbers and eliminate the equality intransitivity 
> mess and its nasty effect on sets.

Totally agree.


> It still strikes me as a bit crazy for Python to say that 0.0 == 0 and 
> 0 == Decimal(0) but that 0.0 != Decimal(0). Who would expect such a 
> thing?

The same person that would expect both

0 == "0"
0.0 == "0.0"

to be False... i.e. anyone that hasn't coded in Perl for too many years.

Glenn


From chambon.pascal at wanadoo.fr  Fri Mar 19 19:58:42 2010
From: chambon.pascal at wanadoo.fr (Pascal Chambon)
Date: Fri, 19 Mar 2010 19:58:42 +0100
Subject: [Python-Dev] Attribute lookup ambiguity
Message-ID: <4BA3C962.7080500@wanadoo.fr>

Hello

I've already crossed a bunch of articles detailing python's attribute 
lookup semantic (__dict__, descriptors, order of base class 
traversing...), but I have never seen, so far, an explanation of WHICH 
method did waht, exactly.

I assumed that getattr(a, b) was the same as a.__getattribute__(b), and 
that this __getattribute__ method (or the hidden routine replacing it 
when we don't override it in our class) was in charge of doing the whole 
job of traversing the object tree, checking descriptors, binding 
methods, calling __getattr__ on failure etc.

However, the test case below shows that __getattribute__ does NOT call 
__getattr__ on failure. So it seems it's an upper levl machinery, in 
getattr(), which is in chrge of that last action.

Is that on purpose ? Considering that __getattribute__ (at lest, 
object.__getattribute__) does 90% of the hard job, why are these 10% left ?
Can we find somewhere the details of "who must do what" when customizing 
attribute access ?
Shouldn't we inform people about the fact that __getattribute__ isn't 
sufficient in itself to lookup an attribute ?

Thanks for the attention,
regards,
Pascal



=======
INPUT
=======

class A(object):

    def __getattribute__(self, name):
        print "A getattribute", name
        return object.__getattribute__(self, name)

    def __getattr__(self, name):
        print "A getattr", name
        return "hello A"


class B(A):


    def __getattribute__(self, name):
        print "B getattribute", name
        return A.__getattribute__(self, name)

    
    def __getattr__(self, name):
        print "B getattr", name
        return "hello B"
    
    

print A().obj
print "---"
print B().obj
print "---"
print getattr(B(), "obj")
print "-----"
print object.__getattribute__(B(), "obj") # DOES NOT CALL __getattr__() !!!


===========
OUTPUT
===========

A getattribute obj
A getattr obj
hello A
---
B getattribute obj
A getattribute obj
B getattr obj
hello B
---
B getattribute obj
A getattribute obj
B getattr obj
hello B
-----
Traceback (most recent call last):
  File "C:\Users\Pakal\Desktop\test_object_model.py", line 34, in <module>
    print object.__getattribute__(B(), "obj") # DOES NOT CALL 
__getattr__() !!!???
AttributeError: 'B' object has no attribute 'obj'

From brett at python.org  Fri Mar 19 20:27:46 2010
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Mar 2010 12:27:46 -0700
Subject: [Python-Dev] 3.1.2 tagging delayed a little
In-Reply-To: <1afaf6161003182020l1dd73856k30942c590ebcfb8f@mail.gmail.com>
References: <1afaf6161003182020l1dd73856k30942c590ebcfb8f@mail.gmail.com>
Message-ID: <bbaeab101003191227o4d4283efh1363b2b22732dc51@mail.gmail.com>

On Thu, Mar 18, 2010 at 20:20, Benjamin Peterson <benjamin at python.org> wrote:
> Pending the resolution of a few Mac OS issues (#8068, #8069, and
> #8133), I'm not going to tag the release at the moment. Hopefully,
> we'll still be able to have a release in the next week.

Do you want issue8133 to be a release blocker? If you do I can make
sure it passes by the end of this weekend.

-Brett


>
> --
> Regards,
> Benjamin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>

From rdmurray at bitdance.com  Fri Mar 19 20:35:33 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 19 Mar 2010 15:35:33 -0400
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <46EF5EEA-CA98-44C5-92AD-0BC83E39090D@zope.com>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
	<20100319142205.E11FC1FDA39@kimball.webabinitio.net>
	<ho0bg1$q6c$1@dough.gmane.org>
	<46EF5EEA-CA98-44C5-92AD-0BC83E39090D@zope.com>
Message-ID: <20100319193533.E63481FDAFD@kimball.webabinitio.net>

On Fri, 19 Mar 2010 14:09:28 -0400, Zvezdan Petkovic <zvezdan at zope.com> wrote:
> On Mar 19, 2010, at 1:15 PM, Terry Reedy wrote:
> > On 3/19/2010 10:22 AM, R. David Murray wrote:
> >> This can be done by anyone just by saying, eg: 'see issue 1234' (roundup
> >> turns that into a link),
> > 
> > That should be 'see issue #1234' to get the autolink.
> > From http://wiki.python.org/moin/TrackerDocs/
> 
> The list in the docs is not complete then.
> David is right.  The words 'issue 1234' will become a link in roundup automatically.  As well as 'issue1234'.

I've updated the wiki page.

--
R. David Murray                                      www.bitdance.com

From rdmurray at bitdance.com  Fri Mar 19 20:46:35 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 19 Mar 2010 15:46:35 -0400
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <20100319163958.GB8716@amk-desktop.matrixgroup.net>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
	<20100319142205.E11FC1FDA39@kimball.webabinitio.net>
	<20100319163958.GB8716@amk-desktop.matrixgroup.net>
Message-ID: <20100319194635.3DE491FDAE7@kimball.webabinitio.net>

On Fri, 19 Mar 2010 12:39:58 -0400, "A.M. Kuchling" <amk at amk.ca> wrote:
> On Fri, Mar 19, 2010 at 10:22:05AM -0400, R. David Murray wrote:
> > > Real world example with issue8151. It is an issue with a trivial patch
> > > in it. Everything what is needed is to dispatch it to stable `commit
> > > queue` and port to trunk. It is not 'easy' - it is 'trivial', but I
> > > have no means to mark it as 'easy' either, so even this trivial fix
> > > lies in tracker for three days waiting for review.
> > 
> > Three days?  That's nothing :)
> 
> I think Anatoly has a point, though; there are issues where the people
> discussing the fix have come to a consensus, but none of them are
> committers, or the committers involved are too busy.  There's a patch
> ready to apply, but no committer knows about it.

Yes, this is a general problem with the workflow.

This is one reason for the creation of the 'committers.rst' file in py3k:
submitters can find committers there to add to the nosy list of issues
that are ready for commit, so that they do notice.

However, as you note, sometimes 'ready' bugs sit in the tracker for
a while even when one or more committers is already nosy.  As someone
else said, that's the nature of a volunteer driven process.  The best
solution as far as I know is to expand the pool of volunteers, so I
hope Anatoloy is inspired to continue to contribute so he can eventually
become a committer!

> > To avoid issues involving C coding, try restricting your search to
> > 'Library' issues.  There is C code in some library modules, but less so
> > than in the core ;).  It would be nice if you could search on more than
> 
> Actually Library should pretty much always be pure-Python.  Issues
> related to C modules such as socket are classified under the
> 'Extension modules' component.  Of course, sometimes the ultimate
> cause will migrate (a bug in SocketServer.py is traced to a flaw in
> socket, for example).

Good point.  Or at least, that's the theory.  In practice bugs sometimes
get miscategorized.  So, Anatoly, if you find bugs like that you can ask that
they be properly recategorized.

--
R. David Murray                                      www.bitdance.com

From dickinsm at gmail.com  Fri Mar 19 20:50:38 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Fri, 19 Mar 2010 19:50:38 +0000
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float 	comparisons in py3k.
In-Reply-To: <ho0gjk$dd4$1@dough.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<ho0gjk$dd4$1@dough.gmane.org>
Message-ID: <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>

On Fri, Mar 19, 2010 at 6:43 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 3/19/2010 2:11 PM, Antoine Pitrou wrote:
>>
>> Raymond Hettinger<raymond.hettinger<at> ?gmail.com> ?writes:
>>>
>>> The reason to prefer an exception is that decimal/float comparisons
>>> are more likely to be a programmer error than an intended behavior.
>
> If you really believe that, then equality comparisons should also be
> disabled by raising NotImplemented or whatever.

Hah.  This is a very good point, and one I'd somehow missed up until
now.  I don't think we *can* reasonably make equality comparisons
raise NotImplemented (in either 2.x or 3.x), since that messes up
containment tests:  something like "1.0 in {2, Decimal(3)}" would
raise a TypeError instead of correctly returning False.  (So the patch
I've put on the issue tracker is wrong, since it does raise TypeError
for equality and inequality, as well as for <, >, <= and >=.)

> Clearly, someone who writes
> 'if somefloat == somedecimal:'assumes (now wrongly) that the test might be
> true. This is just as buggy as an order comparison. Raising an exception
> would consistently isolate decimals from other numbers and eliminate the
> equality intransitivity mess and its nasty effect on sets.
>
> It still strikes me as a bit crazy for Python to say that 0.0 == 0 and 0 ==
> Decimal(0) but that 0.0 != Decimal(0). Who would expect such a thing?

Agreed.  A solution to the original problem that still has 0.0 ==
Decimal(0) evaluating to False isn't much of a solution.

This puts me firmly in the 'make numeric comparisons give the right
answer' camp.

Thanks,

Mark

From dickinsm at gmail.com  Fri Mar 19 20:55:58 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Fri, 19 Mar 2010 19:55:58 +0000
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float 	comparisons in py3k.
In-Reply-To: <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<ho0gjk$dd4$1@dough.gmane.org>
	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>
Message-ID: <5c6f2a5d1003191255u14d74227j583ec140fea534ab@mail.gmail.com>

On Fri, Mar 19, 2010 at 7:50 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> So the patch
> I've put on the issue tracker is wrong, since it does raise TypeError ...

s/I've put/I have yet to put/

I really shouldn't admit to errors in things that I haven't even been
made public yet.  :)

Mark

From v+python at g.nevcal.com  Fri Mar 19 21:05:32 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 13:05:32 -0700
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float 	comparisons in py3k.
In-Reply-To: <5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>
	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>
Message-ID: <4BA3D90C.5060009@g.nevcal.com>

On 3/19/2010 12:50 PM, Mark Dickinson wrote:
> Hah.  This is a very good point, and one I'd somehow missed up until
> now.  I don't think we*can*  reasonably make equality comparisons
> raise NotImplemented (in either 2.x or 3.x), since that messes up
> containment tests:  something like "1.0 in {2, Decimal(3)}" would
> raise a TypeError instead of correctly returning False.  (So the patch
> I've put on the issue tracker is wrong, since it does raise TypeError
> for equality and inequality, as well as for<,>,<= and>=.)
>    

Sounds to me like containment checking is wrong; that if it gets an 
exception during the comparison that it should assume unequal, rather 
than aborting, and continue to the next entry.

Wouldn't the same issue arise for containment tests when disparate, 
incomparable objects are included in the same set?  Why is this different?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/486d0dbe/attachment.html>

From fuzzyman at voidspace.org.uk  Fri Mar 19 21:23:43 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 19 Mar 2010 20:23:43 +0000
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA3C962.7080500@wanadoo.fr>
References: <4BA3C962.7080500@wanadoo.fr>
Message-ID: <4BA3DD4F.1080209@voidspace.org.uk>

On 19/03/2010 18:58, Pascal Chambon wrote:
> Hello
>
> I've already crossed a bunch of articles detailing python's attribute 
> lookup semantic (__dict__, descriptors, order of base class 
> traversing...), but I have never seen, so far, an explanation of WHICH 
> method did waht, exactly.
>
> I assumed that getattr(a, b) was the same as a.__getattribute__(b), 
> and that this __getattribute__ method (or the hidden routine replacing 
> it when we don't override it in our class) was in charge of doing the 
> whole job of traversing the object tree, checking descriptors, binding 
> methods, calling __getattr__ on failure etc.
>
> However, the test case below shows that __getattribute__ does NOT call 
> __getattr__ on failure. So it seems it's an upper levl machinery, in 
> getattr(), which is in chrge of that last action.

Python 3 has the behavior you are asking for. It would be a backwards 
incompatible change to do it in Python 2 as __getattribute__ *not* 
calling __getattr__ is the documented behaviour.

Python 3.2a0 (py3k:78770, Mar 7 2010, 20:32:50)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
 >>> class x:
... def __getattribute__(s, name):
... print ('__getattribute__', name)
... raise AttributeError
... def __getattr__(s, name):
... print ('__getattr__', name)
...
 >>> a = x()
 >>> a.b
__getattribute__ b
__getattr__ b


This list is not really an appropriate place to ask questions like this 
though, comp.lang.python would be better.

All the best,

Michael Fooord

>
> Is that on purpose ? Considering that __getattribute__ (at lest, 
> object.__getattribute__) does 90% of the hard job, why are these 10% 
> left ?
> Can we find somewhere the details of "who must do what" when 
> customizing attribute access ?
> Shouldn't we inform people about the fact that __getattribute__ isn't 
> sufficient in itself to lookup an attribute ?
>
> Thanks for the attention,
> regards,
> Pascal
>
>
>
> =======
> INPUT
> =======
>
> class A(object):
>
> def __getattribute__(self, name):
> print "A getattribute", name
> return object.__getattribute__(self, name)
>
> def __getattr__(self, name):
> print "A getattr", name
> return "hello A"
>
>
> class B(A):
>
>
> def __getattribute__(self, name):
> print "B getattribute", name
> return A.__getattribute__(self, name)
>
> def __getattr__(self, name):
> print "B getattr", name
> return "hello B"
>
> print A().obj
> print "---"
> print B().obj
> print "---"
> print getattr(B(), "obj")
> print "-----"
> print object.__getattribute__(B(), "obj") # DOES NOT CALL 
> __getattr__() !!!
>
>
> ===========
> OUTPUT
> ===========
>
> A getattribute obj
> A getattr obj
> hello A
> ---
> B getattribute obj
> A getattribute obj
> B getattr obj
> hello B
> ---
> B getattribute obj
> A getattribute obj
> B getattr obj
> hello B
> -----
> Traceback (most recent call last):
> File "C:\Users\Pakal\Desktop\test_object_model.py", line 34, in <module>
> print object.__getattribute__(B(), "obj") # DOES NOT CALL 
> __getattr__() !!!???
> AttributeError: 'B' object has no attribute 'obj'
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk 
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From rdmurray at bitdance.com  Fri Mar 19 21:33:56 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 19 Mar 2010 16:33:56 -0400
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float comparisons in py3k.
In-Reply-To: <4BA3CD93.8080502@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<ho0gjk$dd4$1@dough.gmane.org> <4BA3CD93.8080502@g.nevcal.com>
Message-ID: <20100319203356.DED911FDAE6@kimball.webabinitio.net>

On Fri, 19 Mar 2010 12:16:35 -0700, Glenn Linderman <v+python at g.nevcal.com> wrote:
> On 3/19/2010 11:43 AM, Terry Reedy wrote:
> > It still strikes me as a bit crazy for Python to say that 0.0 == 0 and 
> > 0 == Decimal(0) but that 0.0 != Decimal(0). Who would expect such a 
> > thing?
> 
> The same person that would expect both
> 
> 0 == "0"
> 0.0 == "0.0"
> 
> to be False... i.e. anyone that hasn't coded in Perl for too many years.

No, those two situations are not comparable.

"if a equals b and b equals c, then a equals c" is what most people
will expect.  In fact, only mathematicians are likely to know
that any other situation could possibly be valid.  Programmers are
generally going to view it as a bug.

--
R. David Murray                                      www.bitdance.com

From benjamin at python.org  Fri Mar 19 21:56:44 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 19 Mar 2010 15:56:44 -0500
Subject: [Python-Dev] 3.1.2 tagging delayed a little
In-Reply-To: <bbaeab101003191227o4d4283efh1363b2b22732dc51@mail.gmail.com>
References: <1afaf6161003182020l1dd73856k30942c590ebcfb8f@mail.gmail.com>
	<bbaeab101003191227o4d4283efh1363b2b22732dc51@mail.gmail.com>
Message-ID: <1afaf6161003191356l517eda26yd978a986ea6d4dc6@mail.gmail.com>

2010/3/19 Brett Cannon <brett at python.org>:
> On Thu, Mar 18, 2010 at 20:20, Benjamin Peterson <benjamin at python.org> wrote:
>> Pending the resolution of a few Mac OS issues (#8068, #8069, and
>> #8133), I'm not going to tag the release at the moment. Hopefully,
>> we'll still be able to have a release in the next week.
>
> Do you want issue8133 to be a release blocker? If you do I can make
> sure it passes by the end of this weekend.

That would be wonderful.



-- 
Regards,
Benjamin

From guido at python.org  Fri Mar 19 22:06:59 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 19 Mar 2010 14:06:59 -0700
Subject: [Python-Dev] Decimal &amp; amp; amp; lt; -&amp; amp; amp; gt;
	float 	comparisons in py3k.
In-Reply-To: <loom.20100319T195334-851@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> 
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com> 
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com> 
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> 
	<loom.20100319T190158-232@post.gmane.org>
	<7A59AEF6-A4F9-4D81-BE9A-EAD89053AD7F@gmail.com> 
	<loom.20100319T195334-851@post.gmane.org>
Message-ID: <ca471dc21003191406s52775930w67a697bd6eaf5706@mail.gmail.com>

Antoine, I think your email client turns replies whose subject
contains '&' into new subjects containing sequences like this:

  &amp;amp; amp; lt; -&amp;amp; amp; gt;

There's a number of new threads with ever-more occurrences of "amp" in
the subject, and you are the first poster in each thread (and the
first post's subject already starts with "Re:").

-- 
--Guido van Rossum (python.org/~guido)

From aditya at wefoundland.com  Fri Mar 19 22:10:53 2010
From: aditya at wefoundland.com (aditya bhargava)
Date: Fri, 19 Mar 2010 16:10:53 -0500
Subject: [Python-Dev] "expected a character buffer object" messages
Message-ID: <4d91314c1003191410r4fe5e9adv264a90d2a68abb18@mail.gmail.com>

I found an old thread on Python-Dev that suggested replacing the error
"TypeError: expected a character buffer object" with something more helpful
for non-experts like "expected a string or character buffer". Here's the
thread: http://mail.python.org/pipermail/python-dev/2006-May/065301.html

I think this is an excellent idea, but besides this ticket:
http://bugs.python.org/issue6780
I haven't seen anything further about it. Does anyone know the status on
this? It would be nice to have a more descriptive error.

Aditya
-- 
wefoundland.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/5d439745/attachment.html>

From solipsis at pitrou.net  Fri Mar 19 22:48:04 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Mar 2010 21:48:04 +0000 (UTC)
Subject: [Python-Dev] Decimal [...] float comparisons in py3k.
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<7A59AEF6-A4F9-4D81-BE9A-EAD89053AD7F@gmail.com>
	<loom.20100319T195334-851@post.gmane.org>
	<ca471dc21003191406s52775930w67a697bd6eaf5706@mail.gmail.com>
Message-ID: <loom.20100319T224605-636@post.gmane.org>

Guido van Rossum <guido <at> python.org> writes:
> 
> Antoine, I think your email client turns replies whose subject
> contains '&' into new subjects containing sequences like this:
> 
>   &amp;amp; amp; lt; -&amp;amp; amp; gt;
> 
> There's a number of new threads with ever-more occurrences of "amp" in
> the subject, and you are the first poster in each thread (and the
> first post's subject already starts with "Re:").

Hmm, indeed. It's the gmane Web posting interface, actually.

Regards

Antoine.



From guido at python.org  Fri Mar 19 22:50:04 2010
From: guido at python.org (Guido van Rossum)
Date: Fri, 19 Mar 2010 14:50:04 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
Message-ID: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>

I'd like to reboot this thread. I've been spinning this topic in my
head for most of the morning, and I think we should seriously
reconsider allowing mixed arithmetic involving Decimal, not just mixed
comparisons. [Quick summary: embed Decimal in the numeric tower but
add a context flag to disallow implicit mixing of float and Decimal.]

I tried to find the argumentation against it in PEP 327 (Decimal Data
Type) and found that it didn't make much of an argument against mixed
arithmetic beyond "it's not needed" and "it's not urgent". (It even
states that initially Decimal.from_float() was omitted for simplicity
-- but it got added later.)

We now also have PEP 3141 (A Type Hierarchy for Numbers) which
proposes a numeric tower. It has an explicit exclusion for Decimal,
but that exclusion is provisional: "After consultation with its
authors it has been decided that the ``Decimal`` type should not at
this time be made part of the numeric tower." That was a compromise
because at the time some contributors to Decimal were fiercely opposed
to including Decimal into the numeric tower, and I didn't want to have
a endless discussion at the time (there were many more pressing issues
to be resolved).

However now the subject is coming up again, and my gut keeps telling
me that Decimal ought to be properly embedded in Python's numeric
tower. Decimal is already *touching* the numeric tower by allowing
mixed arithmetic with ints. This causes the anomaly that Mark
mentioned earlier: the three values 1, 1.0 and Decimal(1) do not
satisfy the rule "if x == y and y == z then it follows that x == z".
We have 1.0 == 1 == Decimal(1) but 1 == 1.0 != Decimal(1). This also
causes problems with hashing, where {Decimal(1), 1, 1.0} !=
{Decimal(1), 1.0, 1}.

I'd like to look at the issue by comparing the benefits and drawbacks
of properly embedding Decimal into the numeric tower. As advantages, I
see consistent behavior in situations like the above and more
intuitive behavior for beginners. Also, this would be a possible road
towards eventually supporting a language extension where floating
point literals produce Decimal values instead of binary floats. (A
possible syntax could be "from __options__ import decimal_float",
which would work similar to "from __future__ import ..." except it's a
permanent part of the language rather than a forward compatibility
feature.)

As a downside, there is the worry that inadvertent mixing of Decimal
and float can compromise the correctness of programs in a way that is
hard to detect. But the anomalies above indicate that not fixing the
situation can *also* compromise correctness in a similar way. Maybe a
way out would be to add a new flag to the decimal Context class
indicating whether to disallow mixing Decimal and float -- that way
programs that care about this can force the issue, while the default
behavior can be more intuitive. Possibly the flag should not affect
comparisons.

There is one choice which I'm not sure about. Should a mixed
float/Decimal operation return a float or a Decimal? I note that
Fraction (which *is* properly embedded in the numeric tower) supports
this and returns a float result in this case. While I earlier proposed
to return the most "complicated" type of the two, i.e. Decimal, I now
think it may also make sense to return a float, being the most "fuzzy"
type in the numeric tower. This would also make checking for
accidental floats easier, since floats now propagate throughout the
computation (like NaN) and a simple assertion that the result is a
Decimal instance suffices to check that no floats were implicitly
mixed into the computation.

The implementation of __hash__ will be complicated, and it may make
sense to tweak the hash function of float, Fraction and Decimal to
make it easier to ensure that for values that can be represented in
either type the hash matches the equality. But this sounds a
worthwhile price to pay for proper embedding in the numeric tower.

-- 
--Guido van Rossum (python.org/~guido)

From solipsis at pitrou.net  Fri Mar 19 23:02:00 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Mar 2010 22:02:00 +0000 (UTC)
Subject: [Python-Dev] containment checking
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>
	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>
	<4BA3D90C.5060009@g.nevcal.com>
Message-ID: <loom.20100319T224859-46@post.gmane.org>

Glenn Linderman <v+python <at> g.nevcal.com> writes:
> 
> Sounds to me like containment checking is wrong; that if it gets an
> exception during the comparison that it should assume unequal, rather
> than aborting, and continue to the next entry.

Well as the Zen says:

Errors should never pass silently.
Unless explicitly silenced.

If there's a bug in your __eq__ method, rather than getting wrong containment
results, you get the proper exception.

> Wouldn't the same issue arise for containment tests when disparate,
> incomparable objects are included in the same set?? Why is this
> different?

That's why equality testing almost never fails between standard types. We do
have an (IMO unfortunate) exception in the datetime module, though:

>>> from datetime import tzinfo, timedelta, datetime
>>> ZERO = timedelta(0)
>>> class UTC(tzinfo):     # UTC class ripped off from the official doc
...     """UTC"""
...     def utcoffset(self, dt):
...         return ZERO
...     def tzname(self, dt):
...         return "UTC"
...     def dst(self, dt):
...         return ZERO
... 
>>> utc = UTC()
>>> a = datetime.now()
>>> b = datetime.now(utc)
>>> a == b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't compare offset-naive and offset-aware datetimes


Regards

Antoine.



From raymond.hettinger at gmail.com  Fri Mar 19 23:36:38 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 19 Mar 2010 15:36:38 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
Message-ID: <28258875-7132-4FA0-9926-24A6346FF943@gmail.com>


On Mar 19, 2010, at 2:50 PM, Guido van Rossum wrote:

> I'd like to reboot this thread. I've been spinning this topic in my
> head for most of the morning, and I think we should seriously
> reconsider allowing mixed arithmetic involving Decimal, not just mixed
> comparisons. [Quick summary: embed Decimal in the numeric tower but
> add a context flag to disallow implicit mixing of float and Decimal.]


Making decimals first class citizens would sure help eliminate 
some special cases and anomalies. 

If a context flag were added, am wondering whether it should simply
provide a warning rather than flat-out disallowing the transaction.
The whole point is to highlight accidental mixing.

If the mixed arithmetic were allowed, then the decimal constructor
should be changed to match (i.e. accept float inputs instead of
requiring Decimal.from_float()).


Raymond



From barry at python.org  Fri Mar 19 23:56:55 2010
From: barry at python.org (Barry Warsaw)
Date: Fri, 19 Mar 2010 18:56:55 -0400
Subject: [Python-Dev] RELEASED Python 2.6.5
Message-ID: <20100319185655.26e1bf1a@heresy>

On behalf of the Python community, I'm happy to announce the availability of
Python 2.6.5 final.  This is the latest production-ready version in the Python
2.6 series.

Python 2.6.5 fixes dozens of issues in the core, built-in modules, libraries,
and documentation since Python 2.6.4 was released back in October 2009.  We
highly recommend that you upgrade to Python 2.6.5.

Please see the NEWS file for all the gory details.

    http://www.python.org/download/releases/2.6.5/NEWS.txt

Source tarballs and the Windows installers can be downloaded from the Python
2.6.5 page.  The Mac OS X disk image will be uploaded soon.

    http://www.python.org/download/releases/2.6.5/

For more information on Python 2.6 in general, please see

    http://docs.python.org/whatsnew/2.6.html

Please report bugs for any Python version in the Python tracker.

    http://bugs.python.org

Enjoy,
-Barry

Barry Warsaw
barry at python.org
Python 2.6 Release Manager
(on behalf of the entire python-dev team)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/1f244f4d/attachment.pgp>

From martin at v.loewis.de  Sat Mar 20 00:00:48 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 20 Mar 2010 00:00:48 +0100
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <4BA388E5.1030703@gmail.com>
References: <20100319023630.GC1617@idyll.org>
	<4BA31858.3010904@gmail.com>	<bad82a81003190457w5c0e1770hd7c0f4af3f0a8746@mail.gmail.com>
	<4BA388E5.1030703@gmail.com>
Message-ID: <4BA40220.30701@v.loewis.de>

> Whether this is worth weeks of work or not will depend on a given
> student's knowledge about Python 3, and I'd suspect that the GSoC would
> be an opportunity for a number of applicant to actually learn the
> intricacies of Python 3.
> Developing Python 3-specific features could be used to increase the
> amount of work on the project, but I am uncertain of whether this is
> worth a full GSoC.

I'd say this would definitely make a GSoC project; any non-trivial
porting will be.

As you get experienced with porting, it can indeed go fairly quickly.
However, the learning curve is not to be underestimated: the first few
changes will rather be in the "one change per week" range.

Regards,
Martin

From tjreedy at udel.edu  Sat Mar 20 00:03:59 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 19 Mar 2010 19:03:59 -0400
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float comparisons in py3k.
In-Reply-To: <4BA3CD93.8080502@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>
	<4BA3CD93.8080502@g.nevcal.com>
Message-ID: <ho0vsv$1g6$1@dough.gmane.org>

On 3/19/2010 3:16 PM, Glenn Linderman wrote:

> I perceive that the whole thread is about _all_ comparison operators with
> one float and one decimal operand currently producing an exception (3.x)

Not true for equality comparison. That returns False regardless of 
value, just as order comparisons return what they do in 2.x regardless 
of value. I claim that the former is at least as bad as the latter for 
numbers.

> or a type-based ordering (2.x).

3.x mixes type-based and value-based equality testing for Decimals and 
other numbers.

Terry Jan Reedy


From dickinsm at gmail.com  Sat Mar 20 00:13:29 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Fri, 19 Mar 2010 23:13:29 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
Message-ID: <5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>

Just a couple of quick side comments on this;  I haven't got my head
around the whole mixed-operations idea yet.

On Fri, Mar 19, 2010 at 9:50 PM, Guido van Rossum <guido at python.org> wrote:
> There is one choice which I'm not sure about. Should a mixed
> float/Decimal operation return a float or a Decimal?

I'll just say that it's much easier to return a Decimal if you want to
be able to make guarantees about rounding behaviour, basically because
floats can be converted losslessly to Decimals.  I also like the fact
that the decimal module offers more control (rounding mode, precision,
flags, wider exponent range) than float.

In general, the correct semantics for an arithmetic operation are to
produce a result that's equivalent to what would have been obtained by
performing the operation to infinite precision and then doing a single
round to fit that result into the output type format.  This is
trivially easy to do if mixed float and Decimal operations produce
Decimal, and much much harder if they produce floats;  if mixed-type
operations produced floats we'd probably have to go with algorithms
that involve two rounds (i.e., first coerce the Decimal to a float,
then do the operation as usual on the two floats), and there would
likely be some (small) numeric surprises as a result.

> The implementation of __hash__ will be complicated, and it may make
> sense to tweak the hash function of float, Fraction and Decimal to
> make it easier to ensure that for values that can be represented in
> either type the hash matches the equality. But this sounds a
> worthwhile price to pay for proper embedding in the numeric tower.

I don't think this is going to be a problem.  I've implemented most of
the scheme I outlined earlier (it's working for ints, floats and
Decimals;  still need to implement it for Fractions and complex
numbers) and it seems to work just fine, with essentially no more code
than was there before.  I'll post a proof-of-concept patch when I've
filled in the missing bits.

Mark

From facundobatista at gmail.com  Sat Mar 20 00:17:24 2010
From: facundobatista at gmail.com (Facundo Batista)
Date: Fri, 19 Mar 2010 19:17:24 -0400
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
Message-ID: <e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>

On Fri, Mar 19, 2010 at 5:50 PM, Guido van Rossum <guido at python.org> wrote:

> As a downside, there is the worry that inadvertent mixing of Decimal
> and float can compromise the correctness of programs in a way that is
> hard to detect. But the anomalies above indicate that not fixing the

Decimal already has something that we can use in this case, and fits
very nice here: Signals.

Signals represent conditions that arise during computation. Each
corresponds to one context flag and one context trap enabler.

So, if we add a signal like "MixedWithFloats", users will have a flag
in the context that they could check to see if a float was mixed in
the operations executed (and if the user set the trap accordingly, an
exception will be raised when the signal happens).

OTOH, returning a float the first time both are mixed is easy to
check... but if it has downsides, and we prefer to return a Decimal in
that case, note that we have a mechanism in Decimal we can use.

Furthermore, in case we want to ease the transition we can do the following:

- add this signal

- set *by default* the trap to raise an exception when float and
Decimal is mixed

So, the behaviour will be the same as we have now, but users can
easily change it.

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From v+python at g.nevcal.com  Sat Mar 20 00:50:17 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 16:50:17 -0700
Subject: [Python-Dev] containment checking
In-Reply-To: <loom.20100319T224859-46@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>	<4BA3D90C.5060009@g.nevcal.com>
	<loom.20100319T224859-46@post.gmane.org>
Message-ID: <4BA40DB9.3020302@g.nevcal.com>

On 3/19/2010 3:02 PM, Antoine Pitrou wrote:
> Glenn Linderman<v+python<at>  g.nevcal.com>  writes:
>    
>> >  
>> >  Sounds to me like containment checking is wrong; that if it gets an
>> >  exception during the comparison that it should assume unequal, rather
>> >  than aborting, and continue to the next entry.
>>      
> Well as the Zen says:
>
> Errors should never pass silently.
> Unless explicitly silenced.
>
> If there's a bug in your __eq__ method, rather than getting wrong containment
> results, you get the proper exception.
>    

If there's a bug in your __eq__ method, it may or may not raise an 
exception, which may or may not get you wrong containment results.  But 
it will probably get you buggy results, somehow or another.  That's what 
design, code reviews, and testing are for.

If the your __eq__ method uses exceptions (the only available method of 
out-of-band signalling for binary operators; not all exceptions are 
errors) to declare that it can't perform the comparison and produce a 
boolean result, that is a case where producing an exception is not an 
error, so your quoted Zen doesn't apply.

Glenn


From solipsis at pitrou.net  Sat Mar 20 00:58:05 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 19 Mar 2010 23:58:05 +0000 (UTC)
Subject: [Python-Dev] containment checking
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>	<4BA3D90C.5060009@g.nevcal.com>
	<loom.20100319T224859-46@post.gmane.org>
	<4BA40DB9.3020302@g.nevcal.com>
Message-ID: <loom.20100320T005523-673@post.gmane.org>

Glenn Linderman <v+python <at> g.nevcal.com> writes:
> 
> If there's a bug in your __eq__ method, it may or may not raise an 
> exception, which may or may not get you wrong containment results.  But 
> it will probably get you buggy results, somehow or another.  That's what 
> design, code reviews, and testing are for.

We'll have to "agree to disagree" then. If you want error silencing by default,
Python is not the language you are looking for.



From greg.ewing at canterbury.ac.nz  Sat Mar 20 01:10:25 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 20 Mar 2010 13:10:25 +1300
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA3680D.8070909@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
	<4BA3680D.8070909@gmail.com>
Message-ID: <4BA41271.3010003@canterbury.ac.nz>

Nick Coghlan wrote:
> Mark Dickinson wrote:

> It seems to me that given the existing conflation of numeric equivalence
> and containment testing, going the whole hog and fixing the set
> membership problem for all of our rational types would be the right
> thing to do.

Isn't this only solving half the problem, though? Even if
you find that the hashes are equal, you still have to decide
whether the values themselves are equal.

Is there some similarly clever way of comparing two
rational numbers without having explicit access to the
numerators and denominators?

-- 
Greg

From v+python at g.nevcal.com  Sat Mar 20 01:13:54 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 17:13:54 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
Message-ID: <4BA41342.8040603@g.nevcal.com>

On 3/19/2010 2:50 PM, Guido van Rossum wrote:
> I'd like to reboot this thread.

I'll go along with that idea!


>   I've been spinning this topic in my
> head for most of the morning, and I think we should seriously
> reconsider allowing mixed arithmetic involving Decimal, not just mixed
> comparisons. [Quick summary: embed Decimal in the numeric tower but
> add a context flag to disallow implicit mixing of float and Decimal.]
>    

As long as there is a way to avoid implicit mixing of float and Decimal, 
or to easily detect (preferably with an exception) the implicit mixing, 
then I think that solves the concerns of people trying to write 
numerically correct code using Decimal.  And if Mark (or someone) can 
solve the hashing anomoly problem without huge expense, then it could be 
a winner.

> I tried to find the argumentation against it in PEP 327
>    

Should Aahz be consulted, as some of the objections in PEP 327 are 
attributed to him, but he is pretty scarce around here these days?

> Also, this would be a possible road
> towards eventually supporting a language extension where floating
> point literals produce Decimal values instead of binary floats. (A
> possible syntax could be "from __options__ import decimal_float",
> which would work similar to "from __future__ import ..." except it's a
> permanent part of the language rather than a forward compatibility
> feature.)
>    

Nice touch here... rather than being forced to quote Decimal values as 
strings and convert from string, or use a tuple to represent the parts, 
both of which are warts.  Not sure what context would be used, though.

Glenn

From glenn at nevcal.com  Sat Mar 20 01:15:06 2010
From: glenn at nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 17:15:06 -0700
Subject: [Python-Dev] containment checking
In-Reply-To: <loom.20100320T005523-673@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>	<4BA3D90C.5060009@g.nevcal.com>	<loom.20100319T224859-46@post.gmane.org>	<4BA40DB9.3020302@g.nevcal.com>
	<loom.20100320T005523-673@post.gmane.org>
Message-ID: <4BA4138A.9090003@nevcal.com>

On 3/19/2010 4:58 PM, Antoine Pitrou wrote:
> Glenn Linderman<v+python<at>  g.nevcal.com>  writes:
>    
>> >  
>> >  If there's a bug in your __eq__ method, it may or may not raise an
>> >  exception, which may or may not get you wrong containment results.  But
>> >  it will probably get you buggy results, somehow or another.  That's what
>> >  design, code reviews, and testing are for.
>>      
> We'll have to "agree to disagree" then. If you want error silencing by default,
> Python is not the language you are looking for.
>    

We can agree to disagree, if you like.  But taken to the limit, the Zen 
you quoted would prevent the try except clause from being used.

From greg.ewing at canterbury.ac.nz  Sat Mar 20 01:27:09 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 20 Mar 2010 13:27:09 +1300
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <loom.20100319T190158-232@post.gmane.org>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
Message-ID: <4BA4165D.2070808@canterbury.ac.nz>

Antoine Pitrou wrote:

> We forbid comparisons when there is a real danger or ambiguity, such as unicode
> vs. bytes. There is no such danger or ambiguity when comparing a decimal with a
> float.

So do you think that float("0.1") and Decimal("0.1") should be
equal or not, and why?

-- 
Greg

From fuzzyman at voidspace.org.uk  Sat Mar 20 01:18:09 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 20 Mar 2010 00:18:09 +0000
Subject: [Python-Dev] containment checking
In-Reply-To: <4BA4138A.9090003@nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>	<4BA3D90C.5060009@g.nevcal.com>	<loom.20100319T224859-46@post.gmane.org>	<4BA40DB9.3020302@g.nevcal.com>	<loom.20100320T005523-673@post.gmane.org>
	<4BA4138A.9090003@nevcal.com>
Message-ID: <4BA41441.8030404@voidspace.org.uk>

On 20/03/2010 00:15, Glenn Linderman wrote:
> On 3/19/2010 4:58 PM, Antoine Pitrou wrote:
>> Glenn Linderman<v+python<at> g.nevcal.com> writes:
>>> > > If there's a bug in your __eq__ method, it may or may not raise an
>>> > exception, which may or may not get you wrong containment results. 
>>> But
>>> > it will probably get you buggy results, somehow or another. That's 
>>> what
>>> > design, code reviews, and testing are for.
>> We'll have to "agree to disagree" then. If you want error silencing 
>> by default,
>> Python is not the language you are looking for.
>
> We can agree to disagree, if you like. But taken to the limit, the Zen 
> you quoted would prevent the try except clause from being used.

No, that is what "unless explicitly silenced" means - you are proposing 
to silence them *without* an explicit try except clause.

Michael

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk 
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From v+python at g.nevcal.com  Sat Mar 20 01:19:28 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 17:19:28 -0700
Subject: [Python-Dev] containment checking
In-Reply-To: <4BA41441.8030404@voidspace.org.uk>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>	<4BA3D90C.5060009@g.nevcal.com>	<loom.20100319T224859-46@post.gmane.org>	<4BA40DB9.3020302@g.nevcal.com>	<loom.20100320T005523-673@post.gmane.org>
	<4BA4138A.9090003@nevcal.com> <4BA41441.8030404@voidspace.org.uk>
Message-ID: <4BA41490.1000803@g.nevcal.com>

On 3/19/2010 5:18 PM, Michael Foord wrote:
>>>> will probably get you buggy results, somehow or another. That's what
>>>> > design, code reviews, and testing are for.
>>> We'll have to "agree to disagree" then. If you want error silencing 
>>> by default,
>>> Python is not the language you are looking for.
>>
>> We can agree to disagree, if you like. But taken to the limit, the 
>> Zen you quoted would prevent the try except clause from being used.
>
> No, that is what "unless explicitly silenced" means - you are 
> proposing to silence them *without* an explicit try except clause.
>
> Michael 

Who, me?  The containment checking code would contain the try/except, I 
was proposing.

Glenn

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/aa578514/attachment.html>

From fuzzyman at voidspace.org.uk  Sat Mar 20 01:20:57 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 20 Mar 2010 00:20:57 +0000
Subject: [Python-Dev] containment checking
In-Reply-To: <4BA41490.1000803@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>	<4BA3D90C.5060009@g.nevcal.com>	<loom.20100319T224859-46@post.gmane.org>	<4BA40DB9.3020302@g.nevcal.com>	<loom.20100320T005523-673@post.gmane.org>
	<4BA4138A.9090003@nevcal.com> <4BA41441.8030404@voidspace.org.uk>
	<4BA41490.1000803@g.nevcal.com>
Message-ID: <4BA414E9.2000105@voidspace.org.uk>

On 20/03/2010 00:19, Glenn Linderman wrote:
> On 3/19/2010 5:18 PM, Michael Foord wrote:
>>>>> will probably get you buggy results, somehow or another. That's what
>>>>> > design, code reviews, and testing are for.
>>>> We'll have to "agree to disagree" then. If you want error silencing 
>>>> by default,
>>>> Python is not the language you are looking for.
>>>
>>> We can agree to disagree, if you like. But taken to the limit, the 
>>> Zen you quoted would prevent the try except clause from being used.
>>
>> No, that is what "unless explicitly silenced" means - you are 
>> proposing to silence them *without* an explicit try except clause.
>>
>> Michael 
>
> Who, me?  The containment checking code would contain the try/except, 
> I was proposing.

Explicit by the programmer. That is what explicit means... Caught and 
silenced for you by Python is implicit.

Michael

>
> Glenn
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100320/59527487/attachment.html>

From solipsis at pitrou.net  Sat Mar 20 01:34:39 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 20 Mar 2010 00:34:39 +0000 (UTC)
Subject: [Python-Dev] 0.1
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<4BA4165D.2070808@canterbury.ac.nz>
Message-ID: <loom.20100320T013221-223@post.gmane.org>

Greg Ewing <greg.ewing <at> canterbury.ac.nz> writes:
> 
> So do you think that float("0.1") and Decimal("0.1") should be
> equal or not, and why?

Given that float("0.1") is not the mathematical 0.1, while Decimal("0.1") is,
I'd say no.



From greg.ewing at canterbury.ac.nz  Sat Mar 20 01:50:50 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 20 Mar 2010 13:50:50 +1300
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA3D90C.5060009@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<ho0gjk$dd4$1@dough.gmane.org>
	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>
	<4BA3D90C.5060009@g.nevcal.com>
Message-ID: <4BA41BEA.6070309@canterbury.ac.nz>

Glenn Linderman wrote:

> Sounds to me like containment checking is wrong; that if it gets an 
> exception during the comparison that it should assume unequal, rather 
> than aborting, and continue to the next entry.

What exception would it catch, though? Catching something as
generic as TypeError would be a very bad idea, I think -- there
would be too much potential for it to mask bugs.

It might be acceptable if there were a special subclass of
TypeError for this, such as NotComparableError.

What happens in Py3 here, BTW? It must have the same problem
if it refuses to compare some things for equality.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sat Mar 20 02:22:16 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 20 Mar 2010 14:22:16 +1300
Subject: [Python-Dev] Decimal &amp; lt; -&amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA3C59F.7030508@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4BA3C59F.7030508@g.nevcal.com>
Message-ID: <4BA42348.4050203@canterbury.ac.nz>

Glenn Linderman wrote:
> In order to do 
> implicit comparisons, one must do an implicit coercion.  Hence the PEP 
> actually already prohibits implicit comparisons, as well as implicit 
> arithmetic.

Not necessarily -- you could compare them as though they
had both been converted to the equivalent rational numbers,
which can be done without loss of precision.

-- 
Greg

From v+python at g.nevcal.com  Sat Mar 20 03:20:54 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 19:20:54 -0700
Subject: [Python-Dev] containment checking
In-Reply-To: <4BA414E9.2000105@voidspace.org.uk>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>	<5c6f2a5d1003191250sd61106fma0db83880fda0b1e@mail.gmail.com>	<4BA3D90C.5060009@g.nevcal.com>	<loom.20100319T224859-46@post.gmane.org>	<4BA40DB9.3020302@g.nevcal.com>	<loom.20100320T005523-673@post.gmane.org>
	<4BA4138A.9090003@nevcal.com> <4BA41441.8030404@voidspace.org.uk>
	<4BA41490.1000803@g.nevcal.com> <4BA414E9.2000105@voidspace.org.uk>
Message-ID: <4BA43106.3000600@g.nevcal.com>

On 3/19/2010 5:20 PM, Michael Foord wrote:
> On 20/03/2010 00:19, Glenn Linderman wrote:
>> On 3/19/2010 5:18 PM, Michael Foord wrote:
>>>>>> will probably get you buggy results, somehow or another. That's what
>>>>>> > design, code reviews, and testing are for.
>>>>> We'll have to "agree to disagree" then. If you want error 
>>>>> silencing by default,
>>>>> Python is not the language you are looking for.
>>>>
>>>> We can agree to disagree, if you like. But taken to the limit, the 
>>>> Zen you quoted would prevent the try except clause from being used.
>>>
>>> No, that is what "unless explicitly silenced" means - you are 
>>> proposing to silence them *without* an explicit try except clause.
>>>
>>> Michael 
>>
>> Who, me?  The containment checking code would contain the try/except, 
>> I was proposing.
>
> Explicit by the programmer. That is what explicit means... Caught and 
> silenced for you by Python is implicit.

Should I really believe that there are no try/except clauses in the 
Python source code (nor their C equivalent, if (errno == blahblahblah) 
... )?

I mean, I haven't read very much of the source code... but that 
statement makes me want to download and grep...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100319/adb13467/attachment.html>

From steve at pearwood.info  Sat Mar 20 03:40:47 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 20 Mar 2010 13:40:47 +1100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
Message-ID: <201003201340.47883.steve@pearwood.info>

On Sat, 20 Mar 2010 08:50:04 am Guido van Rossum wrote:
> I'd like to reboot this thread. I've been spinning this topic in my
> head for most of the morning, and I think we should seriously
> reconsider allowing mixed arithmetic involving Decimal, not just
> mixed comparisons. [Quick summary: embed Decimal in the numeric tower
> but add a context flag to disallow implicit mixing of float and
> Decimal.]

The last few days, I've argued against changing the prohibition against 
mixed arithmetic operations. But you've inspired me to go take a look 
at a module I've been neglecting, fractions, and I've learned that the 
Fraction type already fully supports arithmetic and comparisons with 
floats and ints. I'm extremely impressed -- I had no idea the numeric 
tower in 2.6 was this advanced. (I still do most of my work with 2.5.)

Decimal appears to be the odd one:

>>> f = Fraction(0)
>>> d = Decimal(0)
>>> 0 == 0.0 == 0j == f
True
>>> 0 == 0.0 == 0j == f == d
False

Not just odd in the sense of "different", but also odd in the sense 
of "weird":

>>> d == 0 == 0.0 == 0j == f
True


[...]
> I'd like to look at the issue by comparing the benefits and drawbacks
> of properly embedding Decimal into the numeric tower. As advantages,
> I see consistent behavior in situations like the above and more
> intuitive behavior for beginners. Also, this would be a possible road
> towards eventually supporting a language extension where floating
> point literals produce Decimal values instead of binary floats. (A
> possible syntax could be "from __options__ import decimal_float",
> which would work similar to "from __future__ import ..." except it's
> a permanent part of the language rather than a forward compatibility
> feature.)

That's far more ambitious than I was willing to even imagine, but now 
that you've suggested it, I like it.



-- 
Steven D'Aprano

From jafo at tummy.com  Sat Mar 20 03:40:18 2010
From: jafo at tummy.com (Sean Reifschneider)
Date: Fri, 19 Mar 2010 20:40:18 -0600
Subject: [Python-Dev] Needs Review: 2to3 fixer for sys.exitfunc to atexit.
Message-ID: <20100320024018.GA7310@tummy.com>

Issue 2356 ( http://bugs.python.org/issue2356 ) looks like it has a patch
candidate, but just needs review to put in place.  Seems like it may be
low hanging fruit critical that just needs a quick review to either apply
or at least get back to the folks who are working on a patch.

Thanks,
Sean
-- 
 The most important thing in communication is to hear what isn't being said.
                 -- Peter Drucker
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability


From benjamin at python.org  Sat Mar 20 03:45:44 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 19 Mar 2010 21:45:44 -0500
Subject: [Python-Dev] Needs Review: 2to3 fixer for sys.exitfunc to
	atexit.
In-Reply-To: <20100320024018.GA7310@tummy.com>
References: <20100320024018.GA7310@tummy.com>
Message-ID: <1afaf6161003191945w20bfaeabvb003c67e51d7cbb5@mail.gmail.com>

2010/3/19 Sean Reifschneider <jafo at tummy.com>:
> Issue 2356 ( http://bugs.python.org/issue2356 ) looks like it has a patch
> candidate, but just needs review to put in place. ?Seems like it may be
> low hanging fruit critical that just needs a quick review to either apply
> or at least get back to the folks who are working on a patch.

"critical"? I'm not sure that a new 2to3 fixer or py3k warning is a
critical issue.



-- 
Regards,
Benjamin

From ctb at msu.edu  Sat Mar 20 04:13:26 2010
From: ctb at msu.edu (C. Titus Brown)
Date: Fri, 19 Mar 2010 20:13:26 -0700
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <4BA40220.30701@v.loewis.de>
References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com>
	<bad82a81003190457w5c0e1770hd7c0f4af3f0a8746@mail.gmail.com>
	<4BA388E5.1030703@gmail.com> <4BA40220.30701@v.loewis.de>
Message-ID: <20100320031326.GM1623@idyll.org>

On Sat, Mar 20, 2010 at 12:00:48AM +0100, "Martin v. L?wis" wrote:
> > Whether this is worth weeks of work or not will depend on a given
> > student's knowledge about Python 3, and I'd suspect that the GSoC would
> > be an opportunity for a number of applicant to actually learn the
> > intricacies of Python 3.
> > Developing Python 3-specific features could be used to increase the
> > amount of work on the project, but I am uncertain of whether this is
> > worth a full GSoC.
> 
> I'd say this would definitely make a GSoC project; any non-trivial
> porting will be.

Sounds good to me.  Line up a good student and bob's your uncle.

--titus
-- 
C. Titus Brown, ctb at msu.edu

From alexander.belopolsky at gmail.com  Sat Mar 20 04:20:52 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 19 Mar 2010 23:20:52 -0400
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <20100319194635.3DE491FDAE7@kimball.webabinitio.net>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
	<20100319142205.E11FC1FDA39@kimball.webabinitio.net>
	<20100319163958.GB8716@amk-desktop.matrixgroup.net>
	<20100319194635.3DE491FDAE7@kimball.webabinitio.net>
Message-ID: <d38f5331003192020u28494633o172ef1d9e9b9e4f@mail.gmail.com>

I am generally happy with the tracker workflow and in my experience,
issues that are lingering are lingering for a good reason.

I've decided to add my two cents to this thread because I've just had
a negative experience with how issue 8154 and my input to it were
handled.

I will not go into details here beyond referring to
http://bugs.python.org/issue8154, but if you follow the link, you'll
see that there was not a consensus on how the issue should be
addressed and even whether or not it was a bug.  Nevertheless the
patch was committed both to the trunk and to 2.6 without any answer to
my concerns and without  even an rNNN link to the committed revision.

I think it would be nice if committers would not cut the discussion
short without at least a note explaining their decision.

From ncoghlan at gmail.com  Sat Mar 20 05:20:49 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 20 Mar 2010 14:20:49 +1000
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA3CD93.8080502@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>
	<4BA3CD93.8080502@g.nevcal.com>
Message-ID: <4BA44D21.1070401@gmail.com>

Glenn Linderman wrote:
> The same person that would expect both
> 
> 0 == "0"
> 0.0 == "0.0"
> 
> to be False... i.e. anyone that hasn't coded in Perl for too many years.

Completely different - that is comparing numbers to strings. What this
discussion is about is comparison between the different kinds of real
number in the interpreter core and standard library (ints, floats,
decimal.Decimal, fractions.Fraction).

For arithmetic, decimal.Decimal deliberately fails to interoperate with
binary floating point numbers (hence why it isn't registered as an
example of the numbers.Real ABC).

For comparison, it only sort of fails to interoperate - it gives either
an exception or silently returns the wrong answer, depending on Python
version and the exact operator involved.

Fractions are nowhere near as purist about things - they will happily
allow themselves to be implicitly converted to floating point values.

So currently (for arithmetic):

int op int -> int (or potentially float due to division)
int op float, float op int -> float
int op Fraction, Fraction op int -> Fraction
int op Decimal, Decimal op int -> Decimal

Fraction op Fraction -> Fraction
Fraction op float, float op Fraction -> float
Fraction op Decimal, Decimal op Fraction -> TypeError

Decimal op Decimal -> Decimal
Decimal op float, float op Decimal -> TypeError

float op float -> float

Nobody is arguing in this thread for any changes to that. I just want to
contrast it with the situation for the comparison operators:

int op int -> bool
int op float, float op int -> bool
int op Fraction, Fraction op int -> bool
int op Decimal, Decimal op int -> bool

Fraction op Fraction -> bool
Fraction op float, float op Fraction -> bool
Fraction op Decimal, Decimal op Fraction -> TypeError

Decimal op Decimal -> bool
Decimal op float, float op Decimal -> TypeError

float op float -> bool

In the case of floats and Decimals, there's no ambiguity here that
creates any temptation to guess - to determine a true/false result for a
comparison, floats can be converted explicitly to Decimals without any
loss of accuracy. For Fractions, the precedent has already been set by
allowing implicit (potentially lossy) conversion to binary floats - a
lossy conversion to Decimal wouldn't be any worse.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From v+python at g.nevcal.com  Sat Mar 20 07:14:24 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Fri, 19 Mar 2010 23:14:24 -0700
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA44D21.1070401@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>
	<4BA3CD93.8080502@g.nevcal.com> <4BA44D21.1070401@gmail.com>
Message-ID: <4BA467C0.1000300@g.nevcal.com>

On 3/19/2010 9:20 PM, Nick Coghlan wrote:
> Glenn Linderman wrote:
>    
>> >  The same person that would expect both
>> >  
>> >  0 == "0"
>> >  0.0 == "0.0"
>> >  
>> >  to be False... i.e. anyone that hasn't coded in Perl for too many years.
>>      
> Completely different - that is comparing numbers to strings.

One can argue that either way, that it is completely different, or 
completely the same.  Is the map the territory, or is the territory the 
map?  The human representation of the numbers in string form is 
meaningful numerically, and there are explicit conversions between them 
that prove that it is so.

But it is completely different, because Python doesn't do implicit 
coercion of strings to numbers, so they don't belong in the tree.

But it is completely the same, because Python doesn't do implicit 
coercion of Decimal to numbers, so they don't belong in the tree.

Completely different, because Python also doesn't do numeric operations 
on strings either... so the analogy only goes so far, admittedly.  Even 
Perl, which will implicitly coerce string to numbers, has very limited 
numeric operations that are performed on the string form.

Thanks, though for the nice chart from an internals guy, you confirmed 
all that I thought I knew about how this presently works, from PEP and 
manual reading, a bit of experimentation, and the discussions here.

I have no problem with Guido's latest proposal in the rebooted thread, 
as long as there is a way to wall off the Decimal type from creeping 
errors due to implicit conversions and inaccurate types.  It is 
presently an excellent feature of Python to have a way to be sure that 
conversions are explicit to retain proper accuracy.   Few languages have 
that without requiring the user to write the whole class himself (or 
beg, borrow, or buy it).  As he implicitly points out, while declaring 
that mixed arithmetic might be appropriate, that it either needs to be 
done correctly and completely, or not at all.

Glenn

From dickinsm at gmail.com  Sat Mar 20 12:06:30 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 20 Mar 2010 11:06:30 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
	<99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com>
Message-ID: <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com>

On Fri, Mar 19, 2010 at 1:17 PM, Case Vanhorsen <casevh at gmail.com> wrote:
> On Fri, Mar 19, 2010 at 3:07 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>> On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>>> Making hashes of int,
>>> float, Decimal *and* Fraction all compatible with one another,
>>> efficient for ints and floats, and not grossly inefficient for
>>> Fractions and Decimals, is kinda hairy, though I have a couple of
>>> ideas of how it could be done.
>>
>> To elaborate, here's a cute scheme for the above, which is actually
>> remarkably close to what Python already does for ints and floats, and
>> which doesn't require any of the numeric types to figure out whether
>> it's exactly equal to an instance of some other numeric type.

> Will this change the result of hashing a long? I know that both gmpy
> and SAGE use their own hash implementations for performance reasons. I
> understand that CPython's hash function is an implementation detail,
> but there are external modules that rely on the existing hash
> behavior.

Yes, it would change the hash of a long.

What external modules are there that rely on existing hash behaviour?
And exactly what behaviour do they rely on?

Mark

From dickinsm at gmail.com  Sat Mar 20 12:16:39 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 20 Mar 2010 11:16:39 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <4BA41271.3010003@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
	<4BA3680D.8070909@gmail.com> <4BA41271.3010003@canterbury.ac.nz>
Message-ID: <5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com>

On Sat, Mar 20, 2010 at 12:10 AM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>>
>> Mark Dickinson wrote:
>
>> It seems to me that given the existing conflation of numeric equivalence
>> and containment testing, going the whole hog and fixing the set
>> membership problem for all of our rational types would be the right
>> thing to do.
>
> Isn't this only solving half the problem, though?

Yes.

> Even if
> you find that the hashes are equal, you still have to decide
> whether the values themselves are equal.

True.  The reason I was concentrating on the hashes is that it's not
immediately obvious that it's even *possible* to find a decent hash
function that's efficient to compute and gives equal results for
numerically equal inputs (regardless of type); this is especially true
if you don't want to significantly slow down the existing hashes for
int and float.  But once that problem is solved, it shouldn't be too
hard to implement all the comparisons.

It *is* kinda messy, because as far as I can see the oddities of the
various types mean that you end up producing specialized code for
comparing each pair of types (one block of code for float<->Fraction
comparisons, another for float<->Decimal, yet another for
Decimal<->Fraction, and so on), but it's doable.

Mark

From p.f.moore at gmail.com  Sat Mar 20 12:41:36 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 20 Mar 2010 11:41:36 +0000
Subject: [Python-Dev] Fraction arithmetic (Was: Decimal ... float
	comparisons in py3k)
Message-ID: <79990c6b1003200441k69f1899fsfd1e8b40871a95cc@mail.gmail.com>

On 20 March 2010 04:20, Nick Coghlan <ncoghlan at gmail.com> wrote:
> In the case of floats and Decimals, there's no ambiguity here that
> creates any temptation to guess - to determine a true/false result for a
> comparison, floats can be converted explicitly to Decimals without any
> loss of accuracy. For Fractions, the precedent has already been set by
> allowing implicit (potentially lossy) conversion to binary floats - a
> lossy conversion to Decimal wouldn't be any worse.

Hmm, given that a float can be converted losslessly to a fraction, why
was the decision taken to convert the fraction to a float rather than
the other way round?

I don't see a PEP for the fractions module, and my google-fu has
failed to find anything. Was there a discussion on this?

Paul.

From dickinsm at gmail.com  Sat Mar 20 12:58:38 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 20 Mar 2010 11:58:38 +0000
Subject: [Python-Dev] Fraction arithmetic (Was: Decimal ... float
	comparisons in py3k)
In-Reply-To: <79990c6b1003200441k69f1899fsfd1e8b40871a95cc@mail.gmail.com>
References: <79990c6b1003200441k69f1899fsfd1e8b40871a95cc@mail.gmail.com>
Message-ID: <5c6f2a5d1003200458q64c033b7u796c12b19eb04d7e@mail.gmail.com>

On Sat, Mar 20, 2010 at 11:41 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 20 March 2010 04:20, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> In the case of floats and Decimals, there's no ambiguity here that
>> creates any temptation to guess - to determine a true/false result for a
>> comparison, floats can be converted explicitly to Decimals without any
>> loss of accuracy. For Fractions, the precedent has already been set by
>> allowing implicit (potentially lossy) conversion to binary floats - a
>> lossy conversion to Decimal wouldn't be any worse.
>
> Hmm, given that a float can be converted losslessly to a fraction, why
> was the decision taken to convert the fraction to a float rather than
> the other way round?

I'm not sure of the actual reason for this decision, but one argument
I've seen used for other languages is that it's desirable for the
inexactness of the float type to be contagious:  rationals are
perceived as exact, while floats are perceived as approximations.

Note that this only applies to arithmetic operations:  for
comparisons, an exact conversion *is* performed.  This is much like
what currently happens with ints and floats in the core:  a mixed-type
arithmetic operation between an int and a float first converts the int
to a float (possibly changing the value in the process).  A mixed-type
comparison makes an exact comparison without doing such a conversion.
For example (in any non-ancient version of Python):

>>> n = 2**53 + 1
>>> x = 2.**53
>>> n > x   # compares exact values;  no conversion performed
True
>>> n - x    # converts n to a float before subtracting
0.0

> I don't see a PEP for the fractions module, and my google-fu has
> failed to find anything. Was there a discussion on this?

There's PEP 3141 (http://www.python.org/dev/peps/pep-3141/), which was
the motivation for adding the fractions module in the first place, and
there's the issue tracker item for the fractions module
(http://bugs.python.org/issue1682).

Mark

From chambon.pascal at wanadoo.fr  Sat Mar 20 13:00:30 2010
From: chambon.pascal at wanadoo.fr (Pascal Chambon)
Date: Sat, 20 Mar 2010 13:00:30 +0100
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA3DD4F.1080209@voidspace.org.uk>
References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk>
Message-ID: <4BA4B8DE.8060109@wanadoo.fr>

Michael Foord a ?crit :
>
> On 19/03/2010 18:58, Pascal Chambon wrote:
>> Hello
>>
>> I've already crossed a bunch of articles detailing python's attribute 
>> lookup semantic (__dict__, descriptors, order of base class 
>> traversing...), but I have never seen, so far, an explanation of 
>> WHICH method did waht, exactly.
>>
>> I assumed that getattr(a, b) was the same as a.__getattribute__(b), 
>> and that this __getattribute__ method (or the hidden routine 
>> replacing it when we don't override it in our class) was in charge of 
>> doing the whole job of traversing the object tree, checking 
>> descriptors, binding methods, calling __getattr__ on failure etc.
>>
>> However, the test case below shows that __getattribute__ does NOT 
>> call __getattr__ on failure. So it seems it's an upper levl 
>> machinery, in getattr(), which is in chrge of that last action.
>
> Python 3 has the behavior you are asking for. It would be a backwards 
> incompatible change to do it in Python 2 as __getattribute__ *not* 
> calling __getattr__ is the documented behaviour.
>
> Python 3.2a0 (py3k:78770, Mar 7 2010, 20:32:50)
> [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
> >>> class x:
> ... def __getattribute__(s, name):
> ... print ('__getattribute__', name)
> ... raise AttributeError
> ... def __getattr__(s, name):
> ... print ('__getattr__', name)
> ...
> >>> a = x()
> >>> a.b
> __getattribute__ b
> __getattr__ b
I'm confused there, because the script you gave behaves the same in 
python 2.6. And according to the doc, it's normal, getattr() reacts to 
an AttributeError from __getattribute__, by calling __getattr__ :

"""
Python 2.6.5 documentation

object.__getattribute__(/self/, /name/)

    Called unconditionally to implement attribute accesses for instances
    of the class. If the class also defines __getattr__()
    <http://docs.python.org/reference/datamodel.html#object.__getattr__>,
    the latter will not be called unless __getattribute__()
    <http://docs.python.org/reference/datamodel.html#object.__getattribute__>
    either calls it explicitly or raises an AttributeError
    <http://docs.python.org/library/exceptions.html#exceptions.AttributeError>.
    This method should return the (computed) attribute value or raise an
    AttributeError
    <http://docs.python.org/library/exceptions.html#exceptions.AttributeError>
    exception. In order to avoid infinite recursion in this method, its
    implementation should always call the base class method with the
    same name to access any attributes it needs, for example,
    object.__getattribute__(self, name).

"""

But the point which for me is still unclear, is : does the default 
implementation of __getattribute__ (the one of "object") call 
__getattr__ by himself, or does it rely on its caller for that, by 
raising an AttributeError ? For Python2, it's blatantly the latter case 
which is favoured, but since it looks like an implementation detail at 
the moment, I propose we settle it (and document it) once for all.

>
>
> This list is not really an appropriate place to ask questions like 
> this though, comp.lang.python would be better.
>
> All the best,
>
> Michael Fooord
Sorry if I misposted, I just (wrongly ?) assumed that it was more an 
undecided, implementation-specific point (since the doc gave possible 
behaviours for __getattribute__, without precising which one was the 
default one), and thus targetted the hands-in-core-code audience only.

Regards,
Pascal


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100320/851651d7/attachment-0001.html>

From solipsis at pitrou.net  Sat Mar 20 16:11:12 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 20 Mar 2010 15:11:12 +0000 (UTC)
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
Message-ID: <loom.20100320T160613-470@post.gmane.org>

Mark Dickinson <dickinsm <at> gmail.com> writes:
> 
> On Fri, Mar 19, 2010 at 9:50 PM, Guido van Rossum <guido <at> python.org>
wrote:
> > There is one choice which I'm not sure about. Should a mixed
> > float/Decimal operation return a float or a Decimal?
> 
> I'll just say that it's much easier to return a Decimal if you want to
> be able to make guarantees about rounding behaviour, basically because
> floats can be converted losslessly to Decimals.  I also like the fact
> that the decimal module offers more control (rounding mode, precision,
> flags, wider exponent range) than float.

A problem, though, is that decimals are much slower than floats. If you have a
decimal creeping in some part of a calculation it could degrade performance
quite a bit.

Regards

Antoine.



From casevh at gmail.com  Sat Mar 20 16:17:56 2010
From: casevh at gmail.com (Case Vanhorsen)
Date: Sat, 20 Mar 2010 08:17:56 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
	<99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com>
	<5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com>
Message-ID: <99e0b9531003200817j6c47c915rf0d77f2be75d306f@mail.gmail.com>

On Sat, Mar 20, 2010 at 4:06 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Fri, Mar 19, 2010 at 1:17 PM, Case Vanhorsen <casevh at gmail.com> wrote:
>> On Fri, Mar 19, 2010 at 3:07 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>>> On Fri, Mar 19, 2010 at 9:37 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>>>> Making hashes of int,
>>>> float, Decimal *and* Fraction all compatible with one another,
>>>> efficient for ints and floats, and not grossly inefficient for
>>>> Fractions and Decimals, is kinda hairy, though I have a couple of
>>>> ideas of how it could be done.
>>>
>>> To elaborate, here's a cute scheme for the above, which is actually
>>> remarkably close to what Python already does for ints and floats, and
>>> which doesn't require any of the numeric types to figure out whether
>>> it's exactly equal to an instance of some other numeric type.
>
>> Will this change the result of hashing a long? I know that both gmpy
>> and SAGE use their own hash implementations for performance reasons. I
>> understand that CPython's hash function is an implementation detail,
>> but there are external modules that rely on the existing hash
>> behavior.
>
> Yes, it would change the hash of a long.
>
> What external modules are there that rely on existing hash behaviour?

I'm only aware of  gmpy and SAGE.

> And exactly what behaviour do they rely on?

Instead of calculating hash(long(mpz)), they calculate hash(mpz)
directly. It avoids creation of a temporary object that could be quite
large and is faster than the two-step process. I would need to modify
the code so that it continues to produce the same result.

casevh
> Mark
>

From rdmurray at bitdance.com  Sat Mar 20 16:30:08 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 20 Mar 2010 11:30:08 -0400
Subject: [Python-Dev] Tracker reviews workflow and flags
In-Reply-To: <d38f5331003192020u28494633o172ef1d9e9b9e4f@mail.gmail.com>
References: <d34314101003190109r8ac59edm8e037fbc42b6a885@mail.gmail.com>
	<20100319142205.E11FC1FDA39@kimball.webabinitio.net>
	<20100319163958.GB8716@amk-desktop.matrixgroup.net>
	<20100319194635.3DE491FDAE7@kimball.webabinitio.net>
	<d38f5331003192020u28494633o172ef1d9e9b9e4f@mail.gmail.com>
Message-ID: <20100320153009.0F0C01FD9F8@kimball.webabinitio.net>

On Fri, 19 Mar 2010 23:20:52 -0400, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:
> I will not go into details here beyond referring to
> http://bugs.python.org/issue8154, but if you follow the link, you'll
> see that there was not a consensus on how the issue should be
> addressed and even whether or not it was a bug.  Nevertheless the
> patch was committed both to the trunk and to 2.6 without any answer to
> my concerns and without  even an rNNN link to the committed revision.
> 
> I think it would be nice if committers would not cut the discussion
> short without at least a note explaining their decision.

I think the reason this happened in this bug is that it *appeared*
as though the same problem had been encountered and already fixed in
python3, and all that was needed was a backport.  We were groping our
way toward deciding that that wasn't actually the case when Matthias
committed the patch.  Possibly my fault, since I said that I agreed that
it should be fixed.  As I've now noted on the tracker (and in the Ubuntu
tracker just for good measure), I don't think that commit was
appropriate, at least for 2.6, and probably not for 2.7 either.

I also hope that Matthias will put in commit numbers in the future,
since it is very helpful.  Even better, I hope that we can automate this
after the switch to Mercurial (but someone will need to write the code
to do it, of course...)

--David

From tjreedy at udel.edu  Sat Mar 20 17:29:10 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 20 Mar 2010 12:29:10 -0400
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>	<99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com>
	<5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com>
Message-ID: <ho2t4l$oar$1@dough.gmane.org>

On 3/20/2010 7:06 AM, Mark Dickinson wrote:

>> Will this change the result of hashing a long? I know that both gmpy
>> and SAGE use their own hash implementations for performance reasons. I
>> understand that CPython's hash function is an implementation detail,
>> but there are external modules that rely on the existing hash
>> behavior.
>
> Yes, it would change the hash of a long.
>
> What external modules are there that rely on existing hash behaviour?
> And exactly what behaviour do they rely on?

Depending on specifics of CPython's hash is a 'do at your own risk' 
enterprise, like CPython bytecode hacks.


From solipsis at pitrou.net  Sat Mar 20 17:45:04 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 20 Mar 2010 16:45:04 +0000 (UTC)
Subject: [Python-Dev] Bug? syslog.openlog using ident &quot; python&quot;
	by default.
References: <20100318115844.GA7374@tummy.com>
Message-ID: <loom.20100320T173557-128@post.gmane.org>

Sean Reifschneider <jafo <at> tummy.com> writes:
> 
> I would propose changing the Python syslog() call to do the C equivalent of:
> 
>    if openlog_hasnt_been_called_before_now:
>       if sys.argv[0]:
>          syslog.openlog(os.path.basename(sys.argv[0]))
> 
> In other words, if there's a script name and openlog hasn't already been
> called, call openlog with the basename of the script name.

This looks fine to me. The compatibility breakage doesn't look too serious,
given that "python" is an inappropriate name and therefore people would have
changed it anyway.

By the way, why does logging have its own syslog implementation?

Regards

Antoine.



From tjreedy at udel.edu  Sat Mar 20 18:02:02 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 20 Mar 2010 13:02:02 -0400
Subject: [Python-Dev] Tracker opened/closed issue list: missing?
Message-ID: <ho2v29$us2$1@dough.gmane.org>

I did not receive the usual Friday tracker post on issues opened and 
closed during the past week.
Did anyone else?
I am reading via gmane.

Terry Jan Reedy


From rhamph at gmail.com  Sat Mar 20 18:04:15 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Sat, 20 Mar 2010 11:04:15 -0600
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <loom.20100320T160613-470@post.gmane.org>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
Message-ID: <aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>

On Sat, Mar 20, 2010 at 09:11, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Mark Dickinson <dickinsm <at> gmail.com> writes:
>>
>> On Fri, Mar 19, 2010 at 9:50 PM, Guido van Rossum <guido <at> python.org>
> wrote:
>> > There is one choice which I'm not sure about. Should a mixed
>> > float/Decimal operation return a float or a Decimal?
>>
>> I'll just say that it's much easier to return a Decimal if you want to
>> be able to make guarantees about rounding behaviour, basically because
>> floats can be converted losslessly to Decimals. ?I also like the fact
>> that the decimal module offers more control (rounding mode, precision,
>> flags, wider exponent range) than float.
>
> A problem, though, is that decimals are much slower than floats. If you have a
> decimal creeping in some part of a calculation it could degrade performance
> quite a bit.

For a little context, we have this numeric tower:

int -> Fraction -> float -> complex

And coincidentally (or not), we have an unspoken rule that you can go
right, but never left (int/int -> float, int/Fraction -> Fraction,
Fraction/float -> float, Fraction/complex -> complex, etc).  This
gives us a preference for fast, inexact results.

Decimal is more precise, and pays a performance cost for it.  It also
seems odd to stick it between float and complex (nobody's planning a
ComplexDecimal, right?)  That suggests it should go between Fraction
and float.  Decimal/float -> float.


-- 
Adam Olsen, aka Rhamphoryncus

From dickinsm at gmail.com  Sat Mar 20 18:05:56 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 20 Mar 2010 17:05:56 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <99e0b9531003200817j6c47c915rf0d77f2be75d306f@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
	<99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com>
	<5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com>
	<99e0b9531003200817j6c47c915rf0d77f2be75d306f@mail.gmail.com>
Message-ID: <5c6f2a5d1003201005n363b8b4w154ce799312f307f@mail.gmail.com>

On Sat, Mar 20, 2010 at 3:17 PM, Case Vanhorsen <casevh at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 4:06 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>> What external modules are there that rely on existing hash behaviour?
>
> I'm only aware of ?gmpy and SAGE.
>
>> And exactly what behaviour do they rely on?
>
> Instead of calculating hash(long(mpz)), they calculate hash(mpz)
> directly. It avoids creation of a temporary object that could be quite
> large and is faster than the two-step process. I would need to modify
> the code so that it continues to produce the same result.

Does gmpy only do this for Python 2.6?  Or does it use different
algorithms for 2.4/2.5 and 2.6?  As far as I can tell, there was no
reasonable way to compute long_hash directly at all before the
algorithm was changed for 2.6, unless you imitate exactly what Python
was doing (break up into 15-bit pieces, and do all the rotation and
addition exactly the same way), in which case you might as well be
calling long_hash directly.

Mark

From brian.curtin at gmail.com  Sat Mar 20 18:11:17 2010
From: brian.curtin at gmail.com (Brian Curtin)
Date: Sat, 20 Mar 2010 11:11:17 -0600
Subject: [Python-Dev] Tracker opened/closed issue list: missing?
In-Reply-To: <ho2v29$us2$1@dough.gmane.org>
References: <ho2v29$us2$1@dough.gmane.org>
Message-ID: <cf9f31f21003201011n612b1d5fg41d8c049fe46a2d5@mail.gmail.com>

On Sat, Mar 20, 2010 at 11:02, Terry Reedy <tjreedy at udel.edu> wrote:

> I did not receive the usual Friday tracker post on issues opened and closed
> during the past week.
> Did anyone else?
> I am reading via gmane.
>
> Terry Jan Reedy
>

I actually haven't seen that mail in a few weeks (subscribed to the list via
my gmail account).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100320/d7ce67ef/attachment.html>

From casevh at gmail.com  Sat Mar 20 18:47:40 2010
From: casevh at gmail.com (Case Vanhorsen)
Date: Sat, 20 Mar 2010 10:47:40 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003201005n363b8b4w154ce799312f307f@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
	<99e0b9531003190617w28c5da2etf8b35c54f223a036@mail.gmail.com>
	<5c6f2a5d1003200406o7b7c8a6am839925e0f07d0222@mail.gmail.com>
	<99e0b9531003200817j6c47c915rf0d77f2be75d306f@mail.gmail.com>
	<5c6f2a5d1003201005n363b8b4w154ce799312f307f@mail.gmail.com>
Message-ID: <99e0b9531003201047n31d853f2mc85de92e17655720@mail.gmail.com>

On Sat, Mar 20, 2010 at 10:05 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 3:17 PM, Case Vanhorsen <casevh at gmail.com> wrote:
>> On Sat, Mar 20, 2010 at 4:06 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>>> What external modules are there that rely on existing hash behaviour?
>>
>> I'm only aware of ?gmpy and SAGE.
>>
>>> And exactly what behaviour do they rely on?
>>
>> Instead of calculating hash(long(mpz)), they calculate hash(mpz)
>> directly. It avoids creation of a temporary object that could be quite
>> large and is faster than the two-step process. I would need to modify
>> the code so that it continues to produce the same result.
>
> Does gmpy only do this for Python 2.6? ?Or does it use different
> algorithms for 2.4/2.5 and 2.6? ?As far as I can tell, there was no
> reasonable way to compute long_hash directly at all before the
> algorithm was changed for 2.6, unless you imitate exactly what Python
> was doing (break up into 15-bit pieces, and do all the rotation and
> addition exactly the same way), in which case you might as well be
> calling long_hash directly.
>
> Mark
>

It does the later: it converts from GMP's internal format to CPython's
long format and calculates the hash along the way. I may (should :) )
revert back to converting to long and then calling long_hash. The
majority of the speed increase came from the conversion improvements,
not the hash calculation.

I am in favor of any change that makes 2.7 and 3.2 behave the same.

casevh

From rdmurray at bitdance.com  Sat Mar 20 18:49:10 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 20 Mar 2010 13:49:10 -0400
Subject: [Python-Dev] Tracker opened/closed issue list: missing?
In-Reply-To: <cf9f31f21003201011n612b1d5fg41d8c049fe46a2d5@mail.gmail.com>
References: <ho2v29$us2$1@dough.gmane.org>
	<cf9f31f21003201011n612b1d5fg41d8c049fe46a2d5@mail.gmail.com>
Message-ID: <20100320174910.7E2D61FCDF1@kimball.webabinitio.net>

On Sat, 20 Mar 2010 11:11:17 -0600, Brian Curtin <brian.curtin at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 11:02, Terry Reedy <tjreedy at udel.edu> wrote:
> > I did not receive the usual Friday tracker post on issues opened and closed
> > during the past week.
> > Did anyone else?
> > I am reading via gmane.
> >
> > Terry Jan Reedy
> >
> 
> I actually haven't seen that mail in a few weeks (subscribed to the list via
> my gmail account).

Darn.  I bet I broke it when I added the 'languishing' status.  I even
thought about that at the time but didn't follow up on it.  I'll see
if I can figure out where the script lives.

--
R. David Murray                                      www.bitdance.com

From guido at python.org  Sat Mar 20 20:04:08 2010
From: guido at python.org (Guido van Rossum)
Date: Sat, 20 Mar 2010 12:04:08 -0700
Subject: [Python-Dev] Fraction arithmetic (Was: Decimal ... float
	comparisons in py3k)
In-Reply-To: <5c6f2a5d1003200458q64c033b7u796c12b19eb04d7e@mail.gmail.com>
References: <79990c6b1003200441k69f1899fsfd1e8b40871a95cc@mail.gmail.com> 
	<5c6f2a5d1003200458q64c033b7u796c12b19eb04d7e@mail.gmail.com>
Message-ID: <ca471dc21003201204m4153e4bq39c01ba7df3c1f00@mail.gmail.com>

On Sat, Mar 20, 2010 at 4:58 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 11:41 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 20 March 2010 04:20, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> In the case of floats and Decimals, there's no ambiguity here that
>>> creates any temptation to guess - to determine a true/false result for a
>>> comparison, floats can be converted explicitly to Decimals without any
>>> loss of accuracy. For Fractions, the precedent has already been set by
>>> allowing implicit (potentially lossy) conversion to binary floats - a
>>> lossy conversion to Decimal wouldn't be any worse.
>>
>> Hmm, given that a float can be converted losslessly to a fraction, why
>> was the decision taken to convert the fraction to a float rather than
>> the other way round?
>
> I'm not sure of the actual reason for this decision, but one argument
> I've seen used for other languages is that it's desirable for the
> inexactness of the float type to be contagious: ?rationals are
> perceived as exact, while floats are perceived as approximations.

You hit the nail on the head.

> Note that this only applies to arithmetic operations: ?for
> comparisons, an exact conversion *is* performed. ?This is much like
> what currently happens with ints and floats in the core: ?a mixed-type
> arithmetic operation between an int and a float first converts the int
> to a float (possibly changing the value in the process). ?A mixed-type
> comparison makes an exact comparison without doing such a conversion.
> For example (in any non-ancient version of Python):
>
>>>> n = 2**53 + 1
>>>> x = 2.**53
>>>> n > x ? # compares exact values; ?no conversion performed
> True
>>>> n - x ? ?# converts n to a float before subtracting
> 0.0
>
>> I don't see a PEP for the fractions module, and my google-fu has
>> failed to find anything. Was there a discussion on this?
>
> There's PEP 3141 (http://www.python.org/dev/peps/pep-3141/), which was
> the motivation for adding the fractions module in the first place, and
> there's the issue tracker item for the fractions module
> (http://bugs.python.org/issue1682).

-- 
--Guido van Rossum (python.org/~guido)

From martin at v.loewis.de  Sat Mar 20 20:29:50 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 20 Mar 2010 20:29:50 +0100
Subject: [Python-Dev] Tracker opened/closed issue list: missing?
In-Reply-To: <20100320174910.7E2D61FCDF1@kimball.webabinitio.net>
References: <ho2v29$us2$1@dough.gmane.org>	<cf9f31f21003201011n612b1d5fg41d8c049fe46a2d5@mail.gmail.com>
	<20100320174910.7E2D61FCDF1@kimball.webabinitio.net>
Message-ID: <4BA5222E.3020008@v.loewis.de>


> Darn.  I bet I broke it when I added the 'languishing' status.  I even
> thought about that at the time but didn't follow up on it.  I'll see
> if I can figure out where the script lives.

My guess is that the Debian upgrade killed it.

Regards,
Martin


From guido at python.org  Sat Mar 20 20:56:24 2010
From: guido at python.org (Guido van Rossum)
Date: Sat, 20 Mar 2010 12:56:24 -0700
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> 
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com> 
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com> 
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com> 
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com> 
	<4BA3680D.8070909@gmail.com> <4BA41271.3010003@canterbury.ac.nz> 
	<5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com>
Message-ID: <ca471dc21003201256s687634an9d3bf1578af6b143@mail.gmail.com>

On Sat, Mar 20, 2010 at 4:16 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> True. ?The reason I was concentrating on the hashes is that it's not
> immediately obvious that it's even *possible* to find a decent hash
> function that's efficient to compute and gives equal results for
> numerically equal inputs (regardless of type); this is especially true
> if you don't want to significantly slow down the existing hashes for
> int and float. ?But once that problem is solved, it shouldn't be too
> hard to implement all the comparisons.
>
> It *is* kinda messy, because as far as I can see the oddities of the
> various types mean that you end up producing specialized code for
> comparing each pair of types (one block of code for float<->Fraction
> comparisons, another for float<->Decimal, yet another for
> Decimal<->Fraction, and so on), but it's doable.

I propose to reduce all hashes to the hash of a normalized fraction,
which we can define as a combination of the hashes for the numerator
and the denominator. Then all we have to do is figure fairly efficient
ways to convert floats and decimals to normalized fractions (not
necessarily Fractions). I may be naive but this seems doable: for a
float, the denominator is always a power of 2 and removing factors of
2 from the denominator is easy (just right-shift until the last bit is
zero). For Decimal, the unnormalized denominator is always a power of
10, and the normalization is a bit messier, but doesn't seem
excessively so. The resulting numerator and denominator may be large
numbers, but for typical use of Decimal and float they will rarely be
excessively large, and I'm not too worried about slowing things down
when they are (everything slows down when you're using really large
integers anyway).

-- 
--Guido van Rossum (python.org/~guido)

From dickinsm at gmail.com  Sat Mar 20 21:38:46 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 20 Mar 2010 20:38:46 +0000
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <ca471dc21003201256s687634an9d3bf1578af6b143@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
	<4BA3680D.8070909@gmail.com> <4BA41271.3010003@canterbury.ac.nz>
	<5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com>
	<ca471dc21003201256s687634an9d3bf1578af6b143@mail.gmail.com>
Message-ID: <5c6f2a5d1003201338p18aebd40y2c345b6dc8b861a6@mail.gmail.com>

On Sat, Mar 20, 2010 at 7:56 PM, Guido van Rossum <guido at python.org> wrote:
> I propose to reduce all hashes to the hash of a normalized fraction,
> which we can define as a combination of the hashes for the numerator
> and the denominator. Then all we have to do is figure fairly efficient
> ways to convert floats and decimals to normalized fractions (not
> necessarily Fractions). I may be naive but this seems doable: for a
> float, the denominator is always a power of 2 and removing factors of
> 2 from the denominator is easy (just right-shift until the last bit is
> zero). For Decimal, the unnormalized denominator is always a power of
> 10, and the normalization is a bit messier, but doesn't seem
> excessively so. The resulting numerator and denominator may be large
> numbers, but for typical use of Decimal and float they will rarely be
> excessively large, and I'm not too worried about slowing things down
> when they are (everything slows down when you're using really large
> integers anyway).

I *am* worried about slowing things down for large Decimals:  if you
can't put Decimal('1e1234567') into a dict or set without waiting for
an hour for the hash computation to complete (because it's busy
computing 10**1234567), I consider that a problem.

But it's solvable!  I've just put a patch on the bug tracker:

http://bugs.python.org/issue8188

It demonstrates how hashes can be implemented efficiently and
compatibly for all numeric types, even large Decimals like the above.
It needs a little tidying up, but it works.

Mark

From brett at python.org  Sat Mar 20 21:40:19 2010
From: brett at python.org (Brett Cannon)
Date: Sat, 20 Mar 2010 13:40:19 -0700
Subject: [Python-Dev] Silencing DeprecationWarning by default in Python
	2.7 is silencing division warnings
In-Reply-To: <bbaeab101003061143s4095b346h1e973db3fe78d6ee@mail.gmail.com>
References: <bbaeab101003061143s4095b346h1e973db3fe78d6ee@mail.gmail.com>
Message-ID: <bbaeab101003201340u3e974benf572e0e24b5d696@mail.gmail.com>

On Sat, Mar 6, 2010 at 12:43, Brett Cannon <brett at python.org> wrote:
> I see two possible fixes for this. One is to not silence DeprecationWarning
> if Py_DivisionWarningFlag is set to >= 1. The other is to introduce a new
> subclass of DeprecationWarning called IntegerDivisionWarning and have that
> added to the warnings filter so that if it is triggered it is handled
> separately from what DeprecationWarning triggers.
> The former means that you might get more than you bargained for in terms of
> warnings as you are suddenly switching on all DeprecationWarnings on top of
> division warnings. The latter means that you now have to explicit care about
> IntegerDivisionWarning on top of DeprecationWarning (to minimize this I
> could have IntegerDivisionWarning added to the warnings filter only in the
> case of when Py_DivisionWarningFlag is set instead of blindly adding it).
> Thoughts?
> -Brett

Just to follow up on this with my current thinking, I am leaning
towards having Py_DivisionWarningFlag disable the suppression of
DeprecationWarning. Since the silencing is new in 2.7, people using -Q
will not see any different semantics than they were with Python 2.6.
Plus if they are bothering to be notified about division warnings they
probably want to know about other warnings as well.

From greg.ewing at canterbury.ac.nz  Sat Mar 20 23:43:01 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 21 Mar 2010 10:43:01 +1200
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA467C0.1000300@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<ho0gjk$dd4$1@dough.gmane.org>
	<4BA3CD93.8080502@g.nevcal.com> <4BA44D21.1070401@gmail.com>
	<4BA467C0.1000300@g.nevcal.com>
Message-ID: <4BA54F75.4050004@canterbury.ac.nz>

Glenn Linderman wrote:

> One can argue that either way, that it is completely different, or 
> completely the same.

An important difference is that there is no intermediate type
that can be compared with both ints and strings.

Another relevant difference is that numbers are just one of
many possible things that a string could be interpreted as
representing. Decimals, on the other hand, are just numbers
and nothing else.

(Even strings which happen to look like numbers might not be --
e.g. telephone "numbers", which are better thought of as strings
with a limited character set... despite some misguided souls
occasionally storing them in floating point. :-)

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun Mar 21 00:20:56 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 21 Mar 2010 11:20:56 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
Message-ID: <4BA55858.2000800@canterbury.ac.nz>

Adam Olsen wrote:

> For a little context, we have this numeric tower:
> 
> int -> Fraction -> float -> complex
> 
> Decimal is more precise, and pays a performance cost for it.  It also
> seems odd to stick it between float and complex (nobody's planning a
> ComplexDecimal, right?)  That suggests it should go between Fraction
> and float.  Decimal/float -> float.

There are two ways in which that linear tower is overly
simplistic:

* It conflates the notions of exactness and width. They're
really orthogonal concepts, and to reflect this you would
need two parallel towers, with exact and inexact versions
of each type.

* Decimal and float really belong side-by-side in the
tower, rather than one above the other. Neither of them is
inherently any more precise or exact than the other.

There doesn't seem to be any good solution here. For every
use case in which Decimal+float->float appears better, there
seems to be another one for which Decimal+float->Decimal
appears better.

-- 
Greg

From jyasskin at gmail.com  Sun Mar 21 00:27:54 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sat, 20 Mar 2010 16:27:54 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA55858.2000800@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
Message-ID: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>

On Sat, Mar 20, 2010 at 4:20 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Adam Olsen wrote:
>
>> For a little context, we have this numeric tower:
>>
>> int -> Fraction -> float -> complex
>>
>> Decimal is more precise, and pays a performance cost for it. ?It also
>> seems odd to stick it between float and complex (nobody's planning a
>> ComplexDecimal, right?) ?That suggests it should go between Fraction
>> and float. ?Decimal/float -> float.
>
> There are two ways in which that linear tower is overly
> simplistic:
>
> * It conflates the notions of exactness and width. They're
> really orthogonal concepts, and to reflect this you would
> need two parallel towers, with exact and inexact versions
> of each type.

It's representing the mathematical concepts
  Integral -> Rational -> Real -> Complex

When designing it, I tried to include a notion of exact/inexact types,
but we couldn't find anything practical to do with them, so we took
them out. It's reasonably easy to design inexact Integral and Rational
types, but pretty hard to design a useful, exact Real type (things
like '==' get uncomputable quickly), so we probably couldn't actually
implement two whole parallel towers.

> * Decimal and float really belong side-by-side in the
> tower, rather than one above the other.

Yep.

From dickinsm at gmail.com  Sun Mar 21 00:46:09 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 20 Mar 2010 23:46:09 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA55858.2000800@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
Message-ID: <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com>

On Sat, Mar 20, 2010 at 11:20 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> * Decimal and float really belong side-by-side in the
> tower, rather than one above the other. Neither of them is
> inherently any more precise or exact than the other.

Except that float is fixed-width (typically 53 bits of precision),
while Decimal allows a user-specified, arbitrarily large, precision;
so in that sense the two floating-point types aren't on an equal
footing.

Mark

From guido at python.org  Sun Mar 21 00:52:42 2010
From: guido at python.org (Guido van Rossum)
Date: Sat, 20 Mar 2010 16:52:42 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com> 
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com> 
	<4BA55858.2000800@canterbury.ac.nz>
	<5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com>
Message-ID: <ca471dc21003201652o6a6649a3v68cbc415334838f5@mail.gmail.com>

On Sat, Mar 20, 2010 at 4:46 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 11:20 PM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
>> * Decimal and float really belong side-by-side in the
>> tower, rather than one above the other. Neither of them is
>> inherently any more precise or exact than the other.
>
> Except that float is fixed-width (typically 53 bits of precision),
> while Decimal allows a user-specified, arbitrarily large, precision;
> so in that sense the two floating-point types aren't on an equal
> footing.

But this doesn't really help deciding which should be positioned at
the end does it? Both Fraction and Decimal can represent every float
value (barring NaN/Inf). I wonder if we need to look at use cases or
see what other languages do for guidance.

-- 
--Guido van Rossum (python.org/~guido)

From rhamph at gmail.com  Sun Mar 21 00:58:28 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Sat, 20 Mar 2010 17:58:28 -0600
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA55858.2000800@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
Message-ID: <aac2c7cb1003201658k3f0d17d9k16ca5f58aa735ee4@mail.gmail.com>

On Sat, Mar 20, 2010 at 17:20, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> There are two ways in which that linear tower is overly
> simplistic:
>
> * It conflates the notions of exactness and width. They're
> really orthogonal concepts, and to reflect this you would
> need two parallel towers, with exact and inexact versions
> of each type.
>
> * Decimal and float really belong side-by-side in the
> tower, rather than one above the other. Neither of them is
> inherently any more precise or exact than the other.
>
> There doesn't seem to be any good solution here. For every
> use case in which Decimal+float->float appears better, there
> seems to be another one for which Decimal+float->Decimal
> appears better.

Sure, from a purist point of view my post is completely wrong.  It
doesn't correspond to the mathematical reality.

What it does correspond to is the code.  Only going rightward through
the types is what we have today.  A linear progression is a lot
simpler to understand than any sort of cycle; parallel progressions
isn't even on the table.

float has been the king of inexact types right for a long time.  All
other things being equal, that's good enough for me.


-- 
Adam Olsen, aka Rhamphoryncus

From ncoghlan at gmail.com  Sun Mar 21 02:10:08 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Mar 2010 11:10:08 +1000
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA4165D.2070808@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>
	<4BA4165D.2070808@canterbury.ac.nz>
Message-ID: <4BA571F0.3090800@gmail.com>

Greg Ewing wrote:
> Antoine Pitrou wrote:
> 
>> We forbid comparisons when there is a real danger or ambiguity, such
>> as unicode
>> vs. bytes. There is no such danger or ambiguity when comparing a
>> decimal with a
>> float.
> 
> So do you think that float("0.1") and Decimal("0.1") should be
> equal or not, and why?

Note that Antoine's point was that float("0.1") and
Decimal.from_float(0.1) should compare equal. The latter exactly matches
the underlying binary floating point value:

>>> dec.from_float(0.1)
Decimal('0.1000000000000000055511151231257827021181583404541015625')

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sun Mar 21 02:17:09 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Mar 2010 11:17:09 +1000
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA467C0.1000300@g.nevcal.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<ho0gjk$dd4$1@dough.gmane.org>
	<4BA3CD93.8080502@g.nevcal.com> <4BA44D21.1070401@gmail.com>
	<4BA467C0.1000300@g.nevcal.com>
Message-ID: <4BA57395.3040803@gmail.com>

Glenn Linderman wrote:
> Thanks, though for the nice chart from an internals guy, you confirmed
> all that I thought I knew about how this presently works, from PEP and
> manual reading, a bit of experimentation, and the discussions here.

I'll confess to a bit of interpreter prompt experimentation to confirm
the behaviour, especially in the relation to Fractions - I didn't know
all of that off the top of my head. The implicit coercion from Fraction
-> float definitely surprised me a bit.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From fuzzyman at voidspace.org.uk  Sun Mar 21 02:30:23 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sun, 21 Mar 2010 01:30:23 +0000
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA4B8DE.8060109@wanadoo.fr>
References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk>
	<4BA4B8DE.8060109@wanadoo.fr>
Message-ID: <4BA576AF.2050208@voidspace.org.uk>

On 20/03/2010 12:00, Pascal Chambon wrote:
> Michael Foord a ?crit :
>>
>> On 19/03/2010 18:58, Pascal Chambon wrote:
>>> Hello
>>>
>>> I've already crossed a bunch of articles detailing python's 
>>> attribute lookup semantic (__dict__, descriptors, order of base 
>>> class traversing...), but I have never seen, so far, an explanation 
>>> of WHICH method did waht, exactly.
>>>
>>> I assumed that getattr(a, b) was the same as a.__getattribute__(b), 
>>> and that this __getattribute__ method (or the hidden routine 
>>> replacing it when we don't override it in our class) was in charge 
>>> of doing the whole job of traversing the object tree, checking 
>>> descriptors, binding methods, calling __getattr__ on failure etc.
>>>
>>> However, the test case below shows that __getattribute__ does NOT 
>>> call __getattr__ on failure. So it seems it's an upper levl 
>>> machinery, in getattr(), which is in chrge of that last action.
>>
>> Python 3 has the behavior you are asking for. It would be a backwards 
>> incompatible change to do it in Python 2 as __getattribute__ *not* 
>> calling __getattr__ is the documented behaviour.
>>
>> Python 3.2a0 (py3k:78770, Mar 7 2010, 20:32:50)
>> [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
>> >>> class x:
>> ... def __getattribute__(s, name):
>> ... print ('__getattribute__', name)
>> ... raise AttributeError
>> ... def __getattr__(s, name):
>> ... print ('__getattr__', name)
>> ...
>> >>> a = x()
>> >>> a.b
>> __getattribute__ b
>> __getattr__ b
> I'm confused there, because the script you gave behaves the same in 
> python 2.6. And according to the doc, it's normal, getattr() reacts to 
> an AttributeError from __getattribute__, by calling __getattr__ :
>
> """
> Python 2.6.5 documentation
>
> object.__getattribute__(/self/, /name/)
>
>     Called unconditionally to implement attribute accesses for
>     instances of the class. If the class also defines __getattr__()
>     <http://docs.python.org/reference/datamodel.html#object.__getattr__>,
>     the latter will not be called unless __getattribute__()
>     <http://docs.python.org/reference/datamodel.html#object.__getattribute__>
>     either calls it explicitly or raises an AttributeError
>     <http://docs.python.org/library/exceptions.html#exceptions.AttributeError>.
>     This method should return the (computed) attribute value or raise
>     an AttributeError
>     <http://docs.python.org/library/exceptions.html#exceptions.AttributeError>
>     exception. In order to avoid infinite recursion in this method,
>     its implementation should always call the base class method with
>     the same name to access any attributes it needs, for example,
>     object.__getattribute__(self, name).
>
> """
>
> But the point which for me is still unclear, is : does the default 
> implementation of __getattribute__ (the one of "object") call 
> __getattr__ by himself, or does it rely on its caller for that, by 
> raising an AttributeError ? For Python2, it's blatantly the latter 
> case which is favoured, but since it looks like an implementation 
> detail at the moment, I propose we settle it (and document it) once 
> for all.

Ah right, my apologies. So it is still documented behaviour - 
__getattr__ is obviously called by the Python runtime and not by 
__getattribute__. (It isn't just by getattr as the same behaviour is 
shown when doing a normal attribute lookup and not via the getattr 
function.)

>
>>
>>
>> This list is not really an appropriate place to ask questions like 
>> this though, comp.lang.python would be better.
>>
>> All the best,
>>
>> Michael Fooord
> Sorry if I misposted, I just (wrongly ?) assumed that it was more an 
> undecided, implementation-specific point (since the doc gave possible 
> behaviours for __getattribute__, without precising which one was the 
> default one), and thus targetted the hands-in-core-code audience only.
>

Well, the documentation you pointed to specifies that __getattr__ will 
be called if __getattribute__ raises an AttributeError, it just doesn't 
specify that it is done by object.__getattribute__ (which it isn't).

Michael Foord

> Regards,
> Pascal
>
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100321/1e20ff61/attachment-0001.html>

From ncoghlan at gmail.com  Sun Mar 21 02:54:34 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Mar 2010 11:54:34 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
Message-ID: <4BA57C5A.9060403@gmail.com>

Guido van Rossum wrote:
> I think we should seriously reconsider allowing mixed arithmetic 
> involving Decimal, not just mixed comparisons.

I'm glad I'm not the only one that started wondering that. I
wasn't quite game enough to actually suggest it though :)

> There is one choice which I'm not sure about. Should a mixed 
> float/Decimal operation return a float or a Decimal? I note that 
> Fraction (which *is* properly embedded in the numeric tower) supports
>  this and returns a float result in this case. While I earlier 
> proposed to return the most "complicated" type of the two, i.e. 
> Decimal, I now think it may also make sense to return a float, being
>  the most "fuzzy" type in the numeric tower. This would also make 
> checking for accidental floats easier, since floats now propagate 
> throughout the computation (like NaN) and a simple assertion that the
>  result is a Decimal instance suffices to check that no floats were 
> implicitly mixed into the computation.

To blend a couple of different ideas from the thread reboot together:

I suggest a 'linearised' numeric tower that looks like:

int -> Decimal -> Fraction -> float -> complex

As Adam stated, this is a pragmatic tower stating which implicit
coercions are defined by the code, not a formal mathematical relationship.

Note that this would involve adding mixed Fraction/Decimal arithmetic as
well as Decimal/float arithmetic. I placed Decimal to the left of
Fraction to keep Decimal's dependencies clear and because Decimal ->
Fraction conversions appear straightforward (using power of 10
denominators) without introducing the precision issues that would arise
in Decimal -> Fraction conversions.

I also like the idea of adding the decimal context signal that Facundo
suggests (e.g. under the name ImplicitCoercionToBinaryFloat).

So "quick and dirty don't need a perfect answer" operations end up with
ordinary binary floats, while more precise code can enable the signal
trap to ensure the calculation fails noisily if binary floats are
introduced.

Allowing implicit operations would also finally allow Decimal to be
registered with numbers.Real.

Whatever we decide to do will need to be codified in a PEP though.
"Cleaning Up Python's Numeric Tower" or something along those lines.

> The implementation of __hash__ will be complicated, and it may make 
> sense to tweak the hash function of float, Fraction and Decimal to 
> make it easier to ensure that for values that can be represented in 
> either type the hash matches the equality. But this sounds a 
> worthwhile price to pay for proper embedding in the numeric tower.

And Mark appears to already have a good answer to that problem.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sun Mar 21 02:56:17 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Mar 2010 11:56:17 +1000
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA576AF.2050208@voidspace.org.uk>
References: <4BA3C962.7080500@wanadoo.fr>
	<4BA3DD4F.1080209@voidspace.org.uk>	<4BA4B8DE.8060109@wanadoo.fr>
	<4BA576AF.2050208@voidspace.org.uk>
Message-ID: <4BA57CC1.1050404@gmail.com>

Michael Foord wrote:
> Well, the documentation you pointed to specifies that __getattr__ will
> be called if __getattribute__ raises an AttributeError, it just doesn't
> specify that it is done by object.__getattribute__ (which it isn't).

And as for why not: because __getattribute__ implementations need to be
able to call object.__getattribute__ without triggering the fallback
behaviour.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From raymond.hettinger at gmail.com  Sun Mar 21 03:44:09 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 20 Mar 2010 19:44:09 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA57C5A.9060403@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
Message-ID: <043F9D0F-AFC7-4D09-8514-0C7AD59BCD9B@gmail.com>


On Mar 20, 2010, at 6:54 PM, Nick Coghlan wrote:
> 
> I suggest a 'linearised' numeric tower that looks like:
> 
> int -> Decimal -> Fraction -> float -> complex

Is that a typo?  Shouldn't Decimal and float go between Fraction and complex?

The abstract numeric tower is:

        Number Complex Real Rational Integral

where both Decimal and float have operations associated with reals.



Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100320/bf390cc3/attachment.html>

From raymond.hettinger at gmail.com  Sun Mar 21 04:46:15 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 20 Mar 2010 20:46:15 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
Message-ID: <B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>


On Mar 20, 2010, at 4:27 PM, Jeffrey Yasskin wrote:
> 
> When designing it, I tried to include a notion of exact/inexact types,
> but we couldn't find anything practical to do with them, so we took
> them out. 

The were also other reasons that they were taken out.  The notion of inexactness is a taint, not a property of a type.  

The design documents for the decimal spec aimed for the ability to do either inexact or exact calculations (and to encompass integer and fixed-point arithmetic).  That is in-part why decimal is suitable for accounting work.  A person can forbid any inexactness by setting a context flag that would raise an exception if any inexact calculation occurs.  Another design principle for decimal is the notion that numbers are always exact, it is the results of operations that are subject to rounding.  That is why the decimal constructor is not context sensitive.

So, as Jeffrey says, the notion of exactness and inexactness is not built in to the numeric tower.  Instead, it is about which operations or methods can be expected to apply to a given type (i.e. both float and Decimal support the necessary methods to register as a Real).   

In terms of interoperability of concrete types, we're in the fortunate position that any float can be exactly converted to a Decimal and any Decimal can be exactly converted to a Fraction.


Raymond





From greg.ewing at canterbury.ac.nz  Sun Mar 21 05:40:19 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 21 Mar 2010 16:40:19 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com>
Message-ID: <4BA5A333.9070404@canterbury.ac.nz>

Mark Dickinson wrote:

> Except that float is fixed-width (typically 53 bits of precision),
> while Decimal allows a user-specified, arbitrarily large, precision;

Yes, but it still has *some* fixed limit at any given
moment, so the result of an operation on Decimals always
has the potential to produce an inexact result. It's
not like an int or Fraction where the result can expand
to whatever size is needed.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun Mar 21 05:48:18 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 21 Mar 2010 16:48:18 +1200
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA571F0.3090800@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<4BA4165D.2070808@canterbury.ac.nz> <4BA571F0.3090800@gmail.com>
Message-ID: <4BA5A512.5060005@canterbury.ac.nz>

Nick Coghlan wrote:

> Note that Antoine's point was that float("0.1") and
> Decimal.from_float(0.1) should compare equal.

That would mean that Decimal("0.1") != float("0.1"), which might be
surprising to someone who didn't realise they were mixing floats
and decimals.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun Mar 21 05:54:18 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 21 Mar 2010 16:54:18 +1200
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA576AF.2050208@voidspace.org.uk>
References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk>
	<4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk>
Message-ID: <4BA5A67A.9000703@canterbury.ac.nz>

Michael Foord wrote:

> Well, the documentation you pointed to specifies that __getattr__ will 
> be called if __getattribute__ raises an AttributeError, it just doesn't 
> specify that it is done by object.__getattribute__ (which it isn't).

If __getattribute__ raises an exception, it won't get a chance to
do anything else, so something outside of __getattribute__ must
catch the AttributeError and calling __getattr__. So I think the
docs *are* specifying the behaviour here, if only by implication.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun Mar 21 06:00:47 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 21 Mar 2010 17:00:47 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA57C5A.9060403@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
Message-ID: <4BA5A7FF.7080501@canterbury.ac.nz>

Nick Coghlan wrote:

> int -> Decimal -> Fraction -> float -> complex

I don't think it's a good idea to put Decimal below Fraction,
because Decimal has to be considered an implicitly inexact
type like float, and we don't want to coerce from an inexact
type to an exact one.

-- 
Greg

From rhamph at gmail.com  Sun Mar 21 06:54:17 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Sat, 20 Mar 2010 23:54:17 -0600
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float 	comparisons in py3k.
In-Reply-To: <4BA5A512.5060005@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<4BA4165D.2070808@canterbury.ac.nz> <4BA571F0.3090800@gmail.com>
	<4BA5A512.5060005@canterbury.ac.nz>
Message-ID: <aac2c7cb1003202254g2db16cb3ta7665b6efd2f99a3@mail.gmail.com>

On Sat, Mar 20, 2010 at 22:48, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>
>> Note that Antoine's point was that float("0.1") and
>> Decimal.from_float(0.1) should compare equal.
>
> That would mean that Decimal("0.1") != float("0.1"), which might be
> surprising to someone who didn't realise they were mixing floats
> and decimals.

Much like Fraction("0.1") != float("0.1") is surprising.

The only way to get rid of surprises around float is to get rid of
float, and that ain't happening.


-- 
Adam Olsen, aka Rhamphoryncus

From ncoghlan at gmail.com  Sun Mar 21 07:58:24 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Mar 2010 16:58:24 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <043F9D0F-AFC7-4D09-8514-0C7AD59BCD9B@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<043F9D0F-AFC7-4D09-8514-0C7AD59BCD9B@gmail.com>
Message-ID: <4BA5C390.8030606@gmail.com>

Raymond Hettinger wrote:
> 
> On Mar 20, 2010, at 6:54 PM, Nick Coghlan wrote:
>>
>> I suggest a 'linearised' numeric tower that looks like:
>>
>> int -> Decimal -> Fraction -> float -> complex
> 
> Is that a typo?  Shouldn't Decimal and float go between Fraction and
> complex?
> 
> The abstract numeric tower is:
> 
>         Number Complex Real Rational Integral
> 
> where both Decimal and float have operations associated with reals.

I don't actually mind either way - the pragmatic tower is about coding
convenience rather than numeric purity (and mixing Fractions and
Decimals in the same algorithm is somewhat nonsensical - they're
designed for two completely different problem domains).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Sun Mar 21 07:59:49 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 21 Mar 2010 16:59:49 +1000
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA5A512.5060005@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>	<201003180759.25229.steve@pearwood.info>	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>	<201003182323.19458.steve@pearwood.info>	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>	<4BA29FB1.9020909@gmail.com>
	<4BA2B681.6090709@g.nevcal.com>	<loom.20100319T021726-50@post.gmane.org>
	<4BA2DCB8.2040406@nevcal.com>	<loom.20100319T124539-150@post.gmane.org>	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>	<loom.20100319T190158-232@post.gmane.org>	<4BA4165D.2070808@canterbury.ac.nz>
	<4BA571F0.3090800@gmail.com> <4BA5A512.5060005@canterbury.ac.nz>
Message-ID: <4BA5C3E5.6080805@gmail.com>

Greg Ewing wrote:
> Nick Coghlan wrote:
> 
>> Note that Antoine's point was that float("0.1") and
>> Decimal.from_float(0.1) should compare equal.
> 
> That would mean that Decimal("0.1") != float("0.1"), which might be
> surprising to someone who didn't realise they were mixing floats
> and decimals.

That's fine - binary floats *are* surprising. That's why Decimal exists
in the first place.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From webmaster at python.org  Sun Mar 21 14:55:18 2010
From: webmaster at python.org (webmaster at python.org)
Date: Sun, 21 Mar 2010 13:55:18 +0000
Subject: [Python-Dev] Fwd: Broken link to download (Mac OS X)
Message-ID: <4BA62546.10906@python.org>

Hey all,

This seems to happen whenever we do a new release (we've had a couple of 
emails to webmaster at python.org about it since 2.6.5 was released). The 
main download page for Python has a broken link for the Mac installer 
(because it hasn't been built yet I assume):

     http://python.org/download/

The link 404s, with no explanation or alternate link - so for the casual 
user who wants to install Python 2.6 on Mac OS X they are 
sorely-out-of-luck.

Not being able to provide a mac installer at the same time as other 
platforms is one thing (and I accept that is unavoidable), breaking the 
download links for Mac users for unspecified lengths of time is just bad 
practise. If we create a new stable release without a Mac installer can 
we at least provide a brief explanation and link to the *previous 
version* until the new version is ready?

All the best,

Michael

-------- Original Message --------
Subject: 	Broken link to down
Date: 	Sun, 21 Mar 2010 13:40:36 +0000
From: 	Ben Hodgson <ben at benhodgson.com>
To: 	webmaster at python.org



Hey there,

In case you don't know, the link on http://www.python.org/download/ to 
the Python 2.6.5 Mac Installer Disk Image 
(http://www.python.org/ftp/python/2.6.5/python-2.6.5_macosx10.3.dmg) is 
broken.

Cheers,
Ben

--
Ben Hodgson
http://benhodgson.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100321/b72a46e1/attachment.html>

From guido at python.org  Sun Mar 21 17:53:14 2010
From: guido at python.org (Guido van Rossum)
Date: Sun, 21 Mar 2010 09:53:14 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA57C5A.9060403@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<4BA57C5A.9060403@gmail.com>
Message-ID: <ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>

On Sat, Mar 20, 2010 at 6:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Guido van Rossum wrote:
>> I think we should seriously reconsider allowing mixed arithmetic
>> involving Decimal, not just mixed comparisons.
>
> I'm glad I'm not the only one that started wondering that. I
> wasn't quite game enough to actually suggest it though :)
>
>> There is one choice which I'm not sure about. Should a mixed
>> float/Decimal operation return a float or a Decimal? I note that
>> Fraction (which *is* properly embedded in the numeric tower) supports
>> ?this and returns a float result in this case. While I earlier
>> proposed to return the most "complicated" type of the two, i.e.
>> Decimal, I now think it may also make sense to return a float, being
>> ?the most "fuzzy" type in the numeric tower. This would also make
>> checking for accidental floats easier, since floats now propagate
>> throughout the computation (like NaN) and a simple assertion that the
>> ?result is a Decimal instance suffices to check that no floats were
>> implicitly mixed into the computation.
>
> To blend a couple of different ideas from the thread reboot together:
>
> I suggest a 'linearised' numeric tower that looks like:
>
> int -> Decimal -> Fraction -> float -> complex
>
> As Adam stated, this is a pragmatic tower stating which implicit
> coercions are defined by the code, not a formal mathematical relationship.
>
> Note that this would involve adding mixed Fraction/Decimal arithmetic as
> well as Decimal/float arithmetic.

Yes, that was my intention too.

> I placed Decimal to the left of
> Fraction to keep Decimal's dependencies clear and because Decimal ->
> Fraction conversions appear straightforward (using power of 10
> denominators) without introducing the precision issues that would arise
> in Decimal -> Fraction conversions.

This is just wrong. Decimal is much more like float than like int or
Fraction, due to its rounding behavior. (You cannot produce exact
results for 1/3 using either Decimal or float.)

Clearly the difficult decision is whether Decimal is between Fraction
and float, or between float and complex (I prefer not to say "to the
left/right of" since PEP 3141 orders the types in the opposite way as
people have been doing here). Both are floating point types that
sometimes produce rounded results (e.g. 1/3); the difference is that
they use different bases and that Decimal has configurable precision
when rounding.

I would make float the last station before complex, whereas Mark would
prefer Decimal to go there. I defer to Mark, who has thought a lot
more about Decimal than I have.

> I also like the idea of adding the decimal context signal that Facundo
> suggests (e.g. under the name ImplicitCoercionToBinaryFloat).

Yes, this is what I was trying to say (but didn't find the right words for).

> So "quick and dirty don't need a perfect answer" operations end up with
> ordinary binary floats, while more precise code can enable the signal
> trap to ensure the calculation fails noisily if binary floats are
> introduced.

But does this break a tie for the relative ordering of Decimal and
float in the tower?

> Allowing implicit operations would also finally allow Decimal to be
> registered with numbers.Real.

Right, that's one aspect of what I meant by "embedded in the numeric tower".

> Whatever we decide to do will need to be codified in a PEP though.
> "Cleaning Up Python's Numeric Tower" or something along those lines.

The cleanup is really just specific to Decimal -- int, Fraction and
float are already properly embedded in the tower (PEP 3141 doesn't
advertise Fraction enough, since it predates it). That we may have to
change the __hash__ implementation for the other types is merely a
compromise towards efficiency.

>> The implementation of __hash__ will be complicated, and it may make
>> sense to tweak the hash function of float, Fraction and Decimal to
>> make it easier to ensure that for values that can be represented in
>> either type the hash matches the equality. But this sounds a
>> worthwhile price to pay for proper embedding in the numeric tower.
>
> And Mark appears to already have a good answer to that problem.

Which I still have to review. (Mark, if you're there, could you make a
brief post here on the mathematical definition of the new hash you're
proposing, and why it is both efficient to compute and good (enough)
as a hash function?)

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Sun Mar 21 18:02:30 2010
From: guido at python.org (Guido van Rossum)
Date: Sun, 21 Mar 2010 10:02:30 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
Message-ID: <ca471dc21003211002v754f1e1i487ae0b2aab51a03@mail.gmail.com>

On Sun, Mar 21, 2010 at 9:53 AM, Guido van Rossum <guido at python.org> wrote:
> Which I still have to review. (Mark, if you're there, could you make a
> brief post here on the mathematical definition of the new hash you're
> proposing, and why it is both efficient to compute and good (enough)
> as a hash function?)

Never mind, Mark. I found your explanation here:
http://codereview.appspot.com/660042/diff/19001/11009?column_width=80

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Sun Mar 21 18:07:56 2010
From: guido at python.org (Guido van Rossum)
Date: Sun, 21 Mar 2010 10:07:56 -0700
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float 	comparisons in py3k.
In-Reply-To: <4BA5C3E5.6080805@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com> 
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com> 
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com> 
	<loom.20100319T190158-232@post.gmane.org>
	<4BA4165D.2070808@canterbury.ac.nz> 
	<4BA571F0.3090800@gmail.com> <4BA5A512.5060005@canterbury.ac.nz> 
	<4BA5C3E5.6080805@gmail.com>
Message-ID: <ca471dc21003211007o4ac8cac4n458306ac5bfca114@mail.gmail.com>

On Sat, Mar 20, 2010 at 11:59 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Greg Ewing wrote:
>> Nick Coghlan wrote:
>>
>>> Note that Antoine's point was that float("0.1") and
>>> Decimal.from_float(0.1) should compare equal.
>>
>> That would mean that Decimal("0.1") != float("0.1"), which might be
>> surprising to someone who didn't realise they were mixing floats
>> and decimals.
>
> That's fine - binary floats *are* surprising. That's why Decimal exists
> in the first place.

Decimals can be just as surprising:

>>> Decimal(1) / Decimal(3) * Decimal(3) == Decimal(1)
False
>>>


-- 
--Guido van Rossum (python.org/~guido)

From benjamin at python.org  Sun Mar 21 18:12:08 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 21 Mar 2010 12:12:08 -0500
Subject: [Python-Dev] [RELEASED] Python 3.1.2
Message-ID: <1afaf6161003211012r30bc0e77ub5dfe9f248668525@mail.gmail.com>

On behalf of the Python development team, I'm joyful to announce the second
bugfix release of the Python 3.1 series, Python 3.1.2.

This bug fix release fixes numerous issues found in 3.1.1, and is considered a
production release.

The Python 3.1 version series focuses on the stabilization and optimization of
the features and changes that Python 3.0 introduced.  For example, the new I/O
system has been rewritten in C for speed.  File system APIs that use unicode
strings now handle paths with undecodable bytes in them. Other features include
an ordered dictionary implementation, a condensed syntax for nested with
statements, and support for ttk Tile in Tkinter.  For a more extensive list of
changes in 3.1, see http://doc.python.org/3.1/whatsnew/3.1.html or Misc/NEWS in
the Python distribution.

To download Python 3.1.2 visit:

     http://www.python.org/download/releases/3.1.2/

A list of changes in 3.1.2 can be found here:

     http://svn.python.org/projects/python/tags/r312/Misc/NEWS

The 3.1 documentation can be found at:

     http://docs.python.org/3.1

Bugs can always be reported to:

     http://bugs.python.org


Enjoy!

--
Benjamin Peterson
Release Manager
benjamin at python.org
(on behalf of the entire python-dev team and 3.1.2's contributors)

From raymond.hettinger at gmail.com  Sun Mar 21 19:25:34 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Mar 2010 11:25:34 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
Message-ID: <E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>


>> 
>> Note that this would involve adding mixed Fraction/Decimal arithmetic as
>> well as Decimal/float arithmetic.
> 
> Yes, that was my intention too.

+1


> 
>> I placed Decimal to the left of
>> Fraction to keep Decimal's dependencies clear and because Decimal ->
>> Fraction conversions appear straightforward (using power of 10
>> denominators) without introducing the precision issues that would arise
>> in Decimal -> Fraction conversions.
> 
> This is just wrong. Decimal is much more like float than like int or
> Fraction, due to its rounding behavior. (You cannot produce exact
> results for 1/3 using either Decimal or float.)


Right.  We should be guided by:

    fractions are a superset of decimals which are a superset of binary floats.

And by:

    binary floats and decimal floats both implement all of the operations
    for the Real abstract base class.

> 
> Yes, this is what I was trying to say (but didn't find the right words for).
> 
>> So "quick and dirty don't need a perfect answer" operations end up with
>> ordinary binary floats, while more precise code can enable the signal
>> trap to ensure the calculation fails noisily if binary floats are
>> introduced.
> 
> But does this break a tie for the relative ordering of Decimal and
> float in the tower?
> 

It seems to me that Decimals and floats should be considered at
the same level (i.e. both implement Real).

Mixed Decimal and float should coerce to Decimal because it can be
done losslessly.

There is no need to embed a notion of "imperfect answer".
Numbers themselves are exact and many mixed operations
can be exact is the coercions go the right way.

Some folks who have had bad experiences with representation
error (i.e. 1.1 cannot be exactly represented as a binary float) or
with round-off error (i.e. 1.0 / 7.0 must be rounded) tend to think
of both binary or decimal floats as necessarily inexact.  But that
is not the case, exact accounting work is perfectly feasable
with decimals.  Remember, the notion of inexactness is a taint,
not an intrinsic property of a type.  Even the Scheme numeric
tower recognizes this.  LIkewise, the decimal specification also
spells-out this notion as basic to its design.


> 
>> Whatever we decide to do will need to be codified in a PEP though.
>> "Cleaning Up Python's Numeric Tower" or something along those lines.
> 
> The cleanup is really just specific to Decimal -- int, Fraction and
> float are already properly embedded in the tower (PEP 3141 doesn't
> advertise Fraction enough, since it predates it). That we may have to
> change the __hash__ implementation for the other types is merely a
> compromise towards efficiency.

I believe that no "clean-up" is necessary.  Decimal already implements
the Real ABC.   All that is necessary is the common __hash__ algorithm
and removing the restriction between decimal/float interaction so that
any two instances of Real can interoperate with one another.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100321/b5740c93/attachment.html>

From raymond.hettinger at gmail.com  Sun Mar 21 19:31:06 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Mar 2010 11:31:06 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA5A333.9070404@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5c6f2a5d1003201646h11f38bc0jcbd621a6591c709c@mail.gmail.com>
	<4BA5A333.9070404@canterbury.ac.nz>
Message-ID: <834D7F8C-2F04-4927-8333-77868535A374@gmail.com>


On Mar 20, 2010, at 9:40 PM, Greg Ewing wrote:

> Mark Dickinson wrote:
> 
>> Except that float is fixed-width (typically 53 bits of precision),
>> while Decimal allows a user-specified, arbitrarily large, precision;
> 
> Yes, but it still has *some* fixed limit at any given
> moment, so the result of an operation on Decimals always
> has the potential to produce an inexact result. It's
> not like an int or Fraction where the result can expand
> to whatever size is needed.

I'm thinking that I need to do more work on the Decimal
documentation.  There still seems to be a profound
misunderstanding of its capabilities (i.e. that an Inexact
flag can be set to preclude any inexact operations,
that the precision can be automatically extended as
needed during a calculation, or that many types of 
calculations can be done exactly especially if 
floor division is used).  If rounded is needed, it
can be controlled explicitly.


Raymond

From raymond.hettinger at gmail.com  Sun Mar 21 19:45:18 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Mar 2010 11:45:18 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003211002v754f1e1i487ae0b2aab51a03@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<ca471dc21003211002v754f1e1i487ae0b2aab51a03@mail.gmail.com>
Message-ID: <341D3926-0D43-41AA-85BE-170A67BC4169@gmail.com>


On Mar 21, 2010, at 10:02 AM, Guido van Rossum wrote:

> On Sun, Mar 21, 2010 at 9:53 AM, Guido van Rossum <guido at python.org> wrote:
>> Which I still have to review. (Mark, if you're there, could you make a
>> brief post here on the mathematical definition of the new hash you're
>> proposing, and why it is both efficient to compute and good (enough)
>> as a hash function?)
> 
> Never mind, Mark. I found your explanation here:
> http://codereview.appspot.com/660042/diff/19001/11009?column_width=80

I'm often dazzled by Mark's brilliance,
but this is an especially nice bit of reasoning.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100321/f7ebb1d4/attachment.html>

From rdmurray at bitdance.com  Sun Mar 21 19:50:50 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 21 Mar 2010 14:50:50 -0400
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
Message-ID: <20100321185050.D9C051FDBF4@kimball.webabinitio.net>

On Sun, 21 Mar 2010 11:25:34 -0700, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> It seems to me that Decimals and floats should be considered at
> the same level (i.e. both implement Real).
> 
> Mixed Decimal and float should coerce to Decimal because it can be
> done losslessly.
> 
> There is no need to embed a notion of "imperfect answer".
> Numbers themselves are exact and many mixed operations
> can be exact is the coercions go the right way.

I think the concern here is rather about operations such as:

    1.1 + Decimal('1.1')

The calculation may produce an "exact" result, but it won't be the exact
result expected, because the conversion from string (at the program text
file level) to float was lossy.  Thus the desire for some mechanism to
know that floats and decimals have been mixed anywhere in the calculations
that led up to whatever result number you are looking at.  And to have
doing so trigger an exception if requested by the programmer.

--
R. David Murray                                      www.bitdance.com

From rhamph at gmail.com  Sun Mar 21 20:21:46 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Sun, 21 Mar 2010 13:21:46 -0600
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA5C390.8030606@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<043F9D0F-AFC7-4D09-8514-0C7AD59BCD9B@gmail.com>
	<4BA5C390.8030606@gmail.com>
Message-ID: <aac2c7cb1003211221y68f00c4fo44e21a7cef077ada@mail.gmail.com>

On Sun, Mar 21, 2010 at 00:58, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I don't actually mind either way - the pragmatic tower is about coding
> convenience rather than numeric purity (and mixing Fractions and
> Decimals in the same algorithm is somewhat nonsensical - they're
> designed for two completely different problem domains).

I think the rule I've been going on is that ideal types (int,
Fraction) are on one end and pragmatic types (float, complex) are on
the other.  Since Decimal can be used exactly it clearly bridges both
groups.

*However*, there's other possible types out there, and would they fit
into my system?  I've just taken a look at sympy and although it's
clearly an ideal type, it also allows mixing with float and complex,
both producing sympy types.  That puts it clearly past float and
complex in the tower.

I have no ideal where Decimal should go.


-- 
Adam Olsen, aka Rhamphoryncus

From raymond.hettinger at gmail.com  Sun Mar 21 20:31:57 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Mar 2010 12:31:57 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <20100321185050.D9C051FDBF4@kimball.webabinitio.net>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
	<20100321185050.D9C051FDBF4@kimball.webabinitio.net>
Message-ID: <B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>


On Mar 21, 2010, at 11:50 AM, R. David Murray wrote:

> On Sun, 21 Mar 2010 11:25:34 -0700, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
>> It seems to me that Decimals and floats should be considered at
>> the same level (i.e. both implement Real).
>> 
>> Mixed Decimal and float should coerce to Decimal because it can be
>> done losslessly.
>> 
>> There is no need to embed a notion of "imperfect answer".
>> Numbers themselves are exact and many mixed operations
>> can be exact is the coercions go the right way.
> 
> I think the concern here is rather about operations such as:
> 
>    1.1 + Decimal('1.1')
> 
> The calculation may produce an "exact" result, but it won't be the exact
> result expected, because the conversion from string (at the program text
> file level) to float was lossy.  Thus the desire for some mechanism to
> know that floats and decimals have been mixed anywhere in the calculations
> that led up to whatever result number you are looking at.  And to have
> doing so trigger an exception if requested by the programmer.

That makes sense.  That's why Guido proposed a context flag in
decimal to issue a warning for implicit mixing of decimals and floats.

What I was talking about was a different issue.  The question of
where to stack decimals in the hierarchy was erroneously 
being steered by the concept that both decimal and binary floats
are intrinsically inexact.  But that would be incorrect, inexactness
is a taint, the numbers themselves are always exact.

I really like Guido's idea of a context flag to control whether
mixing of decimal and binary floats will issue a warning.
The default should be to issue the warning (because unless
you know what you're doing, it is most likely an error).


Raymond


From greg.ewing at canterbury.ac.nz  Sun Mar 21 22:47:53 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 22 Mar 2010 09:47:53 +1200
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <4BA5C3E5.6080805@gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<201003180759.25229.steve@pearwood.info>
	<453BC86C-93DA-4C39-914B-BF7DFEEA99B9@gmail.com>
	<201003182323.19458.steve@pearwood.info>
	<06AFBC1C-4900-41DE-9A7A-D69F80C93E3A@gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com> <4BA2B681.6090709@g.nevcal.com>
	<loom.20100319T021726-50@post.gmane.org> <4BA2DCB8.2040406@nevcal.com>
	<loom.20100319T124539-150@post.gmane.org>
	<4C8F1869-5918-4BBF-BCC4-B8B236EB28C7@gmail.com>
	<loom.20100319T190158-232@post.gmane.org>
	<4BA4165D.2070808@canterbury.ac.nz>
	<4BA571F0.3090800@gmail.com> <4BA5A512.5060005@canterbury.ac.nz>
	<4BA5C3E5.6080805@gmail.com>
Message-ID: <4BA69409.4040209@canterbury.ac.nz>

Nick Coghlan wrote:

> That's fine - binary floats *are* surprising. That's why Decimal exists
> in the first place.

This argument could equally well be used the other way --
someone using Decimal is doing so precisely because they
*don't* want to be surprised, in which case they would
probably prefer to get an exception.

The fundamental problem here is that there are two possible
reasons for a mixed float-decimal operation:

1) The user is thinking in terms of floats and has
happened to get a Decimal mixed in somehow.

2) The user is thinking in terms of Decimals and has
happened to get a float mixed in somehow.

There is no way of distinguishing between these
automatically.

-- 
Greg

From tjreedy at udel.edu  Sun Mar 21 23:10:44 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 21 Mar 2010 18:10:44 -0400
Subject: [Python-Dev] [RELEASED] Python 3.1.2
In-Reply-To: <1afaf6161003211012r30bc0e77ub5dfe9f248668525@mail.gmail.com>
References: <1afaf6161003211012r30bc0e77ub5dfe9f248668525@mail.gmail.com>
Message-ID: <ho65h1$bv7$3@dough.gmane.org>

On 3/21/2010 1:12 PM, Benjamin Peterson wrote:
> On behalf of the Python development team, I'm joyful to announce the second
> bugfix release of the Python 3.1 series, Python 3.1.2.

Thanks for the work.

> This bug fix release fixes numerous issues found in 3.1.1, and is considered a
> production release.
>
> The Python 3.1 version series focuses on the stabilization and optimization of
> the features and changes that Python 3.0 introduced.  For example, the new I/O
> system has been rewritten in C for speed.  File system APIs that use unicode
> strings now handle paths with undecodable bytes in them. Other features include
> an ordered dictionary implementation, a condensed syntax for nested with
> statements, and support for ttk Tile in Tkinter.  For a more extensive list of
> changes in 3.1, see http://doc.python.org/3.1/whatsnew/3.1.html or Misc/NEWS in
> the Python distribution.
>
> To download Python 3.1.2 visit:
>
>       http://www.python.org/download/releases/3.1.2/
>
> A list of changes in 3.1.2 can be found here:
>
>       http://svn.python.org/projects/python/tags/r312/Misc/NEWS

There seem to be about a hundred fixes.

> The 3.1 documentation can be found at:
>
>       http://docs.python.org/3.1
>
> Bugs can always be reported to:
>
>       http://bugs.python.org


From greg.ewing at canterbury.ac.nz  Sun Mar 21 23:41:49 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 22 Mar 2010 10:41:49 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
Message-ID: <4BA6A0AD.7090207@canterbury.ac.nz>

Raymond Hettinger wrote:
> 
> Remember, the notion of inexactness is a taint,
> not an intrinsic property of a type.  Even the Scheme numeric
> tower recognizes this.  LIkewise, the decimal specification also
> spells-out this notion as basic to its design.

I'm not sure it really does, otherwise every decimal
value would have a flag indicating whether it was
tainted with inexactness, and this flag would propagate
through calculations.

-- 
Greg

From steve at pearwood.info  Sun Mar 21 23:32:20 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 22 Mar 2010 09:32:20 +1100
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
	float comparisons in py3k.
In-Reply-To: <4BA69409.4040209@canterbury.ac.nz>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<4BA5C3E5.6080805@gmail.com> <4BA69409.4040209@canterbury.ac.nz>
Message-ID: <201003220932.22259.steve@pearwood.info>

On Mon, 22 Mar 2010 08:47:53 am Greg Ewing wrote:
> Nick Coghlan wrote:
> > That's fine - binary floats *are* surprising. That's why Decimal
> > exists in the first place.
>
> This argument could equally well be used the other way --
> someone using Decimal is doing so precisely because they
> *don't* want to be surprised, in which case they would
> probably prefer to get an exception.

Then they're in for a terrible, terrible disappointment. Rounding issues 
don't go away because you're using Decimal instead of float, and I 
can't imagine anyone would like an exception in the following cases:

>>> Decimal(1)/Decimal(3)*Decimal(3) == Decimal(1)
False
>>> Decimal(2).sqrt()**Decimal(2) == Decimal(2)
False
>>> Decimal(10**28)+Decimal(1)-Decimal(10**28) == Decimal(1)
False

Rounding isn't the only surprise:

>>> x = Decimal("NAN"); x == x
False

Decimals are floats, but using radix 10 instead of radix 2.


-- 
Steven D'Aprano

From guido at python.org  Sun Mar 21 23:35:51 2010
From: guido at python.org (Guido van Rossum)
Date: Sun, 21 Mar 2010 15:35:51 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com> 
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
Message-ID: <ca471dc21003211535t26c61b11t9aa2fc3cf20e7948@mail.gmail.com>

On Sun, Mar 21, 2010 at 11:25 AM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> Right. ?We should be guided by:
> ?? ?fractions are a superset of decimals which are a superset of binary floats.

But mixed Fraction-float operations return floats, not Fractions.

> And by:
> ?? ?binary floats and decimal floats both implement all of the operations
> ?? ?for the Real abstract base class.

Sure, but that doesn't help us decide what mixed Decimal-float
operations should return.

> It seems to me that Decimals and floats should be considered at
> the same level (i.e. both implement Real).

Agreed, but doesn't help. (Except against the idea that Decimal goes
on the "integer" side of Fraction, which is just wrong.)

> Mixed Decimal and float should coerce to Decimal because it can be
> done losslessly.

But mixed Fraction-float returns float even though returning Fraction
could be done losslessly.

The real criterion should be what's more useful, not what can be done
losslessly.

> There is no need to embed a notion of "imperfect answer".
> Numbers themselves are exact and many mixed operations
> can be exact if the coercions go the right way.

Division cannot, in general (I consider floor division a bastard child
of the integers). And for multiplication it seems that rounding at
some point becomes necessary since the alternative would be to use
infinite precision.

> Some folks who have had bad experiences with representation
> error (i.e. 1.1 cannot be exactly represented as a binary float) or
> with round-off error (i.e. 1.0 / 7.0 must be rounded) tend to think
> of both binary or decimal floats as necessarily inexact. ?But that
> is not the case, exact accounting work is perfectly feasable
> with decimals. ?Remember, the notion of inexactness is a taint,
> not an intrinsic property of a type. ?Even the Scheme numeric
> tower recognizes this. ?LIkewise, the decimal specification also
> spells-out this notion as basic to its design.

I really don't think advertising Decimal as having exact operations is
the right thing to do. Sure, it is the right data type for all
accounting operations -- but that is a very specialized use case,
where certain round-off errors are desirable (since nobody wants
fractional pennies in their bill).

> I believe that no "clean-up" is necessary. ?Decimal already implements
> the Real ABC. ? All that is necessary is the common __hash__ algorithm
> and removing the restriction between decimal/float interaction so that
> any two instances of Real can interoperate with one another.

Call it by any name you want. We're looking at revising the hash
function and allowing mixed operations between Decimal and float, with
signal that warns about these operations. I see two open issues:
whether mixed Decimal-float operations should return Decimal and
float, and whether the warning about such operations should be on or
off by default. My gut tells me that the signal should be off by
default. My gut doesn't say much about whether the result should be
float or Decimal, but I want the reasoning to reach a decision to be
sound.

--
--Guido van Rossum (python.org/~guido)

From greg.ewing at canterbury.ac.nz  Sun Mar 21 23:50:26 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 22 Mar 2010 10:50:26 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
Message-ID: <4BA6A2B2.6050207@canterbury.ac.nz>

Raymond Hettinger wrote:

> Since decimal also allows arbitrary sizes, all long ints can be
> exactly represented (this was even one of the design goals
> for the decimal module).

There may be something we need to clarify here. I've been
imagining that the implicit conversions to Decimal that
we're talking about would be done to whatever precision
is set in the context. Am I wrong about that? Is the intention
to always use enough digits to get an exact representation?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon Mar 22 00:07:53 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 22 Mar 2010 11:07:53 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
	<20100321185050.D9C051FDBF4@kimball.webabinitio.net>
	<B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
Message-ID: <4BA6A6C9.3020708@canterbury.ac.nz>

Raymond Hettinger wrote:
> The question of
> where to stack decimals in the hierarchy was erroneously 
> being steered by the concept that both decimal and binary floats
> are intrinsically inexact.  But that would be incorrect, inexactness
> is a taint, the numbers themselves are always exact.

I don't think that's correct. "Numbers are always exact" is
a simplification due to choosing not to attach an inexactness
flag to every value. Without such a flag, we don't really know
whether any given value is exact or not, we can only guess.

The reason for regarding certain types as "implicitly inexact"
is something like this: If you start with exact ints, and do
only int operations with them, you must end up with exact
ints. But the same is not true of float or Decimal: even if
you start with exact values, you can end up with inexact ones.

> I really like Guido's idea of a context flag to control whether
> mixing of decimal and binary floats will issue a warning.

Personally I feel that far too much stuff concerning decimals
is controlled by implicit context parameters. It gives me the
uneasy feeling that I don't know what the heck any given
decimal operation is going to do. It's probably justified in
this case, though.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Mon Mar 22 00:08:11 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 22 Mar 2010 11:08:11 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
	<20100321185050.D9C051FDBF4@kimball.webabinitio.net>
	<B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
Message-ID: <4BA6A6DB.3010801@canterbury.ac.nz>

Raymond Hettinger wrote:
> The question of
> where to stack decimals in the hierarchy was erroneously 
> being steered by the concept that both decimal and binary floats
> are intrinsically inexact.  But that would be incorrect, inexactness
> is a taint, the numbers themselves are always exact.

I don't think that's correct. "Numbers are always exact" is
a simplification due to choosing not to attach an inexactness
flag to every value. Without such a flag, we don't really know
whether any given value is exact or not, we can only guess.

The reason for regarding certain types as "implicitly inexact"
is something like this: If you start with exact ints, and do
only int operations with them, you must end up with exact
ints. But the same is not true of float or Decimal: even if
you start with exact values, you can end up with inexact ones.

> I really like Guido's idea of a context flag to control whether
> mixing of decimal and binary floats will issue a warning.

Personally I feel that far too much stuff concerning decimals
is controlled by implicit context parameters. It gives me the
uneasy feeling that I don't know what the heck any given
decimal operation is going to do. It's probably justified in
this case, though.

-- 
Greg


From raymond.hettinger at gmail.com  Sun Mar 21 23:58:14 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Mar 2010 15:58:14 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003211535t26c61b11t9aa2fc3cf20e7948@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
	<ca471dc21003211535t26c61b11t9aa2fc3cf20e7948@mail.gmail.com>
Message-ID: <7FBF6CE1-9132-47BF-8397-76EFD6C63712@gmail.com>


On Mar 21, 2010, at 3:35 PM, Guido van Rossum wrote:
> 
>> It seems to me that Decimals and floats should be considered at
>> the same level (i.e. both implement Real).
> 
> Agreed, but doesn't help. (Except against the idea that Decimal goes
> on the "integer" side of Fraction, which is just wrong.)

Woohoo!  Glad you agree.
I was concerned that idea
was gathering a following.


> 
>> There is no need to embed a notion of "imperfect answer".
>> Numbers themselves are exact and many mixed operations
>> can be exact if the coercions go the right way.
> 
> Division cannot, in general (I consider floor division a bastard child
> of the integers). And for multiplication it seems that rounding at
> some point becomes necessary since the alternative would be to use
> infinite precision.

Perception here is probably dictated by the use case of the beholder :-)
In the accounting world, such operations are very common (part of
getting three apples for one dollar recorded as 0.33, 0.33, and 0.34
in the individual apple accounts).  Cost allocation tools use floor
division all the time.  Amortization computations always have to 
take into account that real payments and interest charges do not have 
fractional pennies (or sometimes fractional dollars) and have to 
readjust the amortization accordingly (usually with small roundings
to the interest charges and a small adjustment to the final payment).


> Call it by any name you want. We're looking at revising the hash
> function and allowing mixed operations between Decimal and float, with
> signal that warns about these operations. I see two open issues:
> whether mixed Decimal-float operations should return Decimal and
> float, and whether the warning about such operations should be on or
> off by default. My gut tells me that the signal should be off by
> default. My gut doesn't say much about whether the result should be
> float or Decimal, but I want the reasoning to reach a decision to be
> sound.

My vote is for:

    decimal + float --> decimal
   and to emit a warning by default.

Reasoning for coercion to decimal:

1) decimal + float can be done losslessly and meaningfully

2) decimal/float comparisons can give a useful and non-confusing result for:

             1.1 == Decimal('1.1')  

This needs to return False since the two values:

  a) have different hash values (under either proposal) and

  b) are in-fact, not equal

But, if the decimal were coerced to a binary float, the relation would return True:

    >>> float(decimal.Decimal('1.1')) == 1.1
    True

This is bad.


Reasoning for emitting a warning by default:

1) Real actual use cases for mixed decimal / float operations are rare.
2) But accidental mixed decimal / float is an easy mistake to make.
3) Implicit coercion hides the error.
4) A warning flag gives you a chance to catch your error.
5) A warning is educational (it makes sure that you understand
what your program is doing)
6). A warning is easily silenced either through a) the warnings module,
     b) setting a context flag in decimal, or c) by making the coercion explicit
     using Decimal.from_float().


Raymond




   


From steve at pearwood.info  Sun Mar 21 23:59:53 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 22 Mar 2010 09:59:53 +1100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<20100321185050.D9C051FDBF4@kimball.webabinitio.net>
	<B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
Message-ID: <201003220959.53306.steve@pearwood.info>

On Mon, 22 Mar 2010 06:31:57 am Raymond Hettinger wrote:
> I really like Guido's idea of a context flag to control whether
> mixing of decimal and binary floats will issue a warning.
> The default should be to issue the warning (because unless
> you know what you're doing, it is most likely an error).

When you say "warning", do you mean warning.warn(), or an exception?

I'd like to put in a vote for allowing naive users with low requirements 
for accuracy and precision to be able to type something like this in 
the interactive interpreter:

>>> Decimal(1) + 1.0

and get two (in whatever type is decided on) without having to change 
the context or deal with an exception.

Yes, this means that they may be surprised if they perform an operation 
which suffers from rounding errors, but that's no worse than what 
happens with floats.

If naive users are going to use the interpreter as a calculator, they're 
going to start off using floats and ints simply because they require 
less typing. My idea is to allow a gentle learning curve with Decimal 
(and Fraction) without scaring them off with exceptions or excessive 
warnings: a single warning per session would be okay, a warning after 
every operation would be excessive in my opinion, and exceptions by 
default would be right out.



-- 
Steven D'Aprano

From greg.ewing at canterbury.ac.nz  Mon Mar 22 00:12:25 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 22 Mar 2010 11:12:25 +1200
Subject: [Python-Dev] Decimal &amp; amp; lt; -&amp; amp; gt;
 float comparisons in py3k.
In-Reply-To: <201003220932.22259.steve@pearwood.info>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<4BA5C3E5.6080805@gmail.com> <4BA69409.4040209@canterbury.ac.nz>
	<201003220932.22259.steve@pearwood.info>
Message-ID: <4BA6A7D9.6030502@canterbury.ac.nz>

Steven D'Aprano wrote:

> Then they're in for a terrible, terrible disappointment. Rounding issues 
> don't go away because you're using Decimal instead of float,

No, but what I mean is that they prefer to be surprised in
unsurprising ways, so to speak.

Everyone knows that floating point numbers have limited
precision. What really surprises people is when *binary*
floating point numbers behave differently from the decimal
ones they're used to.

-- 
Greg

From raymond.hettinger at gmail.com  Mon Mar 22 00:11:13 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Mar 2010 16:11:13 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA6A2B2.6050207@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
Message-ID: <465AE465-278C-4627-B285-BACF62FB731A@gmail.com>


On Mar 21, 2010, at 3:50 PM, Greg Ewing wrote:

> Raymond Hettinger wrote:
> 
>> Since decimal also allows arbitrary sizes, all long ints can be
>> exactly represented (this was even one of the design goals
>> for the decimal module).
> 
> There may be something we need to clarify here. I've been
> imagining that the implicit conversions to Decimal that
> we're talking about would be done to whatever precision
> is set in the context. Am I wrong about that?

Yes.  The conversion to decimal is independent of the
current context.

The background docs for the decimal module makes its design
intention clear.   All numbers are exact.  Rounding and context
adjustments only apply to the *results (of operations.  

The module is designed that way.  Its background documents
confirm that viewpoint.  And its operations are designed to support
that world view (i.e. unary plus is an operation that can change a
value).  Also, we confirmed that point-of-view with the person
who wrote the spec. 


> Is the intention
> to always use enough digits to get an exact representation?

Yes.   That is in-fact what Decimal.fromfloat() does.

That agrees with the decimal constructor itself which is not context sensitive:

     >>> decimal.getcontext().prec = 5
     >>> decimal.Decimal('3.1415926535')    # Notice this value doesn't get rounded
     Decimal('3.1415926535')

     >>> decimal.Decimal('3.1415926535') + 0   # This result does get rounded.
     Decimal('3.1416')

    >>> # Also, rounding does not get applied during an equality check
    >>> decimal.getcontext().prec = 5
    >>> decimal.Decimal('3.1415926535') == decimal.Decimal('3.1416')
    False


Raymond


P.S.  Thanks for asking the question. 




From raymond.hettinger at gmail.com  Mon Mar 22 00:16:00 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Mar 2010 16:16:00 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <201003220959.53306.steve@pearwood.info>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<20100321185050.D9C051FDBF4@kimball.webabinitio.net>
	<B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
	<201003220959.53306.steve@pearwood.info>
Message-ID: <2042CE43-567E-43AB-A76B-23CE410F56DC@gmail.com>


On Mar 21, 2010, at 3:59 PM, Steven D'Aprano wrote:

> On Mon, 22 Mar 2010 06:31:57 am Raymond Hettinger wrote:
>> I really like Guido's idea of a context flag to control whether
>> mixing of decimal and binary floats will issue a warning.
>> The default should be to issue the warning (because unless
>> you know what you're doing, it is most likely an error).
> 
> When you say "warning", do you mean warning.warn(), or an exception?


I'm not sure I understand your question.  I did mean warnings.warn().
But that does raise a catchable exception or it can be suppressed
through the warnings module.  It should probably be set to 
warn no more than once.


Raymond

From guido at python.org  Mon Mar 22 02:24:26 2010
From: guido at python.org (Guido van Rossum)
Date: Sun, 21 Mar 2010 18:24:26 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <2042CE43-567E-43AB-A76B-23CE410F56DC@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<20100321185050.D9C051FDBF4@kimball.webabinitio.net>
	<B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com> 
	<201003220959.53306.steve@pearwood.info>
	<2042CE43-567E-43AB-A76B-23CE410F56DC@gmail.com>
Message-ID: <ca471dc21003211824q3bd518d2h8c303455eebc3007@mail.gmail.com>

On Sun, Mar 21, 2010 at 4:16 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 21, 2010, at 3:59 PM, Steven D'Aprano wrote:
>
>> On Mon, 22 Mar 2010 06:31:57 am Raymond Hettinger wrote:
>>> I really like Guido's idea of a context flag to control whether
>>> mixing of decimal and binary floats will issue a warning.
>>> The default should be to issue the warning (because unless
>>> you know what you're doing, it is most likely an error).
>>
>> When you say "warning", do you mean warning.warn(), or an exception?
>
>
> I'm not sure I understand your question. ?I did mean warnings.warn().
> But that does raise a catchable exception or it can be suppressed
> through the warnings module. ?It should probably be set to
> warn no more than once.

I would hope it could use whatever mechanism is already used for other
conditions in the decimal module such as Underflow, Inexact, Rounded
etc. But I have to admit I don't know exactly what those do. It
appears they can either raise an exception or call a handle() method
on the given exception. Are you thinking of putting the warn() call
inside that handle() method?

-- 
--Guido van Rossum (python.org/~guido)

From raymond.hettinger at gmail.com  Mon Mar 22 04:10:31 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 21 Mar 2010 20:10:31 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003211824q3bd518d2h8c303455eebc3007@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<20100321185050.D9C051FDBF4@kimball.webabinitio.net>
	<B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
	<201003220959.53306.steve@pearwood.info>
	<2042CE43-567E-43AB-A76B-23CE410F56DC@gmail.com>
	<ca471dc21003211824q3bd518d2h8c303455eebc3007@mail.gmail.com>
Message-ID: <9C6B548B-E117-4FA9-9023-915E66D9C22F@gmail.com>


On Mar 21, 2010, at 6:24 PM, Guido van Rossum wrote:

> On Sun, Mar 21, 2010 at 4:16 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>> 
>> On Mar 21, 2010, at 3:59 PM, Steven D'Aprano wrote:
>> 
>>> On Mon, 22 Mar 2010 06:31:57 am Raymond Hettinger wrote:
>>>> I really like Guido's idea of a context flag to control whether
>>>> mixing of decimal and binary floats will issue a warning.
>>>> The default should be to issue the warning (because unless
>>>> you know what you're doing, it is most likely an error).
>>> 
>>> When you say "warning", do you mean warning.warn(), or an exception?
>> 
>> 
>> I'm not sure I understand your question.  I did mean warnings.warn().
>> But that does raise a catchable exception or it can be suppressed
>> through the warnings module.  It should probably be set to
>> warn no more than once.
> 
> I would hope it could use whatever mechanism is already used for other
> conditions in the decimal module such as Underflow, Inexact, Rounded
> etc. But I have to admit I don't know exactly what those do. It
> appears they can either raise an exception or call a handle() method
> on the given exception. Are you thinking of putting the warn() call
> inside that handle() method?

Yes.


Raymond

From rhamph at gmail.com  Mon Mar 22 06:51:47 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Sun, 21 Mar 2010 23:51:47 -0600
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <201003220959.53306.steve@pearwood.info>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<20100321185050.D9C051FDBF4@kimball.webabinitio.net>
	<B147B2DE-ADBC-49B3-9C39-349136B9AB84@gmail.com>
	<201003220959.53306.steve@pearwood.info>
Message-ID: <aac2c7cb1003212251s2232164p3f5318348a53444a@mail.gmail.com>

On Sun, Mar 21, 2010 at 16:59, Steven D'Aprano <steve at pearwood.info> wrote:
> If naive users are going to use the interpreter as a calculator, they're
> going to start off using floats and ints simply because they require
> less typing. My idea is to allow a gentle learning curve with Decimal
> (and Fraction) without scaring them off with exceptions or excessive
> warnings: a single warning per session would be okay, a warning after
> every operation would be excessive in my opinion, and exceptions by
> default would be right out.

That strikes me as a passive-aggressive way of saying we tolerate it
for interactive use, but don't you dare mix them for real programs.

A warning should be regarded as a bug in real programs ? unless it's a
transitional measure ? so it might as well be an exception.  Don't
guess and all that.


-- 
Adam Olsen, aka Rhamphoryncus

From dickinsm at gmail.com  Mon Mar 22 10:23:00 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Mon, 22 Mar 2010 09:23:00 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA6A2B2.6050207@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
Message-ID: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>

On Sun, Mar 21, 2010 at 10:50 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Raymond Hettinger wrote:
>
>> Since decimal also allows arbitrary sizes, all long ints can be
>> exactly represented (this was even one of the design goals
>> for the decimal module).
>
> There may be something we need to clarify here. I've been
> imagining that the implicit conversions to Decimal that
> we're talking about would be done to whatever precision
> is set in the context. Am I wrong about that? Is the intention
> to always use enough digits to get an exact representation?

I've been thinking about this, too.

Currently, Decimal + integer -> Decimal converts the integer
losslessly, with no reference to the Decimal context.

But the Fraction type is going to mess this up:  for Decimal +
Fraction ->  Decimal, I don't see any other sensible option than to
convert the Fraction using the current context, since lossless
conversion isn't generally possible.

So with the above two conventions, we'd potentially end up with
Decimal('1.23') + 314159 giving a different result from
Decimal('1.23') + Fraction(314159, 1)  (depending on the context).

It may be that we should reconsider Decimal + int interactions, making
the implicit int->Decimal conversion lossy.  Decimal would then match
the way that float behaves:  float + int and float + Fraction both do
a lossy conversion of the non-Fraction argument to Fraction.  But then
we're changing established behaviour of the Decimal module, which
could cause problems for existing users.

Note that comparisons are a separate issue:  those always need to be
done exactly (at least for equality, and once you're doing it for
equality it makes sense to make the other comaprisons exact as well),
else the rule that x == y implies hash(x) == hash(y) would become
untenable.  Again, this is the pattern that already exists for
int<->float and Fraction<->float interactions: comparisons are exact,
but arithmetic operations involve a lossy conversion.

Mark

From ncoghlan at gmail.com  Mon Mar 22 11:32:35 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 Mar 2010 20:32:35 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>	<loom.20100320T160613-470@post.gmane.org>	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>	<4BA55858.2000800@canterbury.ac.nz>	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>	<4BA5AA34.8050709@canterbury.ac.nz>	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
Message-ID: <4BA74743.90204@gmail.com>

Mark Dickinson wrote:
> But the Fraction type is going to mess this up:  for Decimal +
> Fraction ->  Decimal, I don't see any other sensible option than to
> convert the Fraction using the current context, since lossless
> conversion isn't generally possible.

Be able to duck this question was precisely why I put Decimal to the
left of Fraction in my pragmatic tower, btw. There are all sorts of good
reasons why that order was mathematically wrong (Real vs Rational,
representability of NaN/Inf, etc), but from the point of view of
providing clearly defined behaviour for an atypical operation, it should
be a lot easier.

Cheers,
Nick.
-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon Mar 22 11:41:10 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 Mar 2010 20:41:10 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA6A0AD.7090207@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<4BA57C5A.9060403@gmail.com>	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
	<4BA6A0AD.7090207@canterbury.ac.nz>
Message-ID: <4BA74946.9060302@gmail.com>

Greg Ewing wrote:
> Raymond Hettinger wrote:
>>
>> Remember, the notion of inexactness is a taint,
>> not an intrinsic property of a type.  Even the Scheme numeric
>> tower recognizes this.  LIkewise, the decimal specification also
>> spells-out this notion as basic to its design.
> 
> I'm not sure it really does, otherwise every decimal
> value would have a flag indicating whether it was
> tainted with inexactness, and this flag would propagate
> through calculations.

http://docs.python.org/library/decimal.html#decimal.Inexact

(Part of the thread context rather than the individual decimal values,
but if you use it properly it tells you whenever an inexact operation
has occurred in the current thread)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon Mar 22 11:51:33 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 Mar 2010 20:51:33 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <7FBF6CE1-9132-47BF-8397-76EFD6C63712@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
	<ca471dc21003211535t26c61b11t9aa2fc3cf20e7948@mail.gmail.com>
	<7FBF6CE1-9132-47BF-8397-76EFD6C63712@gmail.com>
Message-ID: <4BA74BB5.8030807@gmail.com>

Raymond Hettinger wrote:
> On Mar 21, 2010, at 3:35 PM, Guido van Rossum wrote:
>>> It seems to me that Decimals and floats should be considered at
>>> the same level (i.e. both implement Real).
>> Agreed, but doesn't help. (Except against the idea that Decimal goes
>> on the "integer" side of Fraction, which is just wrong.)
> 
> Woohoo!  Glad you agree.
> I was concerned that idea
> was gathering a following.

Heck no, it was just a random late night thought from me, and even I
thought it was a somewhat dubious idea. I don't mind at all that it
since has been knocked soundly (and deservedly) on the head :)

> Reasoning for emitting a warning by default:
> 
> 1) Real actual use cases for mixed decimal / float operations are rare.
> 2) But accidental mixed decimal / float is an easy mistake to make.
> 3) Implicit coercion hides the error.
> 4) A warning flag gives you a chance to catch your error.
> 5) A warning is educational (it makes sure that you understand
> what your program is doing)
> 6). A warning is easily silenced either through a) the warnings module,
>      b) setting a context flag in decimal, or c) by making the coercion explicit
>      using Decimal.from_float().

I'll add another one to that list:

(7) For backwards compatible changes, it is easy to go from exception ->
warning -> no warning (if we later decide to take that second step).
Going from exception -> no warning -> warning (if we were to change our
minds the other way) is a lot less user-friendly.

(I was going to try to play devil's advocate and argue in favour of
float results and/or no warning, but I got nuthin' - Raymond's points
made too much sense to me).

A warning is nice in that you can mix decimals and floats at the
interpreter prompt with a single warning per session, but the warning
can still act as a pointer into the weird and wonderful world of binary
vs decimal floating point.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From barry at python.org  Mon Mar 22 14:56:04 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 22 Mar 2010 09:56:04 -0400
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
Message-ID: <20100322095604.3371039e@heresy>

I have a pretty good start on PEP 3147 implementation [1], but I've
encountered a situation that I'd like to get some feedback on.  Here's the
test case illustrating the problem.  From test_import.py:

    def test_writable_directory(self):
        # The umask is not conducive to creating a writable __pycache__
        # directory.
        with umask(0o222):
            __import__(TESTFN)
        self.assertTrue(os.path.exists('__pycache__'))
        self.assertTrue(os.path.exists(os.path.join(
            '__pycache__', '{}.{}.pyc'.format(TESTFN, self.tag))))

The __pycache__ directory does not exist before the import, and the import
machinery creates the directory, but the umask leaves the directory unwritable
by anybody.  So of course when the import machinery goes to write the .pyc
file inside __pycache__, it fails.  This does not cause an ImportError though,
just like if today the package directory were unwritable.

This might be different than today's situation though because once the
unwritable __pycache__ directory is created, nothing is going to change that
without explicit user interaction, and that might be difficult after the
fact.

I'm not sure what the right answer is.  Some possible choices:

* Tough luck
* Force the umask so that the directory is writable, but then the question is,
  by whom?  ugo+w or something less?
* Copy the permissions from the parent directory and ignore umask
* Raise an exception or refuse to create __pycache__ if it's not writable
  (again, by whom?)

Or maybe you have a better idea?  What's the equivalent situation on Windows
and how would things work there?

-Barry

[1] https://edge.launchpad.net/~barry/python/pep3147

P.S. I'm down to only 8 unit test failures.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/80ada87a/attachment.pgp>

From solipsis at pitrou.net  Mon Mar 22 15:30:12 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Mar 2010 14:30:12 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?=
	=?utf-8?q?s_and_umask?=
References: <20100322095604.3371039e@heresy>
Message-ID: <loom.20100322T150829-761@post.gmane.org>

Barry Warsaw <barry <at> python.org> writes:
> 
> I'm not sure what the right answer is.  Some possible choices:
> 
> * Tough luck
> * Force the umask so that the directory is writable, but then the question is,
>   by whom?  ugo+w or something less?
> * Copy the permissions from the parent directory and ignore umask
> * Raise an exception or refuse to create __pycache__ if it's not writable
>   (again, by whom?)

Welcome to a problem PHP has known for years.

The problem with solution 3 is that a non-root user can't change the owner of a
directory, therefore copying the permissions doesn't ensure the directory will
be writable by the right users.

Solution 2 (chmod 777) is obviously too laxist. Anyone could put fake bytecode
files in a root-owned application. Less laxist chmods have the same problem as
solution 3, and can still be insecure.

I think solution 4 would be the best compromise. In the face of ambiguity,
refuse to guess. But, rather than :
    refuse to create __pycache__ if it's not writable,
the test should be :
    refuse to create __pycache__ if I can't create it with the same ownership
and permissions as the parent directory (and raise an ImportWarning).


In light of this issue, I'm -0.5 on __pycache__ becoming the default caching
mechanism. The directory ownership/permissions issue is too much of a mess,
especially for Web applications (think __pycache__ files created by the Apache
user).

__pycache__ should only be created if explicitly activated (for example by
distutils when installing stuff). Otherwise, if not present, the "legacy"
mechanism (writing an untagged pyc file along the py file) should be used.

Actually, __pycache__ creation doesn't have to be part of the import mechanism.
It can be part of distutils instead (or whatever third-party tool such as
distribute, or distro-specific packaging scripts). This would relax complexity
of core Python a bit.

Regards

Antoine.



From barry at python.org  Mon Mar 22 16:21:08 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 22 Mar 2010 11:21:08 -0400
Subject: [Python-Dev] __file__ and bytecode-only
In-Reply-To: <4B9F52FB.2090701@gmail.com>
References: <fb6fbf561003031637y70d48a56w9f21b5bf99e4d76f@mail.gmail.com>
	<20100303204923.6d597f03@heresy.wooz.org>
	<4B8FB6D2.4040501@gmail.com>
	<20100312145312.1f0a0aa0@heresy.wooz.org>
	<4B9AC4CA.3070302@voidspace.org.uk>
	<20100312175657.287bb456@heresy.wooz.org>
	<4B9B9E65.4080902@gmail.com>
	<20100315111654.1aa7acf5@heresy.wooz.org>
	<4B9F52FB.2090701@gmail.com>
Message-ID: <20100322112108.68d47556@heresy>

On Mar 16, 2010, at 07:44 PM, Nick Coghlan wrote:

>In Python 3.1, *invoking* py_compile.compile() will create 2.x style
>bytecode. Similarly, when force==False, compileall.compile_dir() and
>compileall.compile_path() will check for 2.x style bytecode in order to
>decide whether or not to compile the module.
>
>The question for 3.2 is what bytecode layout py_compile.compile() should
>generate. For the precompile-a-system-library use case it should clearly
>generate a PEP 3147 layout and this probably makes sense as the default
>behaviour in 3.2.
>
>However, for production of bytecode-only packages, it would be
>convenient to be able to explicitly invoke the 2.x style behaviour
>without having to specify the target filename explicitly using the
>'cfile' parameter (which isn't exposed at the compileall layer anyway).

My working branch modifies py_compile to produce PEP 3147 layout by default.
I agree that it should support traditional pyc output as an option, and
compileall should support this as well.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/c94658c9/attachment.pgp>

From martin at v.loewis.de  Mon Mar 22 16:52:03 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 22 Mar 2010 16:52:03 +0100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <20100322095604.3371039e@heresy>
References: <20100322095604.3371039e@heresy>
Message-ID: <4BA79223.1030806@v.loewis.de>

> Or maybe you have a better idea?  What's the equivalent situation on Windows
> and how would things work there?

On Windows, this problem is easy: create the directory with no
specification of an ACL, and it will (usually) inherit the ACL of the
parent directory. IOW, the same users will have write access to
__pycache__ as to the parent directory.

It is possible to defeat this machinery, by setting ACL entries on the
parent that explicitly don't get inherited, or that only apply to
subdirectories. In this case, I would say "tough luck": this is what the
users requested, so it may have a reason.

Regards,
Martin

From martin at v.loewis.de  Mon Mar 22 16:58:47 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 22 Mar 2010 16:58:47 +0100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <loom.20100322T150829-761@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
Message-ID: <4BA793B7.3060209@v.loewis.de>

>     refuse to create __pycache__ if I can't create it with the same ownership
> and permissions as the parent directory (and raise an ImportWarning).

I don't think an ImportWarning should be raised: AFAICT, we are not
raising one, either, when the pyc file cannot be created (and it may
well be the case that you don't have write permission at all, which
would trigger lots of ImportWarnings).

Regards,
Martin


From raymond.hettinger at gmail.com  Mon Mar 22 17:39:23 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 22 Mar 2010 09:39:23 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
Message-ID: <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>


On Mar 22, 2010, at 2:23 AM, Mark Dickinson wrote:
> Note that comparisons are a separate issue:  those always need to be
> done exactly (at least for equality, and once you're doing it for
> equality it makes sense to make the other comaprisons exact as well),
> else the rule that x == y implies hash(x) == hash(y) would become
> untenable.  Again, this is the pattern that already exists for
> int<->float and Fraction<->float interactions: comparisons are exact,
> but arithmetic operations involve a lossy conversion.

My instinct says that we're asking for trouble if comparisons have
different coercion rules than arithmetic operations.

That would suggest that we follow the chain of lossless conversions:

   Fraction + float --> Fraction
   Fraction + decimal --> Fraction
   Decimal + float --> Decimal

That way, arithmetic coercions match comparison coercions.
We preserve get Mark's fast, clean new universal hash function.
The equality/hash homomorphism is preserved.
And a decimal context is not needed for any of the coercions
(it is needed for the addition between two decimals once coercion
has happened, but that is completely normal for Decimal so there
are no surprises). 


Raymond



From guido at python.org  Mon Mar 22 18:00:34 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Mar 2010 09:00:34 -0800
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com> 
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com> 
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz> 
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz> 
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> 
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
Message-ID: <ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>

On Mon, Mar 22, 2010 at 8:39 AM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> My instinct says that we're asking for trouble if comparisons have
> different coercion rules than arithmetic operations.

Sorry, no, this is a station we passed long ago when float-long
comparison was fixed to do the comparison exactly (but other mixed ops
continue to convert the long to a float). Otherwise you'll get a
violation of "if x < y and y < z then it follows that x < z".

(FWIW, my instinct used to be the same as yours, but I got better. :-)

> That would suggest that we follow the chain of lossless conversions:
>
> ? Fraction + float --> Fraction

Besides the obvious backwards compatibility problem, I also just
really think that this would be a mistake. Fraction corresponds to
Rational in PEP 3141's numeric tower and float corresponds to Real (as
does Decimal). The convention is to only move to the more general type
in the numeric tower.

Now we all know that floats (being a fixed number of bits) are
actually representable as rationals as well, but *that's not how we
think about them!*

> ? Fraction + decimal --> Fraction

This also violates the natural direction in the numeric tower.

> ? Decimal + float --> Decimal

If everybody associated with the Decimal implementation wants this I
won't stop you; as I repeatedly said my intuition about this one (as
opposed to the other two above) is very weak.

> That way, arithmetic coercions match comparison coercions.

Which is not a requirement.

> We preserve get Mark's fast, clean new universal hash function.

That hash function (or another that matches the requirements) is a given.

> The equality/hash homomorphism is preserved.

That is also a given (except I don't know if homomorphism is the right
word to use).

> And a decimal context is not needed for any of the coercions
> (it is needed for the addition between two decimals once coercion
> has happened, but that is completely normal for Decimal so there
> are no surprises).

Doesn't strike me as a deciding argument.

-- 
--Guido van Rossum (python.org/~guido)

From solipsis at pitrou.net  Mon Mar 22 18:14:29 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Mar 2010 17:14:29 +0000 (UTC)
Subject: [Python-Dev] __pycache__ creation
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
Message-ID: <ho88hk$n52$1@dough.gmane.org>


Oh, and by the way, there can be a race condition between __pycache__ 
creation and deletion (if it fails the test), where an attacker can stuff 
a hostile pyc file in the directory in the meantime (and the deletion 
then fails because the directory isn't empty).

IMO, all these issues militate for putting __pycache__ creation out of 
the interpreter core, and in the hands of third-party package-time/
install-time tools (or distutils).



Le Mon, 22 Mar 2010 14:30:12 +0000, Antoine Pitrou a ?crit?:
> 
> __pycache__ should only be created if explicitly activated (for example
> by distutils when installing stuff). Otherwise, if not present, the
> "legacy" mechanism (writing an untagged pyc file along the py file)
> should be used.
> 
> Actually, __pycache__ creation doesn't have to be part of the import
> mechanism. It can be part of distutils instead (or whatever third-party
> tool such as distribute, or distro-specific packaging scripts). This
> would relax complexity of core Python a bit.
> 
> Regards
> 
> Antoine.



From raymond.hettinger at gmail.com  Mon Mar 22 18:56:41 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 22 Mar 2010 10:56:41 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
Message-ID: <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>


On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote:
> 
>>   Decimal + float --> Decimal
> 
> If everybody associated with the Decimal implementation wants this I
> won't stop you; as I repeatedly said my intuition about this one (as
> opposed to the other two above) is very weak.

That's my vote.
I believe Nick chimed-in in agreement.
Mark, do you concur?


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/9fb8d175/attachment.html>

From ijmorlan at uwaterloo.ca  Mon Mar 22 18:28:25 2010
From: ijmorlan at uwaterloo.ca (Isaac Morland)
Date: Mon, 22 Mar 2010 13:28:25 -0400 (EDT)
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <ho88hk$n52$1@dough.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
Message-ID: <Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>

On Mon, 22 Mar 2010, Antoine Pitrou wrote:

> Oh, and by the way, there can be a race condition between __pycache__
> creation and deletion (if it fails the test), where an attacker can stuff
> a hostile pyc file in the directory in the meantime (and the deletion
> then fails because the directory isn't empty).

Would creating it under a different name and then renaming help with this?

> IMO, all these issues militate for putting __pycache__ creation out of
> the interpreter core, and in the hands of third-party package-time/
> install-time tools (or distutils).

Speaking only for myself, but really for anybody who likes tidy source 
directories, I hope some version of the __pycache__ proposal becomes part 
of standard Python, by which I ideally mean it's enabled by default but if 
that is just not a good idea then at most it should be required to set a 
command-line option to get this feature.

If I just want to write some .py code and run it, I don't see why my 
directories need to clutter up with .pyc files.  I've previously suggested 
a Python version of javac's -d ("destination directory") option, but 
putting all the .pyc's in a __pycache__ directory per source directory is 
good enough to make me happy (and is Pythonically simple, in my opinion).

Isaac Morland			CSCF Web Guru
DC 2554C, x36650		WWW Software Specialist

From chambon.pascal at wanadoo.fr  Mon Mar 22 19:06:44 2010
From: chambon.pascal at wanadoo.fr (Pascal Chambon)
Date: Mon, 22 Mar 2010 19:06:44 +0100
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA576AF.2050208@voidspace.org.uk>
References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk>
	<4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk>
Message-ID: <4BA7B1B4.4010000@wanadoo.fr>

Michael Foord a ?crit :
> On 20/03/2010 12:00, Pascal Chambon wrote:
>>
>>
>> But the point which for me is still unclear, is : does the default 
>> implementation of __getattribute__ (the one of "object") call 
>> __getattr__ by himself, or does it rely on its caller for that, by 
>> raising an AttributeError ? For Python2, it's blatantly the latter 
>> case which is favoured, but since it looks like an implementation 
>> detail at the moment, I propose we settle it (and document it) once 
>> for all.
>
> Ah right, my apologies. So it is still documented behaviour - 
> __getattr__ is obviously called by the Python runtime and not by 
> __getattribute__. (It isn't just by getattr as the same behaviour is 
> shown when doing a normal attribute lookup and not via the getattr 
> function.)
>
I really don't see the docs you're referring to ; until I tested myself, 
I think I had no obvious reasons to guess that __getattribute__ relied 
on the upper level caller instead of finishing the hard job himself.


Nick Coghlan a ?crit :
> Michael Foord wrote:
>   
>> Well, the documentation you pointed to specifies that __getattr__ will
>> be called if __getattribute__ raises an AttributeError, it just doesn't
>> specify that it is done by object.__getattribute__ (which it isn't).
>>     
>
> And as for why not: because __getattribute__ implementations need to be
> able to call object.__getattribute__ without triggering the fallback
> behaviour.
>
> Cheers,
> Nick.
>
>   
I guess there are cases in which it is beneficial indeed.

>
> Michael Foord wrote:
>
>> Well, the documentation you pointed to specifies that __getattr__ 
>> will be called if __getattribute__ raises an AttributeError, it just 
>> doesn't specify that it is done by object.__getattribute__ (which it 
>> isn't).
>
> If __getattribute__ raises an exception, it won't get a chance to
> do anything else, so something outside of __getattribute__ must
> catch the AttributeError and calling __getattr__. So I think the
> docs *are* specifying the behaviour here, if only by implication.
>
I don't follow you there - in my mind, the default __getattribute__ 
could simply have wrapped all its operations inside soem kind of 
"try..catch AttributeError:" mechanism, and thus been able to fallback 
to __getattr__ in any way.


If I sum it up properly the semantic is :
-A.obj and getattr(A, "obj") are exactly the same
-They trigger the calling of __getattribute__ on the object (or it's 
python core equivalent)
-By default, this __getattribute__ browse the whole object hierarchy 
according to well known rules (__dict__, type,  type's ancestors..), 
handling descriptor protocols and the like. But it doesn't fallback to 
__getattr__ - it raises an AttributeError instead.
-getattr() falls back to __getattr__ if __getattribute__ fails
-customized __getattribute__ methods have the choice between calling 
__getattr__ by themselves, or delegating it to getattr() by raising an 
exception.

Wouldn't it be worth completing the doc with these point ? They really 
didn't seem obvious to me basically (even though, after analysis, some 
behaviours make more sense than others).
I might submit a patch.

regards,
Pascal


>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/c908270b/attachment-0001.html>

From solipsis at pitrou.net  Mon Mar 22 19:15:01 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Mar 2010 18:15:01 +0000 (UTC)
Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?=
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
Message-ID: <loom.20100322T191016-732@post.gmane.org>

Isaac Morland <ijmorlan <at> uwaterloo.ca> writes:
> 
> > IMO, all these issues militate for putting __pycache__ creation out of
> > the interpreter core, and in the hands of third-party package-time/
> > install-time tools (or distutils).
> 
> Speaking only for myself, but really for anybody who likes tidy source 
> directories, I hope some version of the __pycache__ proposal becomes part 
> of standard Python, by which I ideally mean it's enabled by default but if 
> that is just not a good idea then at most it should be required to set a 
> command-line option to get this feature.

This doesn't contradict by my proposal.

What I am proposing is that the creation of __pycache__ /directories/ be put
outside of the core. It can be part of distutils, or of a separate module, or
delegated to third-party tools. It could even be as simple as
"python -m compileall --pycache", if someone implements it.

Creation of the __pycache__ /contents/ (files inside the directory) would still
be part of core Python, but only if the directory exists and is writable by the
current process.

Regards

Antoine.



From alexander.belopolsky at gmail.com  Mon Mar 22 19:22:43 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 22 Mar 2010 14:22:43 -0400
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
Message-ID: <d38f5331003221122w2f7f9150s266ce22ace81a99e@mail.gmail.com>

On Mon, Mar 22, 2010 at 1:56 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote:
>
> ? Decimal + float --> Decimal
>
> If everybody associated with the Decimal implementation wants this I
> won't stop you; as I repeatedly said my intuition about this one (as
> opposed to the other two above) is very weak.
>
> That's my vote.

I've been lurking on this thread so far, but let me add my +1 to this
option.  My reasoning is that Decimal is a "better" model of Real than
float and mixed operations should not degrade the result.   "Better"
can mean different things to different people, but to me the tie
breaker is the support for contexts.  I would not want precision to
suddenly change in the middle of calculation I add 1.0 instead of 1.

This behavior will also be familiar to users of other "enhanced"
numeric types such as NumPy scalars.   Note that in the older Numeric,
it was the other way around, but after considerable discussion, the
behavior was changed.

From guido at python.org  Mon Mar 22 19:24:20 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Mar 2010 10:24:20 -0800
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <d38f5331003221122w2f7f9150s266ce22ace81a99e@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz> 
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz> 
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> 
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com> 
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<d38f5331003221122w2f7f9150s266ce22ace81a99e@mail.gmail.com>
Message-ID: <ca471dc21003221124p1668f21cx8820e6301aa50a38@mail.gmail.com>

On Mon, Mar 22, 2010 at 10:22 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Mon, Mar 22, 2010 at 1:56 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>>
>> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote:
>>
>> ? Decimal + float --> Decimal
>>
>> If everybody associated with the Decimal implementation wants this I
>> won't stop you; as I repeatedly said my intuition about this one (as
>> opposed to the other two above) is very weak.
>>
>> That's my vote.
>
> I've been lurking on this thread so far, but let me add my +1 to this
> option. ?My reasoning is that Decimal is a "better" model of Real than
> float and mixed operations should not degrade the result. ? "Better"
> can mean different things to different people, but to me the tie
> breaker is the support for contexts. ?I would not want precision to
> suddenly change in the middle of calculation I add 1.0 instead of 1.
>
> This behavior will also be familiar to users of other "enhanced"
> numeric types such as NumPy scalars. ? Note that in the older Numeric,
> it was the other way around, but after considerable discussion, the
> behavior was changed.

Thanks, "better" is a great way to express this.

-- 
--Guido van Rossum (python.org/~guido)

From dickinsm at gmail.com  Mon Mar 22 19:26:10 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Mon, 22 Mar 2010 18:26:10 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
Message-ID: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>

On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote:
>
> ? Decimal + float --> Decimal
>
> If everybody associated with the Decimal implementation wants this I
> won't stop you; as I repeatedly said my intuition about this one (as
> opposed to the other two above) is very weak.
>
> That's my vote.
> I believe Nick chimed-in in agreement.
> Mark, do you concur?

Yes;  Decimal + float -> Decimal works for me; the greater flexibility
of the Decimal type is the main plus for me.  I don't think my own
intuition is particularly strong here, either.  It would be
interesting to hear from major users of the decimal module (I have to
confess to not actually using it myself very much).

I argued earlier for Decimal + float -> Decimal on the basis that the
float->Decimal conversion can be done exactly;  but having thought
about Decimal + Fraction it's no longer clear to me that this makes
sense.  Having Decimal + float -> Decimal round the float using the
current Decimal context still seems like a reasonable option.

Just for the record, I'd also prefer Decimal + Fraction -> Decimal.

I don't want to let the abstractions of the numeric tower get in the
way of the practicalities:  we should modify the abstractions if
necessary!  In particular, it's not clear to me that all numeric types
have to be comparable with each other.  It might make sense for
Decimal + complex mixed-type operations to be disallowed, for example.

Mark

From raymond.hettinger at gmail.com  Mon Mar 22 19:34:51 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 22 Mar 2010 11:34:51 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
Message-ID: <23CC71BC-B7B4-4D51-B1E4-D70B2941391F@gmail.com>


On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote:

> On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>> 
>> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote:
>> 
>>   Decimal + float --> Decimal
>> 
>> If everybody associated with the Decimal implementation wants this I
>> won't stop you; as I repeatedly said my intuition about this one (as
>> opposed to the other two above) is very weak.
>> 
>> That's my vote.
>> I believe Nick chimed-in in agreement.
>> Mark, do you concur?
> 
> Yes;  Decimal + float -> Decimal works for me; the greater flexibility
> of the Decimal type is the main plus for me.



> On Mar 22, 2010, at 11:22 AM, Alexander Belopolsky wrote:
> 
>> I've been lurking on this thread so far, but let me add my +1 to this
>> option.


On Mar 22, 2010, at 11:24 AM, Guido van Rossum wrote:
>> 
> 
> Thanks, "better" is a great way to express this.


OMG ;-)   

It looks like there is a consensus on:  Decimal + float --> Decimal

We have progress.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/ff255cbf/attachment-0001.html>

From rhamph at gmail.com  Mon Mar 22 19:45:41 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Mon, 22 Mar 2010 12:45:41 -0600
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
Message-ID: <aac2c7cb1003221145v29e1bf44m6550e2ef8e504b51@mail.gmail.com>

On Mon, Mar 22, 2010 at 12:26, Mark Dickinson <dickinsm at gmail.com> wrote:
> I don't want to let the abstractions of the numeric tower get in the
> way of the practicalities: ?we should modify the abstractions if
> necessary! ?In particular, it's not clear to me that all numeric types
> have to be comparable with each other. ?It might make sense for
> Decimal + complex mixed-type operations to be disallowed, for example.

Only until a needy user breaks out the duck tape and builds a
ComplexDecimal type. ;)

The nature of the beast is more will be added on later.  So long as
Decimal == complex works right I don't see a problem with Decimal +
complex raising an exception.


-- 
Adam Olsen, aka Rhamphoryncus

From guido at python.org  Mon Mar 22 19:54:11 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Mar 2010 10:54:11 -0800
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz> 
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz> 
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> 
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com> 
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
Message-ID: <ca471dc21003221154i69670c8duab9128618f48ee65@mail.gmail.com>

On Mon, Mar 22, 2010 at 10:26 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Mon, Mar 22, 2010 at 5:56 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>>
>> On Mar 22, 2010, at 10:00 AM, Guido van Rossum wrote:
>>
>> ? Decimal + float --> Decimal
>>
>> If everybody associated with the Decimal implementation wants this I
>> won't stop you; as I repeatedly said my intuition about this one (as
>> opposed to the other two above) is very weak.
>>
>> That's my vote.
>> I believe Nick chimed-in in agreement.
>> Mark, do you concur?
>
> Yes; ?Decimal + float -> Decimal works for me; the greater flexibility
> of the Decimal type is the main plus for me. ?I don't think my own
> intuition is particularly strong here, either. ?It would be
> interesting to hear from major users of the decimal module (I have to
> confess to not actually using it myself very much).

You're unlikely to hear from them here... :-( Though maybe Raymond qualifies.

> I argued earlier for Decimal + float -> Decimal on the basis that the
> float->Decimal conversion can be done exactly; ?but having thought
> about Decimal + Fraction it's no longer clear to me that this makes
> sense. ?Having Decimal + float -> Decimal round the float using the
> current Decimal context still seems like a reasonable option.

So now we have a second-order decision to make -- whether
Decimal+float should convert the float to Decimal using the current
context's precision, or do it exactly. I think we should just follow
Decimal.from_float() here, which AFAIK does the conversion exactly --
the operation will already round properly if needed, and I don't think
it is a good idea to round twice in the same operation. (Even though
this is what long+float does -- but the situation is different there
since float has no variable precision.)

> Just for the record, I'd also prefer Decimal + Fraction -> Decimal.

I think that is pretty much uncontested. Though there are second-order
details to decide here too: Fraction + float converts the Fraction to
a float first and then just adds the floats; but but Fraction /
Decimal cannot always be computed exactly as a Decimal so here I'd
suggest rounding first because we have to in some cases (it's better
to be consistent, except for comparisons, which should always be done
exactly if at all possible).

> I don't want to let the abstractions of the numeric tower get in the
> way of the practicalities: ?we should modify the abstractions if
> necessary! ?In particular, it's not clear to me that all numeric types
> have to be comparable with each other. ?It might make sense for
> Decimal + complex mixed-type operations to be disallowed, for example.

One of the reasons for having a numeric tower is actually that unless
you do at least comparisons right, you get into inconsistencies like a
== b, b ==c, a != c (and similar for orderings). Of course complex
doesn't play along with the orderings, but it does play along with ==.

Real and Complex are distinct stations in the numeric tower, and the
tower defines mixed Real-Complex operations as returning Complex
results, but it allows raising exceptions as well. Since Decimal(-1)
** Decimal('0.5') already raises an exception instead of returning a
complex number, I'm fine with following that example -- complex
numbers are definitely a specialty area.

PS. PEP 3141 defines some miscellaneous operations that any number in
the numeric tower ought to implement, e.g. x.imag, x.real and
x.conjugate().

-- 
--Guido van Rossum (python.org/~guido)

From raymond.hettinger at gmail.com  Mon Mar 22 20:00:37 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 22 Mar 2010 12:00:37 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
Message-ID: <C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>


On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote:
> 
> Just for the record, I'd also prefer Decimal + Fraction -> Decimal.


Guido was persuasive on why float + Fraction --> float,
so this makes sense for the same reasons.

For the implementation, is there a way to avoid the double rounding
in   myfloat + myfrac.numerator / myfrac.denominator?

Perhaps translate it to:

      f = Fractions.from_decimal(myfloat) + myfract   # Lossless, exact addition
      return f.numerator / f.denominator           # Only one decimal context rounding applied.

Elsewhere in the decimal module, there is a fundamental notion
that numbers are exact and only the results of operations are
rounded.   For example, it is possible in decimal to add together
two high precision numbers but do so in a low precision context.


Raymond


From rrr at ronadam.com  Mon Mar 22 20:02:42 2010
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 22 Mar 2010 14:02:42 -0500
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <loom.20100322T191016-732@post.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
Message-ID: <ho8esj$qnd$1@dough.gmane.org>



Antoine Pitrou wrote:
> Isaac Morland <ijmorlan <at> uwaterloo.ca> writes:
>>> IMO, all these issues militate for putting __pycache__ creation out of
>>> the interpreter core, and in the hands of third-party package-time/
>>> install-time tools (or distutils).
>> Speaking only for myself, but really for anybody who likes tidy source 
>> directories, I hope some version of the __pycache__ proposal becomes part 
>> of standard Python, by which I ideally mean it's enabled by default but if 
>> that is just not a good idea then at most it should be required to set a 
>> command-line option to get this feature.
> 
> This doesn't contradict by my proposal.
> 
> What I am proposing is that the creation of __pycache__ /directories/ be put
> outside of the core. It can be part of distutils, or of a separate module, or
> delegated to third-party tools. It could even be as simple as
> "python -m compileall --pycache", if someone implements it.
> 
> Creation of the __pycache__ /contents/ (files inside the directory) would still
> be part of core Python, but only if the directory exists and is writable by the
> current process.

+1

If I understand correctly, we would have the current mode as the default, 
and can trigger __pycache__ behavior simply by manually creating a 
__pycache__ directory and deleting any byte-code files in the 
module/program directory.

I like this, it is easy to understand and can be used without messing with 
flags or environment variables.

Ron










From guido at python.org  Mon Mar 22 20:03:04 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Mar 2010 11:03:04 -0800
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com> 
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> 
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com> 
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> 
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
Message-ID: <ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>

On Mon, Mar 22, 2010 at 11:00 AM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote:
>>
>> Just for the record, I'd also prefer Decimal + Fraction -> Decimal.
>
>
> Guido was persuasive on why float + Fraction --> float,
> so this makes sense for the same reasons.
>
> For the implementation, is there a way to avoid the double rounding
> in ? myfloat + myfrac.numerator / myfrac.denominator?
>
> Perhaps translate it to:
>
> ? ? ?f = Fractions.from_decimal(myfloat) + myfract ? # Lossless, exact addition
> ? ? ?return f.numerator / f.denominator ? ? ? ? ? # Only one decimal context rounding applied.
>
> Elsewhere in the decimal module, there is a fundamental notion
> that numbers are exact and only the results of operations are
> rounded. ? For example, it is possible in decimal to add together
> two high precision numbers but do so in a low precision context.

Works for me.

-- 
--Guido van Rossum (python.org/~guido)

From raymond.hettinger at gmail.com  Mon Mar 22 20:03:55 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 22 Mar 2010 12:03:55 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003221154i69670c8duab9128618f48ee65@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<ca471dc21003221154i69670c8duab9128618f48ee65@mail.gmail.com>
Message-ID: <8909CA79-4204-4F0C-A476-023B20C74CA4@gmail.com>


On Mar 22, 2010, at 11:54 AM, Guido van Rossum wrote:
> 
> So now we have a second-order decision to make -- whether
> Decimal+float should convert the float to Decimal using the current
> context's precision, or do it exactly. I think we should just follow
> Decimal.from_float() here, which AFAIK does the conversion exactly --
> the operation will already round properly if needed, and I don't think
> it is a good idea to round twice in the same operation. (Even though
> this is what long+float does -- but the situation is different there
> since float has no variable precision.)

I concur.  That is consistent with the basic design of the decimal
module which treats inputs as exact and only applies rounding
to the results of operations.

FWIW, Mark and I have both been bitten severely by doubling
rounding in the world of binary floats.  Avoiding double rounding
is a darned good idea.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/a7d01aaa/attachment.html>

From stefan at bytereef.org  Mon Mar 22 20:26:51 2010
From: stefan at bytereef.org (Stefan Krah)
Date: Mon, 22 Mar 2010 20:26:51 +0100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
References: <aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
Message-ID: <20100322192651.GA21459@yoda.bytereef.org>

Guido van Rossum <guido at python.org> wrote:
> > ? Decimal + float --> Decimal
> 
> If everybody associated with the Decimal implementation wants this I
> won't stop you; as I repeatedly said my intuition about this one (as
> opposed to the other two above) is very weak.

I've been following the discussion only passively so far, but I also
think this is the most logical solution.


Stefan Krah




From dickinsm at gmail.com  Mon Mar 22 20:32:54 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Mon, 22 Mar 2010 19:32:54 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
Message-ID: <5c6f2a5d1003221232l69fe8eenf0778d9c4a5e436b@mail.gmail.com>

On Mon, Mar 22, 2010 at 7:00 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote:
>>
>> Just for the record, I'd also prefer Decimal + Fraction -> Decimal.
>
>
> Guido was persuasive on why float + Fraction --> float,
> so this makes sense for the same reasons.
>
> For the implementation, is there a way to avoid the double rounding
> in ? myfloat + myfrac.numerator / myfrac.denominator?
>
> Perhaps translate it to:
>
> ? ? ?f = Fractions.from_decimal(myfloat) + myfract ? # Lossless, exact addition
> ? ? ?return f.numerator / f.denominator ? ? ? ? ? # Only one decimal context rounding applied.

I'm not sure;  I see a couple of problems with this.  (1) It's fine
for the basic arithmetic operations that Fraction already supports,
but what about all the other Decimal methods that don't have Fraction
counterparts.  (2) It bothers me that the Decimal -> Fraction
conversion can be inefficient in cases like Decimal('1e<large>');
currently, all Decimal operations are relatively efficient (no
exponential-time behaviour) provided only that the coefficients don't
get too large;  large exponents aren't a problem.

I think getting this to work would involve a lot of extra code and
significant 'cleverness'.  I'd prefer the simple-to-implement and
simple-to-explain option of rounding the Fraction before performing
the operation, even if this means that the whole operation involves
two rounding operations.  It's not so different from what currently
happens for Fraction+float, or even int+float.

Mark

From raymond.hettinger at gmail.com  Mon Mar 22 20:36:18 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 22 Mar 2010 12:36:18 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>
Message-ID: <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com>

One other thought.

The Decimal constructor should now accept floats as a possible input type.
Formerly, we separated that out to Decimal.from_float() because
decimals weren't interoperable with floats.

This will put decimal and float back on equal footing so that we have both:
    float(some_decimal)        # coerce to binary float
and
    Decimal(some_float)       # coerce to decimal float

That will also save us from odd idioms like:

    d = some_float + Decimal(0)   # coerce to decimal then apply context rounding

This also matches the behavior of other constructors:
    int(some_float)
or
    int(some_decimal)
or
    str(some_float)
or
    str(some_decimal)


Raymond


From stefan at bytereef.org  Mon Mar 22 20:46:28 2010
From: stefan at bytereef.org (Stefan Krah)
Date: Mon, 22 Mar 2010 20:46:28 +0100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
References: <B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
Message-ID: <20100322194628.GB21459@yoda.bytereef.org>

Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> 
> On Mar 22, 2010, at 11:26 AM, Mark Dickinson wrote:
> > 
> > Just for the record, I'd also prefer Decimal + Fraction -> Decimal.
> 
> 
> Guido was persuasive on why float + Fraction --> float,
> so this makes sense for the same reasons.
> 
> For the implementation, is there a way to avoid the double rounding
> in   myfloat + myfrac.numerator / myfrac.denominator?
> 
> Perhaps translate it to:
> 
>       f = Fractions.from_decimal(myfloat) + myfract   # Lossless, exact addition
>       return f.numerator / f.denominator           # Only one decimal context rounding applied.
> 
> Elsewhere in the decimal module, there is a fundamental notion
> that numbers are exact and only the results of operations are
> rounded.   For example, it is possible in decimal to add together
> two high precision numbers but do so in a low precision context.


I don't think this will be practical for huge decimal exponents.

Also, at first glance I wonder how to integrate this cleanly into the
control flow. convert_other() will not work, so there would need to
be a special case for fractions in each function.

Or do you mean to outsource the whole computation to the fractions
module, which calls decimal only for the final division?


Stefan Krah



From guido at python.org  Mon Mar 22 20:52:24 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Mar 2010 11:52:24 -0800
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com> 
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com> 
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> 
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com> 
	<027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com>
Message-ID: <ca471dc21003221252h130b1fdcv4a333bb5c2eb1817@mail.gmail.com>

On Mon, Mar 22, 2010 at 11:36 AM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> One other thought.
>
> The Decimal constructor should now accept floats as a possible input type.
> Formerly, we separated that out to Decimal.from_float() because
> decimals weren't interoperable with floats.

Not sure this follows; Fraction(1.1) raises an exception, you have to
use Fraction.from_float().

> This will put decimal and float back on equal footing so that we have both:
> ? ?float(some_decimal) ? ? ? ?# coerce to binary float
> and
> ? ?Decimal(some_float) ? ? ? # coerce to decimal float

If you really want this I won't stop you.

> That will also save us from odd idioms like:
>
> ? ?d = some_float + Decimal(0) ? # coerce to decimal then apply context rounding

Although this isn't too bad:

  d = Decimal.from_float(some_float)

> This also matches the behavior of other constructors:
> ? ?int(some_float)
> or
> ? ?int(some_decimal)
> or
> ? ?str(some_float)
> or
> ? ?str(some_decimal)

True, I don't care strongly.

-- 
--Guido van Rossum (python.org/~guido)

From pierrebai at hotmail.com  Mon Mar 22 21:02:57 2010
From: pierrebai at hotmail.com (Pierre B.)
Date: Mon, 22 Mar 2010 20:02:57 +0000 (UTC)
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
Message-ID: <loom.20100322T204833-290@post.gmane.org>

Sorry to intervene out of the blue, but I find the suggested rule for 
fractional to decimal conversion not as clean as I'd expect.

If fractions are converted to decimals when doing arithmetics, would it be 
worthwhile to at least provide a minimum of fractional conversion integrity?
What I have in mind is the following rule:

When doing conversion from fraction to decimal, always generate a whole number
of repeating digits, always at least twice.

Examples, with a precision of 5 in Decimal:

1/2 -> 0.50000

1/3 -> 0.33333

1/11 -> 0.090909
# Note that we produced 6 digits, because
# the repeating pattern contains 2 digits.

1/7 -> 0.142857142857
 # Always at least two full patterns.

 The benefits I see are that:

 1. If a number can be represented exactly it will be converted exactly.

 2. The minimum precision requested is respected.

 3. The conversion yields something that will convert back more precisely.  
    Not perfectly, but see the next point.

 4. Since the repeating pattern is present at least twice at the end,
    one can augment the precision of the conversion by detecting the 
    repetition and adding more. This detection is trivial.



From dickinsm at gmail.com  Mon Mar 22 21:12:27 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Mon, 22 Mar 2010 20:12:27 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <loom.20100322T204833-290@post.gmane.org>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<loom.20100322T204833-290@post.gmane.org>
Message-ID: <5c6f2a5d1003221312u19c71ca3i85eea936fc9465fa@mail.gmail.com>

On Mon, Mar 22, 2010 at 8:02 PM, Pierre B. <pierrebai at hotmail.com> wrote:
> Sorry to intervene out of the blue, but I find the suggested rule for
> fractional to decimal conversion not as clean as I'd expect.
>
> If fractions are converted to decimals when doing arithmetics, would it be
> worthwhile to at least provide a minimum of fractional conversion integrity?
> What I have in mind is the following rule:
>
> When doing conversion from fraction to decimal, always generate a whole number
> of repeating digits, always at least twice.
>
> Examples, with a precision of 5 in Decimal:
>
> 1/2 -> 0.50000
>
> 1/3 -> 0.33333
>
> 1/11 -> 0.090909
> # Note that we produced 6 digits, because
> # the repeating pattern contains 2 digits.
>
> 1/7 -> 0.142857142857
> ?# Always at least two full patterns.

And for 1/123123127?  The decimal expansion of this fraction has a
period of over 15 million!

Sorry, but this doesn't seem like a feasible or desirable strategy.

Mark

From pierrebai at hotmail.com  Mon Mar 22 21:14:28 2010
From: pierrebai at hotmail.com (Pierre B.)
Date: Mon, 22 Mar 2010 20:14:28 +0000 (UTC)
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<loom.20100322T204833-290@post.gmane.org>
Message-ID: <loom.20100322T211238-356@post.gmane.org>

Pierre B. <pierrebai <at> hotmail.com> writes:
> 
>  4. Since the repeating pattern is present at least twice at the end,
>     one can augment the precision of the conversion by detecting the 
>     repetition and adding more. This detection is trivial.

Wrong of me. This point is acutally invalid, since it is impossible to
generally differentiate where the pattern ends. The counter example is:

0.177177177177... vs 0.177.17777777...




From barry at python.org  Mon Mar 22 21:16:26 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 22 Mar 2010 16:16:26 -0400
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <loom.20100322T150829-761@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
Message-ID: <20100322161626.6e56efac@heresy>

On Mar 22, 2010, at 02:30 PM, Antoine Pitrou wrote:

>Barry Warsaw <barry <at> python.org> writes:
>> 
>> * Tough luck
>> * Force the umask so that the directory is writable, but then the question is,
>>   by whom?  ugo+w or something less?
>> * Copy the permissions from the parent directory and ignore umask
>> * Raise an exception or refuse to create __pycache__ if it's not writable
>>   (again, by whom?)
>
>Welcome to a problem PHP has known for years.

Yay.

>The problem with solution 3 is that a non-root user can't change the owner of
>a directory, therefore copying the permissions doesn't ensure the directory
>will be writable by the right users.
>
>Solution 2 (chmod 777) is obviously too laxist. Anyone could put fake bytecode
>files in a root-owned application. Less laxist chmods have the same problem as
>solution 3, and can still be insecure.
>
>I think solution 4 would be the best compromise. In the face of ambiguity,
>refuse to guess. But, rather than :
>    refuse to create __pycache__ if it's not writable,
>the test should be :
>    refuse to create __pycache__ if I can't create it with the same ownership
>and permissions as the parent directory (and raise an ImportWarning).

I actually think "tough luck" might be the right answer and that it's not
really all that serious.  Certainly not serious enough not to enable this by
default.

When Python is being installed, either by a from-source 'make install' or by
the distro packager, then you'd expect the umask not to be insane.  In the
latter case, it's a bug and in the former case you screwed up so you should be
able to delete and reinstall, or at the very least execute the same `find`
command that Python's own Makefile calls (in my branch) for 'make clean'.

When you're installing packages, again, I would expect that the system
installer, or you via `easy_install` or whatever, would not have an insane
umask.  As with above, if you *did* have a bad umask, you could certainly also
have a bad umask whenever you ran the third party tool as when you ran
Python.

You could also have a bad umask when you're doing local development, but that
should be easy enough to fix, and besides, I think you'd have other problems
if that were the case.

One possibility would be to instrument compileall to remove __pycache__
directories with a flag.

>In light of this issue, I'm -0.5 on __pycache__ becoming the default caching
>mechanism. The directory ownership/permissions issue is too much of a mess,
>especially for Web applications (think __pycache__ files created by the Apache
>user).
>
>__pycache__ should only be created if explicitly activated (for example by
>distutils when installing stuff). Otherwise, if not present, the "legacy"
>mechanism (writing an untagged pyc file along the py file) should be used.

I don't particularly like this much, because most people won't get the benefit
of it in a local dev environment.  While not the primary use case, it's a
useful enough side benefit, and unless Guido changes his mind, we want to
enable this by default.

>Actually, __pycache__ creation doesn't have to be part of the import
>mechanism.  It can be part of distutils instead (or whatever third-party tool
>such as distribute, or distro-specific packaging scripts). This would relax
>complexity of core Python a bit.

It's not really that complicated. :)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/739eff36/attachment.pgp>

From barry at python.org  Mon Mar 22 21:18:17 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 22 Mar 2010 16:18:17 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <loom.20100322T191016-732@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
Message-ID: <20100322161817.06a8911f@heresy>

On Mar 22, 2010, at 06:15 PM, Antoine Pitrou wrote:

>What I am proposing is that the creation of __pycache__ /directories/ be put
>outside of the core. It can be part of distutils, or of a separate module, or
>delegated to third-party tools. It could even be as simple as
>"python -m compileall --pycache", if someone implements it.

Except that most people won't do this, they'll just run Python.  I wouldn't
count this as "enabled by default".

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/5bee0814/attachment.pgp>

From barry at python.org  Mon Mar 22 21:20:02 2010
From: barry at python.org (Barry Warsaw)
Date: Mon, 22 Mar 2010 16:20:02 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <ho8esj$qnd$1@dough.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<ho8esj$qnd$1@dough.gmane.org>
Message-ID: <20100322162002.6daff86f@heresy>

On Mar 22, 2010, at 02:02 PM, Ron Adam wrote:

>If I understand correctly, we would have the current mode as the default, and
>can trigger __pycache__ behavior simply by manually creating a __pycache__
>directory and deleting any byte-code files in the module/program directory.
>
>I like this, it is easy to understand and can be used without messing with 
>flags or environment variables.

Well, for a package with subpackages, it gets more complicated.  Definitely
not something you're likely to do manually.  Antoine's suggestion of 'python
-m compileall --pycache' would work, but I think it's also obscure enough that
most Python users won't get the benefit.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100322/28503161/attachment-0001.pgp>

From raymond.hettinger at gmail.com  Mon Mar 22 21:33:20 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 22 Mar 2010 13:33:20 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003221252h130b1fdcv4a333bb5c2eb1817@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>
	<027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com>
	<ca471dc21003221252h130b1fdcv4a333bb5c2eb1817@mail.gmail.com>
Message-ID: <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com>

While we're on the topic, I think you should consider allowing the Fraction()
constructor to accept a decimal input.

This corresponds to common schoolbook problems and simple client requests:
   "Express 3.5 as a fraction".

    >>> Fraction(Decimal('3.5'))
   Fraction(7, 2)

Unlike typical binary floats which use full precision, it is not uncommon
to have decimal floats with only a few digits of precision where the
expression as a fraction is both useful and unsurprising.


Raymond

From solipsis at pitrou.net  Mon Mar 22 21:33:14 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Mar 2010 20:33:14 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?=
	=?utf-8?q?s_and_umask?=
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<20100322161626.6e56efac@heresy>
Message-ID: <loom.20100322T212223-259@post.gmane.org>

Barry Warsaw <barry <at> python.org> writes:
> 
> When Python is being installed, either by a from-source 'make install' or by
> the distro packager, then you'd expect the umask not to be insane.  In the
> latter case, it's a bug and in the former case you screwed up so you should be
> able to delete and reinstall, or at the very least execute the same `find`
> command that Python's own Makefile calls (in my branch) for 'make clean'.
> 
> When you're installing packages, again, I would expect that the system
> installer, or you via `easy_install` or whatever, would not have an insane
> umask.

Well, precisely. That's why I suggest that creating the __pycache__ directories
be done *at install time* (or packaging time), and not via the core import
machinery (that is, not at import time). That is, when you *know* you are the
right user, with the right umask.

Which also means that it can and probably should be pulled out of the core (and
written in pure Python, which is also much more practical to maintain, test and
debug).

> I don't particularly like this much, because most people won't get the benefit
> of it in a local dev environment.

AFAICT it has no benefit in a local dev environment (apart from supposedly
"cleaner" directories, which is not something anyone seems to consider a problem
with the current CPythons).

And again, this could be part of the distutils standard "setup.py install"
(or "setup.py develop" with distutils2/distribute) if people care :)

Regards

Antoine.



From guido at python.org  Mon Mar 22 21:38:42 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Mar 2010 12:38:42 -0800
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <20100322162002.6daff86f@heresy>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org> 
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca> 
	<loom.20100322T191016-732@post.gmane.org>
	<ho8esj$qnd$1@dough.gmane.org> <20100322162002.6daff86f@heresy>
Message-ID: <ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>

On Mon, Mar 22, 2010 at 12:20 PM, Barry Warsaw <barry at python.org> wrote:
> On Mar 22, 2010, at 02:02 PM, Ron Adam wrote:
>
>>If I understand correctly, we would have the current mode as the default, and
>>can trigger __pycache__ behavior simply by manually creating a __pycache__
>>directory and deleting any byte-code files in the module/program directory.

Huh? Last time I looked weren't we going to make __pycache__ the
default (and eventually only) behavior?

>>I like this, it is easy to understand and can be used without messing with
>>flags or environment variables.
>
> Well, for a package with subpackages, it gets more complicated. ?Definitely
> not something you're likely to do manually. ?Antoine's suggestion of 'python
> -m compileall --pycache' would work, but I think it's also obscure enough that
> most Python users won't get the benefit.

I see only two reasonable solutions for __pycache__ creation -- either
we change all setup/install scripts (both for core Python and for 3rd
party packages) to always create a __pycache__ subdirectory for every
directory (including package directories) installed; or we somehow
create it the first time it's needed.

But creating it as needed runs into at least similar problems with
ownership as creating .pyc files when first needed (if the parent
directory is root-owned a mere mortal can't create it at all). So even
apart from the security issue (which I haven't thought about deeply) I
think precreation should at least be an easily accessible option both
for the core (where it can be done by compileall) and for 3rd party
packages (where I guess it's up to distutils or whatever install
mechanism is used).

-- 
--Guido van Rossum (python.org/~guido)

From p.f.moore at gmail.com  Mon Mar 22 21:45:05 2010
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 Mar 2010 20:45:05 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003221232l69fe8eenf0778d9c4a5e436b@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<5c6f2a5d1003221232l69fe8eenf0778d9c4a5e436b@mail.gmail.com>
Message-ID: <79990c6b1003221345n9d02b0dwf7d015ebfa812712@mail.gmail.com>

On 22 March 2010 19:32, Mark Dickinson <dickinsm at gmail.com> wrote:
> I think getting this to work would involve a lot of extra code and
> significant 'cleverness'. ?I'd prefer the simple-to-implement and
> simple-to-explain option of rounding the Fraction before performing
> the operation, even if this means that the whole operation involves
> two rounding operations. ?It's not so different from what currently
> happens for Fraction+float, or even int+float.

My instinct throughout this discussion has been that

    a_decimal + a_fraction

is in effect

    a_decimal + a_fraction.numerator / a_fraction.denominator

and as the numerator and denomination can be converted into exact
decimals, my instincts "expect" a double rounding. Anything else
effectively means implementing partial cases of  (a+b/c) treated as a
ternary operation.

So I guess I'm +1 on Mark's suggestion.
Paul.

From guido at python.org  Mon Mar 22 21:44:47 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Mar 2010 12:44:47 -0800
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com> 
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> 
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com> 
	<027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com>
	<ca471dc21003221252h130b1fdcv4a333bb5c2eb1817@mail.gmail.com> 
	<5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com>
Message-ID: <ca471dc21003221344oea72a86yf19b2050ff462704@mail.gmail.com>

On Mon, Mar 22, 2010 at 12:33 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> While we're on the topic, I think you should consider allowing the Fraction()
> constructor to accept a decimal input.
>
> This corresponds to common schoolbook problems and simple client requests:
> ? "Express 3.5 as a fraction".
>
> ? ?>>> Fraction(Decimal('3.5'))
> ? Fraction(7, 2)
>
> Unlike typical binary floats which use full precision, it is not uncommon
> to have decimal floats with only a few digits of precision where the
> expression as a fraction is both useful and unsurprising.

There is already a Fraction.from_decimal() class method. I understand
the reasons for not allowing floats in the unadorned Fraction
constructor don't quite hold for Decimal, but at the same time I'm not
sure how compelling the use case is for such downcasts given that it
can be done easily with an explicit call. (Same actually for requiring
Decimal.from_float().)

-- 
--Guido van Rossum (python.org/~guido)

From dickinsm at gmail.com  Mon Mar 22 21:45:39 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Mon, 22 Mar 2010 20:45:39 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>
	<027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com>
	<ca471dc21003221252h130b1fdcv4a333bb5c2eb1817@mail.gmail.com>
	<5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com>
Message-ID: <5c6f2a5d1003221345n5560ab59m82381d1aaac4481d@mail.gmail.com>

On Mon, Mar 22, 2010 at 8:33 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> While we're on the topic, I think you should consider allowing the Fraction()
> constructor to accept a decimal input.
>
> This corresponds to common schoolbook problems and simple client requests:
> ? "Express 3.5 as a fraction".
>
> ? ?>>> Fraction(Decimal('3.5'))
> ? Fraction(7, 2)
>
> Unlike typical binary floats which use full precision, it is not uncommon
> to have decimal floats with only a few digits of precision where the
> expression as a fraction is both useful and unsurprising.

Sounds fine to me.  Fraction already accepts decimal floating-point
strings, so the implementation of this would be trivial (convert to
string, then call the Fraction constructor).

Mark

From martin at v.loewis.de  Mon Mar 22 21:47:40 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 22 Mar 2010 21:47:40 +0100
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <20100322161817.06a8911f@heresy>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>
	<20100322161817.06a8911f@heresy>
Message-ID: <4BA7D76C.9010502@v.loewis.de>

Barry Warsaw wrote:
> On Mar 22, 2010, at 06:15 PM, Antoine Pitrou wrote:
> 
>> What I am proposing is that the creation of __pycache__ /directories/ be put
>> outside of the core. It can be part of distutils, or of a separate module, or
>> delegated to third-party tools. It could even be as simple as
>> "python -m compileall --pycache", if someone implements it.
> 
> Except that most people won't do this, they'll just run Python.  I wouldn't
> count this as "enabled by default".

Therefore, I'm in favor of having it on by default. If certain use cases
make it problematic (e.g. Apache creating directories which you then
cannot delete), there should be a way to turn it *off*. Perhaps the
existing machinery to turn of byte code generation at all might be
sufficient.

As for the original question (funny umasks), I think my proposal is
"tough luck". Don't try to be super-smart; as Antoine explains, it gets
worse, not better. If the user has arranged that Python will create
unusable directories, the user better changes his setup.

Regards,
Martin

From guido at python.org  Mon Mar 22 21:48:09 2010
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 Mar 2010 12:48:09 -0800
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003221345n5560ab59m82381d1aaac4481d@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com> 
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com> 
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com> 
	<027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com>
	<ca471dc21003221252h130b1fdcv4a333bb5c2eb1817@mail.gmail.com> 
	<5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com>
	<5c6f2a5d1003221345n5560ab59m82381d1aaac4481d@mail.gmail.com>
Message-ID: <ca471dc21003221348q66a12c4j5d7d0a3a1c714479@mail.gmail.com>

On Mon, Mar 22, 2010 at 12:45 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Mon, Mar 22, 2010 at 8:33 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>> While we're on the topic, I think you should consider allowing the Fraction()
>> constructor to accept a decimal input.
>>
>> This corresponds to common schoolbook problems and simple client requests:
>> ? "Express 3.5 as a fraction".
>>
>> ? ?>>> Fraction(Decimal('3.5'))
>> ? Fraction(7, 2)
>>
>> Unlike typical binary floats which use full precision, it is not uncommon
>> to have decimal floats with only a few digits of precision where the
>> expression as a fraction is both useful and unsurprising.
>
> Sounds fine to me. ?Fraction already accepts decimal floating-point
> strings, so the implementation of this would be trivial (convert to
> string, then call the Fraction constructor).

Better, Fraction.from_decimal() already exists. ;-)

-- 
--Guido van Rossum (python.org/~guido)

From dickinsm at gmail.com  Mon Mar 22 21:49:06 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Mon, 22 Mar 2010 20:49:06 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003221344oea72a86yf19b2050ff462704@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>
	<027473D4-AB40-425C-A83E-F0B6F4BDB516@gmail.com>
	<ca471dc21003221252h130b1fdcv4a333bb5c2eb1817@mail.gmail.com>
	<5F54E962-E216-4BF5-AB87-5237FB624DD7@gmail.com>
	<ca471dc21003221344oea72a86yf19b2050ff462704@mail.gmail.com>
Message-ID: <5c6f2a5d1003221349o12e64d76qc5fc8e643a71fa5d@mail.gmail.com>

On Mon, Mar 22, 2010 at 8:44 PM, Guido van Rossum <guido at python.org> wrote:
> On Mon, Mar 22, 2010 at 12:33 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>> While we're on the topic, I think you should consider allowing the Fraction()
>> constructor to accept a decimal input.
>>
>> This corresponds to common schoolbook problems and simple client requests:
>> ? "Express 3.5 as a fraction".
>>
>> ? ?>>> Fraction(Decimal('3.5'))
>> ? Fraction(7, 2)
>
> There is already a Fraction.from_decimal() class method.

So there is;  I'd failed to notice that!  So decimal -> fraction
conversion is implemented twice in the Fractions module---once in
__new__ and once in from_decimal.  Hmm.

Mark

From raymond.hettinger at gmail.com  Mon Mar 22 21:53:25 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 22 Mar 2010 13:53:25 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>
	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>
Message-ID: <926781A9-32A5-4814-952D-F8159A944B03@gmail.com>

For the record, I thought I would take a stab at making a single post
that recaps the trade-offs and reasoning behind the decision
to have Fraction + decimal/float --> decimal/float.

Pros:

* While we know that both decimal and binary floats have a
fixed internal precision and can be converted losslessly to
a rational, that doesn't correspond to the way we think about
them.  We tend to think of floating point values as real numbers,
not as rationals.

* There is a notion of fractions being used for unrounded
arithmetic and floats operations being rounded arithmetic.  
So, it doesn't make sense to create the illusion of an unrounded
result from inputs that we already subject to rounding.

* Backward compatibility.  That is what the fractions module already
does and we haven't have any problems with it.

Cons:

* The coercion logic for comparisons won't match the
coercion logic for arithmetic operations.   The former
strives to be exact and to be consistent with hashing
while the latter goes in the opposite direction.

* Operations such as some_float + some_fraction are
subject to double rounding.  The potentially produces
a different result than the single rounding in:

    float(Fraction.from_float(some_float) + some_fraction)



Raymond

From solipsis at pitrou.net  Mon Mar 22 22:00:30 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Mar 2010 21:00:30 +0000 (UTC)
Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?=
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>
	<20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de>
Message-ID: <loom.20100322T215811-374@post.gmane.org>

Martin v. L?wis <martin <at> v.loewis.de> writes:
> 
> If certain use cases
> make it problematic (e.g. Apache creating directories which you then
> cannot delete), there should be a way to turn it *off*. Perhaps the
> existing machinery to turn of byte code generation at all might be
> sufficient.

Except that you notice the problem when it happens, so turning it off is always
too late.



From martin at v.loewis.de  Mon Mar 22 22:25:35 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 22 Mar 2010 22:25:35 +0100
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <loom.20100322T215811-374@post.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<20100322161817.06a8911f@heresy>
	<4BA7D76C.9010502@v.loewis.de>
	<loom.20100322T215811-374@post.gmane.org>
Message-ID: <4BA7E04F.7070308@v.loewis.de>

Antoine Pitrou wrote:
> Martin v. L?wis <martin <at> v.loewis.de> writes:
>> If certain use cases
>> make it problematic (e.g. Apache creating directories which you then
>> cannot delete), there should be a way to turn it *off*. Perhaps the
>> existing machinery to turn of byte code generation at all might be
>> sufficient.
> 
> Except that you notice the problem when it happens, so turning it off is always
> too late.

Why is it too late? Fix it, and get on.

Regards,
Martin

From brett at python.org  Mon Mar 22 22:28:26 2010
From: brett at python.org (Brett Cannon)
Date: Mon, 22 Mar 2010 14:28:26 -0700
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <20100322095604.3371039e@heresy>
References: <20100322095604.3371039e@heresy>
Message-ID: <bbaeab101003221428q594cfa87h58a91ef188d71e89@mail.gmail.com>

On Mon, Mar 22, 2010 at 06:56, Barry Warsaw <barry at python.org> wrote:
> I have a pretty good start on PEP 3147 implementation [1], but I've
> encountered a situation that I'd like to get some feedback on. ?Here's the
> test case illustrating the problem. ?From test_import.py:
>
> ? ?def test_writable_directory(self):
> ? ? ? ?# The umask is not conducive to creating a writable __pycache__
> ? ? ? ?# directory.
> ? ? ? ?with umask(0o222):
> ? ? ? ? ? ?__import__(TESTFN)
> ? ? ? ?self.assertTrue(os.path.exists('__pycache__'))
> ? ? ? ?self.assertTrue(os.path.exists(os.path.join(
> ? ? ? ? ? ?'__pycache__', '{}.{}.pyc'.format(TESTFN, self.tag))))
>
> The __pycache__ directory does not exist before the import, and the import
> machinery creates the directory, but the umask leaves the directory unwritable
> by anybody. ?So of course when the import machinery goes to write the .pyc
> file inside __pycache__, it fails. ?This does not cause an ImportError though,
> just like if today the package directory were unwritable.
>
> This might be different than today's situation though because once the
> unwritable __pycache__ directory is created, nothing is going to change that
> without explicit user interaction, and that might be difficult after the
> fact.
>
> I'm not sure what the right answer is. ?Some possible choices:
>
> * Tough luck
> * Force the umask so that the directory is writable, but then the question is,
> ?by whom? ?ugo+w or something less?
> * Copy the permissions from the parent directory and ignore umask
> * Raise an exception or refuse to create __pycache__ if it's not writable
> ?(again, by whom?)

I say tough luck. At the moment if you can't write a .pyc it is a
silent failure; don't see the difference as significant enough to
warrant changing the practice.

-Brett

>
> Or maybe you have a better idea? ?What's the equivalent situation on Windows
> and how would things work there?
>
> -Barry
>
> [1] https://edge.launchpad.net/~barry/python/pep3147
>
> P.S. I'm down to only 8 unit test failures.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>

From rdmurray at bitdance.com  Mon Mar 22 22:39:26 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 22 Mar 2010 17:39:26 -0400
Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?=
In-Reply-To: <loom.20100322T191016-732@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
Message-ID: <20100322213926.927511FB7F0@kimball.webabinitio.net>

On Mon, 22 Mar 2010 18:15:01 -0000, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Isaac Morland <ijmorlan <at> uwaterloo.ca> writes:
> > 
> > > IMO, all these issues militate for putting __pycache__ creation out of
> > > the interpreter core, and in the hands of third-party package-time/
> > > install-time tools (or distutils).
> > 
> > Speaking only for myself, but really for anybody who likes tidy source 
> > directories, I hope some version of the __pycache__ proposal becomes part 
> > of standard Python, by which I ideally mean it's enabled by default but if 
> > that is just not a good idea then at most it should be required to set a 
> > command-line option to get this feature.
> 
> This doesn't contradict by my proposal.
> 
> What I am proposing is that the creation of __pycache__ /directories/ be put
> outside of the core. It can be part of distutils, or of a separate module, or
> delegated to third-party tools. It could even be as simple as
> "python -m compileall --pycache", if someone implements it.

Or even as simple as 'mkdir __pycache__', if you are working in your
own library and don't want .pyc clutter.

--
R. David Murray                                      www.bitdance.com

From solipsis at pitrou.net  Mon Mar 22 22:57:03 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Mar 2010 21:57:03 +0000 (UTC)
Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?=
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<20100322161817.06a8911f@heresy>
	<4BA7D76C.9010502@v.loewis.de>
	<loom.20100322T215811-374@post.gmane.org>
	<4BA7E04F.7070308@v.loewis.de>
Message-ID: <loom.20100322T225133-106@post.gmane.org>

Martin v. L?wis <martin <at> v.loewis.de> writes:
> 
> Antoine Pitrou wrote:
> > Martin v. L?wis <martin <at> v.loewis.de> writes:
> >> If certain use cases
> >> make it problematic (e.g. Apache creating directories which you then
> >> cannot delete), there should be a way to turn it *off*. Perhaps the
> >> existing machinery to turn of byte code generation at all might be
> >> sufficient.
> > 
> > Except that you notice the problem when it happens, so turning it off is
always
> > too late.
> 
> Why is it too late? Fix it, and get on.

Sure, but it is annoying, and since it's the kind of things that noone
(including sysadmins) ever thinks about in advance, it's bound to repeat itself
quite often.

It's especially annoying, of course, if you have to ask someone else to remove
the directories for you (or if you have to write custom code and get it executed
by the Apache or WSGI handler...).

Really, it's unfriendly to users and it's certainly not outweighed by the
"benefit" of having "cleaner" source directories.



From martin at v.loewis.de  Mon Mar 22 23:18:02 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 22 Mar 2010 23:18:02 +0100
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <loom.20100322T225133-106@post.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<20100322161817.06a8911f@heresy>	<4BA7D76C.9010502@v.loewis.de>	<loom.20100322T215811-374@post.gmane.org>	<4BA7E04F.7070308@v.loewis.de>
	<loom.20100322T225133-106@post.gmane.org>
Message-ID: <4BA7EC9A.1010206@v.loewis.de>

>> Why is it too late? Fix it, and get on.
> 
> Sure, but it is annoying, and since it's the kind of things that noone
> (including sysadmins) ever thinks about in advance, it's bound to repeat itself
> quite often.
> 
> It's especially annoying, of course, if you have to ask someone else to remove
> the directories for you (or if you have to write custom code and get it executed
> by the Apache or WSGI handler...).
> 
> Really, it's unfriendly to users and it's certainly not outweighed by the
> "benefit" of having "cleaner" source directories.

Whether it is outweighed would also depend on how likely and frequent
the presumed problem is, no?

If Apache creates a folder for me that I cannot remove, most likely,
there was a configuration error in the first place: common practice
tells that you should execute user code under user permissions, not as
www-data. If your code does get run as Apache, this also opens a way of
not asking for help: just put "os.system('chmod +w /.../__pycache__')"
into your code, and have Apache run it again.

So I don't think this is any more unfriendly than creating .pyc files in
the first place, and the advantages of uniformity of this new approach
certainly outweigh the disadvantages.

Regards,
Martin


From ncoghlan at gmail.com  Mon Mar 22 23:19:35 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 23 Mar 2010 08:19:35 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <926781A9-32A5-4814-952D-F8159A944B03@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<4BA5AA34.8050709@canterbury.ac.nz>	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>	<4BA6A2B2.6050207@canterbury.ac.nz>	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>	<C0923A96-1661-4BE8-85FA-BCFCA63B7A84@gmail.com>	<ca471dc21003221203y71706db6ja53ce4e84a8ed823@mail.gmail.com>
	<926781A9-32A5-4814-952D-F8159A944B03@gmail.com>
Message-ID: <4BA7ECF7.5020809@gmail.com>

Raymond Hettinger wrote:
> * The coercion logic for comparisons won't match the
> coercion logic for arithmetic operations.   The former
> strives to be exact and to be consistent with hashing
> while the latter goes in the opposite direction.

Although Guido pointed out that float/long comparisons and coercions
blazed the trail on this particular discrepancy years ago.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Mon Mar 22 23:53:35 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 22 Mar 2010 23:53:35 +0100
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <4BA7EC9A.1010206@v.loewis.de>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>	<20100322161817.06a8911f@heresy>
	<4BA7D76C.9010502@v.loewis.de>	<loom.20100322T215811-374@post.gmane.org>
	<4BA7E04F.7070308@v.loewis.de>
	<loom.20100322T225133-106@post.gmane.org>
	<4BA7EC9A.1010206@v.loewis.de>
Message-ID: <1269298415.5792.11.camel@localhost>

Le lundi 22 mars 2010 ? 23:18 +0100, "Martin v. L?wis" a ?crit :
> If Apache creates a folder for me that I cannot remove, most likely,
> there was a configuration error in the first place: common practice
> tells that you should execute user code under user permissions, not as
> www-data.

I'm sure there can be reasons not to do so.
One of them is mass hosting of Python apps: you don't want to create a
separate process per Python user, and therefore you can't run the code
under the "right" user.

> If your code does get run as Apache, this also opens a way of
> not asking for help: just put "os.system('chmod +w /.../__pycache__')"
> into your code, and have Apache run it again.

This is what I meant by "write custom code and get it executed by the
Apache or WSGI handler". Having the Web server execute ad hoc system
administration code is far from elegant and user-friendly.

> So I don't think this is any more unfriendly than creating .pyc files in
> the first place, and the advantages of uniformity of this new approach
> certainly outweigh the disadvantages.

I don't see the "advantages of uniformity".

What we had initially was a way to solve problems which are specific to
Ubuntu and Debian packagers, and which was only to be activated by those
people on those systems.

We now end with an alleged complete solution to a problem which doesn't
seem to exist, or is at least vastly overblown (the idea that having pyc
files along their source counterparts is a nuisance doesn't seem to be a
common grief against Python).

I would really recommend reexamining it, rather than falling for the
shiny new thing.

Regards

Antoine.



From greg.ewing at canterbury.ac.nz  Tue Mar 23 00:05:47 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 23 Mar 2010 11:05:47 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>
	<loom.20100320T160613-470@post.gmane.org>
	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>
	<4BA55858.2000800@canterbury.ac.nz>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
Message-ID: <4BA7F7CB.7010603@canterbury.ac.nz>

Mark Dickinson wrote:

> But the Fraction type is going to mess this up:  for Decimal +
> Fraction ->  Decimal, I don't see any other sensible option than to
> convert the Fraction using the current context, since lossless
> conversion isn't generally possible.

You could convert the Decimal to a Fraction, do the arithmetic
as Fractions, and convert the result back to Decimal using the
current precision.

I think this would satisfy the principle of performing the
calculation as though infinite precision were available, and
the final rounding is justified, since seen from the outside
we are performing a single operation on two numbers.

Another approach would be to convert the numerators and
denominators separately to Decimal, and then do

   a/b + c/d = (a*d + b*c) / (b*d)

using the usual rules of Decimal arithmetic. This would
incur more than one rounding, but the rules for Decimals
concern operations between two Decimals, and we have a
Decimal and a Fraction here, so all bets could be considered
off.

-- 
Greg

From martin at v.loewis.de  Tue Mar 23 00:02:53 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 23 Mar 2010 00:02:53 +0100
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <1269298415.5792.11.camel@localhost>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<20100322161817.06a8911f@heresy>	<4BA7D76C.9010502@v.loewis.de>	<loom.20100322T215811-374@post.gmane.org>	<4BA7E04F.7070308@v.loewis.de>	<loom.20100322T225133-106@post.gmane.org>	<4BA7EC9A.1010206@v.loewis.de>
	<1269298415.5792.11.camel@localhost>
Message-ID: <4BA7F71D.6030907@v.loewis.de>

> We now end with an alleged complete solution to a problem which doesn't
> seem to exist, or is at least vastly overblown (the idea that having pyc
> files along their source counterparts is a nuisance doesn't seem to be a
> common grief against Python).
> 
> I would really recommend reexamining it, rather than falling for the
> shiny new thing.

I think the appropriate action at this point is to record this specific
objection in the PEP.

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Tue Mar 23 00:26:14 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 23 Mar 2010 11:26:14 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA74946.9060302@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA57C5A.9060403@gmail.com>
	<ca471dc21003210953j7a5fa7cyd8329696346f392e@mail.gmail.com>
	<E696D61B-A7F6-4C49-A204-923A5A52F6DB@gmail.com>
	<4BA6A0AD.7090207@canterbury.ac.nz> <4BA74946.9060302@gmail.com>
Message-ID: <4BA7FC96.6020109@canterbury.ac.nz>

Nick Coghlan wrote:

> http://docs.python.org/library/decimal.html#decimal.Inexact
> 
> (Part of the thread context rather than the individual decimal values,
> but if you use it properly it tells you whenever an inexact operation
> has occurred in the current thread)

My problem was that the statement "All numbers are exact;
inexactness is a taint" appears to be self-contradictory.
Numbers *can* become tainted with inexactness whether you
explicitly keep track of that or not.

However, the excerpt by Cowlishaw posted earlier reveals
where the confusion is coming from, I think. What Cowlishaw
appears to mean by "numbers are exact" is that Decimals
represent particular values, not *intervals*.

This is not really the same thing as the notion of inexact
numbers in the numeric tower. There, it means more like
"this number may not quite represent the value the programmer
had in mind".

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Mar 23 00:58:16 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 23 Mar 2010 11:58:16 +1200
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <loom.20100322T150829-761@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
Message-ID: <4BA80418.6030905@canterbury.ac.nz>

Antoine Pitrou wrote:

> In light of this issue, I'm -0.5 on __pycache__ becoming the default caching
> mechanism. The directory ownership/permissions issue is too much of a mess,
> especially for Web applications (think __pycache__ files created by the Apache
> user).

Doesn't the existing .pyc mechanism have the same problem? Seems
to me it's just as insecure to allow the Apache user to create
.pyc files, since an attacker could overwrite them with arbitrary
bytecode.

The only safe way is to pre-compile under a different user and
make everything read-only to Apache. The same thing would apply
under the __pycache__ regime.

> Actually, __pycache__ creation doesn't have to be part of the import mechanism.
> It can be part of distutils instead (or whatever third-party tool

What about development, or if a user installs by dragging into
site-packages instead of using an installer? I don't like the
idea of being required to use an installation tool in order to
get .pyc files.

-- 
Greg

From solipsis at pitrou.net  Tue Mar 23 01:05:11 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 Mar 2010 00:05:11 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?=
	=?utf-8?q?s_and_umask?=
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
Message-ID: <loom.20100323T005510-312@post.gmane.org>

Greg Ewing <greg.ewing <at> canterbury.ac.nz> writes:
> 
> Doesn't the existing .pyc mechanism have the same problem? Seems
> to me it's just as insecure to allow the Apache user to create
> .pyc files, since an attacker could overwrite them with arbitrary
> bytecode.

The problem is that you can't delete the __pycache__ directory if it doesn't
have the right ownership and if it's non-empty.
This problem doesn't exist with a pyc file situated in a directory you own.

> > Actually, __pycache__ creation doesn't have to be part of the import
mechanism.
> > It can be part of distutils instead (or whatever third-party tool
> 
> What about development,

The main point of the __pycache__ proposal is to solve the needs of
Ubuntu/Debian packagers. If you are developing (rather than deploying or
building packages), you shouldn't have these needs AFAICT.

> or if a user installs by dragging into
> site-packages instead of using an installer?

Well... do people actually do this?

"python setup.py install" is simpler than finding the right place to drag your
package to, and doing the dragging. It also gives you metadata for free. And
there's less risk of screwing up.



From greg.ewing at canterbury.ac.nz  Tue Mar 23 01:23:41 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 23 Mar 2010 12:23:41 +1200
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <ho88hk$n52$1@dough.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
Message-ID: <4BA80A0D.6020704@canterbury.ac.nz>

Antoine Pitrou wrote:
> Oh, and by the way, there can be a race condition between __pycache__ 
> creation and deletion (if it fails the test)

You can check whether the directory would be created with
the right user beforehand, and if not, don't create one
at all.

To exploit a race condition there, the attacker would have
to be capable of either changing the owner of the parent
directory or removing it and replacing it with a different
one, and if he can do that, he can do whatever he wants
anyway.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Mar 23 01:30:15 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 23 Mar 2010 12:30:15 +1200
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA7B1B4.4010000@wanadoo.fr>
References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk>
	<4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk>
	<4BA7B1B4.4010000@wanadoo.fr>
Message-ID: <4BA80B97.6080606@canterbury.ac.nz>

Pascal Chambon wrote:

> I don't follow you there - in my mind, the default __getattribute__ 
> could simply have wrapped all its operations inside soem kind of 
> "try..catch AttributeError:" mechanism, and thus been able to fallback 
> to __getattr__ in any way.

But then it would be incorrect to say that "__getattribute__
raises an exception".

When we say that a function raises an exception, we normally
mean that the exception propagates out of the function and
can be seen by the caller, not that it was raised and caught
somewhere inside the function.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Mar 23 01:33:48 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 23 Mar 2010 12:33:48 +1200
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>
	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
Message-ID: <4BA80C6C.5070809@canterbury.ac.nz>

Mark Dickinson wrote:
> It might make sense for
> Decimal + complex mixed-type operations to be disallowed, for example.

As long as you're allowing Decimal-float comparisons,
Decimal-complex comparison for equality has an obvious
interpretation.

-- 
Greg

From cs at zip.com.au  Tue Mar 23 01:40:02 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 23 Mar 2010 11:40:02 +1100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <20100322095604.3371039e@heresy>
References: <20100322095604.3371039e@heresy>
Message-ID: <20100323004002.GA29460@cskk.homeip.net>

On 22Mar2010 09:56, Barry Warsaw <barry at python.org> wrote:
| I have a pretty good start on PEP 3147 implementation [1], but I've
| encountered a situation that I'd like to get some feedback on.  Here's the
| test case illustrating the problem.  From test_import.py:
| 
|     def test_writable_directory(self):
|         # The umask is not conducive to creating a writable __pycache__
|         # directory.
|         with umask(0o222):
[...]
| The __pycache__ directory does not exist before the import, and the import
| machinery creates the directory, but the umask leaves the directory unwritable
| by anybody.  So of course when the import machinery goes to write the .pyc
| file inside __pycache__, it fails.  This does not cause an ImportError though,
| just like if today the package directory were unwritable.
| 
| This might be different than today's situation though because once the
| unwritable __pycache__ directory is created, nothing is going to change that
| without explicit user interaction, and that might be difficult after the
| fact.

Like any bad/suboptimal permission.

| I'm not sure what the right answer is.  Some possible choices:
| 
| * Tough luck

+1
I'd go with this one myself.

| * Force the umask so that the directory is writable, but then the question is,
|   by whom?  ugo+w or something less?

-2
Racy and dangerous. The umask is a UNIX process global, and other threads may
get bad results if they're active during this window.

| * Copy the permissions from the parent directory and ignore umask

-1
Maybe. But consider that you may not be the owner of the parent: then
the new child will have different ownership than the parent but the same
permission mask. Potentially a bad mix. This approach is very hard to
get right.

| * Raise an exception or refuse to create __pycache__ if it's not writable
|   (again, by whom?)

-3
Bleah. My python program won't run because an obscure (to the user)
directory had unusual permissions?

Tough ove, it's the only way:-)
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

When asked what would I most want to try before doing it, I said Death.
        - Michael Burton, michaelb at compnews.co.uk

From cs at zip.com.au  Tue Mar 23 01:45:19 2010
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 23 Mar 2010 11:45:19 +1100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <20100323004002.GA29460@cskk.homeip.net>
References: <20100323004002.GA29460@cskk.homeip.net>
Message-ID: <20100323004519.GA30414@cskk.homeip.net>

On 23Mar2010 11:40, I wrote:
| | * Raise an exception or refuse to create __pycache__ if it's not writable
| |   (again, by whom?)
| 
| -3
| Bleah. My python program won't run because an obscure (to the user)
| directory had unusual permissions?

Clarification: I'm -3 on the exception. Silent failure to make the
__pycache__ would do, and may be better than silently making a useless
(unwritable) one.  

How about:

  made_it = False
  ok = False
  try:
    if not isdir(__pycache__dir):
      mkdir(__pycache__dir)
      made_it = True
    write pyc content ...
    ok = True
  except OSError, IOerror etc:
    if not ok:
      os.remove pyc content file
      if made_it:
        rmdir(__pycache__dir) but be quiet if this fails,
          eg if it is not empty because another process or thread
             added stuff

So silent tidyup attempt on failure, but no escaping exception.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

The govt MUST regulate the Net NOW! We can't have average people saying
what's on their minds!  - ezwriter at netcom.com

From greg.ewing at canterbury.ac.nz  Tue Mar 23 01:57:27 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 23 Mar 2010 12:57:27 +1200
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <1269298415.5792.11.camel@localhost>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de>
	<loom.20100322T215811-374@post.gmane.org>
	<4BA7E04F.7070308@v.loewis.de>
	<loom.20100322T225133-106@post.gmane.org>
	<4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost>
Message-ID: <4BA811F7.2010003@canterbury.ac.nz>

Antoine Pitrou wrote:
> Having the Web server execute ad hoc system
> administration code is far from elegant and user-friendly.

With the right piece of code, you could create yourself
a setuid-apache shell and solve this problem once and for
all. :-)

-- 
Greg

From rrr at ronadam.com  Tue Mar 23 01:53:28 2010
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 22 Mar 2010 19:53:28 -0500
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <20100322162002.6daff86f@heresy>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<ho8esj$qnd$1@dough.gmane.org>
	<20100322162002.6daff86f@heresy>
Message-ID: <ho93e8$50o$1@dough.gmane.org>



Barry Warsaw wrote:
> On Mar 22, 2010, at 02:02 PM, Ron Adam wrote:
> 
>> If I understand correctly, we would have the current mode as the default, and
>> can trigger __pycache__ behavior simply by manually creating a __pycache__
>> directory and deleting any byte-code files in the module/program directory.
>>
>> I like this, it is easy to understand and can be used without messing with 
>> flags or environment variables.
> 
> Well, for a package with subpackages, it gets more complicated.  Definitely
> not something you're likely to do manually.  Antoine's suggestion of 'python
> -m compileall --pycache' would work, but I think it's also obscure enough that
> most Python users won't get the benefit.


May be a bit more complicated, but it should be easy to write tools to 
handle the repetitive stuff.

When I'm writing python projects I usually only work on one or two packages 
at a time at most.  Creating a couple of directories when I first get 
started on a project is nothing compared with the other 10 or more files of 
 > 1,000 lines of code each.  This has a very low mental hurdle too.

All the other packages I'm importing would probably already be preinstalled 
complete with __pycache__ directories and bytecode files.  As Antoine was 
suggesting that it would be up to the installer's (scrips) to create the 
__pycache__ directories at install time, either directly or by possibly 
issuing a 'python -m compileall --pycache' command?

Ron


From solipsis at pitrou.net  Tue Mar 23 01:58:42 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 Mar 2010 00:58:42 +0000 (UTC)
Subject: [Python-Dev] =?utf-8?q?=5F=5Fpycache=5F=5F_creation?=
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de>
	<loom.20100322T215811-374@post.gmane.org>
	<4BA7E04F.7070308@v.loewis.de>
	<loom.20100322T225133-106@post.gmane.org>
	<4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost>
	<4BA811F7.2010003@canterbury.ac.nz>
Message-ID: <loom.20100323T015317-989@post.gmane.org>

Greg Ewing <greg.ewing <at> canterbury.ac.nz> writes:
> 
> Antoine Pitrou wrote:
> > Having the Web server execute ad hoc system
> > administration code is far from elegant and user-friendly.
> 
> With the right piece of code, you could create yourself
> a setuid-apache shell and solve this problem once and for
> all. 

Well, if I can create a setuid apache shell, I can probably su as root or apache
as well.
("su -c rm -r whatever")

Or are you talking about a Web-based shell?



From rrr at ronadam.com  Tue Mar 23 02:03:30 2010
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 22 Mar 2010 20:03:30 -0500
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>	<ho8esj$qnd$1@dough.gmane.org>
	<20100322162002.6daff86f@heresy>
	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
Message-ID: <ho9413$6gk$1@dough.gmane.org>


Guido van Rossum wrote:
> On Mon, Mar 22, 2010 at 12:20 PM, Barry Warsaw <barry at python.org> wrote:
>> On Mar 22, 2010, at 02:02 PM, Ron Adam wrote:
>>
>>> If I understand correctly, we would have the current mode as the default, and
>>> can trigger __pycache__ behavior simply by manually creating a __pycache__
>>> directory and deleting any byte-code files in the module/program directory.

> Huh? Last time I looked weren't we going to make __pycache__ the
> default (and eventually only) behavior?

I expect that the __pycache__ directories would quickly become the 
recommended "defacto default" for writing and preinitializing modules and 
packages,  with the current behavior of having bytecode in the same 
directory as the .py files only as the fall-back (what I meant by default) 
behavior when the __pycache__ directories do not exist.


> I see only two reasonable solutions for __pycache__ creation -- either
> we change all setup/install scripts (both for core Python and for 3rd
> party packages) to always create a __pycache__ subdirectory for every
> directory (including package directories) installed; or we somehow
> create it the first time it's needed.
 >
> But creating it as needed runs into at least similar problems with
> ownership as creating .pyc files when first needed (if the parent
> directory is root-owned a mere mortal can't create it at all).
 >
> So even apart from the security issue (which I haven't thought about deeply) I
> think precreation should at least be an easily accessible option both
> for the core (where it can be done by compileall) and for 3rd party
> packages (where I guess it's up to distutils or whatever install
> mechanism is used).

Yes, I think that is what Antoine was also getting at.


Is there a need for python to use __pycache__ directories 100% of the time? 
   For 2.x it seems like being flexible would be best, and if 3.x is going 
to be strict about it, it should be strict sooner than later rather than 
have a lot of 3rd party packages break at some point down the road.

Ron


From ben+python at benfinney.id.au  Tue Mar 23 02:28:36 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 23 Mar 2010 12:28:36 +1100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<20100322161626.6e56efac@heresy>
	<loom.20100322T212223-259@post.gmane.org>
Message-ID: <87vdcnrfln.fsf@benfinney.id.au>

Antoine Pitrou <solipsis at pitrou.net> writes:

> Barry Warsaw <barry <at> python.org> writes:
> > 
> > When Python is being installed, either by a from-source 'make
> > install' or by the distro packager, then you'd expect the umask not
> > to be insane. In the latter case, it's a bug and in the former case
> > you screwed up so you should be able to delete and reinstall, or at
> > the very least execute the same `find` command that Python's own
> > Makefile calls (in my branch) for 'make clean'.
> > 
> > When you're installing packages, again, I would expect that the
> > system installer, or you via `easy_install` or whatever, would not
> > have an insane umask.
>
> Well, precisely. That's why I suggest that creating the __pycache__
> directories be done *at install time* (or packaging time), and not via
> the core import machinery (that is, not at import time). That is, when
> you *know* you are the right user, with the right umask.

+1.

Taking advantage of caching directories that have already been set up
correctly in advance at install time is friendly. Littering the runtime
directory with new subdirectories by default is not so friendly.

Perhaps also of note is that the FHS recommends systems use
?/var/cache/foo/? for cached data from applications:

    /var/cache : Application cache data

    Purpose

    /var/cache is intended for cached data from applications. Such data is
    locally generated as a result of time-consuming I/O or calculation. The
    application must be able to regenerate or restore the data. Unlike
    /var/spool, the cached files can be deleted without data loss. The data
    must remain valid between invocations of the application and rebooting
    the system.

    <URL:http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html#VARCACHEAPPLICATIONCACHEDATA>

This would suggest that Python could start using ?/var/cache/python/?
for its cached bytecode tree on systems that implement the FHS.

-- 
 \         ?Simplicity and elegance are unpopular because they require |
  `\           hard work and discipline to achieve and education to be |
_o__)                                appreciated.? ?Edsger W. Dijkstra |
Ben Finney


From tjreedy at udel.edu  Tue Mar 23 04:06:21 2010
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 22 Mar 2010 23:06:21 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <loom.20100322T191016-732@post.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
Message-ID: <ho9b7a$noo$1@dough.gmane.org>

On 3/22/2010 2:15 PM, Antoine Pitrou wrote:

> What I am proposing is that the creation of __pycache__ /directories/ be put
> outside of the core. It can be part of distutils, or of a separate module, or
> delegated to third-party tools. It could even be as simple as
> "python -m compileall --pycache", if someone implements it.
>
> Creation of the __pycache__ /contents/ (files inside the directory) would still
> be part of core Python, but only if the directory exists and is writable by the
> current process.

-1

If, as I have done several times recently, I create a directory and 
insert an empty __init__.py and several real module.py files, I want the 
.pycs to go into __pycache__ *automatically, by default, without me also 
having to remember to create an empty __pycache__ *directory*, *each 
time*.  Ugh.

Terry Jan Reedy


From doko at ubuntu.com  Tue Mar 23 05:55:03 2010
From: doko at ubuntu.com (Matthias Klose)
Date: Tue, 23 Mar 2010 05:55:03 +0100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <87vdcnrfln.fsf@benfinney.id.au>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<20100322161626.6e56efac@heresy>	<loom.20100322T212223-259@post.gmane.org>
	<87vdcnrfln.fsf@benfinney.id.au>
Message-ID: <4BA849A7.8060504@ubuntu.com>

On 23.03.2010 02:28, Ben Finney wrote:
> Antoine Pitrou<solipsis at pitrou.net>  writes:
>
>> Barry Warsaw<barry<at>  python.org>  writes:
>>>
>>> When Python is being installed, either by a from-source 'make
>>> install' or by the distro packager, then you'd expect the umask not
>>> to be insane. In the latter case, it's a bug and in the former case
>>> you screwed up so you should be able to delete and reinstall, or at
>>> the very least execute the same `find` command that Python's own
>>> Makefile calls (in my branch) for 'make clean'.
>>>
>>> When you're installing packages, again, I would expect that the
>>> system installer, or you via `easy_install` or whatever, would not
>>> have an insane umask.
>>
>> Well, precisely. That's why I suggest that creating the __pycache__
>> directories be done *at install time* (or packaging time), and not via
>> the core import machinery (that is, not at import time). That is, when
>> you *know* you are the right user, with the right umask.
>
> +1.
>
> Taking advantage of caching directories that have already been set up
> correctly in advance at install time is friendly. Littering the runtime
> directory with new subdirectories by default is not so friendly.
>
> Perhaps also of note is that the FHS recommends systems use
> ?/var/cache/foo/? for cached data from applications:
>
>      /var/cache : Application cache data
>
>      Purpose
>
>      /var/cache is intended for cached data from applications. Such data is
>      locally generated as a result of time-consuming I/O or calculation. The
>      application must be able to regenerate or restore the data. Unlike
>      /var/spool, the cached files can be deleted without data loss. The data
>      must remain valid between invocations of the application and rebooting
>      the system.
>
>      <URL:http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html#VARCACHEAPPLICATIONCACHEDATA>
>
> This would suggest that Python could start using ?/var/cache/python/?
> for its cached bytecode tree on systems that implement the FHS.

it reads *data*, not code.

From martin at v.loewis.de  Tue Mar 23 09:00:50 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 23 Mar 2010 09:00:50 +0100
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <ho9413$6gk$1@dough.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<ho8esj$qnd$1@dough.gmane.org>	<20100322162002.6daff86f@heresy>	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
	<ho9413$6gk$1@dough.gmane.org>
Message-ID: <4BA87532.10603@v.loewis.de>

> Is there a need for python to use __pycache__ directories 100% of the
> time?   For 2.x it seems like being flexible would be best, and if 3.x
> is going to be strict about it, it should be strict sooner than later
> rather than have a lot of 3rd party packages break at some point down
> the road.

For 2.x, nothing will happen, anyway (except for Linux distributions
perhaps integrating a patch on their own): 2.7b1 is about to be
released, after which point 2.x will not see any new features.

Regards,
Martin

From dickinsm at gmail.com  Tue Mar 23 10:16:03 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 23 Mar 2010 09:16:03 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA80C6C.5070809@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA5AA34.8050709@canterbury.ac.nz>
	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>
	<4BA6A2B2.6050207@canterbury.ac.nz>
	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<3D792060-F4B9-42EE-8FDC-18E919D7EC17@gmail.com>
	<ca471dc21003221000y1266b879i12720288926465be@mail.gmail.com>
	<90AFBA1F-96D7-4EB9-814C-B313776A1B98@gmail.com>
	<5c6f2a5d1003221126o260da583i94c1eb37cf101713@mail.gmail.com>
	<4BA80C6C.5070809@canterbury.ac.nz>
Message-ID: <5c6f2a5d1003230216o50f86e5el18380add2c03a044@mail.gmail.com>

On Tue, Mar 23, 2010 at 12:33 AM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Mark Dickinson wrote:
>>
>> It might make sense for
>> Decimal + complex mixed-type operations to be disallowed, for example.
>
> As long as you're allowing Decimal-float comparisons,
> Decimal-complex comparison for equality has an obvious
> interpretation.

Agreed.  Decimal-to-complex equality and inequality comparisons should
be permitted.  Order comparisons would raise TypeError (just as
complex-to-complex and float-to-complex comparisons do at the moment.)

Mark

From steve at pearwood.info  Tue Mar 23 12:02:11 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 23 Mar 2010 22:02:11 +1100
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy>
	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
Message-ID: <201003232202.11749.steve@pearwood.info>

On Tue, 23 Mar 2010 07:38:42 am Guido van Rossum wrote:
> But creating it as needed runs into at least similar problems with
> ownership as creating .pyc files when first needed (if the parent
> directory is root-owned a mere mortal can't create it at all).

Isn't that a feature though?

I don't see why this is a problem, perhaps somebody can explain what I'm 
missing.

Current system:
If the user doesn't have write permission in the appropriate directory, 
no .pyc file is created and the import uses the .py file only.

New system (proposal):
If the user doesn't have write permission in the appropriate directory, 
no __pycache__ folder is created and the import uses the .py file only. 
If the user has write permission and the __pycache__ folder is created, 
but the umask is screwy and no .pyc files can be created, no .pyc file 
is created and the import uses the .py file only despite the existence 
of an empty __pycache__ folder.

Why bother to delete the unwritable __pycache__ directory? If you can't 
write to it, neither can any process with the same or fewer 
permissions. If some other process has more permissions, it could write 
something nasty to the directory, but surely it's not Python's 
responsibility to be secure in the face of a compromised root or admin 
account?

As I see it, the only question is, should we warn the user that we can't 
write to the newly-created __pycache__ directory? I'm +0 on that. To be 
clear:

Can't create __pycache__? That's a feature. Fail silently and fall back 
to using the .py file alone.

Existing __pycache__ directory, but can't write to it? That's a feature 
too, treat it just like the above.

No __pycache__ directory, and creating it succeeds, but then import 
can't write to the freshly created directory? That's just weird, so I'm 
+0 on raising a warning and -1 on raising an exception.


-- 
Steven D'Aprano

From stefan at bytereef.org  Tue Mar 23 13:09:59 2010
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 23 Mar 2010 13:09:59 +0100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
Message-ID: <20100323120959.GA486@mail.bytereef.org>

Facundo Batista <facundobatista at gmail.com> wrote:
> On Fri, Mar 19, 2010 at 5:50 PM, Guido van Rossum <guido at python.org> wrote:
> 
> > As a downside, there is the worry that inadvertent mixing of Decimal
> > and float can compromise the correctness of programs in a way that is
> > hard to detect. But the anomalies above indicate that not fixing the
> 
> Decimal already has something that we can use in this case, and fits
> very nice here: Signals.

I like the simplicity of having a single signal (e.g. CoercionError), but
a strictness context flag could offer greater control for people who only
want pure decimal/integer operations.


For example:

  strictness 0: completely promiscuous behaviour

  strictness 1: current py3k behaviour

  strictness 2: current py3k behaviour + pure equality comparisons

  strictness 3: current py3k behaviour + pure equality comparisons +
                disallow NaN equality comparisons [1]


Just as an illustration, here is a quick and dirty diff using the
DefaultContext for simplicity:

Index: Lib/decimal.py
===================================================================
--- Lib/decimal.py      (revision 78352)
+++ Lib/decimal.py      (working copy)
@@ -3765,8 +3765,8 @@
     def __init__(self, prec=None, rounding=None,
                  traps=None, flags=None,
                  Emin=None, Emax=None,
-                 capitals=None, _clamp=0,
-                 _ignored_flags=None):
+                 capitals=None, strictness=1,
+                 _clamp=0, _ignored_flags=None):
         if flags is None:
             flags = []
         if _ignored_flags is None:
@@ -5785,7 +5785,9 @@
         return other
     if isinstance(other, int):
         return Decimal(other)
-    if raiseit:
+    if isinstance(other, float) and DefaultContext.strictness == 0:
+        return Decimal.from_float(other)
+    if raiseit or DefaultContext.strictness > 1:
         raise TypeError("Unable to convert %s to Decimal" % other)
     return NotImplemented
 
@@ -5800,7 +5802,8 @@
         flags=[],
         Emax=999999999,
         Emin=-999999999,
-        capitals=1
+        capitals=1,
+        strictness=1
 )



Stefan Krah


[1] See: http://mail.python.org/pipermail/python-dev/2009-November/093910.html,
         http://mail.python.org/pipermail/python-dev/2009-November/093952.html



From dickinsm at gmail.com  Tue Mar 23 14:20:03 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 23 Mar 2010 13:20:03 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <20100323120959.GA486@mail.bytereef.org>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
Message-ID: <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com>

On Tue, Mar 23, 2010 at 12:09 PM, Stefan Krah <stefan at bytereef.org> wrote:
> Facundo Batista <facundobatista at gmail.com> wrote:
>> On Fri, Mar 19, 2010 at 5:50 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> > As a downside, there is the worry that inadvertent mixing of Decimal
>> > and float can compromise the correctness of programs in a way that is
>> > hard to detect. But the anomalies above indicate that not fixing the
>>
>> Decimal already has something that we can use in this case, and fits
>> very nice here: Signals.
>
> I like the simplicity of having a single signal (e.g. CoercionError), but
> a strictness context flag could offer greater control for people who only
> want pure decimal/integer operations.

Sounds worth considering.

> For example:
>
> ?strictness 0: completely promiscuous behaviour
>
> ?strictness 1: current py3k behaviour
>
> ?strictness 2: current py3k behaviour + pure equality comparisons

Can you explain what you mean by "+ pure equality comparisons" here?
If I'm understanding correctly, this is a mode that's *more* strict
than the current py3k behaviour;  what's it disallowing that the
current py3k behaviour allows?

> ?strictness 3: current py3k behaviour + pure equality comparisons +
> ? ? ? ? ? ? ? ?disallow NaN equality comparisons [1]

Sorry, no.  I think there are good reasons for the current NaN
equality behaviour:  2.0 really *isn't* a NaN, and Decimal(2) ==
Decimal('nan') should return False rather than raising an exception.
And the decimal module provides compare and compare_signal for those
who want complete standards-backed control here.  Besides, this seems
to me to be an orthogonal issue to the issue of mixing Decimal with
other numeric types.

Mark

From ben+python at benfinney.id.au  Tue Mar 23 14:35:43 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 24 Mar 2010 00:35:43 +1100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<20100322161626.6e56efac@heresy>
	<loom.20100322T212223-259@post.gmane.org>
	<87vdcnrfln.fsf@benfinney.id.au> <4BA849A7.8060504@ubuntu.com>
Message-ID: <87iq8nqhxs.fsf@benfinney.id.au>

Matthias Klose <doko at ubuntu.com> writes:

> On 23.03.2010 02:28, Ben Finney wrote:
> > Perhaps also of note is that the FHS recommends systems use
> > ?/var/cache/foo/? for cached data from applications:
> >
> >      /var/cache : Application cache data
> >
> >      Purpose
> >
> >      /var/cache is intended for cached data from applications. Such data is
> >      locally generated as a result of time-consuming I/O or calculation. The
> >      application must be able to regenerate or restore the data. Unlike
> >      /var/spool, the cached files can be deleted without data loss. The data
> >      must remain valid between invocations of the application and rebooting
> >      the system.
> >
> >      <URL:http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html#VARCACHEAPPLICATIONCACHEDATA>
> >
> > This would suggest that Python could start using ?/var/cache/python/?
> > for its cached bytecode tree on systems that implement the FHS.
>
> it reads *data*, not code.

So what? There's no implication that data-which-happens-to-be-code is
unsuitable for storage in ?/var/cache/foo/?. Easily-regenerated Python
byte code for caching meets the description quite well, AFAICT.

-- 
 \      ?It seems intuitively obvious to me, which means that it might |
  `\                                           be wrong.? ?Chris Torek |
_o__)                                                                  |
Ben Finney


From ncoghlan at gmail.com  Tue Mar 23 15:36:57 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 00:36:57 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA7F7CB.7010603@canterbury.ac.nz>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<5c6f2a5d1003191613w4ee93d31v4abf8e8480204c13@mail.gmail.com>	<loom.20100320T160613-470@post.gmane.org>	<aac2c7cb1003201004r38b5ea87s393c7401ddb4cc0d@mail.gmail.com>	<4BA55858.2000800@canterbury.ac.nz>	<5d44f72f1003201627x72bd244fv8e9b44fc921c9fe6@mail.gmail.com>	<B0EAB835-77E3-4FD4-B65D-B579259CFB1F@gmail.com>	<4BA5AA34.8050709@canterbury.ac.nz>	<077AA3B1-4B9F-4F80-BEAC-5C51A57F7CAB@gmail.com>	<4BA6A2B2.6050207@canterbury.ac.nz>	<5c6f2a5d1003220223k12850877j97e7fef880dde864@mail.gmail.com>
	<4BA7F7CB.7010603@canterbury.ac.nz>
Message-ID: <4BA8D209.2040609@gmail.com>

Greg Ewing wrote:
> Mark Dickinson wrote:
> 
>> But the Fraction type is going to mess this up:  for Decimal +
>> Fraction ->  Decimal, I don't see any other sensible option than to
>> convert the Fraction using the current context, since lossless
>> conversion isn't generally possible.
> 
> You could convert the Decimal to a Fraction, do the arithmetic
> as Fractions, and convert the result back to Decimal using the
> current precision.

It gets rather messy implementation-wise if you do it that way. As
Stefan pointed out, given the Decimal module's existing execution model
of invoking "convert_other" to create a Decimal instance and then
proceeding from there, it seems most straightforward to adopt the same
approach for floats and Fractions (i.e. keep the changes to handle the
new types inside the existing argument conversion algorithm).

Such a change should make Decimal/float and Decimal/Fraction arithmetic
work in a comprehensible way, albeit with double-rounding in the case of
the latter.

Comparisons would still need to be handled separately to avoid the lossy
conversions.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Tue Mar 23 15:45:15 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 00:45:15 +1000
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <loom.20100323T005510-312@post.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<4BA80418.6030905@canterbury.ac.nz>
	<loom.20100323T005510-312@post.gmane.org>
Message-ID: <4BA8D3FB.5050603@gmail.com>

Antoine Pitrou wrote:
>> or if a user installs by dragging into
>> site-packages instead of using an installer?
> 
> Well... do people actually do this?

Yes. We do it all the time with unpackaged only-for-internal-use Python
code.

I wouldn't expect it to work with random packages downloaded from the
'net, but it works fine for our own stuff (since it expects to be used
this way).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From brian.curtin at gmail.com  Tue Mar 23 15:50:21 2010
From: brian.curtin at gmail.com (Brian Curtin)
Date: Tue, 23 Mar 2010 08:50:21 -0600
Subject: [Python-Dev] Request for commit access
Message-ID: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>

Hi all,

Having been active in bug triage and patch writing/reviewing since late
2009, it was suggested in the python-dev IRC channel that I request commit
access to the repository. I'm primarily a Windows user and have worked with
many of the other active contributors to diagnose issues and test patches
when they don't have direct access to Windows.

Brian Curtin

p.s. My contributor form in on file as of 2010-01-31.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/f5009148/attachment.html>

From steve at pearwood.info  Tue Mar 23 15:55:23 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 24 Mar 2010 01:55:23 +1100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <87iq8nqhxs.fsf@benfinney.id.au>
References: <20100322095604.3371039e@heresy> <4BA849A7.8060504@ubuntu.com>
	<87iq8nqhxs.fsf@benfinney.id.au>
Message-ID: <201003240155.24604.steve@pearwood.info>

On Wed, 24 Mar 2010 12:35:43 am Ben Finney wrote:
> Matthias Klose <doko at ubuntu.com> writes:
> > On 23.03.2010 02:28, Ben Finney wrote:
> > > Perhaps also of note is that the FHS recommends systems use
> > > ?/var/cache/foo/? for cached data from applications:
> > >
> > >      /var/cache : Application cache data
> > >
> > >      Purpose
> > >
> > >      /var/cache is intended for cached data from applications.
> > > Such data is locally generated as a result of time-consuming I/O
> > > or calculation. The application must be able to regenerate or
> > > restore the data. Unlike /var/spool, the cached files can be
> > > deleted without data loss. The data must remain valid between
> > > invocations of the application and rebooting the system.
> > >
> > >     
> > > <URL:http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html
> > >#VARCACHEAPPLICATIONCACHEDATA>
> > >
> > > This would suggest that Python could start using
> > > ?/var/cache/python/? for its cached bytecode tree on systems that
> > > implement the FHS.
> >
> > it reads *data*, not code.
>
> So what? There's no implication that data-which-happens-to-be-code is
> unsuitable for storage in ?/var/cache/foo/?. Easily-regenerated
> Python byte code for caching meets the description quite well,
> AFAICT.


While I strongly approve of the concept of a central cache directory for 
many things, I don't think that .pyc files fit the bill.

Since there is no privileged python user that runs all Python code, and 
since any unprivileged user needs to be able to write .pyc files, the 
cache directory needs to be world-writable. But since the data being 
stored is code, that opens up a fairly nasty security hole: user fred 
could overwrite the cached .pyc files used by user barney and cause 
barney to run any arbitrary code fred likes.

The alternative would be to have individual caches for every user. Apart 
from being wasteful of disk space ("but who cares, bigger disks are 
cheap") that just complicates everything. You would need:

/var/cache/python/<user>/

which would essentially make it impossible to ship pre-compiled .pyc 
files, since the packaging system couldn't predict what usernames (note 
plural) to store them under.

It is not true that one can necessarily delete the cached files without 
data loss. Python still supports .pyc-only packages, and AFAIK there 
are no plans to stop that, so deleting the .pyc file may delete the 
module.


-- 
Steven D'Aprano

From ezio.melotti at gmail.com  Tue Mar 23 15:58:26 2010
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Tue, 23 Mar 2010 16:58:26 +0200
Subject: [Python-Dev] Request for commit access
In-Reply-To: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
Message-ID: <4BA8D712.9070802@gmail.com>

On 23/03/2010 16.50, Brian Curtin wrote:
> Hi all,
>
> Having been active in bug triage and patch writing/reviewing since 
> late 2009, it was suggested in the python-dev IRC channel that I 
> request commit access to the repository. I'm primarily a Windows user 
> and have worked with many of the other active contributors to diagnose 
> issues and test patches when they don't have direct access to Windows.
>
> Brian Curtin
>
> p.s. My contributor form in on file as of 2010-01-31. 

+1

From dirkjan at ochtman.nl  Tue Mar 23 16:05:00 2010
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Tue, 23 Mar 2010 16:05:00 +0100
Subject: [Python-Dev] Request for commit access
In-Reply-To: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
Message-ID: <ea2499da1003230805i3702368ajb032ccfbd6a8d7b1@mail.gmail.com>

On Tue, Mar 23, 2010 at 15:50, Brian Curtin <brian.curtin at gmail.com> wrote:
> Having been active in bug triage and patch writing/reviewing since late
> 2009, it was suggested in the python-dev IRC channel that I request commit
> access to the repository. I'm primarily a Windows user and have worked with
> many of the other active contributors to diagnose issues and test patches
> when they don't have direct access to Windows.

I have usefully cooperated with Brian, so +1.

Cheers,

Dirkjan

From eric at trueblade.com  Tue Mar 23 16:09:10 2010
From: eric at trueblade.com (Eric Smith)
Date: Tue, 23 Mar 2010 11:09:10 -0400
Subject: [Python-Dev] Request for commit access
In-Reply-To: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
Message-ID: <4BA8D996.1060305@trueblade.com>

Brian Curtin wrote:
> Hi all,
> 
> Having been active in bug triage and patch writing/reviewing since late 
> 2009, it was suggested in the python-dev IRC channel that I request 
> commit access to the repository. I'm primarily a Windows user and have 
> worked with many of the other active contributors to diagnose issues and 
> test patches when they don't have direct access to Windows.
> 
> Brian Curtin

Having worked with Brian at PyCon and via email, I support this. 
Especially if it can be done soon so he can work on committing 
http://bugs.python.org/issue1578269, which I have not had time to focus on.

-- 
Eric.

From stefan at bytereef.org  Tue Mar 23 16:09:50 2010
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 23 Mar 2010 16:09:50 +0100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com>
Message-ID: <20100323150950.GA1562@yoda.bytereef.org>

Mark Dickinson <dickinsm at gmail.com> wrote:
> > I like the simplicity of having a single signal (e.g. CoercionError), but
> > a strictness context flag could offer greater control for people who only
> > want pure decimal/integer operations.
> 
> Sounds worth considering.
> 
> > For example:
> >
> > ?strictness 0: completely promiscuous behaviour
> >
> > ?strictness 1: current py3k behaviour
> >
> > ?strictness 2: current py3k behaviour + pure equality comparisons
> 
> Can you explain what you mean by "+ pure equality comparisons" here?
> If I'm understanding correctly, this is a mode that's *more* strict
> than the current py3k behaviour;  what's it disallowing that the
> current py3k behaviour allows?


It's disallowing all comparisons between e.g. float and decimal. The idea
is that the context can provide a cheap way of enforcing types for people
who like it:


>>> from decimal import *
>>> DefaultContext.strictness = 1
>>> Decimal(9) == 9.0
False
>>> Decimal(9) in [1, 4.0 ,9]
True


>>> DefaultContext.strictness = 2
>>> Decimal(9) == 9.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stefan/svn/py3k/Lib/decimal.py", line 858, in __eq__
    other = _convert_other(other)
  File "/home/stefan/svn/py3k/Lib/decimal.py", line 5791, in _convert_other
    raise TypeError("Unable to convert %s to Decimal" % other)
TypeError: Unable to convert 9.0 to Decimal
>>> Decimal(9) in [1, 4.0 ,9]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/stefan/svn/py3k/Lib/decimal.py", line 858, in __eq__
    other = _convert_other(other)
  File "/home/stefan/svn/py3k/Lib/decimal.py", line 5791, in _convert_other
    raise TypeError("Unable to convert %s to Decimal" % other)
TypeError: Unable to convert 4.0 to Decimal
>>> Decimal(9) in [1, 4 ,9]
True


This mode could help catch bugs like:

n = 7 / 3 # Programmer thinks this is an integer
x = Decimal(100)
while x != n:
    pass # do something
    x -= 1



> > ?strictness 3: current py3k behaviour + pure equality comparisons +
> > ? ? ? ? ? ? ? ?disallow NaN equality comparisons [1]
> 
> Sorry, no.  I think there are good reasons for the current NaN
> equality behaviour:  2.0 really *isn't* a NaN, and Decimal(2) ==
> Decimal('nan') should return False rather than raising an exception.
> And the decimal module provides compare and compare_signal for those
> who want complete standards-backed control here.

I'd like to make it an option for people who don't want to write:

while x.compare_signal(7) != 0

And I think that an sNaN should really signal by default.


> Besides, this seems
> to me to be an orthogonal issue to the issue of mixing Decimal with
> other numeric types.

Yes, it would kind of overload the strictness parameter. I see it as another
type of strictness, so I brought it up here. Level 3 would be a bit like
the highest warning level of a compiler.

But of course there's no need to discuss NaNs further in this thread other
than to show a possible use of the flag.


Stefan Krah




From rdmurray at bitdance.com  Tue Mar 23 16:27:48 2010
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 23 Mar 2010 11:27:48 -0400
Subject: [Python-Dev] Request for commit access
In-Reply-To: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
Message-ID: <20100323152748.50ECE1FD33F@kimball.webabinitio.net>

On Tue, 23 Mar 2010 08:50:21 -0600, Brian Curtin wrote:
> Having been active in bug triage and patch writing/reviewing since late
> 2009, it was suggested in the python-dev IRC channel that I request commit
> access to the repository. I'm primarily a Windows user and have worked with
> many of the other active contributors to diagnose issues and test patches
> when they don't have direct access to Windows.
> 
> Brian Curtin

+1 from me too.  I've been working with Brian in IRC, and also at the Pycon
sprint.  Brian is careful with his code and very responsive to feedback.
And we need more Windows devs :)

--
R. David Murray                                      www.bitdance.com

From bob at redivi.com  Tue Mar 23 16:43:11 2010
From: bob at redivi.com (Bob Ippolito)
Date: Tue, 23 Mar 2010 11:43:11 -0400
Subject: [Python-Dev] Decimal <-> float comparisons in py3k.
In-Reply-To: <5c6f2a5d1003201338p18aebd40y2c345b6dc8b861a6@mail.gmail.com>
References: <5c6f2a5d1003160741qd83eef0ua9f8184ff8440191@mail.gmail.com>
	<5c6f2a5d1003181141g7076d6bam348298d86af6bf62@mail.gmail.com>
	<4BA29FB1.9020909@gmail.com>
	<5c6f2a5d1003190237v5f33a822m7b38c20c8b84ff70@mail.gmail.com>
	<5c6f2a5d1003190307v595a8bbax55ae1e50536b2c4d@mail.gmail.com>
	<4BA3680D.8070909@gmail.com> <4BA41271.3010003@canterbury.ac.nz>
	<5c6f2a5d1003200416i59ce880fk9cec234caaac2eb0@mail.gmail.com>
	<ca471dc21003201256s687634an9d3bf1578af6b143@mail.gmail.com>
	<5c6f2a5d1003201338p18aebd40y2c345b6dc8b861a6@mail.gmail.com>
Message-ID: <6a36e7291003230843k3fa44402ob806d5fa47245436@mail.gmail.com>

On Sat, Mar 20, 2010 at 4:38 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Sat, Mar 20, 2010 at 7:56 PM, Guido van Rossum <guido at python.org> wrote:
>> I propose to reduce all hashes to the hash of a normalized fraction,
>> which we can define as a combination of the hashes for the numerator
>> and the denominator. Then all we have to do is figure fairly efficient
>> ways to convert floats and decimals to normalized fractions (not
>> necessarily Fractions). I may be naive but this seems doable: for a
>> float, the denominator is always a power of 2 and removing factors of
>> 2 from the denominator is easy (just right-shift until the last bit is
>> zero). For Decimal, the unnormalized denominator is always a power of
>> 10, and the normalization is a bit messier, but doesn't seem
>> excessively so. The resulting numerator and denominator may be large
>> numbers, but for typical use of Decimal and float they will rarely be
>> excessively large, and I'm not too worried about slowing things down
>> when they are (everything slows down when you're using really large
>> integers anyway).
>
> I *am* worried about slowing things down for large Decimals: ?if you
> can't put Decimal('1e1234567') into a dict or set without waiting for
> an hour for the hash computation to complete (because it's busy
> computing 10**1234567), I consider that a problem.
>
> But it's solvable! ?I've just put a patch on the bug tracker:
>
> http://bugs.python.org/issue8188
>
> It demonstrates how hashes can be implemented efficiently and
> compatibly for all numeric types, even large Decimals like the above.
> It needs a little tidying up, but it works.

I was interested in how the implementation worked yesterday,
especially given the lack of explanation in the margins of
numeric_hash3.patch. numeric_hash4.patch has much better comments, but
I didn't see this patch until after I had sufficiently deciphered the
previous patch and wrote most of this:
http://bob.pythonmac.org/archives/2010/03/23/py3k-unified-numeric-hash/

I'm not really qualified to review the patch, what little formal math
training I had has atrophied quite a bit over the years, but as far as
I can tell it seems to work. The results also seem to match the Python
implementations that I created.

-bob

From raymond.hettinger at gmail.com  Tue Mar 23 18:01:14 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 23 Mar 2010 10:01:14 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <20100323120959.GA486@mail.bytereef.org>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
Message-ID: <E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>


On Mar 23, 2010, at 5:09 AM, Stefan Krah wrote:
> 
> I like the simplicity of having a single signal (e.g. CoercionError), but
> a strictness context flag could offer greater control for people who only
> want pure decimal/integer operations.
> 
> 
> For example:
> 
>  strictness 0: completely promiscuous behaviour
> 
>  strictness 1: current py3k behaviour
> 
>  strictness 2: current py3k behaviour + pure equality comparisons
> 
>  strictness 3: current py3k behaviour + pure equality comparisons +
>                disallow NaN equality comparisons [1]
> 

The decimal module is already drowning in complexity,
so it would be best to keep it simple:  one boolean flag
that if set would warn about any implicit decimal/float
interaction.


Raymond

From dickinsm at gmail.com  Tue Mar 23 18:31:40 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 23 Mar 2010 17:31:40 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <20100323150950.GA1562@yoda.bytereef.org>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com>
	<20100323150950.GA1562@yoda.bytereef.org>
Message-ID: <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com>

On Tue, Mar 23, 2010 at 3:09 PM, Stefan Krah <stefan at bytereef.org> wrote:
> Mark Dickinson <dickinsm at gmail.com> wrote:
>> [Stefan]
>> >
>> > ?strictness 2: current py3k behaviour + pure equality comparisons
>>
>> Can you explain what you mean by "+ pure equality comparisons" here?
>> If I'm understanding correctly, this is a mode that's *more* strict
>> than the current py3k behaviour; ?what's it disallowing that the
>> current py3k behaviour allows?
>
> It's disallowing all comparisons between e.g. float and decimal. The idea
> is that the context can provide a cheap way of enforcing types for people
> who like it:

>>>> DefaultContext.strictness = 2
>>>> Decimal(9) == 9.0
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> ?File "/home/stefan/svn/py3k/Lib/decimal.py", line 858, in __eq__
> ? ?other = _convert_other(other)
> ?File "/home/stefan/svn/py3k/Lib/decimal.py", line 5791, in _convert_other
> ? ?raise TypeError("Unable to convert %s to Decimal" % other)
> TypeError: Unable to convert 9.0 to Decimal

Hmm.  It seems to me that deliberately making an __eq__ method between
hashable types raise an exception isn't something that should be done
lightly, since it can *really* screw up sets and dicts.  For example,
with your proposal, {9.0, Decimal(x)} would either raise or not,
depending on whether Decimal(x) happened to hash equal to 9.0 (if they
don't hash equal, then __eq__ will never be called).  If the hash is
regarded as essentially a black box (which is what it should be for
most users) then you can easily end up with code that almost always
works, but *very* occasionally and unpredicatably raises an exception.

> And I think that an sNaN should really signal by default.

Agreed, notwithstanding the above comments.  Though to avoid the
problems described above, I think the only way to make this acceptable
would be to prevent hashing of signaling nans.  (Which the decimal
module current does; it also prevents hashing of quiet NaNs, but I
can't see any good rationale for that.)

Mark

From rhamph at gmail.com  Tue Mar 23 18:48:20 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Tue, 23 Mar 2010 11:48:20 -0600
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com>
	<20100323150950.GA1562@yoda.bytereef.org>
	<5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com>
Message-ID: <aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>

On Tue, Mar 23, 2010 at 11:31, Mark Dickinson <dickinsm at gmail.com> wrote:
> Agreed, notwithstanding the above comments. ?Though to avoid the
> problems described above, I think the only way to make this acceptable
> would be to prevent hashing of signaling nans. ?(Which the decimal
> module current does; it also prevents hashing of quiet NaNs, but I
> can't see any good rationale for that.)

a = Decimal('nan')
a != a

They don't follow the behaviour required for being hashable.

float NaN should stop being hashable as well.


-- 
Adam Olsen, aka Rhamphoryncus

From dickinsm at gmail.com  Tue Mar 23 19:04:37 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 23 Mar 2010 18:04:37 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com>
	<20100323150950.GA1562@yoda.bytereef.org>
	<5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com>
	<aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>
Message-ID: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>

On Tue, Mar 23, 2010 at 5:48 PM, Adam Olsen <rhamph at gmail.com> wrote:
> a = Decimal('nan')
> a != a
>
> They don't follow the behaviour required for being hashable.

What's this required behaviour?  The only rule I'm aware of is that if
a == b then hash(a) == hash(b).  That's not violated here.

Note that containment tests check identity before equality, so there's
no problem with putting (float) nans in sets or dicts:

>>> x = float('nan')
>>> s = {x}
>>> x in s
True

Mark

From rhamph at gmail.com  Tue Mar 23 19:07:44 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Tue, 23 Mar 2010 12:07:44 -0600
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com>
	<20100323150950.GA1562@yoda.bytereef.org>
	<5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com>
	<aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>
	<5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>
Message-ID: <aac2c7cb1003231107i1f348bd3k39362f3634078a60@mail.gmail.com>

On Tue, Mar 23, 2010 at 12:04, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Tue, Mar 23, 2010 at 5:48 PM, Adam Olsen <rhamph at gmail.com> wrote:
>> a = Decimal('nan')
>> a != a
>>
>> They don't follow the behaviour required for being hashable.
>
> What's this required behaviour? ?The only rule I'm aware of is that if
> a == b then hash(a) == hash(b). ?That's not violated here.
>
> Note that containment tests check identity before equality, so there's
> no problem with putting (float) nans in sets or dicts:
>
>>>> x = float('nan')
>>>> s = {x}
>>>> x in s
> True

Ergh, I thought that got changed.  Nevermind then.


-- 
Adam Olsen, aka Rhamphoryncus

From dickinsm at gmail.com  Tue Mar 23 19:55:36 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 23 Mar 2010 18:55:36 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <aac2c7cb1003231107i1f348bd3k39362f3634078a60@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com>
	<20100323150950.GA1562@yoda.bytereef.org>
	<5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com>
	<aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>
	<5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>
	<aac2c7cb1003231107i1f348bd3k39362f3634078a60@mail.gmail.com>
Message-ID: <5c6f2a5d1003231155n211ee51fke7b62ced6f5d36c5@mail.gmail.com>

On Tue, Mar 23, 2010 at 6:07 PM, Adam Olsen <rhamph at gmail.com> wrote:
> On Tue, Mar 23, 2010 at 12:04, Mark Dickinson <dickinsm at gmail.com> wrote:
>> Note that containment tests check identity before equality, so there's
>> no problem with putting (float) nans in sets or dicts:
>>
>>>>> x = float('nan')
>>>>> s = {x}
>>>>> x in s
>> True
>
> Ergh, I thought that got changed. ?Nevermind then.

Hmm.  I think you're right:  it did get changed at some point early in
py3k's history;  I seem to recall that the identity-checking behaviour
got restored before 3.1 was released, though.  There was an issue
about this somewhere, but I'm failing to find it.

Mark

From rowen at uw.edu  Tue Mar 23 20:49:26 2010
From: rowen at uw.edu (Russell E. Owen)
Date: Tue, 23 Mar 2010 12:49:26 -0700
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
Message-ID: <rowen-C41123.12492623032010@news.gmane.org>

In article <4BA80418.6030905 at canterbury.ac.nz>,
 Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Antoine Pitrou wrote:
> 
> > In light of this issue, I'm -0.5 on __pycache__ becoming the default 
> > caching
> > mechanism. The directory ownership/permissions issue is too much of a mess,
> > especially for Web applications (think __pycache__ files created by the 
> > Apache
> > user).
> 
> Doesn't the existing .pyc mechanism have the same problem? Seems
> to me it's just as insecure to allow the Apache user to create
> .pyc files, since an attacker could overwrite them with arbitrary
> bytecode.
> 
> The only safe way is to pre-compile under a different user and
> make everything read-only to Apache. The same thing would apply
> under the __pycache__ regime.

This does sound like a bit security hole both in existing Python and the 
new __pycache__ proposed mechanism. It seems like this is the time to 
address it, while changing the caching mechanism.

If .pyc files are to be shared, it seems essential to (by default) 
generate them at install time and make them read-only for unprivileged 
users.

This in turn implies that we may have to give up some support for 
dragging python modules into site-packages, e.g. not generate .pyc files 
for such modules. At least if we go that route it will mostly affect 
power users, who can presumably cope.

-- Russell


From guido at python.org  Tue Mar 23 21:19:14 2010
From: guido at python.org (Guido van Rossum)
Date: Tue, 23 Mar 2010 12:19:14 -0800
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003231155n211ee51fke7b62ced6f5d36c5@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com> 
	<20100323120959.GA486@mail.bytereef.org>
	<5c6f2a5d1003230620g4564f18cib0228ab2afa583e@mail.gmail.com> 
	<20100323150950.GA1562@yoda.bytereef.org>
	<5c6f2a5d1003231031s3482b1a7w6a58d8c1afeb34e8@mail.gmail.com> 
	<aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com> 
	<5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com> 
	<aac2c7cb1003231107i1f348bd3k39362f3634078a60@mail.gmail.com> 
	<5c6f2a5d1003231155n211ee51fke7b62ced6f5d36c5@mail.gmail.com>
Message-ID: <ca471dc21003231319u4cc44091y2b35c7726272e9b5@mail.gmail.com>

On Tue, Mar 23, 2010 at 10:55 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Tue, Mar 23, 2010 at 6:07 PM, Adam Olsen <rhamph at gmail.com> wrote:
>> On Tue, Mar 23, 2010 at 12:04, Mark Dickinson <dickinsm at gmail.com> wrote:
>>> Note that containment tests check identity before equality, so there's
>>> no problem with putting (float) nans in sets or dicts:
>>>
>>>>>> x = float('nan')
>>>>>> s = {x}
>>>>>> x in s
>>> True
>>
>> Ergh, I thought that got changed. ?Nevermind then.
>
> Hmm. ?I think you're right: ?it did get changed at some point early in
> py3k's history; ?I seem to recall that the identity-checking behaviour
> got restored before 3.1 was released, though. ?There was an issue
> about this somewhere, but I'm failing to find it.

Raymond and I don't see this the same way. It looks like he won. :-)

-- 
--Guido van Rossum (python.org/~guido)

From henning.vonbargen at arcor.de  Tue Mar 23 21:27:47 2010
From: henning.vonbargen at arcor.de (Henning von Bargen)
Date: Tue, 23 Mar 2010 21:27:47 +0100
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <mailman.5002.1269355520.23597.python-dev@python.org>
References: <mailman.5002.1269355520.23597.python-dev@python.org>
Message-ID: <4BA92443.4050204@arcor.de>

Antoine Pitrou wrote:
 > > or if a user installs by dragging into
 > > site-packages instead of using an installer?
 >
 > Well... do people actually do this?

Why not?

And it is also common to just set the PYTHONPATH environment variable
instead of using setup.py install or copy the files to the
lib/site-packages directory.

I prefer it that way because it gives a clean separation between
standard library files and application specific files.
For example, if I want to use another Python version, I can just
delete the python26 directory because I know it doesn't contain
any application-specific code, just the standard library files.

Considering the complexity, the risks and the benefit of the
whole __pycache__ thingie, I'm -1...

Henning

From chambon.pascal at wanadoo.fr  Tue Mar 23 21:57:22 2010
From: chambon.pascal at wanadoo.fr (Pascal Chambon)
Date: Tue, 23 Mar 2010 21:57:22 +0100
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA80B97.6080606@canterbury.ac.nz>
References: <4BA3C962.7080500@wanadoo.fr>
	<4BA3DD4F.1080209@voidspace.org.uk>	<4BA4B8DE.8060109@wanadoo.fr>
	<4BA576AF.2050208@voidspace.org.uk>	<4BA7B1B4.4010000@wanadoo.fr>
	<4BA80B97.6080606@canterbury.ac.nz>
Message-ID: <4BA92B32.8050803@wanadoo.fr>

Greg Ewing a ?crit :
>
> Pascal Chambon wrote:
>
>> I don't follow you there - in my mind, the default __getattribute__ 
>> could simply have wrapped all its operations inside soem kind of 
>> "try..catch AttributeError:" mechanism, and thus been able to 
>> fallback to __getattr__ in any way.
>
> But then it would be incorrect to say that "__getattribute__
> raises an exception".
>
> When we say that a function raises an exception, we normally
> mean that the exception propagates out of the function and
> can be seen by the caller, not that it was raised and caught
> somewhere inside the function.
>
Indeed, but I've never run into any doc mentionning that the default 
__getattribute__ raised in exception instead of forwarding to 
__getattr__ by itself.
All I've found is "If the class also defines __getattr__() 
<http://docs.python.org/reference/datamodel.html#object.__getattr__>, 
the latter will not be called unless __getattribute__() 
<http://docs.python.org/reference/datamodel.html#object.__getattribute__> 
either calls it explicitly or raises an AttributeError 
<http://docs.python.org/library/exceptions.html#exceptions.AttributeError>"; 
that sentence which simply offers two alternatives for the behaviour of 
customized __gettattribute__ methods, without giving any hint on the 
behaviourthat was chosen when implementing object.__gettattribute__.

Or am I missing some other doc which I'm supposed to know  :?

"In the face of ambiguity, refuse the temptation to guess", as we say 
anyway, so I propose we patch the doc to clarify this point for newcomers ^^

Regards,
Pascal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/088e9338/attachment.html>

From greg.ewing at canterbury.ac.nz  Tue Mar 23 22:41:20 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 24 Mar 2010 09:41:20 +1200
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <loom.20100323T005510-312@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
	<loom.20100323T005510-312@post.gmane.org>
Message-ID: <4BA93580.9000402@canterbury.ac.nz>

Antoine Pitrou wrote:

> The main point of the __pycache__ proposal is to solve the needs of
> Ubuntu/Debian packagers. If you are developing (rather than deploying or
> building packages), you shouldn't have these needs AFAICT.

Maybe it's one point, but I'm not sure it's the *main* one.
Personally I would benefit most from it during development.
I hardly ever look in the directories of installed packages,
so I don't care what they look like.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Mar 23 22:48:51 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 24 Mar 2010 09:48:51 +1200
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <loom.20100323T015317-989@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de>
	<loom.20100322T215811-374@post.gmane.org>
	<4BA7E04F.7070308@v.loewis.de>
	<loom.20100322T225133-106@post.gmane.org>
	<4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost>
	<4BA811F7.2010003@canterbury.ac.nz>
	<loom.20100323T015317-989@post.gmane.org>
Message-ID: <4BA93743.90006@canterbury.ac.nz>

Antoine Pitrou wrote:

> Well, if I can create a setuid apache shell, I can probably su as root or apache
> as well.
> ("su -c rm -r whatever")
> 
> Or are you talking about a Web-based shell?

I'm just saying that if there is any way of running code of
your choice as the apache user, you can get it to make a
copy of /bin/sh and suid it.

Of course, if you have permission to su apache, then this
is not necessary. But then you wouldn't have to go through
web server contortions to fix apache-generated botchups
either.

-- 
Greg

From steve at pearwood.info  Tue Mar 23 23:00:30 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 24 Mar 2010 09:00:30 +1100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>
	<5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>
Message-ID: <201003240900.30334.steve@pearwood.info>

On Wed, 24 Mar 2010 05:04:37 am Mark Dickinson wrote:
> On Tue, Mar 23, 2010 at 5:48 PM, Adam Olsen <rhamph at gmail.com> wrote:
> > a = Decimal('nan')
> > a != a
> >
> > They don't follow the behaviour required for being hashable.
>
> What's this required behaviour?  The only rule I'm aware of is that
> if a == b then hash(a) == hash(b).  That's not violated here.
>
> Note that containment tests check identity before equality, so
> there's no problem with putting (float) nans in sets or dicts:
> >>> x = float('nan')
> >>> s = {x}
> >>> x in s
> True


As usual though, NANs are unintuitive:

>>> d = {float('nan'): 1}
>>> d[float('nan')] = 2
>>> d
{nan: 1, nan: 2}


I suspect that's a feature, not a bug.


-- 
Steven D'Aprano

From greg.ewing at canterbury.ac.nz  Tue Mar 23 23:14:12 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 24 Mar 2010 10:14:12 +1200
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <201003232202.11749.steve@pearwood.info>
References: <20100322095604.3371039e@heresy> <20100322162002.6daff86f@heresy>
	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
	<201003232202.11749.steve@pearwood.info>
Message-ID: <4BA93D34.3090407@canterbury.ac.nz>

Steven D'Aprano wrote:

> If the user has write permission and the __pycache__ folder is created, 
> but the umask is screwy and no .pyc files can be created, no .pyc file 
> is created and the import uses the .py file only despite the existence 
> of an empty __pycache__ folder.

Sounds okay to me.

-- 
Greg



From greg.ewing at canterbury.ac.nz  Tue Mar 23 23:37:12 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 24 Mar 2010 10:37:12 +1200
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <rowen-C41123.12492623032010@news.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
	<rowen-C41123.12492623032010@news.gmane.org>
Message-ID: <4BA94298.2060904@canterbury.ac.nz>

Russell E. Owen wrote:

> If .pyc files are to be shared, it seems essential to (by default) 
> generate them at install time and make them read-only for unprivileged 
> users.
> 
> This in turn implies that we may have to give up some support for 
> dragging python modules into site-packages

No, I don't think so. Currently, when you install a package (by
whatever means) in a directory that's not writable by the people
who will be running it, there are two possibilities:

1) Precompiled .pyc files are generated at installation time,
which are then read-only to users.

2) No .pyc files are installed, in which case none can or will be
created by users either, since they don't have write permission
to the directory.

None of this would change if __pycache__ directories were used.
The only difference would be that there would be an additional
mode for failure to create .pyc files, i.e. __pycache__ could
be created but nothing could be written to it because of a
umask issue.

If you install in a shared site-packages by dragging, you
already have to be careful about setting the permissions. You'd
just have to be sure to extended that diligence to any
contained __pycache__ directories.

-- 
Greg

From barry at python.org  Tue Mar 23 23:32:12 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 23 Mar 2010 18:32:12 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<ho8esj$qnd$1@dough.gmane.org> <20100322162002.6daff86f@heresy>
	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
Message-ID: <20100323183212.7463e098@heresy>

On Mar 22, 2010, at 12:38 PM, Guido van Rossum wrote:

>Huh? Last time I looked weren't we going to make __pycache__ the
>default (and eventually only) behavior?

We definitely agreed it would be the default in Python 3.2.

My recollection is that we agreed it would be the only on-demand way of
writing pyc files, but that Python would read a lone .pyc file where the
source would be if the source is missing, and that py_compile/compileall would
support optional creation of those lone .pyc files.

>I see only two reasonable solutions for __pycache__ creation -- either
>we change all setup/install scripts (both for core Python and for 3rd
>party packages) to always create a __pycache__ subdirectory for every
>directory (including package directories) installed; or we somehow
>create it the first time it's needed.
>
>But creating it as needed runs into at least similar problems with
>ownership as creating .pyc files when first needed (if the parent
>directory is root-owned a mere mortal can't create it at all). So even
>apart from the security issue (which I haven't thought about deeply) I
>think precreation should at least be an easily accessible option both
>for the core (where it can be done by compileall) and for 3rd party
>packages (where I guess it's up to distutils or whatever install
>mechanism is used).

So you're +1 on Tough Luck?  I think that's the best answer.  You will be able
to arrange pre-creation though compileall with the right layout and presumably
the right umask if you want.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/77db9189/attachment.pgp>

From martin at v.loewis.de  Tue Mar 23 23:34:42 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 23 Mar 2010 23:34:42 +0100
Subject: [Python-Dev] Request for commit access
In-Reply-To: <20100323152748.50ECE1FD33F@kimball.webabinitio.net>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
	<20100323152748.50ECE1FD33F@kimball.webabinitio.net>
Message-ID: <4BA94202.6060502@v.loewis.de>

>> Having been active in bug triage and patch writing/reviewing since late
>> 2009, it was suggested in the python-dev IRC channel that I request commit
>> access to the repository. I'm primarily a Windows user and have worked with
>> many of the other active contributors to diagnose issues and test patches
>> when they don't have direct access to Windows.
>>
>> Brian Curtin
> 
> +1 from me too.  I've been working with Brian in IRC, and also at the Pycon
> sprint.  Brian is careful with his code and very responsive to feedback.
> And we need more Windows devs :)

Procedurally, I wonder where people got the notion from that you can or
need to apply for commit access. IIUC, it used to be the case that you
would be recommended for commit access, by some (more or less senior)
fellow committer. That person then would work on actually getting commit
access to the new committer - perhaps by first asking other people in
private, to avoid any public embarrassment if access is ultimately
denied. IMO, that committer should then also mentor the new guy, both by
helping out in difficult cases, and by closely following commits to see
whether (possibly unstated) conventions are being followed.

I'm not really picking on Brian here specifically, I just want to point
out that I dislike this (apparent) change in process, primarily because
of the risk of embarrassment.

With all that said: I'm also fine with giving access to Brian (in
particular due to the endorsements that have already been posted). I
still would like somebody to step forward as a mentor, and I would need
Brian to send me his SSH key.

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Tue Mar 23 23:47:00 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 24 Mar 2010 10:47:00 +1200
Subject: [Python-Dev] Attribute lookup ambiguity
In-Reply-To: <4BA92B32.8050803@wanadoo.fr>
References: <4BA3C962.7080500@wanadoo.fr> <4BA3DD4F.1080209@voidspace.org.uk>
	<4BA4B8DE.8060109@wanadoo.fr> <4BA576AF.2050208@voidspace.org.uk>
	<4BA7B1B4.4010000@wanadoo.fr> <4BA80B97.6080606@canterbury.ac.nz>
	<4BA92B32.8050803@wanadoo.fr>
Message-ID: <4BA944E4.9000403@canterbury.ac.nz>

Pascal Chambon wrote:

> All I've found is "If the class also defines __getattr__(), 
> the latter will not be called unless __getattribute__()
> either calls it explicitly or raises an AttributeError

Hmmm. Well, it still implies that there is some mechanism
outside of __getattribute__ that will catch an AttributeError
and call __getattr__ for you. Given the existence of that
mechanism, it seems unlikely that the same thing would be
implemented over again in the standard __getattribute__
method.

So I don't think it requires very much guesswork to infer
that the standard __getattribute__ won't call __getattr__ on
its own.

I concede that the wording could be improved to remove any
possibility of doubt, though.

-- 
Greg

From barry at python.org  Tue Mar 23 23:37:16 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 23 Mar 2010 18:37:16 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <4BA7D76C.9010502@v.loewis.de>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de>
Message-ID: <20100323183716.1e9aac6f@heresy>

On Mar 22, 2010, at 09:47 PM, Martin v. L?wis wrote:

>Therefore, I'm in favor of having it on by default. If certain use cases
>make it problematic (e.g. Apache creating directories which you then
>cannot delete), there should be a way to turn it *off*. Perhaps the
>existing machinery to turn of byte code generation at all might be
>sufficient.

That's what I'm thinking.  Of course it will always be possible to run
compileall and get the __pycache__ directories pre-created, presumably with
the right umask.

>As for the original question (funny umasks), I think my proposal is
>"tough luck". Don't try to be super-smart; as Antoine explains, it gets
>worse, not better. If the user has arranged that Python will create
>unusable directories, the user better changes his setup.

I completely agree; +1

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/38d3f9c5/attachment.pgp>

From barry at python.org  Tue Mar 23 23:42:22 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 23 Mar 2010 18:42:22 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <loom.20100322T225133-106@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de>
	<loom.20100322T215811-374@post.gmane.org>
	<4BA7E04F.7070308@v.loewis.de>
	<loom.20100322T225133-106@post.gmane.org>
Message-ID: <20100323184222.28cb9a3a@heresy>

On Mar 22, 2010, at 09:57 PM, Antoine Pitrou wrote:

>It's especially annoying, of course, if you have to ask someone else to
>remove the directories for you (or if you have to write custom code and get
>it executed by the Apache or WSGI handler...).

compileall should probably grow a --clean option which would be essentially
equivalent to

    find . -name '__pycache__' | xargs rmdir

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/01e831b2/attachment.pgp>

From solipsis at pitrou.net  Tue Mar 23 23:57:48 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 Mar 2010 22:57:48 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?=
	=?utf-8?q?s_and_umask?=
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
	<loom.20100323T005510-312@post.gmane.org>
	<4BA93580.9000402@canterbury.ac.nz>
Message-ID: <loom.20100323T235432-758@post.gmane.org>

Greg Ewing <greg.ewing <at> canterbury.ac.nz> writes:
> > The main point of the __pycache__ proposal is to solve the needs of
> > Ubuntu/Debian packagers. If you are developing (rather than deploying or
> > building packages), you shouldn't have these needs AFAICT.
> 
> Maybe it's one point, but I'm not sure it's the *main* one.

It's the only reason the PEP was originally designed, and proposed.

> Personally I would benefit most from it during development.

Why? What benefit would it bring to you?

> I hardly ever look in the directories of installed packages,
> so I don't care what they look like.

Neither do I, but Ubuntu/Debian packagers want to share the source code of
third-party libraries across Python versions while keeping distinct bytecode
files (obviously).
Again, that's the original point of the PEP as proposed by Barry.



From barry at python.org  Wed Mar 24 00:01:22 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 23 Mar 2010 19:01:22 -0400
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <loom.20100322T212223-259@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<20100322161626.6e56efac@heresy>
	<loom.20100322T212223-259@post.gmane.org>
Message-ID: <20100323190122.54513b4a@heresy>

On Mar 22, 2010, at 08:33 PM, Antoine Pitrou wrote:

>Well, precisely. That's why I suggest that creating the __pycache__
>directories be done *at install time* (or packaging time), and not via the
>core import machinery (that is, not at import time). That is, when you *know*
>you are the right user, with the right umask.

I don't think they're mutually exclusive.  We will definitely give users the
tool to do compilation at install time via compileall.  That needn't preclude
on-demand creation, which will generally Just Work.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/002307ce/attachment.pgp>

From barry at python.org  Wed Mar 24 00:02:47 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 23 Mar 2010 19:02:47 -0400
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <87iq8nqhxs.fsf@benfinney.id.au>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<20100322161626.6e56efac@heresy>
	<loom.20100322T212223-259@post.gmane.org>
	<87vdcnrfln.fsf@benfinney.id.au> <4BA849A7.8060504@ubuntu.com>
	<87iq8nqhxs.fsf@benfinney.id.au>
Message-ID: <20100323190247.29be43c7@heresy>

On Mar 24, 2010, at 12:35 AM, Ben Finney wrote:

>So what? There's no implication that data-which-happens-to-be-code is
>unsuitable for storage in ?/var/cache/foo/?. Easily-regenerated Python
>byte code for caching meets the description quite well, AFAICT.

pyc files don't go there now, so why would PEP 3147 change that?

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/4fcc82d0/attachment.pgp>

From solipsis at pitrou.net  Wed Mar 24 00:01:43 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 Mar 2010 23:01:43 +0000 (UTC)
Subject: [Python-Dev] Request for commit access
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
	<20100323152748.50ECE1FD33F@kimball.webabinitio.net>
	<4BA94202.6060502@v.loewis.de>
Message-ID: <loom.20100323T235922-727@post.gmane.org>

Martin v. L?wis <martin <at> v.loewis.de> writes:
> 
> Procedurally, I wonder where people got the notion from that you can or
> need to apply for commit access. IIUC, it used to be the case that you
> would be recommended for commit access, by some (more or less senior)
> fellow committer. That person then would work on actually getting commit
> access to the new committer - perhaps by first asking other people in
> private, to avoid any public embarrassment if access is ultimately
> denied. IMO, that committer should then also mentor the new guy, both by
> helping out in difficult cases, and by closely following commits to see
> whether (possibly unstated) conventions are being followed.
> 
> I'm not really picking on Brian here specifically, I just want to point
> out that I dislike this (apparent) change in process, primarily because
> of the risk of embarrassment.

For the record, I'm not opposing any point you are making, but all this is not
clearly written out, and I think that's why people (including me) lately have
been thinking that the candidate for commit rights had to declare himself on
this mailing-list.

If the process goes differently, it probably deserves documenting somewhere.

Regards

Antoine.



From barry at python.org  Wed Mar 24 00:07:24 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 23 Mar 2010 19:07:24 -0400
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <rowen-C41123.12492623032010@news.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
	<rowen-C41123.12492623032010@news.gmane.org>
Message-ID: <20100323190724.46d5dc03@heresy>

On Mar 23, 2010, at 12:49 PM, Russell E. Owen wrote:

>If .pyc files are to be shared, it seems essential to (by default) 
>generate them at install time and make them read-only for unprivileged 
>users.

I think in practice this is what's almost always going to happen for system
Python source, either via your distribution's installer or distutils.  I think
most on-demand creation of pyc files will be when you're developing your own
code.  I don't want to have to run a separate step to get the benefit of
__pycache__, but I admit it's outside the scope of the PEP's original
intention.  I leave it to the BDFL.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/fc7697e0/attachment.pgp>

From ben+python at benfinney.id.au  Wed Mar 24 00:16:30 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 24 Mar 2010 10:16:30 +1100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
	<loom.20100323T005510-312@post.gmane.org>
Message-ID: <87eijar5m9.fsf@benfinney.id.au>

Antoine Pitrou <solipsis at pitrou.net> writes:

Steven D'Aprano <steve at pearwood.info> writes:

> On Wed, 24 Mar 2010 12:35:43 am Ben Finney wrote:
> > > On 23.03.2010 02:28, Ben Finney wrote:
> > > > <URL:http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html
> > > >#VARCACHEAPPLICATIONCACHEDATA>
> > > >
> > > > This would suggest that Python could start using
> > > > ?/var/cache/python/? for its cached bytecode tree on systems
> > > > that implement the FHS.
> > [?] There's no implication that data-which-happens-to-be-code is
> > unsuitable for storage in ?/var/cache/foo/?. Easily-regenerated
> > Python byte code for caching meets the description quite well,
> > AFAICT.
>
> While I strongly approve of the concept of a central cache directory
> for many things, I don't think that .pyc files fit the bill.
>
> Since there is no privileged python user that runs all Python code,
> and since any unprivileged user needs to be able to write .pyc files,

Hold up; my understanding is that, as Antoine Pitrou says:

> The main point of the __pycache__ proposal is to solve the needs of
> Ubuntu/Debian packagers. If you are developing (rather than deploying
> or building packages), you shouldn't have these needs AFAICT.

So, the packaging system will, by definition, have access to write to
FHS directories and those directories don't need to be world-writable.

-- 
 \         ?Pinky, are you pondering what I'm pondering?? ?I think so, |
  `\         Brain, but how will we get a pair of Abe Vigoda's pants?? |
_o__)                                           ?_Pinky and The Brain_ |
Ben Finney


From barry at python.org  Wed Mar 24 00:18:25 2010
From: barry at python.org (Barry Warsaw)
Date: Tue, 23 Mar 2010 19:18:25 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <4BA7F71D.6030907@v.loewis.de>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<20100322161817.06a8911f@heresy> <4BA7D76C.9010502@v.loewis.de>
	<loom.20100322T215811-374@post.gmane.org>
	<4BA7E04F.7070308@v.loewis.de>
	<loom.20100322T225133-106@post.gmane.org>
	<4BA7EC9A.1010206@v.loewis.de> <1269298415.5792.11.camel@localhost>
	<4BA7F71D.6030907@v.loewis.de>
Message-ID: <20100323191825.27fe28e5@heresy>

On Mar 23, 2010, at 12:02 AM, Martin v. L?wis wrote:

>I think the appropriate action at this point is to record this specific
>objection in the PEP.

Done.
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100323/f1c5d982/attachment.pgp>

From ijmorlan at uwaterloo.ca  Wed Mar 24 01:50:20 2010
From: ijmorlan at uwaterloo.ca (Isaac Morland)
Date: Tue, 23 Mar 2010 20:50:20 -0400 (EDT)
Subject: [Python-Dev]
 =?utf-8?q?PEP_3147=2C_=5F=5Fpycache=5F=5F_directorie?=
 =?utf-8?q?s_and_umask?=
In-Reply-To: <loom.20100323T235432-758@post.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
	<loom.20100323T005510-312@post.gmane.org>
	<4BA93580.9000402@canterbury.ac.nz>
	<loom.20100323T235432-758@post.gmane.org>
Message-ID: <Pine.GSO.4.64.1003232042080.23175@core.cs.uwaterloo.ca>

On Tue, 23 Mar 2010, Antoine Pitrou wrote:

> Greg Ewing <greg.ewing <at> canterbury.ac.nz> writes:
>>> The main point of the __pycache__ proposal is to solve the needs of
>>> Ubuntu/Debian packagers. If you are developing (rather than deploying or
>>> building packages), you shouldn't have these needs AFAICT.
>>
>> Maybe it's one point, but I'm not sure it's the *main* one.
>
> It's the only reason the PEP was originally designed, and proposed.

At least one additional use case has appeared.  Actually, my use case was 
mentioned long ago, but I didn't really push (e.g. by writing a patch) and 
nobody jumped on it.  But this PEP solves my case too, so it should not be 
ignored just because the immediate impetus for the PEP is another case.

>> Personally I would benefit most from it during development.
>
> Why? What benefit would it bring to you?

I'm sure Greg will jump in if I'm wrong about what he is saying, but the 
benefit to me and to Greg and to others writing .py code is that our 
directories will contain *.py and __pycache__, rather than *.py and *.pyc. 
So it will be much easier to see what is actually there.

Or if we're using SVN and we do "svn status", the only spurious result 
will be "? __pycache__" rather than "? X.pyc" for every X.py in the 
directory.

Or whatever other good effects come from having less junk in our source 
directories.

Directory tidiness is a positive general feature with at least a few 
specific benefits.

Isaac Morland			CSCF Web Guru
DC 2554C, x36650		WWW Software Specialist

From solipsis at pitrou.net  Wed Mar 24 02:05:16 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 24 Mar 2010 02:05:16 +0100
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <Pine.GSO.4.64.1003232042080.23175@core.cs.uwaterloo.ca>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
	<loom.20100323T005510-312@post.gmane.org>
	<4BA93580.9000402@canterbury.ac.nz>
	<loom.20100323T235432-758@post.gmane.org>
	<Pine.GSO.4.64.1003232042080.23175@core.cs.uwaterloo.ca>
Message-ID: <1269392716.4874.3.camel@localhost>

Le mardi 23 mars 2010 ? 20:50 -0400, Isaac Morland a ?crit :
> 
> I'm sure Greg will jump in if I'm wrong about what he is saying, but the 
> benefit to me and to Greg and to others writing .py code is that our 
> directories will contain *.py and __pycache__, rather than *.py and *.pyc. 
> So it will be much easier to see what is actually there.

I don't really get it. I have never had any problem to see "what is
actually here" in a Python source directory, despite the presence of pyc
files.

> Or if we're using SVN and we do "svn status", the only spurious result 
> will be "? __pycache__" rather than "? X.pyc" for every X.py in the 
> directory.

Well, I was assuming that everyone had been using svn:ignore,
or .hgignore, for years.
Similarly, you will configure svn, hg, or any other system, to ignore
__pycache__ directories (or .pyc files) so that they don't appear in
"svn status".

> Directory tidiness is a positive general feature with at least a few 
> specific benefits.

It's still mostly cosmetic and I don't think it's as serious as any
positive or negative system administration effects the change may have.



From rrr at ronadam.com  Wed Mar 24 02:42:44 2010
From: rrr at ronadam.com (Ron Adam)
Date: Tue, 23 Mar 2010 20:42:44 -0500
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <ho9b7a$noo$1@dough.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>
	<ho9b7a$noo$1@dough.gmane.org>
Message-ID: <hobqml$5d4$1@dough.gmane.org>



Terry Reedy wrote:
> On 3/22/2010 2:15 PM, Antoine Pitrou wrote:
> 
>> What I am proposing is that the creation of __pycache__ /directories/ 
>> be put
>> outside of the core. It can be part of distutils, or of a separate 
>> module, or
>> delegated to third-party tools. It could even be as simple as
>> "python -m compileall --pycache", if someone implements it.
>>
>> Creation of the __pycache__ /contents/ (files inside the directory) 
>> would still
>> be part of core Python, but only if the directory exists and is 
>> writable by the
>> current process.
> 
> -1
> 
> If, as I have done several times recently, I create a directory and 
> insert an empty __init__.py and several real module.py files, I want the 
> .pycs to go into __pycache__ *automatically, by default, without me also 
> having to remember to create an empty __pycache__ *directory*, *each 
> time*.  Ugh.

I think I misunderstood this at first.

It looks like, while developing a python 3.2+ program, if you don't create 
an empty __pycache__ directory, everything will still work, you just won't 
get the .pyc files.  That can be a good thing during development because 
you also will not have any problems with old .pyc files hanging around if 
you move or rename files.

The startup time may just be a tad longer, but probably not enough to be 
much of a problem.  If it is a problem you can just create the __pycache__ 
directory, but nothing bad will happen if you don't.

Ron






From lgautier at gmail.com  Wed Mar 24 06:17:00 2010
From: lgautier at gmail.com (Laurent Gautier)
Date: Wed, 24 Mar 2010 06:17:00 +0100
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <20100320031326.GM1623@idyll.org>
References: <20100319023630.GC1617@idyll.org>
	<4BA31858.3010904@gmail.com>	<bad82a81003190457w5c0e1770hd7c0f4af3f0a8746@mail.gmail.com>	<4BA388E5.1030703@gmail.com>
	<4BA40220.30701@v.loewis.de> <20100320031326.GM1623@idyll.org>
Message-ID: <4BA9A04C.4010405@gmail.com>

On 3/20/10 4:13 AM, C. Titus Brown wrote:
> On Sat, Mar 20, 2010 at 12:00:48AM +0100, "Martin v. L?wis" wrote:
>>> Whether this is worth weeks of work or not will depend on a given
>>> student's knowledge about Python 3, and I'd suspect that the GSoC would
>>> be an opportunity for a number of applicant to actually learn the
>>> intricacies of Python 3.
>>> Developing Python 3-specific features could be used to increase the
>>> amount of work on the project, but I am uncertain of whether this is
>>> worth a full GSoC.
>>
>> I'd say this would definitely make a GSoC project; any non-trivial
>> porting will be.
>
> Sounds good to me.  Line up a good student and bob's your uncle.
>
> --titus


I may have a student that is looking interested, and I have inserted the 
porting project to the Wiki table.


Laurent.


PS: Editing that table felt like a test in disguise ;-)

From stephen at xemacs.org  Wed Mar 24 06:36:40 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 24 Mar 2010 14:36:40 +0900
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <201003240900.30334.steve@pearwood.info>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>
	<5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>
	<201003240900.30334.steve@pearwood.info>
Message-ID: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>

Steven D'Aprano writes:

 > As usual though, NANs are unintuitive:
 > 
 > >>> d = {float('nan'): 1}
 > >>> d[float('nan')] = 2
 > >>> d
 > {nan: 1, nan: 2}
 > 
 > 
 > I suspect that's a feature, not a bug.

I don't see how it can be so.  Aren't all of those entries garbage?
To compute a histogram of results for computations on a series of
cases would you not have to test each result for NaN-hood, then hash
on a proxy such as the string "Nan"?

From amauryfa at gmail.com  Wed Mar 24 09:42:52 2010
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 24 Mar 2010 09:42:52 +0100
Subject: [Python-Dev] argparse.py is licensed under the Apache License
Message-ID: <e27efe131003240142v50833b3cu80071e69062cb201@mail.gmail.com>

Hello,

I noticed that the newly added argparse module has an unusual
licence header,?included below. This is the only file in the Python tree
that contains an explicit reference to the Apache License,
and this leads me to some questions:

- Is the Apache license compatible with the Python license?
Will this cause problem for some organizations that redistribute Python,
possibly with proprietary software? Are there additional constraints?

- Does this addition require a paragraph in the python documentation?
http://docs.python.org/license.html#licenses-and-acknowledgements-for-incorporated-software

- The Apache License states that::
?? ?You must cause any modified files to carry prominent notices stating
?? ?that You changed the files
but r78749 already modified the file (to remove a py3k warning)
didn't we break the License?

- Did the contributor sign a Contributor agreement? In this case,
shouldn't the code be marked as "Licensed to PSF under a Contributor Agreement",
as?mentioned in the contribution form?
http://www.python.org/psf/contrib/contrib-form
And then, could this Apache License be removed?


The first lines of Lib/argparse.py are:
# Copyright 2006-2009 Steven J. Bethard <steven.bethard at gmail.com>.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
# use this file except in compliance with the License. You may obtain a copy
# of the License at
#
# ? ? http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

--
Amaury Forgeot d'Arc

From dickinsm at gmail.com  Wed Mar 24 10:51:36 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 09:51:36 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>
	<5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>
	<201003240900.30334.steve@pearwood.info>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>

On Wed, Mar 24, 2010 at 5:36 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Steven D'Aprano writes:
>
> ?> As usual though, NANs are unintuitive:
> ?>
> ?> >>> d = {float('nan'): 1}
> ?> >>> d[float('nan')] = 2
> ?> >>> d
> ?> {nan: 1, nan: 2}
> ?>
> ?>
> ?> I suspect that's a feature, not a bug.

Right:  distinct nans (i.e., those with different id()) are treated as
distinct set elements or dict keys.

> I don't see how it can be so. ?Aren't all of those entries garbage?
> To compute a histogram of results for computations on a series of
> cases would you not have to test each result for NaN-hood, then hash
> on a proxy such as the string "Nan"?

So what alternative behaviour would you suggest, and how would you implement it?

I agree that many aspects of the current treatment of nans aren't
ideal, but I as far as I can see that's unavoidable.  For sane
containment testing, Python's == operator needs to give an equivalence
relation.  Meanwhile IEEE 754 requires that nans compare unequal to
themselves, breaking reflexivity.  So there have to be some
compromises somewhere.

The current compromise at least has the virtue that it doesn't require
special-casing nans anywhere in the general containment-testing and
hashing machinery.

One alternative would be to prohibit putting nans into sets and dicts
by making them unhashable;  I'm not sure what that would gain, though.
 And there would still be some unintuitive behaviour for containment
testing of nans in lists.

Mark

From techtonik at gmail.com  Wed Mar 24 10:59:34 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 24 Mar 2010 11:59:34 +0200
Subject: [Python-Dev] At least one package management tool for 2.7
Message-ID: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>

I wonder if there are many people here who don't use some kind of
"easy_install" package for package management in their Python /
virtualenv installations? I propose to include at least one such
package that is capable to auto-update itself in Python 2.7

C:\~env\Python27>python.exe -m easy_install
C:\~env\Python27\python.exe: No module named easy_install

C:\~env\Python27>python.exe -m pip
C:\~env\Python27\python.exe: No module named pip


It bugs me when I have to troubleshoot things on yet another machine
that doesn't have some kind of `setuptools` installed. Or when I have
to test some bug in my package on different Python version with a
clean install and need some dependencies.

-- 
anatoly t.

From ziade.tarek at gmail.com  Wed Mar 24 11:26:20 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 24 Mar 2010 11:26:20 +0100
Subject: [Python-Dev] At least one package management tool for 2.7
In-Reply-To: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
Message-ID: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>

On Wed, Mar 24, 2010 at 10:59 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> I wonder if there are many people here who don't use some kind of
> "easy_install" package for package management in their Python /
> virtualenv installations? I propose to include at least one such
> package that is capable to auto-update itself in Python 2.7
>
> C:\~env\Python27>python.exe -m easy_install
> C:\~env\Python27\python.exe: No module named easy_install
>
> C:\~env\Python27>python.exe -m pip
> C:\~env\Python27\python.exe: No module named pip
>
>
> It bugs me when I have to troubleshoot things on yet another machine
> that doesn't have some kind of `setuptools` installed. Or when I have
> to test some bug in my package on different Python version with a
> clean install and need some dependencies.


We are working on distutils2 right now to improve the situation, and
Ian has proposed to work on the possible inclusion of virtualenv in
the stldib as well.

I'll talk for distutils2 :

The plan is to provide a distutils2 standalone version that can be
installed from 2.4 to 3.x, and that will provide a basic
installer/uninstaller via -m.

Distutils2 is planned to be reintegrated in the stdlib in Python 3.3,
and my goal is to release it when Python 2.7 final is released.

The open question is: do we want to include a full installer that
takes care of installing / removing dependencies as well ?

I think not. Pip already provides this feature on the top of distutils
(and distutils2 later I guess) and is not hard to install on the top
of Python.

But the "auto-update" story seems interesting, can you expand on this ?

Tarek

-- 
Tarek Ziad? | http://ziade.org

From techtonik at gmail.com  Wed Mar 24 11:31:06 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 24 Mar 2010 12:31:06 +0200
Subject: [Python-Dev] Joel Spolsky on Mercurial
In-Reply-To: <ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
References: <hnu6fu$ufa$1@dough.gmane.org>
	<ea2499da1003190200w42d08f29laf86c41a6396c2b8@mail.gmail.com>
Message-ID: <d34314101003240331q39eb6c6cjef8b372a9d8fbbbf@mail.gmail.com>

On Fri, Mar 19, 2010 at 11:00 AM, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:
>> Having gotten that far, I think this might be worth referencing in new dev
>> docs.
>
> Will do. I finally read hginit yesterday, after having seen people
> rave about it on twitter for a few weeks, and it's a very friendly
> introduction.

I more like hands-on approach that does use real world workflow
instead of academic examples. For example, tutorial like Ondrej Certik
made for SymPy patches using Mercurial queues.
http://docs.sympy.org/sympy-patches-tutorial.html  It goes straight
from the problem in SymPy development you'd like to resolve for
yourself and then shows how Mercurial helps with it. If you have more
time to dig - you may read a book or Joel series.

-- 
anatoly t.

From steve at pearwood.info  Wed Mar 24 12:14:39 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 24 Mar 2010 22:14:39 +1100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
Message-ID: <201003242214.39778.steve@pearwood.info>

On Wed, 24 Mar 2010 08:51:36 pm Mark Dickinson wrote:
> On Wed, Mar 24, 2010 at 5:36 AM, Stephen J. Turnbull 
<stephen at xemacs.org> wrote:
> > Steven D'Aprano writes:
> >
> > ?> As usual though, NANs are unintuitive:
> > ?>
> > ?> >>> d = {float('nan'): 1}
> > ?> >>> d[float('nan')] = 2
> > ?> >>> d
> > ?> {nan: 1, nan: 2}
> > ?>
> > ?>
> > ?> I suspect that's a feature, not a bug.
>
> Right:  distinct nans (i.e., those with different id()) are treated
> as distinct set elements or dict keys.
>
> > I don't see how it can be so. ?Aren't all of those entries garbage?
> > To compute a histogram of results for computations on a series of
> > cases would you not have to test each result for NaN-hood, then
> > hash on a proxy such as the string "Nan"?

Not necessarily -- you could merely ignore any key which is a NaN, or 
you could pass each key through this first:

def intern_nan(x, nan=float('nan')):
    if math.isnan(x):  return nan
    return x

thus ensuring that all NaN keys were the same NaN.

> So what alternative behaviour would you suggest, and how would you
> implement it?
[...]
> One alternative would be to prohibit putting nans into sets and dicts
> by making them unhashable;  I'm not sure what that would gain,
> though. And there would still be some unintuitive behaviour for
> containment testing of nans in lists.

I think that would be worse than the current situation. That would mean 
that dict[some_float] would *nearly always* succeed, but occasionally 
would fail. I can't see that being a good thing.


-- 
Steven D'Aprano

From stephen at xemacs.org  Wed Mar 24 12:15:55 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 24 Mar 2010 20:15:55 +0900
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<aac2c7cb1003231048t757b639cgd564888a63ab5e@mail.gmail.com>
	<5c6f2a5d1003231104y2ca66d53g8b238d9f51dbf954@mail.gmail.com>
	<201003240900.30334.steve@pearwood.info>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
Message-ID: <87eijagec4.fsf@uwakimon.sk.tsukuba.ac.jp>

Mark Dickinson writes:

 > On Wed, Mar 24, 2010 at 5:36 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
 > > Steven D'Aprano writes:

 > > ?> I suspect that's a feature, not a bug.
 > 
 > Right:  distinct nans (i.e., those with different id()) are treated as
 > distinct set elements or dict keys.
 > 
 > > I don't see how it can be so. ?Aren't all of those entries garbage?
 > > To compute a histogram of results for computations on a series of
 > > cases would you not have to test each result for NaN-hood, then hash
 > > on a proxy such as the string "Nan"?
 > 
 > So what alternative behaviour would you suggest, and how would you
 > implement it?

I don't have an alternative behavior to suggest.  I'm not suggesting
that it's a bug, I'm suggesting that it's a wart: useless, ugly, and
in some presumably rare/buggy cases, it could lead to nasty behavior.
The example I have in mind is computing a histogram of function values
for a very large sample of inputs.  (This is a pathological example,
of course: things where NaNs are representable generally won't be used
directly as keys in a dictionary used to represent a histogram.
Rather, they would be mapped to a representative value as the key.)
If there are a lot of NaN's, the dictionary could get unexpectedly
large.

That's not Python's fault, of course:

 > Meanwhile IEEE 754 requires that nans compare unequal to
 > themselves, breaking reflexivity.  So there have to be some
 > compromises somewhere.

Indeed.  IEEE 754 compatibility *is* a feature.

 > One alternative would be to prohibit putting nans into sets and
 > dicts by making them unhashable; I'm not sure what that would gain,
 > though.

I would find that more intuitive.  While NaNs aren't mutable, they're
similar to mutable values in that their value is not deterministic in
a certain sense.

OTOH, since the only example I can think of where I would personally
want to check whether a NaN is in a container is pathological, my
intuition is hardly reliable.

From techtonik at gmail.com  Wed Mar 24 12:20:19 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 24 Mar 2010 13:20:19 +0200
Subject: [Python-Dev] At least one package management tool for 2.7
In-Reply-To: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
Message-ID: <d34314101003240420j57010ac3k71ba34cc0b2902fa@mail.gmail.com>

On Wed, Mar 24, 2010 at 12:26 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
> Distutils2 is planned to be reintegrated in the stdlib in Python 3.3,
> and my goal is to release it when Python 2.7 final is released.

Does that means "after" Python 2.7, because I meant it to be "before"
or at least "with"?

> The open question is: do we want to include a full installer that
> takes care of installing / removing dependencies as well ?

If there is a risk to get nothing at all in 2.7 distribution, because
it just won't be ready/accepted by that time, then I it is certainly
optional.

> But the "auto-update" story seems interesting, can you expand on this ?

Sure. Package management tool should have an ability to update itself
when required regardless of Python release. For example::

    python.exe -m easy_install setuptools

This will get you new version of `setuptools` and `easy_install`
module from it automagically. You do not need to install new version
of `setuptools` manually or copy files from SVN if you want to see
fixes before next Python release. The stuff you would likely need to
do with distutils bugs, which I personally find awkward.

Package management is orthogonal to Python releases, and it is more
oriented at Python users who don't like to wait or follow PEPs. That's
why package management tool such as 'easy_install' has shorter
development cycle, and it should faster react to user feedback. What
can be one of the reasons that no package management tool is included
with Python.

In various README you may often see "requires setuptools > 0.6c9" or
similar. I can't see why package management tool can not detect this
dependency and propose to update itself.

If it is impossible to ship the whole package management system then
at least Python distribution may carry small bootstrap script for it.
When user tries to execute package management tools, it warns him that
these are not installed and gives a hint where to get them::

> python -m easy_install bla-bla-bla
Error: easy_install module is not shipped with this Python release.
Please execute the following command to install the latest version.

python -m easy_bootstrap

-- 
anatoly t.

From techtonik at gmail.com  Wed Mar 24 12:23:18 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 24 Mar 2010 13:23:18 +0200
Subject: [Python-Dev] At least one package management tool for 2.7
In-Reply-To: <d34314101003240420j57010ac3k71ba34cc0b2902fa@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
	<d34314101003240420j57010ac3k71ba34cc0b2902fa@mail.gmail.com>
Message-ID: <d34314101003240423n503b46c1i92580d4723313506@mail.gmail.com>

> Sure. Package management tool should have an ability to update itself when required regardless of Python release. For example::
>
> python.exe -m easy_install setuptools
>

This should be:

    python -m easy_install -U setuptools

P.S. Wave effect. =)
-- 
anatoly t.

From ziade.tarek at gmail.com  Wed Mar 24 12:30:19 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 24 Mar 2010 12:30:19 +0100
Subject: [Python-Dev] At least one package management tool for 2.7
In-Reply-To: <d34314101003240420j57010ac3k71ba34cc0b2902fa@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
	<d34314101003240420j57010ac3k71ba34cc0b2902fa@mail.gmail.com>
Message-ID: <94bdd2611003240430m1fb8f6e5o22fc24d204c043c9@mail.gmail.com>

On Wed, Mar 24, 2010 at 12:20 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 12:26 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>>
>> Distutils2 is planned to be reintegrated in the stdlib in Python 3.3,
>> and my goal is to release it when Python 2.7 final is released.
>
> Does that means "after" Python 2.7, because I meant it to be "before"
> or at least "with"?

The goal is to provide a first version by the time 2.7 is out.

>
>> The open question is: do we want to include a full installer that
>> takes care of installing / removing dependencies as well ?
>
> If there is a risk to get nothing at all in 2.7 distribution, because
> it just won't be ready/accepted by that time, then I it is certainly
> optional.
>

Understand that the distutils2 project is happening outside the stdlib
at this time,
so you will have to install it on the top of Python in any case.

Last, its release cycle will be shorter until it gets back in the
stdlib, that will make it easy
to add features.

>> But the "auto-update" story seems interesting, can you expand on this ?
>
> Sure. Package management tool should have an ability to update itself
> when required regardless of Python release. For example::
>
> ? ?python.exe -m easy_install setuptools
>
> This will get you new version of `setuptools` and `easy_install`
> module from it automagically. You do not need to install new version
> of `setuptools` manually or copy files from SVN if you want to see
> fixes before next Python release. The stuff you would likely need to
> do with distutils bugs, which I personally find awkward.
>
> Package management is orthogonal to Python releases, and it is more
> oriented at Python users who don't like to wait or follow PEPs. That's
> why package management tool such as 'easy_install' has shorter
> development cycle, and it should faster react to user feedback. What
> can be one of the reasons that no package management tool is included
> with Python.
>
> In various README you may often see "requires setuptools > 0.6c9" or
> similar. I can't see why package management tool can not detect this
> dependency and propose to update itself.
>
> If it is impossible to ship the whole package management system then
> at least Python distribution may carry small bootstrap script for it.
> When user tries to execute package management tools, it warns him that
> these are not installed and gives a hint where to get them::
>
>> python -m easy_install bla-bla-bla
> Error: easy_install module is not shipped with this Python release.
> Please execute the following command to install the latest version.
>
> python -m easy_bootstrap

Interesting, can you start a new thread on distutils-SIG for this
part, so we can discuss it there ?

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From ncoghlan at gmail.com  Wed Mar 24 12:47:26 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 21:47:26 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <201003242214.39778.steve@pearwood.info>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info>
Message-ID: <4BA9FBCE.5040908@gmail.com>

Steven D'Aprano wrote:
> On Wed, 24 Mar 2010 08:51:36 pm Mark Dickinson wrote:
>>> I don't see how it can be so.  Aren't all of those entries garbage?
>>> To compute a histogram of results for computations on a series of
>>> cases would you not have to test each result for NaN-hood, then
>>> hash on a proxy such as the string "Nan"?
> 
> Not necessarily -- you could merely ignore any key which is a NaN, or 
> you could pass each key through this first:
> 
> def intern_nan(x, nan=float('nan')):
>     if math.isnan(x):  return nan
>     return x
> 
> thus ensuring that all NaN keys were the same NaN.

Interning NaN certainly seems like it should be sufficient to eliminate
the set/dict membership weirdness.

That is, make it so that the first two lines of the following return
True, while the latter two lines continue to return False:

>>> float("nan") is float("nan")
False
>>> dec("nan") is dec("nan")
False
>>> float("nan") == float("nan")
False
>>> dec("nan") == dec("nan")
False

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed Mar 24 12:48:40 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 21:48:40 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
Message-ID: <4BA9FC18.90102@gmail.com>

Raymond Hettinger wrote:
> The decimal module is already drowning in complexity,
> so it would be best to keep it simple:  one boolean flag
> that if set would warn about any implicit decimal/float
> interaction.

Agreed - those that want exceptions instead can use the usual warnings
module mechanisms to trigger them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From dsdale24 at gmail.com  Wed Mar 24 12:50:44 2010
From: dsdale24 at gmail.com (Darren Dale)
Date: Wed, 24 Mar 2010 07:50:44 -0400
Subject: [Python-Dev] [Distutils] At least one package management tool
	for 2.7
In-Reply-To: <94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
Message-ID: <a08e5f81003240450xd30444n194f9ad13163d8c7@mail.gmail.com>

On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> The open question is: do we want to include a full installer that
> takes care of installing / removing dependencies as well ?
>
> I think not. Pip already provides this feature on the top of distutils
> (and distutils2 later I guess) and is not hard to install on the top
> of Python.

Is pip able to determine and install dependencies recursively, like
easy_install does? Or is it up to the requested package to it specify
its dependencies (and its dependencies dependencies) in a pip
requirements file that is distributed separately?

Darren

From ziade.tarek at gmail.com  Wed Mar 24 12:53:01 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 24 Mar 2010 12:53:01 +0100
Subject: [Python-Dev] [Distutils] At least one package management tool
	for 2.7
In-Reply-To: <a08e5f81003240450xd30444n194f9ad13163d8c7@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
	<a08e5f81003240450xd30444n194f9ad13163d8c7@mail.gmail.com>
Message-ID: <94bdd2611003240453l79f7787ft101459a199a64061@mail.gmail.com>

On Wed, Mar 24, 2010 at 12:50 PM, Darren Dale <dsdale24 at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> The open question is: do we want to include a full installer that
>> takes care of installing / removing dependencies as well ?
>>
>> I think not. Pip already provides this feature on the top of distutils
>> (and distutils2 later I guess) and is not hard to install on the top
>> of Python.
>
> Is pip able to determine and install dependencies recursively, like
> easy_install does? Or is it up to the requested package to it specify
> its dependencies (and its dependencies dependencies) in a pip
> requirements file that is distributed separately?

Pip uses the same standard: the install_requires option in setuptools,
that is put in a requires.txt file when the egg-info is computed.

Notice that we have defined this field in the new version of the
metadata (PEP 345) for future interoperability.

Let's remove python-dev for packaging talks in the next answers

Regards
Tarek

>
> Darren
>



-- 
Tarek Ziad? | http://ziade.org

From ncoghlan at gmail.com  Wed Mar 24 13:02:06 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 22:02:06 +1000
Subject: [Python-Dev] At least one package management tool for 2.7
In-Reply-To: <d34314101003240420j57010ac3k71ba34cc0b2902fa@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
	<d34314101003240420j57010ac3k71ba34cc0b2902fa@mail.gmail.com>
Message-ID: <4BA9FF3E.10009@gmail.com>

anatoly techtonik wrote:
> If it is impossible to ship the whole package management system then 
> at least Python distribution may carry small bootstrap script for it.
>  When user tries to execute package management tools, it warns him
> that these are not installed and gives a hint where to get them::
> 
>> python -m easy_install bla-bla-bla
> Error: easy_install module is not shipped with this Python release. 
> Please execute the following command to install the latest version.
> 
> python -m easy_bootstrap

Note that this idea has come up before and is *much* more likely to get
traction than including a full package management system.

The nature of package management is such that tying an installation
system to the release cycle of the core interpreter is unlikely to end
well. A bootstrapping tool that only knows how to download a single
specific package would be much easier to keep stable.

Even such a scaled back idea is still far from a certainty to be
accepted though, given the number of people who vehemently object to
duplicating any significant part of the functionality of the system
package management tools (i.e. RPM, apt et al).

At this point, the packaging story is in the hands of distutils-sig and
they're pressing forward with several initiatives that will permit the
construction of more robust and reliable Python-specific package
management systems (such as supporting listing and uninstallation of
installed packages).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Wed Mar 24 13:04:48 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 22:04:48 +1000
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <20100323183212.7463e098@heresy>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<ho8esj$qnd$1@dough.gmane.org>
	<20100322162002.6daff86f@heresy>	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
	<20100323183212.7463e098@heresy>
Message-ID: <4BA9FFE0.3010800@gmail.com>

Barry Warsaw wrote:
> On Mar 22, 2010, at 12:38 PM, Guido van Rossum wrote:
> 
>> Huh? Last time I looked weren't we going to make __pycache__ the
>> default (and eventually only) behavior?
> 
> We definitely agreed it would be the default in Python 3.2.
> 
> My recollection is that we agreed it would be the only on-demand way of
> writing pyc files, but that Python would read a lone .pyc file where the
> source would be if the source is missing, and that py_compile/compileall would
> support optional creation of those lone .pyc files.

Yep, that's my recollection as well. I don't recall seeing an update to
state that clearly in the PEP go by on the checkins list though :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From steve at pearwood.info  Wed Mar 24 13:04:56 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 24 Mar 2010 23:04:56 +1100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA9FBCE.5040908@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info>
	<4BA9FBCE.5040908@gmail.com>
Message-ID: <201003242304.56454.steve@pearwood.info>

On Wed, 24 Mar 2010 10:47:26 pm Nick Coghlan wrote:
> Steven D'Aprano wrote:
> > On Wed, 24 Mar 2010 08:51:36 pm Mark Dickinson wrote:
> >>> I don't see how it can be so.  Aren't all of those entries
> >>> garbage? To compute a histogram of results for computations on a
> >>> series of cases would you not have to test each result for
> >>> NaN-hood, then hash on a proxy such as the string "Nan"?
> >
> > Not necessarily -- you could merely ignore any key which is a NaN,
> > or you could pass each key through this first:
> >
> > def intern_nan(x, nan=float('nan')):
> >     if math.isnan(x):  return nan
> >     return x
> >
> > thus ensuring that all NaN keys were the same NaN.
>
> Interning NaN certainly seems like it should be sufficient to
> eliminate the set/dict membership weirdness.

I didn't mean to suggest that Python should do that automatically! I 
meant that the developer could easily intern NaNs if needed.

I wouldn't want Python to automatically intern NaNs, the reason being 
that this would throw away information (at least potentially, depending 
on the C library). According to the relevant IEEE standard, NaNs should 
(may?) carry a payload. For example, Apple's SANE math library back in 
the 1980s exposed this payload: NaNs created from different failures 
would have a consistent payload, allowing the programmer to tell how 
the NaN appeared in the calculation. 

E.g. INF-INF would give you a payload of 123 (or whatever it was), while 
log(-1) would give you a payload of 456. (I've made up the numbers, 
it's been far too many years for me to remember what they were.)

The point is, whether Python currently exposes these payloads or not, we 
shouldn't prohibit it. If programmers want to explicitly fold all NaNs 
into one, it is easy to do so themselves.



-- 
Steven D'Aprano

From ncoghlan at gmail.com  Wed Mar 24 13:07:36 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 22:07:36 +1000
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <hobqml$5d4$1@dough.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<ho9b7a$noo$1@dough.gmane.org>
	<hobqml$5d4$1@dough.gmane.org>
Message-ID: <4BAA0088.2010909@gmail.com>

Ron Adam wrote:
> I think I misunderstood this at first.
> 
> It looks like, while developing a python 3.2+ program, if you don't
> create an empty __pycache__ directory, everything will still work, you
> just won't get the .pyc files.  That can be a good thing during
> development because you also will not have any problems with old .pyc
> files hanging around if you move or rename files.

The behaviour you described (not creating __pycache__ automatically) was
just a suggestion in this thread.

The behaviour in the actual PEP (and what will be implemented for 3.2+)
is to create __pycache__ if it is missing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From barry at python.org  Wed Mar 24 13:09:37 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 24 Mar 2010 08:09:37 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <hobqml$5d4$1@dough.gmane.org>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<ho9b7a$noo$1@dough.gmane.org> <hobqml$5d4$1@dough.gmane.org>
Message-ID: <20100324080937.5343f791@heresy>

On Mar 23, 2010, at 08:42 PM, Ron Adam wrote:

>It looks like, while developing a python 3.2+ program, if you don't create 
>an empty __pycache__ directory, everything will still work, you just won't 
>get the .pyc files.  That can be a good thing during development because 
>you also will not have any problems with old .pyc files hanging around if 
>you move or rename files.

Not quite.  In PEP-3147-land, you do not need to create the empty __pycache__
directory, Python will create it for you on-demand.  If you subsequently move
the .py source file, the __pycache__/...pyc file will be ignored.  So this is
actually better than today because you can't accidentally load stale pyc files
-- if they live inside __pycache__.   For backward compatibility we'll still
support loading lone pyc files in the source file directory (i.e. outside of
__pycache__).

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/70337bb7/attachment.pgp>

From ncoghlan at gmail.com  Wed Mar 24 13:15:00 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 22:15:00 +1000
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <rowen-C41123.12492623032010@news.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<4BA80418.6030905@canterbury.ac.nz>
	<rowen-C41123.12492623032010@news.gmane.org>
Message-ID: <4BAA0244.401@gmail.com>

Russell E. Owen wrote:
> This in turn implies that we may have to give up some support for 
> dragging python modules into site-packages, e.g. not generate .pyc files 
> for such modules. At least if we go that route it will mostly affect 
> power users, who can presumably cope.

And when someone drags a Python module into the per-user site-packages
instead? [1]

Yes, a shared Python needs to be managed carefully. Systems with a
shared Python should also generally have a vaguely competent sysadmin
running them.

An unshared Python and associated packages under PEP 3147 should work
just as well as they do under the existing pyc scheme (only without the
source directory clutter).

Cheers,
Nick.

[1] http://www.python.org/dev/peps/pep-0370/

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From olemis at gmail.com  Wed Mar 24 13:22:23 2010
From: olemis at gmail.com (Olemis Lang)
Date: Wed, 24 Mar 2010 08:22:23 -0400
Subject: [Python-Dev] At least one package management tool for 2.7
In-Reply-To: <d34314101003240423n503b46c1i92580d4723313506@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
	<d34314101003240420j57010ac3k71ba34cc0b2902fa@mail.gmail.com>
	<d34314101003240423n503b46c1i92580d4723313506@mail.gmail.com>
Message-ID: <24ea26601003240522n3f44d8b5k8105a13d4fb614bf@mail.gmail.com>

On Wed, Mar 24, 2010 at 7:23 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>> Sure. Package management tool should have an ability to update itself when required regardless of Python release. For example::
>>
>> python.exe -m easy_install setuptools
>>
>
> This should be:
>
> ? ?python -m easy_install -U setuptools
>

JFTR

More precisely, what I use for CI is

{{{
#!sh

$ easy_install -U setuptools==dev
}}}

but the `python -m` part should work as well ;o)

PS: e.g. that allows to check out code from SVN in order to use
setuptools 0.7.x `test_runner` switch like this

{{{
python -W ignore::DeprecationWarning setup.py test -r ciutils:junitrunner
}}}

;o)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
TracRpc: API v2: Test cases for XML-RPC ... PASS -
http://bitbucket.org/osimons/trac-rpc-mq/changeset/228ef43726b0/

From olemis at gmail.com  Wed Mar 24 13:27:03 2010
From: olemis at gmail.com (Olemis Lang)
Date: Wed, 24 Mar 2010 08:27:03 -0400
Subject: [Python-Dev] [Distutils] At least one package management tool
	for 2.7
In-Reply-To: <a08e5f81003240450xd30444n194f9ad13163d8c7@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
	<a08e5f81003240450xd30444n194f9ad13163d8c7@mail.gmail.com>
Message-ID: <24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com>

On Wed, Mar 24, 2010 at 7:50 AM, Darren Dale <dsdale24 at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 6:26 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> The open question is: do we want to include a full installer that
>> takes care of installing / removing dependencies as well ?
>>
>> I think not. Pip already provides this feature on the top of distutils
>> (and distutils2 later I guess) and is not hard to install on the top
>> of Python.
>
> Is pip able to determine and install dependencies recursively, like
> easy_install does? Or is it up to the requested package to it specify
> its dependencies (and its dependencies dependencies) in a pip
> requirements file that is distributed separately?
>

My experience is that only `install_requires` is needed (unless you
want to create app bundles AFAICR) , but in practice I've noticed that
*some* easy_installable packages are not pip-able (though I had no
time to figure out why :-/ )

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
On adding Hessian (RPC) support for Trac -
http://feedproxy.google.com/~r/simelo-en/~3/Vit6dRudChU/on-adding-hessian-rpc-support-for-trac.html

From barry at python.org  Wed Mar 24 13:32:36 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 24 Mar 2010 08:32:36 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <4BA9FFE0.3010800@gmail.com>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<ho8esj$qnd$1@dough.gmane.org> <20100322162002.6daff86f@heresy>
	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
	<20100323183212.7463e098@heresy> <4BA9FFE0.3010800@gmail.com>
Message-ID: <20100324083236.6e48437e@heresy>

On Mar 24, 2010, at 10:04 PM, Nick Coghlan wrote:

>Yep, that's my recollection as well. I don't recall seeing an update to
>state that clearly in the PEP go by on the checkins list though :)

Check again <wink>.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/b8e24977/attachment-0001.pgp>

From ncoghlan at gmail.com  Wed Mar 24 14:06:54 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Mar 2010 23:06:54 +1000
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <20100324083236.6e48437e@heresy>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<ho8esj$qnd$1@dough.gmane.org>	<20100322162002.6daff86f@heresy>	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>	<20100323183212.7463e098@heresy>	<4BA9FFE0.3010800@gmail.com>
	<20100324083236.6e48437e@heresy>
Message-ID: <4BAA0E6E.8050408@gmail.com>

Barry Warsaw wrote:
> On Mar 24, 2010, at 10:04 PM, Nick Coghlan wrote:
> 
>> Yep, that's my recollection as well. I don't recall seeing an update to
>> state that clearly in the PEP go by on the checkins list though :)
> 
> Check again <wink>.

Ah yes, the recollection of seeing such a message is now quite fresh in
my mind :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From solipsis at pitrou.net  Wed Mar 24 14:19:43 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 24 Mar 2010 13:19:43 +0000 (UTC)
Subject: [Python-Dev] At least one package management tool for 2.7
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
Message-ID: <loom.20100324T141701-302@post.gmane.org>

anatoly techtonik <techtonik <at> gmail.com> writes:
> 
> I wonder if there are many people here who don't use some kind of
> "easy_install" package for package management in their Python /
> virtualenv installations? I propose to include at least one such
> package that is capable to auto-update itself in Python 2.7
> 
> C:\~env\Python27>python.exe -m easy_install
> C:\~env\Python27\python.exe: No module named easy_install
> 
> C:\~env\Python27>python.exe -m pip
> C:\~env\Python27\python.exe: No module named pip

It will not happen in 2.7, which is almost in beta phase.
Based on previous discussions, it will most likely not happen in 3.2 either. The
consensus is that setuptools, which both pip and easy_install depend on, should
not be included into the core.

However, Tarek's work on "distutils2" (a mostly forward-compatible replacement
for distutils) will include features such as dependencies, and make it much
easier to create and perhaps bundle a pip-like utility. So perhaps in 3.3 you
have a chance ;)

Regards

Antoine.



From barry at python.org  Wed Mar 24 14:22:49 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 24 Mar 2010 09:22:49 -0400
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <4BAA0E6E.8050408@gmail.com>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<ho8esj$qnd$1@dough.gmane.org> <20100322162002.6daff86f@heresy>
	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
	<20100323183212.7463e098@heresy> <4BA9FFE0.3010800@gmail.com>
	<20100324083236.6e48437e@heresy> <4BAA0E6E.8050408@gmail.com>
Message-ID: <20100324092249.659bd810@heresy>

On Mar 24, 2010, at 11:06 PM, Nick Coghlan wrote:

>Ah yes, the recollection of seeing such a message is now quite fresh in
>my mind :)

Just don't tell Guido I borrowed his time machine keys!
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/5f8264aa/attachment.pgp>

From stephen at xemacs.org  Wed Mar 24 15:48:51 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 24 Mar 2010 23:48:51 +0900
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <20100324092249.659bd810@heresy>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<ho88hk$n52$1@dough.gmane.org>
	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>
	<loom.20100322T191016-732@post.gmane.org>
	<ho8esj$qnd$1@dough.gmane.org> <20100322162002.6daff86f@heresy>
	<ca471dc21003221338kab1551dw33585fec11fb0f46@mail.gmail.com>
	<20100323183212.7463e098@heresy> <4BA9FFE0.3010800@gmail.com>
	<20100324083236.6e48437e@heresy> <4BAA0E6E.8050408@gmail.com>
	<20100324092249.659bd810@heresy>
Message-ID: <877hp1hj1o.fsf@uwakimon.sk.tsukuba.ac.jp>

Barry Warsaw writes:
 > On Mar 24, 2010, at 11:06 PM, Nick Coghlan wrote:
 > 
 > >Ah yes, the recollection of seeing such a message is now quite fresh in
 > >my mind :)
 > 
 > Just don't tell Guido I borrowed his time machine keys!

Wouldn't that be preferable to revealing you've learned to hotwire it?

From steven.bethard at gmail.com  Wed Mar 24 17:05:36 2010
From: steven.bethard at gmail.com (Steven Bethard)
Date: Wed, 24 Mar 2010 09:05:36 -0700
Subject: [Python-Dev] argparse.py is licensed under the Apache License
In-Reply-To: <e27efe131003240142v50833b3cu80071e69062cb201@mail.gmail.com>
References: <e27efe131003240142v50833b3cu80071e69062cb201@mail.gmail.com>
Message-ID: <d11dcfba1003240905n59882d52ge987a68bb885a6f4@mail.gmail.com>

On Wed, Mar 24, 2010 at 1:42 AM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> I noticed that the newly added argparse module has an unusual
> licence header,?included below. This is the only file in the Python tree
> that contains an explicit reference to the Apache License,
> and this leads me to some questions:

Sorry, I forgot to remove this when moving argparse to the stdlib.
There's probably one test\test_argparse too.

> - Did the contributor sign a Contributor agreement? In this case,
> shouldn't the code be marked as "Licensed to PSF under a Contributor Agreement",
> as?mentioned in the contribution form?
> http://www.python.org/psf/contrib/contrib-form
> And then, could this Apache License be removed?

Yes, I have signed a contributor agreement. I was thinking of just
removing the license entirely, but if it's better to replace it with
"Licensed to PSF under a Contributor Agreement", that's fine too. Let
me know, and I'll take care of this today.

Thanks for catching this!

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

From dickinsm at gmail.com  Wed Mar 24 17:22:29 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 16:22:29 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA9FBCE.5040908@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com>
Message-ID: <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>

On Wed, Mar 24, 2010 at 11:47 AM, Nick Coghlan
> Interning NaN certainly seems like it should be sufficient to eliminate
> the set/dict membership weirdness.
>
> That is, make it so that the first two lines of the following return
> True, while the latter two lines continue to return False:
>
>>>> float("nan") is float("nan")
> False
>>>> dec("nan") is dec("nan")
> False
>>>> float("nan") == float("nan")
> False
>>>> dec("nan") == dec("nan")
> False

Yes;  that could be done.  Though as Steven points out, not all NaNs
are equivalent (possibility of different payloads and different
signs), so float nans with different underlying bit patterns, and
Decimal nans with different string representations, would ideally be
interned separately.  For floats it might be possible to get away with
pretending that there's only one nan.  For decimal, I don't think
that's true, since the payload and sign are part of the standard, and
are very visible (e.g. in the repr of the nan).

The obvious way to do this nan interning for floats would be to put
the interning code into PyFloat_FromDouble.  I'm not sure whether this
would be worth the cost in terms of added code (and possibly reduced
performance, since the nan check would be done every time a float was
returned), but I'd be willing to review a patch.

Mark

From rrr at ronadam.com  Wed Mar 24 17:46:28 2010
From: rrr at ronadam.com (Ron Adam)
Date: Wed, 24 Mar 2010 11:46:28 -0500
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <4BAA0088.2010909@gmail.com>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<ho9b7a$noo$1@dough.gmane.org>	<hobqml$5d4$1@dough.gmane.org>
	<4BAA0088.2010909@gmail.com>
Message-ID: <hodfl5$gio$1@dough.gmane.org>



Nick Coghlan wrote:
> Ron Adam wrote:
>> I think I misunderstood this at first.
>>
>> It looks like, while developing a python 3.2+ program, if you don't
>> create an empty __pycache__ directory, everything will still work, you
>> just won't get the .pyc files.  That can be a good thing during
>> development because you also will not have any problems with old .pyc
>> files hanging around if you move or rename files.
> 
> The behaviour you described (not creating __pycache__ automatically) was
> just a suggestion in this thread.
> 
> The behaviour in the actual PEP (and what will be implemented for 3.2+)
> is to create __pycache__ if it is missing.
> 
> Cheers,
> Nick.

OK  :-)

hmmmm... unless there is a __pycache__ *file* located there first. ;-)

Not that I can think of any good reason to do that at this moment.

Ron


From amentajo at msu.edu  Wed Mar 24 18:16:14 2010
From: amentajo at msu.edu (Joe Amenta)
Date: Wed, 24 Mar 2010 13:16:14 -0400
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <hnv6j0$id6$1@dough.gmane.org>
References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> 
	<hnv6j0$id6$1@dough.gmane.org>
Message-ID: <4dc473a51003241016va2503cexf5c0ec55d5c30bda@mail.gmail.com>

On Fri, Mar 19, 2010 at 2:45 AM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 3/19/2010 2:23 AM, Laurent Gautier wrote:
>
>> On 3/19/10 3:36 AM, C. Titus Brown wrote:
>>
>>> Hi all,
>>>
>>> once again, the PSF has been accepted as a mentoring foundation for
>>> the Google
>>> Summer of Code! This year, we're going to emphasize python 3 porting, so
>>> please think of projects you'd like to see tackled.
>>>
>>
>> Hi,
>>
>>
>> Does this mean that any other python project could potentially see
>> itself ported to Python 3 in the course of this SoC ?
>>
>
> The theme should include both general porting tools and specific projects,
> especially infrastructure projects like numeric, which are blocking the
> porting of other projects. It would be nice if those doing specific projects
> contributed their experience/knowledge to a central pool.
>
>
>  If so, can any project owner submit a request for help,
>>
>
> Any project owner is *always* free to ask for help (on python-list, but now
> here in this thread). Those who can also mentor might be more likely to get
> it. If I were a student, I would consider serious interest from a project
> owner (and a promise to distribute a port, when ready), a prerequisite.
>
>
> > or is there going to be a list
>
>> of projects that would nice to port, or will a voting system of some
>> sort be put in place ?
>>
>
> Like most contributors, students choose projects, within the limits of what
> they can get mentors for, that scratch their itches. They may or may not
> otherwise be swayed by requests and opinions.
>
> My views.
>
> Terry Jan Reedy
>
>

Would anyone be interested in mentoring further lib3to2 work?  I'm planning
on applying again as a student.

--Joe Amenta
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/56cdea3e/attachment.html>

From brett at python.org  Wed Mar 24 18:08:59 2010
From: brett at python.org (Brett Cannon)
Date: Wed, 24 Mar 2010 10:08:59 -0700
Subject: [Python-Dev] argparse.py is licensed under the Apache License
In-Reply-To: <d11dcfba1003240905n59882d52ge987a68bb885a6f4@mail.gmail.com>
References: <e27efe131003240142v50833b3cu80071e69062cb201@mail.gmail.com> 
	<d11dcfba1003240905n59882d52ge987a68bb885a6f4@mail.gmail.com>
Message-ID: <bbaeab101003241008j3dad3388p540e3ca3e835ee56@mail.gmail.com>

On Wed, Mar 24, 2010 at 09:05, Steven Bethard <steven.bethard at gmail.com>wrote:

> On Wed, Mar 24, 2010 at 1:42 AM, Amaury Forgeot d'Arc
> <amauryfa at gmail.com> wrote:
> > I noticed that the newly added argparse module has an unusual
> > licence header, included below. This is the only file in the Python tree
> > that contains an explicit reference to the Apache License,
> > and this leads me to some questions:
>
> Sorry, I forgot to remove this when moving argparse to the stdlib.
> There's probably one test\test_argparse too.
>
> > - Did the contributor sign a Contributor agreement? In this case,
> > shouldn't the code be marked as "Licensed to PSF under a Contributor
> Agreement",
> > as mentioned in the contribution form?
> > http://www.python.org/psf/contrib/contrib-form
> > And then, could this Apache License be removed?
>
> Yes, I have signed a contributor agreement. I was thinking of just
> removing the license entirely, but if it's better to replace it with
> "Licensed to PSF under a Contributor Agreement", that's fine too. Let
> me know, and I'll take care of this today.
>
>

I say just strip it out. If you leave it the PSF board will get an email
from some large corporation at some point in the future about whether you
have actually signed an agreement or not. Much easier to simply remove it so
people can rightfully assume it is licensed under the PSF license.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/61ddc396/attachment.html>

From ianb at colorstudy.com  Wed Mar 24 18:19:27 2010
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 24 Mar 2010 12:19:27 -0500
Subject: [Python-Dev] [Distutils] At least one package management tool
	for 2.7
In-Reply-To: <24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com> 
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com> 
	<a08e5f81003240450xd30444n194f9ad13163d8c7@mail.gmail.com> 
	<24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com>
Message-ID: <b654cd2e1003241019r59d788adp749f18a8db0d790c@mail.gmail.com>

On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang <olemis at gmail.com> wrote:
> My experience is that only `install_requires` is needed (unless you
> want to create app bundles AFAICR) , but in practice I've noticed that
> *some* easy_installable packages are not pip-able (though I had no
> time to figure out why :-/ )

Usually this is because Setuptools is poking at objects to do its
work, while pip tries to work mostly with subprocesses.  Though to
complicate things a bit, pip makes sure the Setuptools monkeypatches
to distutils are applied, so that it's always as though the setup.py
says "from setuptools import setup".  easy_install *also* does this.

But then easy_install starts calling methods and whatnot, while pip just does:

  setup.py install --single-version-externally-managed --no-deps
--record some_tmp_file

The --no-deps keeps Setuptools from resolving dependencies (because it
does so using easy_install), and --single-version-externally-managed
keeps Setuptools doing egg directories.  And --record keeps track of
installed files, which are later moved around to facilitate uninstall.
 But some distributions pay extra attention to those options, or do
other tricky things in their setup.py, and as a result this causes
failures.  Because easy_install is just calling internal methods, it
kind of sidelines those tricks (also people don't tend to give these
options to easy_install, and some don't even exist, so some code paths
just aren't exercised with typical easy_install usage).

Oh, the other reason is the link searching mechanism is slightly
different between the two installers.  Not particularly intentionally,
just incidentally.  Also because pip doesn't install anything but
source packages, some packages are installable via easy_install but
not pip -- usually this is an oversight on the part of the person
doing the packaging, but they just never noticed.

-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://twitter.com/ianbicking

From raymond.hettinger at gmail.com  Wed Mar 24 18:39:54 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 24 Mar 2010 10:39:54 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info>
	<4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
Message-ID: <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>


On Mar 24, 2010, at 9:22 AM, Mark Dickinson wrote:
> 
> The obvious way to do this nan interning for floats would be to put
> the interning code into PyFloat_FromDouble.  I'm not sure whether this
> would be worth the cost in terms of added code (and possibly reduced
> performance, since the nan check would be done every time a float was
> returned), but I'd be willing to review a patch.

-1  

Propagating support for NaNs has already consumed an
enormous amount of your time.  The code for the math
module doubled in complexity as a result of adding Nan/Inf
support.  At each stage, some new weirdness emerges
(such as complex(Nan, Nan), etc).  And each change introduces
the risk of new bugs in code that had been stable for a long time.

IIRC, the original purpose of a NaN was to serve as a placeholder
value in a long series of floating point ops so that the programmer
would not have to introduce edge case tests at each stage
of a calculation.  Yet, I look at the code for the decimal module
and the C code for the math module and see that the opposite
result occurred, the code is littered with is_special(x) tests
and handlers.

In terms of practicality, NaNs work great as a way to mark 
missing values and to propagate through subsequent calculations.   
IMO, their least useful feature is the property of being not equal
to themselves -- that causes more problems than it solves
because it impairs a programmer's ability to reason about
their programs.


Raymond





 





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/4c458fef/attachment-0001.html>

From dsdale24 at gmail.com  Wed Mar 24 18:53:46 2010
From: dsdale24 at gmail.com (Darren Dale)
Date: Wed, 24 Mar 2010 13:53:46 -0400
Subject: [Python-Dev] [Distutils] At least one package management tool
	for 2.7
In-Reply-To: <b654cd2e1003241019r59d788adp749f18a8db0d790c@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
	<a08e5f81003240450xd30444n194f9ad13163d8c7@mail.gmail.com>
	<24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com>
	<b654cd2e1003241019r59d788adp749f18a8db0d790c@mail.gmail.com>
Message-ID: <a08e5f81003241053p340deb66pf748dc2422a6fb35@mail.gmail.com>

On Wed, Mar 24, 2010 at 1:19 PM, Ian Bicking <ianb at colorstudy.com> wrote:
> On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang <olemis at gmail.com> wrote:
>> My experience is that only `install_requires` is needed (unless you
>> want to create app bundles AFAICR) , but in practice I've noticed that
>> *some* easy_installable packages are not pip-able (though I had no
>> time to figure out why :-/ )
>
> Usually this is because Setuptools is poking at objects to do its
> work, while pip tries to work mostly with subprocesses. ?Though to
> complicate things a bit, pip makes sure the Setuptools monkeypatches
> to distutils are applied, so that it's always as though the setup.py
> says "from setuptools import setup". ?easy_install *also* does this.
>
> But then easy_install starts calling methods and whatnot, while pip just does:
>
> ?setup.py install --single-version-externally-managed --no-deps
> --record some_tmp_file
>
> The --no-deps keeps Setuptools from resolving dependencies

Seeking clarification: how can pip recursively install dependencies
*and* keep Setuptools from resolving dependencies?

Darren

From alexander.belopolsky at gmail.com  Wed Mar 24 19:26:12 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 14:26:12 -0400
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
Message-ID: <d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>

On Wed, Mar 24, 2010 at 1:39 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
..
> IMO,?their least useful feature is the property of being not equal
> to themselves -- that causes more problems than it solves
> because it impairs a programmer's ability to reason about
> their programs.


I agree.  An often cited rationale for IEEE 754 behavior is that it
eliminates branching in some high performance numerical algorithms.
While this is likely to be true, I have never seen it benefiting real
life applications, certainly not those written in Python.

I wonder why Python did not follow Java model where Float NaN objects
unlike raw float NaNs compare equal to themselves.    One reason may
be that Python does not have raw floats, but if someone needs IEEE 754
NaNs, one can use numpy scalars or add arithmetics to ctypes numerical
types.

Mark, I wonder if you could describe an algorithm off the top of your
head that relies on NaN == NaN being false.

From jbaker at zeomega.com  Wed Mar 24 19:06:03 2010
From: jbaker at zeomega.com (Jason Baker)
Date: Wed, 24 Mar 2010 13:06:03 -0500
Subject: [Python-Dev] [Distutils] At least one package management tool
	for 2.7
In-Reply-To: <a08e5f81003241053p340deb66pf748dc2422a6fb35@mail.gmail.com>
References: <d34314101003240259n70b823b4p5988bad01d062a8c@mail.gmail.com>
	<94bdd2611003240326u7712700fh5415125208b5fcbf@mail.gmail.com>
	<a08e5f81003240450xd30444n194f9ad13163d8c7@mail.gmail.com>
	<24ea26601003240527t33c038b2ka1b865dcca172374@mail.gmail.com>
	<b654cd2e1003241019r59d788adp749f18a8db0d790c@mail.gmail.com>
	<a08e5f81003241053p340deb66pf748dc2422a6fb35@mail.gmail.com>
Message-ID: <fa34f1941003241106n6a4dfd2cja881241d247f260c@mail.gmail.com>

On Wed, Mar 24, 2010 at 12:53 PM, Darren Dale <dsdale24 at gmail.com> wrote:

> On Wed, Mar 24, 2010 at 1:19 PM, Ian Bicking <ianb at colorstudy.com> wrote:
> > On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang <olemis at gmail.com> wrote:
> >> My experience is that only `install_requires` is needed (unless you
> >> want to create app bundles AFAICR) , but in practice I've noticed that
> >> *some* easy_installable packages are not pip-able (though I had no
> >> time to figure out why :-/ )
> >
> > Usually this is because Setuptools is poking at objects to do its
> > work, while pip tries to work mostly with subprocesses.  Though to
> > complicate things a bit, pip makes sure the Setuptools monkeypatches
> > to distutils are applied, so that it's always as though the setup.py
> > says "from setuptools import setup".  easy_install *also* does this.
> >
> > But then easy_install starts calling methods and whatnot, while pip just
> does:
> >
> >  setup.py install --single-version-externally-managed --no-deps
> > --record some_tmp_file
> >
> > The --no-deps keeps Setuptools from resolving dependencies
>
> Seeking clarification: how can pip recursively install dependencies
> *and* keep Setuptools from resolving dependencies?
>
>
Using the --no-deps option to setup.py
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/2000f593/attachment.html>

From guido at python.org  Wed Mar 24 19:36:41 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Mar 2010 11:36:41 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> 
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> 
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> 
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
Message-ID: <ca471dc21003241136n50086123gaecf680162bda79f@mail.gmail.com>

On Wed, Mar 24, 2010 at 11:26 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> I wonder why Python did not follow Java model where Float NaN objects
> unlike raw float NaNs compare equal to themselves. ? ?One reason may
> be that Python does not have raw floats, but if someone needs IEEE 754
> NaNs, one can use numpy scalars or add arithmetics to ctypes numerical
> types.

Probably because we were blindly following the IEEE standard without
understanding it in every detail.

-- 
--Guido van Rossum (python.org/~guido)

From dickinsm at gmail.com  Wed Mar 24 19:50:41 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 18:50:41 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
Message-ID: <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com>

On Wed, Mar 24, 2010 at 6:26 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> Mark, I wonder if you could describe an algorithm off the top of your
> head that relies on NaN == NaN being false.
>

No, I certainly couldn't!  And I often wonder if the original IEEE 754
committee, given 20/20 foresight, would have made the same decisions
regarding comparisons of nans.  It's certainly not one of my favourite
features of IEEE 754.  (Though sqrt(-0.) -> -0. ranks lower for me.
Grr.)

A bogus application that I've often seen mentioned is that it allows
checking whether a float 'x' is a nan by doing `x == x';  but the
proper way to do this is to have an 'isnan' function or method, so
this isn't particularly convincing.

Slightly more convincing is history:  this is the way that nan
comparisons behave in other languages (Fortran, C) used for numerics.
 If Python were to do something different then a naively translated
algorithm from another language would fail.  It's the behaviour that
numerically-aware people expect, and I'd expect to get complaints from
those people if it changed.

Mark

From alexander.belopolsky at gmail.com  Wed Mar 24 19:55:53 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 14:55:53 -0400
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <ca471dc21003241136n50086123gaecf680162bda79f@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
	<ca471dc21003241136n50086123gaecf680162bda79f@mail.gmail.com>
Message-ID: <d38f5331003241155j985f3ecm7f717f806c52b3ea@mail.gmail.com>

On Wed, Mar 24, 2010 at 2:36 PM, Guido van Rossum <guido at python.org> wrote:
..
> Probably because we were blindly following the IEEE standard without
> understanding it in every detail.

Are you talking about "accidental" support for NaNs in older versions
of Python or about recent efforts to support them properly in math and
decimal modules?

I feel you are too harsh on the developers that worked in this area.
I dare to suggest that the current situation is not due to lack of
understanding of the standard, but due to pragmatic decisions made in
early development and desire for backward compatibility in the recent
versions.

Is this an area where design changes are feasible?  IIRC, NaN support
was never "officially" in the language, but it may have changed with
the decimal module.

From alexander.belopolsky at gmail.com  Wed Mar 24 20:04:22 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 15:04:22 -0400
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
	<5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com>
Message-ID: <d38f5331003241204s370a1e94n958084434e57ba72@mail.gmail.com>

On Wed, Mar 24, 2010 at 2:50 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
..
> ?If Python were to do something different then a naively translated
> algorithm from another language would fail. ?It's the behaviour that
> numerically-aware people expect, and I'd expect to get complaints from
> those people if it changed.

Numerically-aware people are likely to be aware of the differences in
languages that they use.   I think in this day and age you are more
likely to hear from confused Java programmers than from Fortran or
even C folks.

From dickinsm at gmail.com  Wed Mar 24 20:26:11 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 19:26:11 +0000
Subject: [Python-Dev] Why is nan != nan?
Message-ID: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>

[Changing the subject line, since we're way off the original topic]

On Wed, Mar 24, 2010 at 7:04 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 2:50 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> ..
>> ?If Python were to do something different then a naively translated
>> algorithm from another language would fail. ?It's the behaviour that
>> numerically-aware people expect, and I'd expect to get complaints from
>> those people if it changed.
>
> Numerically-aware people are likely to be aware of the differences in
> languages that they use.

Sure.  But I'd still expect them to complain.  :)

Here's an interesting recent blog post on this subject, from the
creator of Eiffel:

http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/

Mark

From steve at pearwood.info  Wed Mar 24 20:34:13 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 25 Mar 2010 06:34:13 +1100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
Message-ID: <201003250634.13861.steve@pearwood.info>

On Thu, 25 Mar 2010 03:22:29 am Mark Dickinson wrote:
> The obvious way to do this nan interning for floats would be to put
> the interning code into PyFloat_FromDouble. ?I'm not sure whether
> this would be worth the cost in terms of added code (and possibly
> reduced performance, since the nan check would be done every time a
> float was returned), but I'd be willing to review a patch.

I hope that it's obvious from my previous post that I do NOT want such 
interning done, but since I put the idea in people's heads, I wish to 
reiterate that I'm against the idea: -1 on interning NaNs. For the rare 
application where it might be useful, it is easy to do in the 
application code.



-- 
Steven D'Aprano

From stefan at bytereef.org  Wed Mar 24 20:36:13 2010
From: stefan at bytereef.org (Stefan Krah)
Date: Wed, 24 Mar 2010 20:36:13 +0100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BA9FC18.90102@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
	<4BA9FC18.90102@gmail.com>
Message-ID: <20100324193613.GA470@mail.bytereef.org>

Nick Coghlan <ncoghlan at gmail.com> wrote:
> Raymond Hettinger wrote:
> > The decimal module is already drowning in complexity,
> > so it would be best to keep it simple:  one boolean flag
> > that if set would warn about any implicit decimal/float
> > interaction.
> 
> Agreed - those that want exceptions instead can use the usual warnings
> module mechanisms to trigger them.


I'm not sure about the warnings module. If lower complexity is a goal,
I would prefer Facundo's original proposal of just adding a single new
signal. Users who just want to know if a NonIntegerConversion has occurred
can check the flags, users who want an exception set the trap.

With the warnings module, users have to know (and deal with) two exception
handling/suppressing mechanisms.


Stefan Krah



From guido at python.org  Wed Mar 24 20:51:13 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Mar 2010 12:51:13 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <d38f5331003241155j985f3ecm7f717f806c52b3ea@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com> 
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com> 
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com> 
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com> 
	<ca471dc21003241136n50086123gaecf680162bda79f@mail.gmail.com> 
	<d38f5331003241155j985f3ecm7f717f806c52b3ea@mail.gmail.com>
Message-ID: <ca471dc21003241251m5aa9f2fela488a4fd8128fa07@mail.gmail.com>

On Wed, Mar 24, 2010 at 11:55 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 2:36 PM, Guido van Rossum <guido at python.org> wrote:
> ..
>> Probably because we were blindly following the IEEE standard without
>> understanding it in every detail.
>
> Are you talking about "accidental" support for NaNs in older versions
> of Python or about recent efforts to support them properly in math and
> decimal modules?

My recollection include recent efforts, such as the math.isnan() function.

I don't recall ever hearing an argument for the peculiar behavior of
NaN in comparisons beyond "this is what the IEEE standard prescribes."

> I feel you are too harsh on the developers that worked in this area.

Maybe. I didn't mean to put down individuals complicit in the decision
-- in fact I would say that I am to blame myself for not probing
deeper.

> I dare to suggest that the current situation is not due to lack of
> understanding of the standard, but due to pragmatic decisions made in
> early development and desire for backward compatibility in the recent
> versions.

I think that originally NaN (and Inf) behavior was so
platform-dependent that it really wouldn't have mattered if we had
changed it.

> Is this an area where design changes are feasible? ?IIRC, NaN support
> was never "officially" in the language, but it may have changed with
> the decimal module.

It has been at least since 2.6 introduced math.isnan(), and ISTR there
was a proposal to add a separate module to deal with NaN and Inf
properly in a pure-python library module.

-- 
--Guido van Rossum (python.org/~guido)

From alexander.belopolsky at gmail.com  Wed Mar 24 20:51:46 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 15:51:46 -0400
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
	<5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com>
Message-ID: <d38f5331003241251q781f48a8w35f4babb192573bc@mail.gmail.com>

On Wed, Mar 24, 2010 at 2:50 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 6:26 PM, Alexander Belopolsky
> <alexander.belopolsky at gmail.com> wrote:
>> Mark, I wonder if you could describe an algorithm off the top of your
>> head that relies on NaN == NaN being false.
>>
>
> No, I certainly couldn't! ?And I often wonder if the original IEEE 754
> committee, given 20/20 foresight, would have made the same decisions
> regarding comparisons of nans. ?It's certainly not one of my favourite
> features of IEEE 754.

I tried to google the rationale for the IEEE 754 decision, but came up
with nothing.

Here are a few representative results:

"""
So without fear let me not stop at the arguments that ?the committee
must have voted on this point and they obviously knew what they were
doing? and ?it is the standard and implemented on zillions of
machines, you cannot change it now?.
"""
- "Reflexivity, and other pillars of civilization" by Bertrand Meyer
http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/

"""
I suppose this simplifies numerical computations in some way, but I
couldn't find an explicitly stated reason, not even in the Lecture
Notes on the Status of IEEE 754 by Kahan which discusses other design
decisions in detail.
"""

- "What is the rationale for all comparisons returning false for
IEEE754 NaN values?"
http://stackoverflow.com/questions/1565164/what-is-the-rationale-for-all-comparisons-returning-false-for-ieee754-nan-values

From arcriley at gmail.com  Wed Mar 24 21:18:53 2010
From: arcriley at gmail.com (Arc Riley)
Date: Wed, 24 Mar 2010 16:18:53 -0400
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <4dc473a51003241016va2503cexf5c0ec55d5c30bda@mail.gmail.com>
References: <20100319023630.GC1617@idyll.org> <4BA31858.3010904@gmail.com> 
	<hnv6j0$id6$1@dough.gmane.org>
	<4dc473a51003241016va2503cexf5c0ec55d5c30bda@mail.gmail.com>
Message-ID: <bad82a81003241318n36f730c5l28e09d04bb67846@mail.gmail.com>

I'm sure we can find you a mentor.

On Wed, Mar 24, 2010 at 1:16 PM, Joe Amenta <amentajo at msu.edu> wrote:

> On Fri, Mar 19, 2010 at 2:45 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>
>> On 3/19/2010 2:23 AM, Laurent Gautier wrote:
>>
>>> On 3/19/10 3:36 AM, C. Titus Brown wrote:
>>>
>>>> Hi all,
>>>>
>>>> once again, the PSF has been accepted as a mentoring foundation for
>>>> the Google
>>>> Summer of Code! This year, we're going to emphasize python 3 porting, so
>>>> please think of projects you'd like to see tackled.
>>>>
>>>
>>> Hi,
>>>
>>>
>>> Does this mean that any other python project could potentially see
>>> itself ported to Python 3 in the course of this SoC ?
>>>
>>
>> The theme should include both general porting tools and specific projects,
>> especially infrastructure projects like numeric, which are blocking the
>> porting of other projects. It would be nice if those doing specific projects
>> contributed their experience/knowledge to a central pool.
>>
>>
>>  If so, can any project owner submit a request for help,
>>>
>>
>> Any project owner is *always* free to ask for help (on python-list, but
>> now here in this thread). Those who can also mentor might be more likely to
>> get it. If I were a student, I would consider serious interest from a
>> project owner (and a promise to distribute a port, when ready), a
>> prerequisite.
>>
>>
>> > or is there going to be a list
>>
>>> of projects that would nice to port, or will a voting system of some
>>> sort be put in place ?
>>>
>>
>> Like most contributors, students choose projects, within the limits of
>> what they can get mentors for, that scratch their itches. They may or may
>> not otherwise be swayed by requests and opinions.
>>
>> My views.
>>
>> Terry Jan Reedy
>>
>>
>
> Would anyone be interested in mentoring further lib3to2 work?  I'm planning
> on applying again as a student.
>
>  --Joe Amenta
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/arcriley%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/93f67e7c/attachment.html>

From raymond.hettinger at gmail.com  Wed Mar 24 21:26:05 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 24 Mar 2010 13:26:05 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <d38f5331003241251q781f48a8w35f4babb192573bc@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<87pr2ugu1j.fsf@uwakimon.sk.tsukuba.ac.jp>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info>
	<4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
	<5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com>
	<d38f5331003241251q781f48a8w35f4babb192573bc@mail.gmail.com>
Message-ID: <56FC12CB-0A9C-46EC-A793-8ACAB8CAC1A3@gmail.com>


On Mar 24, 2010, at 12:51 PM, Alexander Belopolsky wrote:
> 
> - "Reflexivity, and other pillars of civilization" by Bertrand Meyer
> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/

Excellent link. 
Thanks for the research.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/4911564a/attachment.html>

From alexander.belopolsky at gmail.com  Wed Mar 24 21:37:05 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 16:37:05 -0400
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <56FC12CB-0A9C-46EC-A793-8ACAB8CAC1A3@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<5c6f2a5d1003240251n20b4ebdu3ff0aa12580b76aa@mail.gmail.com>
	<201003242214.39778.steve@pearwood.info> <4BA9FBCE.5040908@gmail.com>
	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
	<5c6f2a5d1003241150q24557b16o772953d9aa404f30@mail.gmail.com>
	<d38f5331003241251q781f48a8w35f4babb192573bc@mail.gmail.com>
	<56FC12CB-0A9C-46EC-A793-8ACAB8CAC1A3@gmail.com>
Message-ID: <d38f5331003241337t13a1961cy1fa50574e2c0434c@mail.gmail.com>

Thanks.  Same link reported concurrently by Mark in the "Why is nan !=
nan?" thread.

On Wed, Mar 24, 2010 at 4:26 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 24, 2010, at 12:51 PM, Alexander Belopolsky wrote:
>
> - "Reflexivity, and other pillars of civilization" by Bertrand Meyer
> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/
>
> Excellent link.
> Thanks for the research.
>
> Raymond
>

From raymond.hettinger at gmail.com  Wed Mar 24 21:56:14 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 24 Mar 2010 13:56:14 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <20100324193613.GA470@mail.bytereef.org>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org>
Message-ID: <CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>

FWIW, my viewpoint on this is softening over time
and I no longer feel a need to push for a new context flag.

It is probably simplest for users if implicit coercions didn't come
with control knobs.  We already have Fraction+float-->float
occurring without any exceptions or warnings, and nothing
bad has happened as a result.

Also, I'm reminded of Tim Peter's admonition to resist
extending the decimal spec.  

I used to worry that any decimal/float interactions were
most likely errors and shouldn't pass silently.  Now, I've
just stopped worrying and I feel better already ;-)
Adding a FAQ entry is simpler than building-out
Context object circuitry and documenting it in an
understandable way.

Raymond



On Mar 24, 2010, at 12:36 PM, Stefan Krah wrote:

> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Raymond Hettinger wrote:
>>> The decimal module is already drowning in complexity,
>>> so it would be best to keep it simple:  one boolean flag
>>> that if set would warn about any implicit decimal/float
>>> interaction.
>> 
>> Agreed - those that want exceptions instead can use the usual warnings
>> module mechanisms to trigger them.
> 
> 
> I'm not sure about the warnings module. If lower complexity is a goal,
> I would prefer Facundo's original proposal of just adding a single new
> signal. Users who just want to know if a NonIntegerConversion has occurred
> can check the flags, users who want an exception set the trap.
> 
> With the warnings module, users have to know (and deal with) two exception
> handling/suppressing mechanisms.
> 
> 
> Stefan Krah


From greg.ewing at canterbury.ac.nz  Wed Mar 24 22:13:52 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 25 Mar 2010 09:13:52 +1200
Subject: [Python-Dev] PEP 3147, __pycache__ directories and umask
In-Reply-To: <Pine.GSO.4.64.1003232042080.23175@core.cs.uwaterloo.ca>
References: <20100322095604.3371039e@heresy>
	<loom.20100322T150829-761@post.gmane.org>
	<4BA80418.6030905@canterbury.ac.nz>
	<loom.20100323T005510-312@post.gmane.org>
	<4BA93580.9000402@canterbury.ac.nz>
	<loom.20100323T235432-758@post.gmane.org>
	<Pine.GSO.4.64.1003232042080.23175@core.cs.uwaterloo.ca>
Message-ID: <4BAA8090.9080508@canterbury.ac.nz>

Isaac Morland wrote:
> the 
> benefit to me and to Greg and to others writing .py code is that our 
> directories will contain *.py and __pycache__, rather than *.py and 
> *.pyc. So it will be much easier to see what is actually there.

Yes. When using MacOSX I do most of my work using the
Finder's column view. With two windows open one above the
other, there's room for about a dozen files to be seen
at once. There's no way to filter the view or sort by
anything other than name, so with the current .pyc
scheme, half of that valuable screen space is wasted.

-- 
Greg

From dickinsm at gmail.com  Wed Mar 24 22:09:26 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 21:09:26 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org>
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
Message-ID: <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com>

Slight change of topic.  I've been implementing the extra comparisons
required for the Decimal type and found an anomaly while testing.
Currently in py3k, order comparisons (but not ==, !=) between a
complex number and another complex, float or int raise TypeError:

>>> z = complex(0, 0)
>>> z < int()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: complex() < int()
>>> z < float()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: complex() < float()
>>> z < complex()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: complex() < complex()

But Fraction is the odd man out:  a comparison between a Fraction and
a complex raises a TypeError for complex numbers with nonzero
imaginary component, but returns a boolean value if the complex number
has zero imaginary component:

>>> z < Fraction()
False
>>> complex(0, 1) < Fraction()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: complex() < Fraction()

I'm tempted to call this Fraction behaviour a bug, but maybe it arises
from the numeric integration themes of PEP 3141.  Any ideas?

Mark

From guido at python.org  Wed Mar 24 22:09:59 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Mar 2010 14:09:59 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com> 
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com> 
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> 
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
Message-ID: <ca471dc21003241409y7567f639xd9c3976151117369@mail.gmail.com>

W00t!

On Wed, Mar 24, 2010 at 1:56 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> FWIW, my viewpoint on this is softening over time
> and I no longer feel a need to push for a new context flag.
>
> It is probably simplest for users if implicit coercions didn't come
> with control knobs. ?We already have Fraction+float-->float
> occurring without any exceptions or warnings, and nothing
> bad has happened as a result.
>
> Also, I'm reminded of Tim Peter's admonition to resist
> extending the decimal spec.
>
> I used to worry that any decimal/float interactions were
> most likely errors and shouldn't pass silently. ?Now, I've
> just stopped worrying and I feel better already ;-)
> Adding a FAQ entry is simpler than building-out
> Context object circuitry and documenting it in an
> understandable way.
>
> Raymond
>
>
>
> On Mar 24, 2010, at 12:36 PM, Stefan Krah wrote:
>
>> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Raymond Hettinger wrote:
>>>> The decimal module is already drowning in complexity,
>>>> so it would be best to keep it simple: ?one boolean flag
>>>> that if set would warn about any implicit decimal/float
>>>> interaction.
>>>
>>> Agreed - those that want exceptions instead can use the usual warnings
>>> module mechanisms to trigger them.
>>
>>
>> I'm not sure about the warnings module. If lower complexity is a goal,
>> I would prefer Facundo's original proposal of just adding a single new
>> signal. Users who just want to know if a NonIntegerConversion has occurred
>> can check the flags, users who want an exception set the trap.
>>
>> With the warnings module, users have to know (and deal with) two exception
>> handling/suppressing mechanisms.
>>
>>
>> Stefan Krah
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From dickinsm at gmail.com  Wed Mar 24 22:11:26 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 21:11:26 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org>
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
Message-ID: <5c6f2a5d1003241411y21d9d661p16501c48d4bfcd13@mail.gmail.com>

On Wed, Mar 24, 2010 at 8:56 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> FWIW, my viewpoint on this is softening over time
> and I no longer feel a need to push for a new context flag.
>
> It is probably simplest for users if implicit coercions didn't come
> with control knobs. ?We already have Fraction+float-->float
> occurring without any exceptions or warnings, and nothing
> bad has happened as a result.

I agree with this;  I'd be happy to avoid the control knobs.

Mark

From raymond.hettinger at gmail.com  Wed Mar 24 22:29:27 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 24 Mar 2010 14:29:27 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org>
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
	<5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com>
Message-ID: <341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com>


On Mar 24, 2010, at 2:09 PM, Mark Dickinson wrote:

> Slight change of topic.  I've been implementing the extra comparisons
> required for the Decimal type and found an anomaly while testing.
> Currently in py3k, order comparisons (but not ==, !=) between a
> complex number and another complex, float or int raise TypeError:
> 
>>>> z = complex(0, 0)
>>>> z < int()
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> TypeError: unorderable types: complex() < int()
>>>> z < float()
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> TypeError: unorderable types: complex() < float()
>>>> z < complex()
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> TypeError: unorderable types: complex() < complex()
> 
> But Fraction is the odd man out:  a comparison between a Fraction and
> a complex raises a TypeError for complex numbers with nonzero
> imaginary component, but returns a boolean value if the complex number
> has zero imaginary component:
> 
>>>> z < Fraction()
> False
>>>> complex(0, 1) < Fraction()
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> TypeError: unorderable types: complex() < Fraction()
> 
> I'm tempted to call this Fraction behaviour a bug, but maybe it arises
> from the numeric integration themes of PEP 3141.  Any ideas?

Conceptually, it's a bug.  The numeric tower treats non-complex
numbers as special cases of complex where the imaginary
component is zero (that's why the non-complex types all support
real/imag), and since complex numbers are not allowed to compare
to themselves, they shouldn't compare to anything else either.

To confirm, we should ask Jeffrey Y to opine.


Raymond


From alexander.belopolsky at gmail.com  Wed Mar 24 22:31:57 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 17:31:57 -0400
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
Message-ID: <d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>

On Wed, Mar 24, 2010 at 3:26 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
..
> Here's an interesting recent blog post on this subject, from the
> creator of Eiffel:
>
> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/

It appears that Meyer's view has evolved over the years:

"""
In this context it doesn't particularly shock me that operations on
NaN should cause invariant violations. After all, isn't NaN supposed
to mean "not a number"? If it's not a number, it doesn't have to
satisfy the properties of numbers.
"""
"NaN and floating point exceptions" by Roger Browne quoting an  ISE
Eiffel mailing list post by Bertrand Meyer

http://teameiffel.blogspot.com/2006/04/nan-and-floating-point-exceptions.html

Compare this with the conclusion he reached in "Pillars:"

"It is rather dangerous indeed to depart from the fundamental laws of
mathematics. "


To bring the discussion back on topic for python-dev, I would argue
that reflexivity should hold for hashable values.  Having it otherwise
would lead to unsurmountable problems when storing such values in sets
or using them as keys in dictionaries.  If x == x is False stays for x
= float('nan'), I would argue that hash(float('nan')) should raise
NotImplemented or ValueError.

From v+python at g.nevcal.com  Wed Mar 24 23:09:29 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 24 Mar 2010 15:09:29 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>	<20100323120959.GA486@mail.bytereef.org>	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>	<4BA9FC18.90102@gmail.com>
	<20100324193613.GA470@mail.bytereef.org>
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
Message-ID: <4BAA8D99.4000606@g.nevcal.com>

On 3/24/2010 1:56 PM, Raymond Hettinger wrote:
> FWIW, my viewpoint on this is softening over time
> and I no longer feel a need to push for a new context flag.
>    


To make Decimal useful for people that want to control its numerical 
quality, there must be a way to exclude accidental operations, and 
preferably an option for producing exceptions.  Otherwise the status quo 
in 3.x is preferable to adding it to the numeric tree.

I agree that not worrying can make you feel better, though. :)


> IIRC, the original purpose of a NaN was to serve as a placeholder
> value in a long series of floating point ops so that the programmer
> would not have to introduce edge case tests at each stage
> of a calculation.  Yet, I look at the code for the decimal module
> and the C code for the math module and see that the opposite
> result occurred, the code is littered with is_special(x) tests
> and handlers.

You are looking at the wrong code.  The point is that the code _using_ 
the math module and decimal module doesn't have to be littered with edge 
case tests; not that the implementation, which the good folks at IEEE 
figured would be in custom silicon that could do the edge case checking 
in parallel with other operations and with no net slowdown, would not 
have to be littered with edge cases.


Glenn

From martin at v.loewis.de  Wed Mar 24 23:14:18 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 24 Mar 2010 23:14:18 +0100
Subject: [Python-Dev] argparse.py is licensed under the Apache License
In-Reply-To: <d11dcfba1003240905n59882d52ge987a68bb885a6f4@mail.gmail.com>
References: <e27efe131003240142v50833b3cu80071e69062cb201@mail.gmail.com>
	<d11dcfba1003240905n59882d52ge987a68bb885a6f4@mail.gmail.com>
Message-ID: <4BAA8EBA.8030302@v.loewis.de>

> Yes, I have signed a contributor agreement. I was thinking of just
> removing the license entirely, but if it's better to replace it with
> "Licensed to PSF under a Contributor Agreement", that's fine too. Let
> me know, and I'll take care of this today.

Technically, your *contribution* should have included that line, whereas
the code that we (i.e. the PSF) immediately relicense won't.

IIUC, and IANAL, the line indicates that you do intend your contribution
to be covered by your agreement (just so you couldn't claim we stole it
from your laptop, and you didn't really mean to contribute it).
Of course, committing it to subversion is an even stronger indication
that you wanted to contribute it :-)

The PSF then immediately exercises its right to relicense the code,
currently under the PSF license.

Regards,
Martin

From raymond.hettinger at gmail.com  Wed Mar 24 23:21:53 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 24 Mar 2010 15:21:53 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
Message-ID: <AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>


On Mar 24, 2010, at 2:31 PM, Alexander Belopolsky wrote:

> """
> In this context it doesn't particularly shock me that operations on
> NaN should cause invariant violations. After all, isn't NaN supposed
> to mean "not a number"? If it's not a number, it doesn't have to
> satisfy the properties of numbers.
> """

This sound like a good, universal reply to "bug reports" about
various containers/tools not working with NaNs :-)

Bug report: "Container X or Function Y doesn't behave well when I
give it a NaN as an input".

Closed -- Won't fix:  "It does not shock me that a NaN violates that
tool or container's invariants."

> 
> To bring the discussion back on topic for python-dev, I would argue
> that reflexivity should hold for hashable values.  Having it otherwise
> would lead to unsurmountable problems when storing such values in sets
> or using them as keys in dictionaries.  If x == x is False stays for x
> = float('nan'), I would argue that hash(float('nan')) should raise
> NotImplemented or ValueError.

Hashability isn't the only place where you have a problem.
Even unordered collections are affected:

  >>> class ListBasedSet(collections.Set):
     ''' Alternate set implementation favoring space over speed
         and not requiring the set elements to be hashable. '''
     def __init__(self, iterable):
         self.elements = lst = []
         for value in iterable:
             if value not in lst:
                 lst.append(value)
     def __iter__(self):
         return iter(self.elements)
     def __contains__(self, value):
         return any(value == elem for elem in self.elements)
     def __len__(self):
         return len(self.elements)

        
>>> n = float('Nan')
>>> s = ListBasedSet([n])
>>> n in s                 # unexpected result
False
>>> len(s)                # expected result
1
>>> len(s & s)         # unexpected result
0

If we want to be able to reason about our programs,
then we need to rely on equality relations being
reflexsive, symmetric, and transitive.  Otherwise,
containers and functions can't even make minimal 
guarantees about what they do.

Anything else smells of a Douglas Hofstadter style
or Betrand Russell style logic bomb:

* this sentence is a lie
* this object isn't equal to itself
* this is a set containing sets that are not members of themselves

The property of NaN objects not being equal to themselves
is more harmful than helpful.  We should probably draw the
line at well-defined numeric contexts such as the decimal module
and stop trying to propagate NaN awareness throughout the
entire object model.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100324/a4983fe4/attachment.html>

From guido at python.org  Wed Mar 24 23:23:11 2010
From: guido at python.org (Guido van Rossum)
Date: Wed, 24 Mar 2010 15:23:11 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com> 
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com> 
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com> 
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org> 
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
	<5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com> 
	<341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com>
Message-ID: <ca471dc21003241523i7e0b7fd0va3247fcbe6c59f72@mail.gmail.com>

On Wed, Mar 24, 2010 at 2:29 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 24, 2010, at 2:09 PM, Mark Dickinson wrote:
>
>> Slight change of topic. ?I've been implementing the extra comparisons
>> required for the Decimal type and found an anomaly while testing.
>> Currently in py3k, order comparisons (but not ==, !=) between a
>> complex number and another complex, float or int raise TypeError:
>>
>>>>> z = complex(0, 0)
>>>>> z < int()
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> TypeError: unorderable types: complex() < int()
>>>>> z < float()
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> TypeError: unorderable types: complex() < float()
>>>>> z < complex()
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> TypeError: unorderable types: complex() < complex()
>>
>> But Fraction is the odd man out: ?a comparison between a Fraction and
>> a complex raises a TypeError for complex numbers with nonzero
>> imaginary component, but returns a boolean value if the complex number
>> has zero imaginary component:
>>
>>>>> z < Fraction()
>> False
>>>>> complex(0, 1) < Fraction()
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> TypeError: unorderable types: complex() < Fraction()
>>
>> I'm tempted to call this Fraction behaviour a bug, but maybe it arises
>> from the numeric integration themes of PEP 3141. ?Any ideas?
>
> Conceptually, it's a bug. ?The numeric tower treats non-complex
> numbers as special cases of complex where the imaginary
> component is zero (that's why the non-complex types all support
> real/imag), and since complex numbers are not allowed to compare
> to themselves, they shouldn't compare to anything else either.

That's how I read the PEP too. PEP 3141 doesn't define any ordering
operations on Complex, they only show up on Real.

> To confirm, we should ask Jeffrey Y to opine.

CC'ed him. After all looks like it was he who added it to Fraction. :-)

-- 
--Guido van Rossum (python.org/~guido)

From raymond.hettinger at gmail.com  Wed Mar 24 23:25:09 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 24 Mar 2010 15:25:09 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
Message-ID: <D6657925-4B04-40F5-BAB1-7C27D1F2BD78@rcn.com>


> Even unordered collections are affected:

s/unordered/unhashed, equality-based/


Raymond

From alexander.belopolsky at gmail.com  Wed Mar 24 23:30:17 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 18:30:17 -0400
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
Message-ID: <d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>

On Wed, Mar 24, 2010 at 6:21 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
..
> If we want to be able to reason about our programs,
> then we need to rely on equality relations being
> reflexsive, symmetric, and transitive. ?Otherwise,
> containers and functions can't even make minimal
> guarantees?about what they do.

+1

> .. ?We should probably draw the
> line at well-defined numeric contexts such as the decimal module
> and stop trying to propagate NaN awareness throughout the
> entire object model.

I am not sure what this means in practical terms.   Should
float('nan') == float('nan') return True or should float('nan') raise
an exception to begin with?   I would prefer the former.

From dickinsm at gmail.com  Wed Mar 24 23:31:42 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 22:31:42 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
Message-ID: <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>

On Wed, Mar 24, 2010 at 10:30 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 6:21 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
> ..
>> If we want to be able to reason about our programs,
>> then we need to rely on equality relations being
>> reflexsive, symmetric, and transitive. ?Otherwise,
>> containers and functions can't even make minimal
>> guarantees?about what they do.
>
> +1
>
>> .. ?We should probably draw the
>> line at well-defined numeric contexts such as the decimal module
>> and stop trying to propagate NaN awareness throughout the
>> entire object model.
>
> I am not sure what this means in practical terms. ? Should
> float('nan') == float('nan') return True or should float('nan') raise
> an exception to begin with? ? I would prefer the former.
>

Neither is necessary, because Python doesn't actually use == as the
equivalence relation for containment testing:  the actual equivalence
relation is:  x equivalent to y iff id(x) == id(y) or x == y.  This
restores the missing reflexivity (besides being a useful
optimization).

Mark

From alexander.belopolsky at gmail.com  Wed Mar 24 23:36:49 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 18:36:49 -0400
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
Message-ID: <d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>

On Wed, Mar 24, 2010 at 6:31 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
..
> Neither is necessary, because Python doesn't actually use == as the
> equivalence relation for containment testing: ?the actual equivalence
> relation is: ?x equivalent to y iff id(x) == id(y) or x == y. ?This
> restores the missing reflexivity (besides being a useful
> optimization).

No, it does not:

>>> float('nan') in [float('nan')]
False


It would if NaNs were always interned, but they are not.

From alexander.belopolsky at gmail.com  Wed Mar 24 23:43:03 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 18:43:03 -0400
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
Message-ID: <d38f5331003241543q10f18c14i1fdd86a4052e968e@mail.gmail.com>

Not to mention the following aberrations:

>>> {x for x in [float('nan')] * 10}
{nan}
>>> {float(x) for x in ['nan'] * 10}
{nan, nan, nan, nan, nan, nan, nan, nan, nan, nan}

>>> {float('nan')} | {float('nan')}
{nan, nan}
>>> {float('nan')} & {float('nan')}
set()



On Wed, Mar 24, 2010 at 6:36 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 6:31 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> ..
>> Neither is necessary, because Python doesn't actually use == as the
>> equivalence relation for containment testing: ?the actual equivalence
>> relation is: ?x equivalent to y iff id(x) == id(y) or x == y. ?This
>> restores the missing reflexivity (besides being a useful
>> optimization).
>
> No, it does not:
>
>>>> float('nan') in [float('nan')]
> False
>
>
> It would if NaNs were always interned, but they are not.
>

From dickinsm at gmail.com  Wed Mar 24 23:47:42 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 22:47:42 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
Message-ID: <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>

On Wed, Mar 24, 2010 at 10:36 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 6:31 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> ..
>> Neither is necessary, because Python doesn't actually use == as the
>> equivalence relation for containment testing: ?the actual equivalence
>> relation is: ?x equivalent to y iff id(x) == id(y) or x == y. ?This
>> restores the missing reflexivity (besides being a useful
>> optimization).
>
> No, it does not:
>
>>>> float('nan') in [float('nan')]
> False

Sure, but just think of it as having two different nans there.  (You
could imagine thinking of the id of the nan as part of the payload.)
There's no ideal solution here;  IMO, the compromise that currently
exists is an acceptable one.

Mark

From alexander.belopolsky at gmail.com  Wed Mar 24 23:52:38 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 18:52:38 -0400
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
Message-ID: <d38f5331003241552qf5afd9fsda4cee6f6aa3ce1e@mail.gmail.com>

On Wed, Mar 24, 2010 at 6:47 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
..
> There's no ideal solution here; ?IMO, the compromise that currently
> exists is an acceptable one.

I don't see a compromise.   So far I failed to find a use case that
benefits from NaN violating reflexivity.

From dickinsm at gmail.com  Thu Mar 25 00:02:39 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 23:02:39 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <d38f5331003241552qf5afd9fsda4cee6f6aa3ce1e@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
	<d38f5331003241552qf5afd9fsda4cee6f6aa3ce1e@mail.gmail.com>
Message-ID: <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com>

On Wed, Mar 24, 2010 at 10:52 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 6:47 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> ..
>> There's no ideal solution here; ?IMO, the compromise that currently
>> exists is an acceptable one.
>
> I don't see a compromise. ? So far I failed to find a use case that
> benefits from NaN violating reflexivity.
>

So if I understand correctly, you propose that float('nan') ==
float('nan') return True.  Would you also suggest extending this
behaviour to Decimal nans?

From alexander.belopolsky at gmail.com  Thu Mar 25 00:11:09 2010
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 24 Mar 2010 19:11:09 -0400
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
	<d38f5331003241552qf5afd9fsda4cee6f6aa3ce1e@mail.gmail.com>
	<5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com>
Message-ID: <d38f5331003241611n777587beq99890761849780f3@mail.gmail.com>

On Wed, Mar 24, 2010 at 7:02 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
..
>
> So if I understand correctly, you propose that float('nan') ==
> float('nan') return True. ?Would you also suggest extending this
> behaviour to Decimal nans?
>

yes

From steven.bethard at gmail.com  Thu Mar 25 00:18:02 2010
From: steven.bethard at gmail.com (Steven Bethard)
Date: Wed, 24 Mar 2010 16:18:02 -0700
Subject: [Python-Dev] argparse.py is licensed under the Apache License
In-Reply-To: <bbaeab101003241008j3dad3388p540e3ca3e835ee56@mail.gmail.com>
References: <e27efe131003240142v50833b3cu80071e69062cb201@mail.gmail.com>
	<d11dcfba1003240905n59882d52ge987a68bb885a6f4@mail.gmail.com>
	<bbaeab101003241008j3dad3388p540e3ca3e835ee56@mail.gmail.com>
Message-ID: <d11dcfba1003241618l5abab2d4lda5a230d0fc44086@mail.gmail.com>

On Wed, Mar 24, 2010 at 10:08 AM, Brett Cannon <brett at python.org> wrote:
> On Wed, Mar 24, 2010 at 09:05, Steven Bethard <steven.bethard at gmail.com>
> wrote:
>> On Wed, Mar 24, 2010 at 1:42 AM, Amaury Forgeot d'Arc
>> <amauryfa at gmail.com> wrote:
>> > I noticed that the newly added argparse module has an unusual
>> > licence header,?included below. This is the only file in the Python tree
>> > that contains an explicit reference to the Apache License,
>> > and this leads me to some questions:
>>
>> Sorry, I forgot to remove this when moving argparse to the stdlib.
>> There's probably one test\test_argparse too.
>
> I say just strip it out.

Benjamin beat me to the punch for argparse.py, but I also fixed this
in test_argparse.py.

Steve
-- 
Where did you get that preposterous hypothesis?
Did Steve tell you that?
        --- The Hiphopopotamus

From gruszczy at gmail.com  Thu Mar 25 00:18:21 2010
From: gruszczy at gmail.com (=?UTF-8?Q?Filip_Gruszczy=C5=84ski?=)
Date: Thu, 25 Mar 2010 00:18:21 +0100
Subject: [Python-Dev] Issue 6081: format string using mapping rather than
	kwargs
Message-ID: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>

Hello everyone,

I was told to bring this issue to python-dev, but I was reluctant to
do this until I was confident I can solve the problem. I have managed
to provide the patch that works, so now I can talk about how this
should be done, just as Eric told me. Here is the link to the ticket:
http://bugs.python.org/issue6081.

In the issue Raymond Hettinger proposed adding method to string, that
would allow to explicitly pass subscriptable object, from which
formatting of a string would be done. Two questions arise:
1) whether add such functionality at all?
2) how the method should be called? Should formatting options based on
*args be kept (like using {0} in formatted string) or not and how
argument should be passed. Eric suggested passing only mapping and
only adding *args, if that is found useful by users; I have though
about having both mapping and *args, which would keep it similar to
the old formatting. Another option is to have *args and one keyword
with the mapping (instead of kwargs).

I would appreciate any advice on this topic, even if this ticket would
be dismissed altogether, as I would like to learn as much as possible
on practices on developing Python.

-- 
Filip Gruszczy?ski

From dickinsm at gmail.com  Thu Mar 25 00:31:57 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Wed, 24 Mar 2010 23:31:57 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <d38f5331003241611n777587beq99890761849780f3@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
	<d38f5331003241552qf5afd9fsda4cee6f6aa3ce1e@mail.gmail.com>
	<5c6f2a5d1003241602h6567ad30p80f3cba70483a32@mail.gmail.com>
	<d38f5331003241611n777587beq99890761849780f3@mail.gmail.com>
Message-ID: <5c6f2a5d1003241631m3c6af7e8p93273583cb322b73@mail.gmail.com>

On Wed, Mar 24, 2010 at 11:11 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 7:02 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> ..
>>
>> So if I understand correctly, you propose that float('nan') ==
>> float('nan') return True. ?Would you also suggest extending this
>> behaviour to Decimal nans?
>>
>
> yes
>

Okay.  So now it seems to me that there are many decisions to make:
should any Decimal nan compare equal to any other?  What if the two
nans have different payloads or signs?  How about comparing a
signaling nan with either an arbitrary quiet nan, or with the exact
quiet nan that corresponds to the signaling nan?  How do decimal nans
compare with float nans?  Should Decimal.compare(Decimal('nan'),
Decimal('nan')) return 0 rather than nan?  If not, how do you justify
the difference between == and compare?  If so, how do you justify the
deviation from the standard on which the decimal modulo is based?

In answering all these questions, you effectively end up developing
your own standard, and hoping that all the answers you chose are
sensible, consistent, and useful.

Alternatively, we could do what we're currently doing:  make use of
*existing* standards to answer these questions, and rely on the
expertise of the many who've thought about this in depth.

You say that you don't see any compromise:  I say that there's value
in adhering to (de facto and de jure) standards, and I see a
compromise between standards adherence and Python pragmatics.

Mark

From greg.ewing at canterbury.ac.nz  Thu Mar 25 01:32:49 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 25 Mar 2010 13:32:49 +1300
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org>
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
	<5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com>
	<341CCB69-D532-4AF3-B811-34BA8263DEAD@gmail.com>
Message-ID: <4BAAAF31.5060709@canterbury.ac.nz>

Raymond Hettinger wrote:

> Conceptually, it's a bug.  The numeric tower treats non-complex
> numbers as special cases of complex where the imaginary
> component is zero (that's why the non-complex types all support
> real/imag), and since complex numbers are not allowed to compare
> to themselves, they shouldn't compare to anything else either.

There's a contradiction in there somewhere. If you believe
that a non-complex is equivalent to a complex with zero
imaginary part, then you *should* be able to compare two
complexes provided that their imaginary parts are both
zero.

(I don't think that should be the case, BTW -- complex
numbers live on a two-dimensional plane, and from a
geometrical point of view there's no reason to single
out the x-axis and give it special treatment.)

-- 
Greg


From jyasskin at gmail.com  Thu Mar 25 02:15:58 2010
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Wed, 24 Mar 2010 18:15:58 -0700
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org>
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
	<5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com>
Message-ID: <5d44f72f1003241815g109236fco1ab9607544352cf0@mail.gmail.com>

On Wed, Mar 24, 2010 at 2:09 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> Slight change of topic. ?I've been implementing the extra comparisons
> required for the Decimal type and found an anomaly while testing.
> Currently in py3k, order comparisons (but not ==, !=) between a
> complex number and another complex, float or int raise TypeError:
>
>>>> z = complex(0, 0)
>>>> z < int()
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> TypeError: unorderable types: complex() < int()
>>>> z < float()
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> TypeError: unorderable types: complex() < float()
>>>> z < complex()
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> TypeError: unorderable types: complex() < complex()
>
> But Fraction is the odd man out: ?a comparison between a Fraction and
> a complex raises a TypeError for complex numbers with nonzero
> imaginary component, but returns a boolean value if the complex number
> has zero imaginary component:
>
>>>> z < Fraction()
> False
>>>> complex(0, 1) < Fraction()
> Traceback (most recent call last):
> ?File "<stdin>", line 1, in <module>
> TypeError: unorderable types: complex() < Fraction()
>
> I'm tempted to call this Fraction behaviour a bug, but maybe it arises
> from the numeric integration themes of PEP 3141. ?Any ideas?

I'd call it a bug.

From g.brandl at gmx.net  Thu Mar 25 07:41:26 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 25 Mar 2010 06:41:26 +0000
Subject: [Python-Dev] Request for commit access
In-Reply-To: <loom.20100323T235922-727@post.gmane.org>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>	<20100323152748.50ECE1FD33F@kimball.webabinitio.net>	<4BA94202.6060502@v.loewis.de>
	<loom.20100323T235922-727@post.gmane.org>
Message-ID: <hof0j2$75o$1@dough.gmane.org>

Am 23.03.2010 23:01, schrieb Antoine Pitrou:
> Martin v. L?wis <martin <at> v.loewis.de> writes:
>> 
>> Procedurally, I wonder where people got the notion from that you can or
>> need to apply for commit access. IIUC, it used to be the case that you
>> would be recommended for commit access, by some (more or less senior)
>> fellow committer. That person then would work on actually getting commit
>> access to the new committer - perhaps by first asking other people in
>> private, to avoid any public embarrassment if access is ultimately
>> denied. IMO, that committer should then also mentor the new guy, both by
>> helping out in difficult cases, and by closely following commits to see
>> whether (possibly unstated) conventions are being followed.
>> 
>> I'm not really picking on Brian here specifically, I just want to point
>> out that I dislike this (apparent) change in process, primarily because
>> of the risk of embarrassment.
> 
> For the record, I'm not opposing any point you are making, but all this is not
> clearly written out, and I think that's why people (including me) lately have
> been thinking that the candidate for commit rights had to declare himself on
> this mailing-list.

I don't think it is "wrong" to do it this way -- but of course the hopeful
new committer needs to be prepared for a "no".

I still would like every new committer to have a "mentor" from the pool of
more experienced committers; however for people who are very active on IRC
like Brian, there's usually one of the around to review a patch before it
is committed.

Georg


From g.brandl at gmx.net  Thu Mar 25 07:54:39 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 25 Mar 2010 06:54:39 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
Message-ID: <hof1br$8s5$1@dough.gmane.org>

Am 24.03.2010 22:47, schrieb Mark Dickinson:
> On Wed, Mar 24, 2010 at 10:36 PM, Alexander Belopolsky
> <alexander.belopolsky at gmail.com> wrote:
>> On Wed, Mar 24, 2010 at 6:31 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
>> ..
>>> Neither is necessary, because Python doesn't actually use == as the
>>> equivalence relation for containment testing:  the actual equivalence
>>> relation is:  x equivalent to y iff id(x) == id(y) or x == y.  This
>>> restores the missing reflexivity (besides being a useful
>>> optimization).
>>
>> No, it does not:
>>
>>>>> float('nan') in [float('nan')]
>> False
> 
> Sure, but just think of it as having two different nans there.  (You
> could imagine thinking of the id of the nan as part of the payload.)

That's interesting.  Thinking of each value created by float('nan') as
a different nan makes sense to my naive mind, and it also explains
nicely the behavior present right now.  Each nan comes from a different
operation and therefore is a "different" non-number.

Of course, float being an immutable type, there is some reason to expect
that all values created by float('nan') should be identical, but after all,
datetime is an immutable type as well, but you wouldn't expect

   datetime.now() in [datetime.now()]

to be true.  The only wart left is that you can't distinguish different
nans by their string representation -- this could be remedied by making
it ``"nan-%s" % id(self)``, but that looks a bit ugly to me.

Georg


From mal at egenix.com  Thu Mar 25 10:10:26 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 25 Mar 2010 10:10:26 +0100
Subject: [Python-Dev] [Python-checkins] r79397 - in
 python/trunk:	Doc/c-api/capsule.rst
 Doc/c-api/cobject.rst	Doc/c-api/concrete.rst	Doc/data/refcounts.dat	Doc/extending/extending.rst
 Include/Python.h Include/cStringIO.h	Include/cobject.h	Include/datetime.h
 Include/py_curses.h Include/pycapsule.h	Include/pyexpat.h	Include/ucnhash.h
 Lib/test/test_sys.py	Makefile.pre.in Misc/NEWS	Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c	Modules/_ctypes/ctypes.h	Modules/_cursesmodule.c
 Modules/_elementtree.c	Modules/_testcapimodule.c	Modules/cStringIO.c	Modules/cjkcodecs/cjkcodecs.h
 Modules/cjkcodecs/multibytecodec.c	Modules/cjkcodecs/multibytecodec.h	Modules/datetimemodule.c
 Modules/pyexpat.c	Modules/socketmodule.c	Modules/socketmodule.h
 Modules/unicodedata.c Objects/capsule.c
 Objects/object.c	Objects/unicodeobject.c	PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj	PC/os2emx/python27.def	PC/os2vacpp/python.def
 Python/compile.c Python/getargs.c
In-Reply-To: <20100325005454.B1AC5FCCD@mail.python.org>
References: <20100325005454.B1AC5FCCD@mail.python.org>
Message-ID: <4BAB2882.6070508@egenix.com>

larry.hastings wrote:
> Author: larry.hastings
> Date: Thu Mar 25 01:54:54 2010
> New Revision: 79397
> 
> Log:
> Backported PyCapsule from 3.1, and converted most uses of
> CObject to PyCapsule.

Backporting PyCapsule is fine, but the changes you made to all
those PyCObject uses does not look backwards compatible.

The C APIs exposed by the modules (e.g. the datetime module)
are used in lots of 3rd party extension modules and changing
them from PyCObject to PyCapsule is a major change in the
module API.

> Added:
>    python/trunk/Doc/c-api/capsule.rst
>    python/trunk/Include/pycapsule.h
>    python/trunk/Objects/capsule.c
> Modified:
>    python/trunk/Doc/c-api/cobject.rst
>    python/trunk/Doc/c-api/concrete.rst
>    python/trunk/Doc/data/refcounts.dat
>    python/trunk/Doc/extending/extending.rst
>    python/trunk/Include/Python.h
>    python/trunk/Include/cStringIO.h
>    python/trunk/Include/cobject.h
>    python/trunk/Include/datetime.h
>    python/trunk/Include/py_curses.h
>    python/trunk/Include/pyexpat.h
>    python/trunk/Include/ucnhash.h
>    python/trunk/Lib/test/test_sys.py
>    python/trunk/Makefile.pre.in
>    python/trunk/Misc/NEWS
>    python/trunk/Modules/_ctypes/callproc.c
>    python/trunk/Modules/_ctypes/cfield.c
>    python/trunk/Modules/_ctypes/ctypes.h
>    python/trunk/Modules/_cursesmodule.c
>    python/trunk/Modules/_elementtree.c
>    python/trunk/Modules/_testcapimodule.c
>    python/trunk/Modules/cStringIO.c
>    python/trunk/Modules/cjkcodecs/cjkcodecs.h
>    python/trunk/Modules/cjkcodecs/multibytecodec.c
>    python/trunk/Modules/cjkcodecs/multibytecodec.h
>    python/trunk/Modules/datetimemodule.c
>    python/trunk/Modules/pyexpat.c
>    python/trunk/Modules/socketmodule.c
>    python/trunk/Modules/socketmodule.h
>    python/trunk/Modules/unicodedata.c
>    python/trunk/Objects/object.c
>    python/trunk/Objects/unicodeobject.c
>    python/trunk/PC/VS7.1/pythoncore.vcproj
>    python/trunk/PC/VS8.0/pythoncore.vcproj
>    python/trunk/PC/os2emx/python27.def
>    python/trunk/PC/os2vacpp/python.def
>    python/trunk/Python/compile.c
>    python/trunk/Python/getargs.c

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 25 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
_______________________________________________
Python-checkins mailing list
Python-checkins at python.org
http://mail.python.org/mailman/listinfo/python-checkins

From steve at pearwood.info  Thu Mar 25 11:18:42 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 25 Mar 2010 21:18:42 +1100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
Message-ID: <201003252118.43146.steve@pearwood.info>

On Thu, 25 Mar 2010 05:26:12 am Alexander Belopolsky wrote:
> Mark, I wonder if you could describe an algorithm off the top of your
> head that relies on NaN == NaN being false.

I don't know whether "relies on" is appropriate, but consider:


def myfunc(x, y):
    if x == y:
        return 1.0
    else:
        return something_complicated**(x-y)


Optimising floating point code is fraught with dangers (the above fails 
for x=y=INF as well as NAN) but anything that make Not A Numbers 
pretend to be numbers is a bad thing.

I'd like to turn the question around ... what algorithms are there that 
rely on NaN == NaN being True?



-- 
Steven D'Aprano

From ncoghlan at gmail.com  Thu Mar 25 11:28:03 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Mar 2010 20:28:03 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <201003250634.13861.steve@pearwood.info>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<4BA9FBCE.5040908@gmail.com>	<5c6f2a5d1003240922k679301bcrb211d3ffe7e2c1b@mail.gmail.com>
	<201003250634.13861.steve@pearwood.info>
Message-ID: <4BAB3AB3.4030204@gmail.com>

Steven D'Aprano wrote:
> On Thu, 25 Mar 2010 03:22:29 am Mark Dickinson wrote:
>> The obvious way to do this nan interning for floats would be to put
>> the interning code into PyFloat_FromDouble.  I'm not sure whether
>> this would be worth the cost in terms of added code (and possibly
>> reduced performance, since the nan check would be done every time a
>> float was returned), but I'd be willing to review a patch.
> 
> I hope that it's obvious from my previous post that I do NOT want such 
> interning done, but since I put the idea in people's heads, I wish to 
> reiterate that I'm against the idea: -1 on interning NaNs. For the rare 
> application where it might be useful, it is easy to do in the 
> application code.

Yep, and I'll freely admit I didn't know about the potential additional
state on NaN values, or I wouldn't have suggested interning automatically.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Mar 25 11:31:33 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Mar 2010 20:31:33 +1000
Subject: [Python-Dev] __pycache__ creation
In-Reply-To: <hodfl5$gio$1@dough.gmane.org>
References: <20100322095604.3371039e@heresy>	<loom.20100322T150829-761@post.gmane.org>	<ho88hk$n52$1@dough.gmane.org>	<Pine.GSO.4.64.1003221321390.19415@core.cs.uwaterloo.ca>	<loom.20100322T191016-732@post.gmane.org>	<ho9b7a$noo$1@dough.gmane.org>	<hobqml$5d4$1@dough.gmane.org>	<4BAA0088.2010909@gmail.com>
	<hodfl5$gio$1@dough.gmane.org>
Message-ID: <4BAB3B85.701@gmail.com>

Ron Adam wrote:
> hmmmm... unless there is a __pycache__ *file* located there first. ;-)

Just a specific reason why attempting to create __pycache__ can fail
(which has defined behaviour in the PEP - running directly from the
source without caching the bytecode file).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Mar 25 12:00:35 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Mar 2010 21:00:35 +1000
Subject: [Python-Dev] Issue 6081: format string using mapping rather
 than kwargs
In-Reply-To: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>
Message-ID: <4BAB4253.8090605@gmail.com>

Filip Gruszczy?ski wrote:
> I would appreciate any advice on this topic, even if this ticket would
> be dismissed altogether, as I would like to learn as much as possible
> on practices on developing Python.

Raymond's use case is valid, but the currently proposed method name is
way too long to be usable and the use of *args is a definite misfeature.

I suggest approaching the issue with the exact spec needed to cover the
problem Raymond described:

   Add a method to strings that allows a format string to
   be applied to an existing mapping without implicitly
   converting the mapping to a dict first.

>From that spec, a straightforward API falls out:

    def format_mapping(self, kwds):
      # Method body actually written in C, so it can
      # easily invoke the internal formatting operation
      return do_string_format(self, NULL, kwds)

Sure, you can't mix positional and keyword arguments the way you can
with .format(), but you can't mix and match those with mod-formatting
either.

*If* support for a sequence were to be added (and that's a big if), it
should be added as an explicit second argument, not as a *args argument.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Mar 25 12:22:12 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Mar 2010 21:22:12 +1000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
Message-ID: <4BAB4764.4000500@gmail.com>

Mark Dickinson wrote:
> Here's an interesting recent blog post on this subject, from the
> creator of Eiffel:
> 
> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/

Interesting. So the natural tweak that would arise from that perspective
is for us to restore reflexivity by declaring that any given NaN is
equal to *itself* but not to any other NaN (even one with the same payload).

With NaN (in general) not being interned, that would actually fit the
idea of a NaN implicitly carrying the operation that created the NaN as
part of its definition of equivalence.

So, I'm specifically putting that proposal on the table for both float
and Decimal NaNs in Python:

  "Not a Number" is not a single floating point value. Instead each
  instance is a distinct value representing the precise conditions that
  created it. Thus, two "NaN" values x and y will compare equal iff they
  are the exact same NaN object (i.e. "if isnan(x) then x == y iff
  x is y".

As stated above, such a change would allow us to restore reflexivity
(eliminating a bunch of weirdness) while still honouring the idea of NaN
being a set of values rather than a single value.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Mar 25 12:25:35 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Mar 2010 21:25:35 +1000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <201003252118.43146.steve@pearwood.info>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
	<201003252118.43146.steve@pearwood.info>
Message-ID: <4BAB482F.4030608@gmail.com>

Steven D'Aprano wrote:
> I'd like to turn the question around ... what algorithms are there that 
> rely on NaN == NaN being True?

Absolutely anything that expects "x is y" to imply that "x == y". The
builtin containers enforce this by checking identity before they check
equality, but there are plenty of algorithms that won't.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From gruszczy at gmail.com  Thu Mar 25 12:48:28 2010
From: gruszczy at gmail.com (=?UTF-8?Q?Filip_Gruszczy=C5=84ski?=)
Date: Thu, 25 Mar 2010 12:48:28 +0100
Subject: [Python-Dev] Issue 6081: format string using mapping rather
	than kwargs
In-Reply-To: <4BAB4253.8090605@gmail.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> 
	<4BAB4253.8090605@gmail.com>
Message-ID: <1be78d221003250448v54b39b9fob81b1eb7364fbf73@mail.gmail.com>

> From that spec, a straightforward API falls out:
>
> ? ?def format_mapping(self, kwds):
> ? ? ?# Method body actually written in C, so it can
> ? ? ?# easily invoke the internal formatting operation
> ? ? ?return do_string_format(self, NULL, kwds)

Thanks a lot for the advice, I'll provide according patch hopefully in
the few days.

-- 
Filip Gruszczy?ski

From dickinsm at gmail.com  Thu Mar 25 12:55:57 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 11:55:57 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB4764.4000500@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com>
Message-ID: <5c6f2a5d1003250455o5f1e2d29ld18c9893d11d3e1@mail.gmail.com>

On Thu, Mar 25, 2010 at 11:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Mark Dickinson wrote:
>> Here's an interesting recent blog post on this subject, from the
>> creator of Eiffel:
>>
>> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/
>
> Interesting. So the natural tweak that would arise from that perspective
> is for us to restore reflexivity by declaring that any given NaN is
> equal to *itself* but not to any other NaN (even one with the same payload).
>
> With NaN (in general) not being interned, that would actually fit the
> idea of a NaN implicitly carrying the operation that created the NaN as
> part of its definition of equivalence.
>
> So, I'm specifically putting that proposal on the table for both float
> and Decimal NaNs in Python:
>
> ?"Not a Number" is not a single floating point value. Instead each
> ?instance is a distinct value representing the precise conditions that
> ?created it. Thus, two "NaN" values x and y will compare equal iff they
> ?are the exact same NaN object (i.e. "if isnan(x) then x == y iff
> ?x is y".

In other words, this would make explicit, at the level of ==, what
Python's already doing under the hood (e.g. in
PyObjectRichCompareBool) for membership testing---at least for nans.

> As stated above, such a change would allow us to restore reflexivity
> (eliminating a bunch of weirdness) while still honouring the idea of NaN
> being a set of values rather than a single value.

+0.2 from me.  I could happily live with this change;  but could also
equally live with the existing weirdness.

It's still a little odd for an immutable type to care about object
identity, but I guess oddness comes with the floating-point territory.
 :)

Mark

From eric at trueblade.com  Thu Mar 25 12:58:14 2010
From: eric at trueblade.com (Eric Smith)
Date: Thu, 25 Mar 2010 07:58:14 -0400
Subject: [Python-Dev] Issue 6081: format string using mapping
 rather	than kwargs
In-Reply-To: <1be78d221003250448v54b39b9fob81b1eb7364fbf73@mail.gmail.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>
	<4BAB4253.8090605@gmail.com>
	<1be78d221003250448v54b39b9fob81b1eb7364fbf73@mail.gmail.com>
Message-ID: <4BAB4FD6.80007@trueblade.com>

Filip Gruszczy?ski wrote:
>> From that spec, a straightforward API falls out:
>>
>>    def format_mapping(self, kwds):
>>      # Method body actually written in C, so it can
>>      # easily invoke the internal formatting operation
>>      return do_string_format(self, NULL, kwds)
> 
> Thanks a lot for the advice, I'll provide according patch hopefully in
> the few days.

I think this is basically my patch, with the method renamed, and needing 
tests.

From mal at egenix.com  Thu Mar 25 08:59:49 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 25 Mar 2010 08:59:49 +0100
Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk:
 Doc/c-api/capsule.rst Doc/c-api/cobject.rst
 Doc/c-api/concrete.rst	Doc/data/refcounts.dat Doc/extending/extending.rst
 Include/Python.h Include/cStringIO.h Include/cobject.h	Include/datetime.h
 Include/py_curses.h Include/pycapsule.h	Include/pyexpat.h Include/ucnhash.h
 Lib/test/test_sys.py	Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h	Modules/_cursesmodule.c
 Modules/_elementtree.c	Modules/_testcapimodule.c
 Modules/cStringIO.c	Modules/cjkcodecs/cjkcodecs.h
 Modules/cjkcodecs/multibytecodec.c	Modules/cjkcodecs/multibytecodec.h
 Modules/datetimemodule.c Modules/pyexpat.c
 Modules/socketmodule.c	Modules/socketmodule.h Modules/unicodedata.c
 Objects/capsule.c Objects/object.c
 Objects/unicodeobject.c	PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj	PC/os2emx/python27.def PC/os2vacpp/python.def
 Python/compile.c Python/getargs.c
In-Reply-To: <20100325005454.B1AC5FCCD@mail.python.org>
References: <20100325005454.B1AC5FCCD@mail.python.org>
Message-ID: <4BAB17F5.1000305@egenix.com>

larry.hastings wrote:
> Author: larry.hastings
> Date: Thu Mar 25 01:54:54 2010
> New Revision: 79397
> 
> Log:
> Backported PyCapsule from 3.1, and converted most uses of
> CObject to PyCapsule.

Backporting PyCapsule is fine, but the changes you made to all
those PyCObject uses does not look backwards compatible.

The C APIs exposed by the modules (e.g. the datetime module)
are used in lots of 3rd party extension modules and changing
them from PyCObject to PyCapsule is a major change in the
module API.

> Added:
>    python/trunk/Doc/c-api/capsule.rst
>    python/trunk/Include/pycapsule.h
>    python/trunk/Objects/capsule.c
> Modified:
>    python/trunk/Doc/c-api/cobject.rst
>    python/trunk/Doc/c-api/concrete.rst
>    python/trunk/Doc/data/refcounts.dat
>    python/trunk/Doc/extending/extending.rst
>    python/trunk/Include/Python.h
>    python/trunk/Include/cStringIO.h
>    python/trunk/Include/cobject.h
>    python/trunk/Include/datetime.h
>    python/trunk/Include/py_curses.h
>    python/trunk/Include/pyexpat.h
>    python/trunk/Include/ucnhash.h
>    python/trunk/Lib/test/test_sys.py
>    python/trunk/Makefile.pre.in
>    python/trunk/Misc/NEWS
>    python/trunk/Modules/_ctypes/callproc.c
>    python/trunk/Modules/_ctypes/cfield.c
>    python/trunk/Modules/_ctypes/ctypes.h
>    python/trunk/Modules/_cursesmodule.c
>    python/trunk/Modules/_elementtree.c
>    python/trunk/Modules/_testcapimodule.c
>    python/trunk/Modules/cStringIO.c
>    python/trunk/Modules/cjkcodecs/cjkcodecs.h
>    python/trunk/Modules/cjkcodecs/multibytecodec.c
>    python/trunk/Modules/cjkcodecs/multibytecodec.h
>    python/trunk/Modules/datetimemodule.c
>    python/trunk/Modules/pyexpat.c
>    python/trunk/Modules/socketmodule.c
>    python/trunk/Modules/socketmodule.h
>    python/trunk/Modules/unicodedata.c
>    python/trunk/Objects/object.c
>    python/trunk/Objects/unicodeobject.c
>    python/trunk/PC/VS7.1/pythoncore.vcproj
>    python/trunk/PC/VS8.0/pythoncore.vcproj
>    python/trunk/PC/os2emx/python27.def
>    python/trunk/PC/os2vacpp/python.def
>    python/trunk/Python/compile.c
>    python/trunk/Python/getargs.c

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 25 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ncoghlan at gmail.com  Thu Mar 25 13:10:27 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 Mar 2010 22:10:27 +1000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003250455o5f1e2d29ld18c9893d11d3e1@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	
	<4BAB4764.4000500@gmail.com>
	<5c6f2a5d1003250455o5f1e2d29ld18c9893d11d3e1@mail.gmail.com>
Message-ID: <4BAB52B3.2010702@gmail.com>

Mark Dickinson wrote:
> +0.2 from me.  I could happily live with this change;  but could also
> equally live with the existing weirdness.
> 
> It's still a little odd for an immutable type to care about object
> identity, but I guess oddness comes with the floating-point territory.
>  :)

The trick for me came in thinking of NaN as a set of values rather than
a single value - at that point, the different id values just reflect the
multitude of members of that set.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From dickinsm at gmail.com  Thu Mar 25 13:24:14 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 12:24:14 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB4764.4000500@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com>
Message-ID: <5c6f2a5d1003250524i2f5414cdta7a8d9defa5c3aad@mail.gmail.com>

On Thu, Mar 25, 2010 at 11:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> So, I'm specifically putting that proposal on the table for both float
> and Decimal NaNs in Python:
>
> ?"Not a Number" is not a single floating point value. Instead each
> ?instance is a distinct value representing the precise conditions that
> ?created it. Thus, two "NaN" values x and y will compare equal iff they
> ?are the exact same NaN object (i.e. "if isnan(x) then x == y iff
> ?x is y".

I'd also suggest that the language make no guarantees about whether
two distinct calls to float('nan') or Decimal('nan') (or any other
function call returning a nan) return identical values or not, but
leave implementations free to do what's convenient or efficient.

For example, with the current decimal module:   Decimal('nan') returns
a new nan each time, but Decimal(-1).sqrt() always returns the same
nan object (assuming that InvalidOperation isn't trapped).  I think
it's fine to regard this as an implementation detail.

Python 2.6.2 (r262:71600, Aug 26 2009, 09:40:44)
[GCC 4.2.1 (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from decimal import *
>>> getcontext().traps[InvalidOperation] = 0
>>> x, y = Decimal('nan'), Decimal('nan')
>>> id(x), id(y)
(47309953516000, 47309930620880)
>>> x, y = Decimal(-1).sqrt(), Decimal(-1).sqrt()
>>> id(x), id(y)
(9922272, 9922272)

Mark

From jcea at jcea.es  Thu Mar 25 13:36:28 2010
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 25 Mar 2010 13:36:28 +0100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <hof1br$8s5$1@dough.gmane.org>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
	<hof1br$8s5$1@dough.gmane.org>
Message-ID: <4BAB58CC.9070004@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/25/2010 07:54 AM, Georg Brandl wrote:
>>>>>> float('nan') in [float('nan')]
>>> False
>>
>> Sure, but just think of it as having two different nans there.  (You
>> could imagine thinking of the id of the nan as part of the payload.)
> 
> That's interesting.  Thinking of each value created by float('nan') as
> a different nan makes sense to my naive mind, and it also explains
> nicely the behavior present right now.  Each nan comes from a different
> operation and therefore is a "different" non-number.

Infinites are "not equal" for a good reason, for example.

1/0 and 2/0 are both infinites, but one is "greater" than the other. Or
(1/0)^(1/0), an infinite infinitelly "bigger".

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS6tYzJlgi5GaxT1NAQL7kgP+LqjzNKOkSOZ+gTvgKfUTrF1poNP1VMC4
1LTkCcpFQYUoc4d8kk6lmzN7RdBesidbnVC2SApKTdNTAfbKMB3hjkTIzoxbx9wf
sLb5IUSqhtc+xJ+JQFepQwA7YLa64AVI23/wZcJCkqCBIg6S5DuGxhWErr3TXVgF
GqcZjhvD2lA=
=AVOO
-----END PGP SIGNATURE-----

From jcea at jcea.es  Thu Mar 25 13:39:25 2010
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 25 Mar 2010 13:39:25 +0100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB4764.4000500@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com>
Message-ID: <4BAB597D.1060508@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/25/2010 12:22 PM, Nick Coghlan wrote:
>   "Not a Number" is not a single floating point value. Instead each
>   instance is a distinct value representing the precise conditions that
>   created it. Thus, two "NaN" values x and y will compare equal iff they
>   are the exact same NaN object (i.e. "if isnan(x) then x == y iff
>   x is y".
> 
> As stated above, such a change would allow us to restore reflexivity
> (eliminating a bunch of weirdness) while still honouring the idea of NaN
> being a set of values rather than a single value.

Sounds good.

But IEEE 754 was created by pretty clever guys and sure they had a
reason for define things in the way they are. Probably we are missing
something.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS6tZfZlgi5GaxT1NAQJZtQP/bxi6l5TaiiOzv+no6cLaVXbkWXb9v6OL
jmejzrlAosXzzd/4CuinN2mdFs7bd9Y3O9gHoQ2nUfbfWQc4SwxpxjK67j10PODJ
MMz7wXgz075A8S7gUlpwWznByU2VfAys6ZVxZCv/uogW9SXIHqEBC/sXwWN5Hwvn
uHImzIL4bfs=
=RI3T
-----END PGP SIGNATURE-----

From eric at trueblade.com  Thu Mar 25 13:57:00 2010
From: eric at trueblade.com (Eric Smith)
Date: Thu, 25 Mar 2010 08:57:00 -0400
Subject: [Python-Dev] Issue 6081: format string using mapping rather
 than kwargs
In-Reply-To: <4BAB4253.8090605@gmail.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>
	<4BAB4253.8090605@gmail.com>
Message-ID: <4BAB5D9C.1010001@trueblade.com>

Nick Coghlan wrote:
> Filip Gruszczy?ski wrote:
>> I would appreciate any advice on this topic, even if this ticket would
>> be dismissed altogether, as I would like to learn as much as possible
>> on practices on developing Python.
> 
> Raymond's use case is valid, but the currently proposed method name is
> way too long to be usable and the use of *args is a definite misfeature.
> 
> I suggest approaching the issue with the exact spec needed to cover the
> problem Raymond described:
> 
>    Add a method to strings that allows a format string to
>    be applied to an existing mapping without implicitly
>    converting the mapping to a dict first.

It occurs to me that Raymond's use case could be satisfied using 
existing Python, by slightly changing the format string. After all, 
str.format() supports mapping lookup already:

$ ./python.exe
Python 2.6.5+ (release26-maint:79421, Mar 25 2010, 08:51:39)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> class Default(dict):
...  def __missing__(self, key):
...   return key
...
 >>> s = '{m[name]} was born in {m[country]}'
 >>> s.format(m=Default(name='Guido'))
'Guido was born in country'
 >>>

Considering that, maybe the best thing is to do nothing. I'll update the 
issue with this note.

-- 
Eric.

From dickinsm at gmail.com  Thu Mar 25 14:17:44 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 13:17:44 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB58CC.9070004@jcea.es>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
	<hof1br$8s5$1@dough.gmane.org> <4BAB58CC.9070004@jcea.es>
Message-ID: <5c6f2a5d1003250617n57008aa8p9fa0ccccaa21eaa0@mail.gmail.com>

On Thu, Mar 25, 2010 at 12:36 PM, Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 03/25/2010 07:54 AM, Georg Brandl wrote:
>>>>>>> float('nan') in [float('nan')]
>>>> False
>>>
>>> Sure, but just think of it as having two different nans there. ?(You
>>> could imagine thinking of the id of the nan as part of the payload.)
>>
>> That's interesting. ?Thinking of each value created by float('nan') as
>> a different nan makes sense to my naive mind, and it also explains
>> nicely the behavior present right now. ?Each nan comes from a different
>> operation and therefore is a "different" non-number.
>
> Infinites are "not equal" for a good reason, for example.

Well, that depends on your mathematical model.  The underlying
mathematical model for IEEE 754 floating-point is the doubly extended
real line:  that is, the set of all real numbers augmented with two
extra elements "infinity" and "-infinity", with the obvious total
order.  This is made explicit in section 3.2 of the standard:

"The mathematical structure underpinning the arithmetic in this
standard is the extended reals, that is, the set
of real numbers together with positive and negative infinity."

This is the same model that one typically uses in a first course in
calculus when studying limits of functions; it's an appropriate model
for dealing with computer approximations to real numbers and
continuous functions.  So the model has precisely two infinities, and
1/0, 2/0 and (1/0)**(1/0) all give the same infinity.   The decision
to make 1/0 "infinity" rather than "-infinity" is admittedly a little
arbitrary.  For floating-point (but not for calculus!), it makes sense
in the light of the decision to have both positive and negative
floating-point zeros;  1/(-0) is -infinity, of course.

Other models of "reals + one or more infinities" are possible, of
course, but they're not relevant to IEEE 754 floating point.  There's
a case for using a floating-point model with a single infinity,
especially for those who care more about algebraic functions
(polynomials, rational functions) than transcendental ones;  however,
IEEE 754 doesn't make provisions for this.

Mark

From dickinsm at gmail.com  Thu Mar 25 14:25:02 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 13:25:02 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB597D.1060508@jcea.es>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
Message-ID: <5c6f2a5d1003250625o40f31568sa795e332866c1cae@mail.gmail.com>

On Thu, Mar 25, 2010 at 12:39 PM, Jesus Cea <jcea at jcea.es> wrote:
>
> But IEEE 754 was created by pretty clever guys and sure they had a
> reason for define things in the way they are. Probably we are missing
> something.

Well, if we are, then nobody seems to know what!  See the Bertrand
Meyer blog post that was linked to up-thread.

Mark

From ncoghlan at gmail.com  Thu Mar 25 15:08:48 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Mar 2010 00:08:48 +1000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB597D.1060508@jcea.es>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<4BAB4764.4000500@gmail.com>
	<4BAB597D.1060508@jcea.es>
Message-ID: <4BAB6E70.3050008@gmail.com>

Jesus Cea wrote:
> But IEEE 754 was created by pretty clever guys and sure they had a
> reason for define things in the way they are. Probably we are missing
> something.

Yes, this is where their "implementable in a hardware circuit" focus
comes in. They were primarily thinking of a floating point
representation where the 32/64 bits are *it* - you can't have "multiple
NaNs" because you don't have the bits available to describe them.

We don't have that limitation - by bring id() into play for NaN equality
tests we have a lot more bits to play with (effectively adding an extra
32 or 64 bits to the floating point value just to say "which NaN is this
one?"). Doubling the size of your storage just to have multiple kinds of
NaN would be insane, but in our case the extra storage is already in use
- we would just be applying an additional interpretation to it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Mar 25 15:16:30 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Mar 2010 00:16:30 +1000
Subject: [Python-Dev] Issue 6081: format string using mapping rather
 than kwargs
In-Reply-To: <4BAB5D9C.1010001@trueblade.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>
	<4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com>
Message-ID: <4BAB703E.10701@gmail.com>

Eric Smith wrote:
> Considering that, maybe the best thing is to do nothing. I'll update the
> issue with this note.

I agree this slightly weakens the case for change, but it's not really
the same thing. Adding a "format_mapping" method allows an arbitrary
mapping to be used with any keyword-based format string. Crafting a
special format string in order to use an arbitrary mapping means that
the same format string can no longer be used with an ordinary keyword
based call - you have to do "format(m=dict(name=y, country=z))" instead.

Moving the decision of "how am I going to be called" to the time of
writing the format string is a bit odd.

On the other hand, the specially crafted format string does have the
virtue of travelling far more easily through any APIs that wrap the
basic format() method (since it doesn't need special treatment to make
its way through the wrapping code).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From steve at pearwood.info  Thu Mar 25 15:19:09 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 26 Mar 2010 01:19:09 +1100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB58CC.9070004@jcea.es>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<hof1br$8s5$1@dough.gmane.org> <4BAB58CC.9070004@jcea.es>
Message-ID: <201003260119.10967.steve@pearwood.info>

On Thu, 25 Mar 2010 11:36:28 pm Jesus Cea wrote:

> Infinites are "not equal" for a good reason, for example.
>
> 1/0 and 2/0 are both infinites, but one is "greater" than the other.
> Or (1/0)^(1/0), an infinite infinitelly "bigger".


I think you're mistaken. In Python 3.1:

>>> x = float('inf')
>>> y = float('inf') + 1
>>> x == y
True
>>> x is y
False

In cardinal arithmetic, there are an infinity of different sized 
infinities, but in ordinal arithmetic there are only two: +inf 
and -inf, corresponding to the infinities on the real number line. (I 
hope that I'm not over-simplifying -- it's been more than a decade 
since I've needed to care about this.) 

But in any case, the IEEE standard doesn't deal with cardinals: it only 
uses two signed infinities.



-- 
Steven D'Aprano

From curt at hagenlocher.org  Thu Mar 25 15:19:24 2010
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Thu, 25 Mar 2010 07:19:24 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB6E70.3050008@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
Message-ID: <d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>

On Thu, Mar 25, 2010 at 7:08 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Jesus Cea wrote:
> > But IEEE 754 was created by pretty clever guys and sure they had a
> > reason for define things in the way they are. Probably we are missing
> > something.
>
> Yes, this is where their "implementable in a hardware circuit" focus
> comes in. They were primarily thinking of a floating point
> representation where the 32/64 bits are *it* - you can't have "multiple
> NaNs" because you don't have the bits available to describe them.
>
Wait, what? I haven't been paying much attention, but this is backwards.
There are multiple representations of NaN in the IEEE encoding; that's
actually part of the problem with saying that NaN = NaN or NaN != NaN. If
you want to ignore the "payload" in the NaN, then you're not just comparing
bits any more.

--
Curt Hagenlocher
curt at hagenlocher.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100325/12b03b63/attachment.html>

From dickinsm at gmail.com  Thu Mar 25 15:22:33 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 14:22:33 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB6E70.3050008@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
Message-ID: <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com>

On Thu, Mar 25, 2010 at 2:08 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Jesus Cea wrote:
>> But IEEE 754 was created by pretty clever guys and sure they had a
>> reason for define things in the way they are. Probably we are missing
>> something.
>
> Yes, this is where their "implementable in a hardware circuit" focus
> comes in. They were primarily thinking of a floating point
> representation where the 32/64 bits are *it* - you can't have "multiple
> NaNs" because you don't have the bits available to describe them.

I'm not so sure about this:  standard 64-bit binary IEEE 754 doubles
allow for 2**53-2 different nans (2**52-2 signaling nans, 2**52 quiet
nans):  anything with bit pattern (msb to lsb)

x1111111 1111xxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx

is an infinity or a nan, and there are only 2 infinities.

Mark

From solipsis at pitrou.net  Thu Mar 25 15:26:05 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 Mar 2010 14:26:05 +0000 (UTC)
Subject: [Python-Dev] Why is nan != nan?
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>
Message-ID: <hofrpt$620$1@dough.gmane.org>

Le Thu, 25 Mar 2010 07:19:24 -0700, Curt Hagenlocher a ?crit?:
> On Thu, Mar 25, 2010 at 7:08 AM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> 
>> Jesus Cea wrote:
>> > But IEEE 754 was created by pretty clever guys and sure they had a
>> > reason for define things in the way they are. Probably we are missing
>> > something.
>>
>> Yes, this is where their "implementable in a hardware circuit" focus
>> comes in. They were primarily thinking of a floating point
>> representation where the 32/64 bits are *it* - you can't have "multiple
>> NaNs" because you don't have the bits available to describe them.
>>
> Wait, what? I haven't been paying much attention, but this is backwards.
> There are multiple representations of NaN in the IEEE encoding; that's
> actually part of the problem with saying that NaN = NaN or NaN != NaN.
> If you want to ignore the "payload" in the NaN, then you're not just
> comparing bits any more.

This sounds a bit sophistic, if the (Python) user doesn't have access to 
the payload anyway.


Antoine.



From dickinsm at gmail.com  Thu Mar 25 15:42:14 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 14:42:14 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <hofrpt$620$1@dough.gmane.org>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>
	<hofrpt$620$1@dough.gmane.org>
Message-ID: <5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com>

On Thu, Mar 25, 2010 at 2:26 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Thu, 25 Mar 2010 07:19:24 -0700, Curt Hagenlocher a ?crit?:
>> Wait, what? I haven't been paying much attention, but this is backwards.
>> There are multiple representations of NaN in the IEEE encoding; that's
>> actually part of the problem with saying that NaN = NaN or NaN != NaN.
>> If you want to ignore the "payload" in the NaN, then you're not just
>> comparing bits any more.
>
> This sounds a bit sophistic, if the (Python) user doesn't have access to
> the payload anyway.

Well, you can get at the payload using the struct module, if you care
enough.  But yes, it's true that Python doesn't take much care with
the payload:  e.g., ideally, an operation on a nan (3.0 + nan,
sqrt(nan), ...) should return exactly the same nan, to make sure that
information in the payload is preserved.  Python doesn't bother, for
floats (though it does for decimal).

Mark

From pierrebai at hotmail.com  Thu Mar 25 15:53:59 2010
From: pierrebai at hotmail.com (Pierre B.)
Date: Thu, 25 Mar 2010 14:53:59 +0000 (UTC)
Subject: [Python-Dev] Why is nan != nan?
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<5c6f2a5d1003250625o40f31568sa795e332866c1cae@mail.gmail.com>
Message-ID: <loom.20100325T154040-984@post.gmane.org>

Mark Dickinson <dickinsm <at> gmail.com> writes:

> 
> On Thu, Mar 25, 2010 at 12:39 PM, Jesus Cea <jcea <at> jcea.es> wrote:
> >
> > But IEEE 754 was created by pretty clever guys and sure they had a
> > reason for define things in the way they are. Probably we are missing
> > something.
> 
> Well, if we are, then nobody seems to know what!  See the Bertrand
> Meyer blog post that was linked to up-thread.

The missing part, IMO, is that allowing a given NaN value to compare equal to
itself only pushes the problem up one level. Any single operation yielding a
NaN will still be unequal to itself. That is, under what is being proposed,
with a function func() returning the same result of some calculation:

x = func()
s1 = (x)
print x in s1
print func() in s1

This would print True and False, even though func() is perfoming the same
calculation and thus logically returning the same NaN.

I think the IEEE NaN represent the fact that you have a number of an undefined
set, but it doesn't specify which.

The only way out, IMO, is to make *all* NaN comparisons yield False, but
identity yield true. No interning necessary. At most, you could make the
identity function return False for the different types of NaN.



From dickinsm at gmail.com  Thu Mar 25 15:54:55 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 14:54:55 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>
	<hofrpt$620$1@dough.gmane.org>
	<5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com>
Message-ID: <5c6f2a5d1003250754t7d7ab207i346825c79951195a@mail.gmail.com>

On Thu, Mar 25, 2010 at 2:42 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Thu, Mar 25, 2010 at 2:26 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> This sounds a bit sophistic, if the (Python) user doesn't have access to
>> the payload anyway.
>
> Well, you can get at the payload using the struct module, if you care
> enough. ?But yes, it's true that Python doesn't take much care with
> the payload: ?e.g., ideally, an operation on a nan (3.0 + nan,
> sqrt(nan), ...) should return exactly the same nan, to make sure that
> information in the payload is preserved. ?Python doesn't bother, for
> floats (though it does for decimal).

Hmm. I take it back.  I was being confused by the fact that sqrt(nan)
returns a nan with a new identity;  but it does apparently preserve
the payload.  An example:

>>> from struct import pack, unpack
>>> from math import sqrt
>>> x = unpack('<d', pack('<Q', (2047 << 52) + 12345))[0]
>>> y = sqrt(x)
>>> bin(unpack('<Q', pack('<d', x))[0])
'0b111111111110000000000000000000000000000000000000011000000111001'
>>> bin(unpack('<Q', pack('<d', y))[0])
'0b111111111111000000000000000000000000000000000000011000000111001'

Here you see that the payload has been preserved.  The bit patterns
aren't quite identical:  the incoming nan was actually a signaling
nan, which got silently (because neither Python nor C understands
signaling nans) 'silenced' by setting bit 51.  So the output is the
corresponding quiet nan, with the same sign and payload.

Mark

From eric at trueblade.com  Thu Mar 25 15:53:42 2010
From: eric at trueblade.com (Eric Smith)
Date: Thu, 25 Mar 2010 10:53:42 -0400
Subject: [Python-Dev] Issue 6081: format string using mapping rather
 than kwargs
In-Reply-To: <4BAB703E.10701@gmail.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>
	<4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com>
	<4BAB703E.10701@gmail.com>
Message-ID: <4BAB78F6.8080905@trueblade.com>

Nick Coghlan wrote:
> Moving the decision of "how am I going to be called" to the time of
> writing the format string is a bit odd.
> 
> On the other hand, the specially crafted format string does have the
> virtue of travelling far more easily through any APIs that wrap the
> basic format() method (since it doesn't need special treatment to make
> its way through the wrapping code).

Completely agree on all points. Now we're just left with "is it worth 
expanding the str api for this?". I don't feel strongly either way.

--
Eric.

From curt at hagenlocher.org  Thu Mar 25 16:01:54 2010
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Thu, 25 Mar 2010 08:01:54 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003250754t7d7ab207i346825c79951195a@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>
	<hofrpt$620$1@dough.gmane.org>
	<5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com>
	<5c6f2a5d1003250754t7d7ab207i346825c79951195a@mail.gmail.com>
Message-ID: <d2155e361003250801y7cc52763ofd04dd5d4ecf5c8a@mail.gmail.com>

On Thu, Mar 25, 2010 at 7:54 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>
> Hmm. I take it back. ?I was being confused by the fact that sqrt(nan)
> returns a nan with a new identity; ?but it does apparently preserve
> the payload. ?An example:

I played with this some a few months ago, and both the FPU and the C
libraries I tested will preserve the payload. I imagine Python just
inherits their behavior.

--
Curt Hagenlocher
curt at hagenlocher.org

From dickinsm at gmail.com  Thu Mar 25 16:05:32 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 15:05:32 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <d2155e361003250801y7cc52763ofd04dd5d4ecf5c8a@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>
	<hofrpt$620$1@dough.gmane.org>
	<5c6f2a5d1003250742g1f945514t8017a897d6f6223f@mail.gmail.com>
	<5c6f2a5d1003250754t7d7ab207i346825c79951195a@mail.gmail.com>
	<d2155e361003250801y7cc52763ofd04dd5d4ecf5c8a@mail.gmail.com>
Message-ID: <5c6f2a5d1003250805q455b2445h41479708e69d6463@mail.gmail.com>

On Thu, Mar 25, 2010 at 3:01 PM, Curt Hagenlocher <curt at hagenlocher.org> wrote:
> On Thu, Mar 25, 2010 at 7:54 AM, Mark Dickinson <dickinsm at gmail.com> wrote:
>>
>> Hmm. I take it back. ?I was being confused by the fact that sqrt(nan)
>> returns a nan with a new identity; ?but it does apparently preserve
>> the payload. ?An example:
>
> I played with this some a few months ago, and both the FPU and the C
> libraries I tested will preserve the payload. I imagine Python just
> inherits their behavior.

Pretty much, yes.  I think we've also taken care to preserve payloads
in functions that have been added to the math library as well (e.g.,
the gamma function).  Not that that's particularly hard:  it's just a
matter of making sure to do "if (isnan(x)) return x;" rather than "if
(isnan(x)) return standard_python_nan;".  If that's not true, then
there's a minor bug to be corrected.

Mark

From ncoghlan at gmail.com  Thu Mar 25 16:05:36 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Mar 2010 01:05:36 +1000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>	
	<4BAB6E70.3050008@gmail.com>
	<5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com>
Message-ID: <4BAB7BC0.8020306@gmail.com>

Mark Dickinson wrote:
> On Thu, Mar 25, 2010 at 2:08 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Jesus Cea wrote:
>>> But IEEE 754 was created by pretty clever guys and sure they had a
>>> reason for define things in the way they are. Probably we are missing
>>> something.
>> Yes, this is where their "implementable in a hardware circuit" focus
>> comes in. They were primarily thinking of a floating point
>> representation where the 32/64 bits are *it* - you can't have "multiple
>> NaNs" because you don't have the bits available to describe them.
> 
> I'm not so sure about this:  standard 64-bit binary IEEE 754 doubles
> allow for 2**53-2 different nans (2**52-2 signaling nans, 2**52 quiet
> nans):  anything with bit pattern (msb to lsb)
> 
> x1111111 1111xxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx
> 
> is an infinity or a nan, and there are only 2 infinities.

I stand corrected :)

It still seems to me that the problems mostly arise when we're trying to
get floats and Decimals to behave like Python *objects* (i.e. with
reflexive equality) rather than like IEEE defined numbers.

It's an extra element that isn't part of the problem the numeric
standards are trying to solve.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From cournape at gmail.com  Thu Mar 25 16:07:10 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 26 Mar 2010 00:07:10 +0900
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB597D.1060508@jcea.es>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
Message-ID: <5b8d13221003250807h21dcd79fj37f65cc81af8e3b7@mail.gmail.com>

On Thu, Mar 25, 2010 at 9:39 PM, Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 03/25/2010 12:22 PM, Nick Coghlan wrote:
>> ? "Not a Number" is not a single floating point value. Instead each
>> ? instance is a distinct value representing the precise conditions that
>> ? created it. Thus, two "NaN" values x and y will compare equal iff they
>> ? are the exact same NaN object (i.e. "if isnan(x) then x == y iff
>> ? x is y".
>>
>> As stated above, such a change would allow us to restore reflexivity
>> (eliminating a bunch of weirdness) while still honouring the idea of NaN
>> being a set of values rather than a single value.
>
> Sounds good.
>
> But IEEE 754 was created by pretty clever guys and sure they had a
> reason for define things in the way they are. Probably we are missing
> something.

Yes, indeed. I don't claim having a deep understanding myself, but up
to now, everytime I thought something in IEE 754 was weird, it ended
up being for good reasons.

I think the fundamental missing point in this discussion about Nan is
exception handling: a lot of NaN quircky behavior becomes much more
natural once you take into account which operations are invalid under
which condition. Unless I am mistaken, python itself does not support
for FPU exception handling.

For example, the reason why x != x for x Nan is because != (and ==)
are about the only operations where you can have NaN as operands
without risking raising an exception, and support for creating and
detecting NaN in languages have been coming only quite lately (e.g.
C99).

Concerning the lack of rationale: a relatively short reference
concerned about FPU exception and NaN handling is from Kahan himself

http://www.eecs.berkeley.edu/~wkahan/ieee754status/ieee754.ps

David

From ncoghlan at gmail.com  Thu Mar 25 16:09:00 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 Mar 2010 01:09:00 +1000
Subject: [Python-Dev] Issue 6081: format string using mapping rather
 than kwargs
In-Reply-To: <4BAB78F6.8080905@trueblade.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>
	<4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com>
	<4BAB703E.10701@gmail.com> <4BAB78F6.8080905@trueblade.com>
Message-ID: <4BAB7C8C.6020004@gmail.com>

Eric Smith wrote:
> Nick Coghlan wrote:
>> Moving the decision of "how am I going to be called" to the time of
>> writing the format string is a bit odd.
>>
>> On the other hand, the specially crafted format string does have the
>> virtue of travelling far more easily through any APIs that wrap the
>> basic format() method (since it doesn't need special treatment to make
>> its way through the wrapping code).
> 
> Completely agree on all points. Now we're just left with "is it worth
> expanding the str api for this?". I don't feel strongly either way.

For something as core as the string API, we better feel darn strongly
about it before we mess with it :)

I'm inclined to leave it alone unless/until Raymond or somebody else
steps up to really champion it.

Cheers,
Nick.

P.S. There's also the language moratorium to consider - since this idea
affect the methods of a builtin type, I believe the moratorium applies.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From dickinsm at gmail.com  Thu Mar 25 16:13:04 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 25 Mar 2010 15:13:04 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB7BC0.8020306@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com>
	<4BAB7BC0.8020306@gmail.com>
Message-ID: <5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com>

On Thu, Mar 25, 2010 at 3:05 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Mark Dickinson wrote:
>> On Thu, Mar 25, 2010 at 2:08 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Jesus Cea wrote:
>>>> But IEEE 754 was created by pretty clever guys and sure they had a
>>>> reason for define things in the way they are. Probably we are missing
>>>> something.
>>> Yes, this is where their "implementable in a hardware circuit" focus
>>> comes in. They were primarily thinking of a floating point
>>> representation where the 32/64 bits are *it* - you can't have "multiple
>>> NaNs" because you don't have the bits available to describe them.
>>
>> I'm not so sure about this: ?standard 64-bit binary IEEE 754 doubles
>> allow for 2**53-2 different nans (2**52-2 signaling nans, 2**52 quiet
>> nans): ?anything with bit pattern (msb to lsb)
>>
>> x1111111 1111xxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx
>>
>> is an infinity or a nan, and there are only 2 infinities.
>
> I stand corrected :)
>
> It still seems to me that the problems mostly arise when we're trying to
> get floats and Decimals to behave like Python *objects* (i.e. with
> reflexive equality) rather than like IEEE defined numbers.
>
> It's an extra element that isn't part of the problem the numeric
> standards are trying to solve.

Agreed.  We don't have to be "missing something";  rather, the IEEE
folks (quite understandably) almost certainly didn't anticipate this
kind of usage.  So I'll concede that it's reasonable to consider
deviating from the standard in the light of this.

Mark

From solipsis at pitrou.net  Thu Mar 25 16:24:38 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 Mar 2010 15:24:38 +0000 (UTC)
Subject: [Python-Dev] Issue 6081: format string using mapping rather
	than kwargs
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>
	<4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com>
	<4BAB703E.10701@gmail.com> <4BAB78F6.8080905@trueblade.com>
	<4BAB7C8C.6020004@gmail.com>
Message-ID: <loom.20100325T162313-699@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> 
> For something as core as the string API, we better feel darn strongly
> about it before we mess with it :)

Having two very similar methods with different names and subtly different APIs
sounds bad IMHO.
Also, the use case doesn't look very strong to me.

> P.S. There's also the language moratorium to consider - since this idea
> affect the methods of a builtin type, I believe the moratorium applies.

So do I.

Regards

Antoine.



From eric at trueblade.com  Thu Mar 25 16:55:54 2010
From: eric at trueblade.com (Eric Smith)
Date: Thu, 25 Mar 2010 11:55:54 -0400
Subject: [Python-Dev] Issue 6081: format string using mapping rather
 than kwargs
In-Reply-To: <4BAB7C8C.6020004@gmail.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com>	<4BAB4253.8090605@gmail.com>
	<4BAB5D9C.1010001@trueblade.com>	<4BAB703E.10701@gmail.com>
	<4BAB78F6.8080905@trueblade.com> <4BAB7C8C.6020004@gmail.com>
Message-ID: <4BAB878A.3080106@trueblade.com>

Nick Coghlan wrote:
>> Completely agree on all points. Now we're just left with "is it worth
>> expanding the str api for this?". I don't feel strongly either way.
> 
> For something as core as the string API, we better feel darn strongly
> about it before we mess with it :)
> 
> I'm inclined to leave it alone unless/until Raymond or somebody else
> steps up to really champion it.

I'm okay with that.

> P.S. There's also the language moratorium to consider - since this idea
> affect the methods of a builtin type, I believe the moratorium applies.

I don't think there's any effort to get this in before the moratorium 
expires.

Eric.


From raymond.hettinger at gmail.com  Thu Mar 25 17:03:04 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Thu, 25 Mar 2010 09:03:04 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAB4764.4000500@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com>
Message-ID: <D4B7353B-4B54-4258-84ED-EB0611389340@gmail.com>


On Mar 25, 2010, at 4:22 AM, Nick Coghlan wrote:

> Mark Dickinson wrote:
>> Here's an interesting recent blog post on this subject, from the
>> creator of Eiffel:
>> 
>> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-civilization/
> 
> Interesting. So the natural tweak that would arise from that perspective
> is for us to restore reflexivity by declaring that any given NaN is
> equal to *itself* but not to any other NaN (even one with the same payload).
> 
> With NaN (in general) not being interned, that would actually fit the
> idea of a NaN implicitly carrying the operation that created the NaN as
> part of its definition of equivalence.
> 
> So, I'm specifically putting that proposal on the table for both float
> and Decimal NaNs in Python:
> 
>  "Not a Number" is not a single floating point value. Instead each
>  instance is a distinct value representing the precise conditions that
>  created it. Thus, two "NaN" values x and y will compare equal iff they
>  are the exact same NaN object (i.e. "if isnan(x) then x == y iff
>  x is y".
> 
> As stated above, such a change would allow us to restore reflexivity
> (eliminating a bunch of weirdness) while still honouring the idea of NaN
> being a set of values rather than a single value.
> 

+1


Raymond

From larry at hastings.org  Thu Mar 25 18:38:54 2010
From: larry at hastings.org (Larry Hastings)
Date: Thu, 25 Mar 2010 10:38:54 -0700
Subject: [Python-Dev] [Python-checkins] r79397 - in
 python/trunk:	Doc/c-api/capsule.rst
 Doc/c-api/cobject.rst	Doc/c-api/concrete.rst	Doc/data/refcounts.dat	Doc/extending/extending.rst
 Include/Python.h Include/cStringIO.h	Include/cobject.h	Include/datetime.h
 Include/py_curses.h Include/pycapsule.h	Include/pyexpat.h	Include/ucnhash.h
 Lib/test/test_sys.py	Makefile.pre.in Misc/NEWS	Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c	Modules/_ctypes/ctypes.h	Modules/_cursesmodule.c
 Modules/_elementtree.c	Modules/_testcapimodule.c	Modules/cStringIO.c	Modules/cjkcodecs/cjkcodecs.h
 Modules/cjkcodecs/multibytecodec.c	Modules/cjkcodecs/multibytecodec.h	Modules/datetimemodule.c
 Modules/pyexpat.c	Modules/socketmodule.c	Modules/socketmodule.h
 Modules/unicodedata.c Objects/capsule.c
 Objects/object.c	Objects/unicodeobject.c	PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj	PC/os2emx/python27.def	PC/os2vacpp/python.def
 Python/compile.c Python/getargs.c
In-Reply-To: <4BAB2882.6070508@egenix.com>
References: <20100325005454.B1AC5FCCD@mail.python.org>
	<4BAB2882.6070508@egenix.com>
Message-ID: <4BAB9FAE.1070206@hastings.org>


M.-A. Lemburg wrote:
> Backporting PyCapsule is fine, but the changes you made to all
> those PyCObject uses does not look backwards compatible.
>
> The C APIs exposed by the modules (e.g. the datetime module)
> are used in lots of 3rd party extension modules and changing
> them from PyCObject to PyCapsule is a major change in the
> module API.

You're right, my changes aren't backwards compatible.  I thought it was 
reasonable for four reasons:

1. The CObject API isn't safe.  It's easy to crash Python 2.6 in just a 
few lines by mixing and matching CObjects.  Switching Python to capsules 
prevents a class of exploits.  I've included a script at the bottom of 
this message that demonstrates three such crashes.  The script runs in 
Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

2. As I just mentioned, Python 3.1 already uses capsules everywhere 
instead of CObjects.  Since part of the purpose of Python 2.7 is to 
prepare developers for the to upgrade to 3.1, getting them to switch to 
capsules now is just one more way they are prepared.

3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we 
ever made a 2.8 I want to remove it completely.

4. When Python publishes an API using a CObject, it describes the thing 
the CObject points to in a header file.  In nearly all cases that header 
file also provides a macro or inline function that does the importing 
work for you.  I changed those to use capsules too.  So if the 
third-party code uses the macro or inline function, all you need do is 
recompile it against 2.7 and it works fine.  Sadly I know of one 
exception: pyexpat.expat_CAPI.  The header file just describes the 
struct pointed to by the CObject, but callers


I can suggest four ways to ameliorate the problem.

First, we could do as Antoine Pitrou suggests on the bug (issue 7992): 
wherever the CObject used to be published as a module attribute to 
expose an API, we could provide both a CObject and a capsule; internally 
Python would only use the capsules.  This would allow third-party 
libraries to run against 2.7 unchanged.  The major problem with this is 
that third-party libraries would still be vulnerable to the 
mix-and-match CObject crash.  A secondary, minor concern: obviously we'd 
store the CObject attribute with the existing name, and the capsule 
attribute would have to get some new name.  But in Python 3.1, these 
attributes already expose a capsule.  Therefore, people who convert to 
using the capsules now would have to convert again when moving to 3.1.

Second, we could make CObject internally support unpacking capsules.  If 
you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and 
return the pointer within.  (We could probably also map the capsule 
"context" to the CObject "desc", if any of the Python use cases needed 
it.)  I wouldn't change anything else about CObjects; creating and using 
them would continue to work as normal.  This would also allow 
third-party libraries to run against Python 2.7 unchanged.  The only 
problem is that it's unsafe, as indeed allowing any use of 
PyCObject_AsVoidPtr() is unsafe.

Third, I've been pondering writing a set of preprocessor macros, shipped 
in their own header file distributed independently of Python and 
released to the public domain, that would make it easy to use either 
CObjects or capsules depending on what version of Python you were 
compiling against.  Obviously, using these macros would require a source 
code change in the third-party library.  But these macros would make it 
a five-minute change.  This could compliment the first or second approaches.

Fourth, we could back out of the changes to published APIs and convert 
them back to CObjects.  -1.


Your thoughts?


/larry/

-----

import sys
def log(message):
    print(message)
    sys.stdout.flush()

def crash1():
    log("Running crash1...")
    try:
        import datetime
        import cStringIO
        cStringIO.cStringIO_CAPI = datetime.datetime_CAPI

        import cPickle
        s = cPickle.dumps([1, 2, 3])
    except ImportError:
        # This test isn't translatable to Python 3.
        pass
    log("Survived crash1!")


def crash2():
    log("Running crash2...")
    try:
        import unicodedata
        import _socket
        _socket.CAPI = unicodedata.ucnhash_CAPI
        import ssl
    except AttributeError:
        # Congratulations, you didn't crash.
        pass
    log("Survived crash2!")


def crash3():
    log("Running crash3...")
    try:
        import unicodedata
        import _multibytecodec
        _multibytecodec.__create_codec(unicodedata.ucnhash_CAPI)

    except ValueError:
        # Congratulations, you didn't crash.
        pass
    log("Survived crash3!")

import sys

if len(sys.argv) > 1:
    if sys.argv[1] == '1':
        crash1()
        sys.exit(0)
    elif sys.argv[1] == '2':
        crash2()
        sys.exit(0)
    elif sys.argv[1] == '3':
        crash3()
        sys.exit(0)

crash1()
crash2()
crash3()

From jcea at jcea.es  Thu Mar 25 18:39:09 2010
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 25 Mar 2010 18:39:09 +0100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003260119.10967.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<hof1br$8s5$1@dough.gmane.org>
	<4BAB58CC.9070004@jcea.es> <201003260119.10967.steve@pearwood.info>
Message-ID: <4BAB9FBD.8010508@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/25/2010 03:19 PM, Steven D'Aprano wrote:
> On Thu, 25 Mar 2010 11:36:28 pm Jesus Cea wrote:
> 
>> Infinites are "not equal" for a good reason, for example.
>>
>> 1/0 and 2/0 are both infinites, but one is "greater" than the other.
>> Or (1/0)^(1/0), an infinite infinitelly "bigger".
> 
> 
> I think you're mistaken. In Python 3.1:

I was refering to mathematical infinites and why inf!=inf is sensible
and a natural consequence calculus and limits (mathematically).

In any case, I am an engineer, not a mathematical or a language designer
:). IANAL.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS6ufvJlgi5GaxT1NAQKrpQP+PbbNP6tNtPi480/ovki89DEDNqvKH4RU
A/R6jLXwm7byF0RQ+3B9gUh8SfANQaTOeCOYufUJHXPDM3BHsmt2S1kcPoW5sUYe
a38nUuD0sTyCV23h2QtzZpNGG7qNa6iHTMEc6vYgY3CdfCw+301xdqH0ZkmXV1B9
OIrj7ec7m+s=
=rTUf
-----END PGP SIGNATURE-----

From jcea at jcea.es  Thu Mar 25 18:47:54 2010
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 25 Mar 2010 18:47:54 +0100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5b8d13221003250807h21dcd79fj37f65cc81af8e3b7@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<4BAB4764.4000500@gmail.com>
	<4BAB597D.1060508@jcea.es>
	<5b8d13221003250807h21dcd79fj37f65cc81af8e3b7@mail.gmail.com>
Message-ID: <4BABA1CA.5050706@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/25/2010 04:07 PM, David Cournapeau wrote:
> Yes, indeed. I don't claim having a deep understanding myself, but up
> to now, everytime I thought something in IEE 754 was weird, it ended
> up being for good reasons.

I was wondering if we could bring the question to news:comp.arch
newsgroup. They have the knowledge, and I know there are people from the
IEEE 754 group lurking there. I only have read-only access, nevertheless.

Another relevant group could be news:comp.arch.arithmetic, but I am not
familiar with it.


- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS6uhyZlgi5GaxT1NAQIZJwP/bAx5vXBLMLI8f724Hf0OtfATpV4SFQ84
RaLHNBPPkE5+cdFxWIv6VmFThtYfKnjutmLNU1TJYFoDwgOvqigYO8hOTFnWlfML
Sx5B3LFdtGSZAfSsd+rMF23wKpbpAy/TicE+B6zg+Qy1LFv1V+OVn/Y3xBPGxVW5
m4yAKWT5T4U=
=mwXx
-----END PGP SIGNATURE-----

From mal at egenix.com  Thu Mar 25 19:05:52 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 25 Mar 2010 19:05:52 +0100
Subject: [Python-Dev] [Python-checkins] r79397 -
 in	python/trunk:	Doc/c-api/capsule.rst
 Doc/c-api/cobject.rst	Doc/c-api/concrete.rst	Doc/data/refcounts.dat	Doc/extending/extending.rst
 Include/Python.h	Include/cStringIO.h	Include/cobject.h	Include/datetime.h
 Include/py_curses.h	Include/pycapsule.h	Include/pyexpat.h	Include/ucnhash.h
 Lib/test/test_sys.py	Makefile.pre.in	Misc/NEWS	Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c	Modules/_ctypes/ctypes.h	Modules/_cursesmodule.c
 Modules/_elementtree.c	Modules/_testcapimodule.c	Modules/cStringIO.c	Modules/cjkcodecs/cjkcodecs.h
 Modules/cjkcodecs/multibytecodec.c	Modules/cjkcodecs/multibytecodec.h	Modules/datetimemodule.c
 Modules/pyexpat.c	Modules/socketmodule.c	Modules/socketmodule.h	Modules/unicodedata.c
 Objects/capsule.c	Objects/object.c	Objects/unicodeobject.c	PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj	PC/os2emx/python27.def	PC/os2vacpp/python.def
 Python/compile.c Python/getargs.c
In-Reply-To: <4BAB9FAE.1070206@hastings.org>
References: <20100325005454.B1AC5FCCD@mail.python.org>	<4BAB2882.6070508@egenix.com>
	<4BAB9FAE.1070206@hastings.org>
Message-ID: <4BABA600.8090103@egenix.com>

Larry Hastings wrote:
> 
> M.-A. Lemburg wrote:
>> Backporting PyCapsule is fine, but the changes you made to all
>> those PyCObject uses does not look backwards compatible.
>>
>> The C APIs exposed by the modules (e.g. the datetime module)
>> are used in lots of 3rd party extension modules and changing
>> them from PyCObject to PyCapsule is a major change in the
>> module API.
> 
> You're right, my changes aren't backwards compatible.  I thought it was
> reasonable for four reasons:
> 
> 1. The CObject API isn't safe.  It's easy to crash Python 2.6 in just a
> few lines by mixing and matching CObjects.  Switching Python to capsules
> prevents a class of exploits.  I've included a script at the bottom of
> this message that demonstrates three such crashes.  The script runs in
> Python 2 and 3, but 3.1 doesn't crash because it's using capsules.
> 
> 2. As I just mentioned, Python 3.1 already uses capsules everywhere
> instead of CObjects.  Since part of the purpose of Python 2.7 is to
> prepare developers for the to upgrade to 3.1, getting them to switch to
> capsules now is just one more way they are prepared.
> 
> 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we
> ever made a 2.8 I want to remove it completely.
> 
> 4. When Python publishes an API using a CObject, it describes the thing
> the CObject points to in a header file.  In nearly all cases that header
> file also provides a macro or inline function that does the importing
> work for you.  I changed those to use capsules too.  So if the
> third-party code uses the macro or inline function, all you need do is
> recompile it against 2.7 and it works fine.  Sadly I know of one
> exception: pyexpat.expat_CAPI.  The header file just describes the
> struct pointed to by the CObject, but callers
> 
> 
> I can suggest four ways to ameliorate the problem.
> 
> First, we could do as Antoine Pitrou suggests on the bug (issue 7992):
> wherever the CObject used to be published as a module attribute to
> expose an API, we could provide both a CObject and a capsule; internally
> Python would only use the capsules.  This would allow third-party
> libraries to run against 2.7 unchanged.  The major problem with this is
> that third-party libraries would still be vulnerable to the
> mix-and-match CObject crash.  A secondary, minor concern: obviously we'd
> store the CObject attribute with the existing name, and the capsule
> attribute would have to get some new name.  But in Python 3.1, these
> attributes already expose a capsule.  Therefore, people who convert to
> using the capsules now would have to convert again when moving to 3.1.
> 
> Second, we could make CObject internally support unpacking capsules.  If
> you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and
> return the pointer within.  (We could probably also map the capsule
> "context" to the CObject "desc", if any of the Python use cases needed
> it.)  I wouldn't change anything else about CObjects; creating and using
> them would continue to work as normal.  This would also allow
> third-party libraries to run against Python 2.7 unchanged.  The only
> problem is that it's unsafe, as indeed allowing any use of
> PyCObject_AsVoidPtr() is unsafe.
> 
> Third, I've been pondering writing a set of preprocessor macros, shipped
> in their own header file distributed independently of Python and
> released to the public domain, that would make it easy to use either
> CObjects or capsules depending on what version of Python you were
> compiling against.  Obviously, using these macros would require a source
> code change in the third-party library.  But these macros would make it
> a five-minute change.  This could compliment the first or second
> approaches.
> 
> Fourth, we could back out of the changes to published APIs and convert
> them back to CObjects.  -1.
> 
> 
> Your thoughts?

More later, have to run. For now, I'm with Antoine on this one:

 * adding support for PyCapsules to 2.7 is fine.

 * exposing the various module C APIs as PyCapsules in addition to
   the existing PyCObject C APIS is fines as well and indeed a good
   idea, since it makes porting 2.7 applications to 3.1 easier.

 * adding more C APIs using PyCapsules in Python 2.7 is fine as well,
   e.g. for the decimal module.

 * removing the PyCObject C APIs is no option in Python 2.7.

Please remember that you're dealing with Python 2.7. Major backwards
incompatible changes are not allowed in that branch and you're pretty
much breaking all 3rd party modules that have just started using e.g
the datetime module C API.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 25 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From solipsis at pitrou.net  Thu Mar 25 19:15:28 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 Mar 2010 18:15:28 +0000 (UTC)
Subject: [Python-Dev] [Python-checkins] PyCapsule backport
References: <20100325005454.B1AC5FCCD@mail.python.org>
	<4BAB2882.6070508@egenix.com> <4BAB9FAE.1070206@hastings.org>
Message-ID: <loom.20100325T190700-671@post.gmane.org>


Hello Larry,

> You're right, my changes aren't backwards compatible.  I thought it was 
> reasonable for four reasons:
> 
> 1. The CObject API isn't safe.  It's easy to crash Python 2.6 in just a 
> few lines by mixing and matching CObjects.  Switching Python to capsules 
> prevents a class of exploits.  I've included a script at the bottom of 
> this message that demonstrates three such crashes.  The script runs in 
> Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

Ok, but this is IMHO not a good enough reason to break the API. No prior
announcement (prior to 2.7) has been made that PyCObject APIs would be replaced
by their PyCapsule equivalents. You must leave people time to adapt.

> 2. As I just mentioned, Python 3.1 already uses capsules everywhere 
> instead of CObjects.  Since part of the purpose of Python 2.7 is to 
> prepare developers for the to upgrade to 3.1, getting them to switch to 
> capsules now is just one more way they are prepared.

Not /forcing them/ to switch, though. If you agree to put a bit more work into
it, and make the PyCObject APIs coexist with the corresponding PyCapsule APIs in
2.7, then I think it would be fine.

The point where we force people to switch their APIs is the 2.x -> 3.x
transition. That's the whole point of breaking compatibility at one defined
point. If we force them to switch when migrating to 2.7 instead, it becomes
messy and quite unfriendly to them.

> 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we 
> ever made a 2.8 I want to remove it completely.

I don't think there is a point in removing it in 2.8. If 2.8 ever exists, its
purpose will be to serve users who don't /want/ to switch to 3.x yet, so
breaking compatibility for them kind of ruins the purpose.

> First, we could do as Antoine Pitrou suggests on the bug (issue 7992): 
> wherever the CObject used to be published as a module attribute to 
> expose an API, we could provide both a CObject and a capsule; internally 
> Python would only use the capsules. [...]
> 
> Second, we could make CObject internally support unpacking capsules.  If 
> you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and 
> return the pointer within. [...]
> 
> Third, I've been pondering writing a set of preprocessor macros, shipped 
> in their own header file distributed independently of Python and 
> released to the public domain, that would make it easy to use either 
> CObjects or capsules depending on what version of Python you were 
> compiling against.  Obviously, using these macros would require a source 
> code change in the third-party library.  But these macros would make it 
> a five-minute change.  This could compliment the first or second approaches.
> 
> Fourth, we could back out of the changes to published APIs and convert 
> them back to CObjects.  -1.

I think solution #1 would be the best one.
#2 looks too complicated.
#3 is not much friendlier than the current compatibility breakage.
#4 is the necessary fallback if you don't want to implement #1.

Regards

Antoine.



From larry at hastings.org  Thu Mar 25 19:16:27 2010
From: larry at hastings.org (Larry Hastings)
Date: Thu, 25 Mar 2010 11:16:27 -0700
Subject: [Python-Dev] [Python-checkins] r79397 -
 in	python/trunk:	Doc/c-api/capsule.rst
 Doc/c-api/cobject.rst	Doc/c-api/concrete.rst	Doc/data/refcounts.dat	Doc/extending/extending.rst
 Include/Python.h	Include/cStringIO.h	Include/cobject.h	Include/datetime.h
 Include/py_curses.h	Include/pycapsule.h	Include/pyexpat.h	Include/ucnhash.h
 Lib/test/test_sys.py	Makefile.pre.in	Misc/NEWS	Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c	Modules/_ctypes/ctypes.h	Modules/_cursesmodule.c
 Modules/_elementtree.c	Modules/_testcapimodule.c	Modules/cStringIO.c	Modules/cjkcodecs/cjkcodecs.h
 Modules/cjkcodecs/multibytecodec.c	Modules/cjkcodecs/multibytecodec.h	Modules/datetimemodule.c
 Modules/pyexpat.c	Modules/socketmodule.c	Modules/socketmodule.h	Modules/unicodedata.c
 Objects/capsule.c	Objects/object.c	Objects/unicodeobject.c	PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj	PC/os2emx/python27.def	PC/os2vacpp/python.def
 Python/compile.c Python/getargs.c
In-Reply-To: <4BABA600.8090103@egenix.com>
References: <20100325005454.B1AC5FCCD@mail.python.org>	<4BAB2882.6070508@egenix.com>
	<4BAB9FAE.1070206@hastings.org> <4BABA600.8090103@egenix.com>
Message-ID: <4BABA87B.2030306@hastings.org>


M.-A. Lemburg wrote:
> [Y]ou're pretty much breaking all 3rd party modules that have just started using e.g the datetime module C API.

Not if they're using PyDateTime_IMPORT.  My understanding is that 
requiring a recompile is okay, and if you use PyDateTime_IMPORT and 
recompile against 2.7 it works fine.  As I already mentioned, the 
exception to this is pyexpat, which provides no such importer macro / 
function.

I shall await your more fully-fleshed out reply,


/larry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100325/0ebf302b/attachment.html>

From fdrake at acm.org  Thu Mar 25 19:26:35 2010
From: fdrake at acm.org (Fred Drake)
Date: Thu, 25 Mar 2010 14:26:35 -0400
Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk:
	Doc/c-api/capsule.rst Doc/c-api/cobject.rst
	Doc/c-api/concrete.rst Doc/data/refcounts.dat
	Doc/extending/extending.rst Include/Python.h
	Include/cStringIO.h Include/cobject.h Include/datetime
Message-ID: <9cee7ab81003251126n28bca927j91f31d1d6d025c08@mail.gmail.com>

On Thu, Mar 25, 2010 at 2:16 PM, Larry Hastings <larry at hastings.org> wrote:
> My understanding is that requiring a recompile is okay

This has always been a point of contention.  I'm not even sure what
the current official position is.


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller

From larry at hastings.org  Thu Mar 25 19:49:39 2010
From: larry at hastings.org (Larry Hastings)
Date: Thu, 25 Mar 2010 11:49:39 -0700
Subject: [Python-Dev] [Python-checkins] PyCapsule backport
In-Reply-To: <loom.20100325T190700-671@post.gmane.org>
References: <20100325005454.B1AC5FCCD@mail.python.org>	<4BAB2882.6070508@egenix.com>
	<4BAB9FAE.1070206@hastings.org>
	<loom.20100325T190700-671@post.gmane.org>
Message-ID: <4BABB043.9080607@hastings.org>


Just a reply to one part of your message, as I really need to get back 
to work for now.

Antoine Pitrou wrote:
> I think solution #1 would be the best one.
> #2 looks too complicated.
>   

#2 is a seven-line diff, below.


/larry/

Index: cobject.c
===================================================================
--- cobject.c   (revision 79396)
+++ cobject.c   (working copy)
@@ -50,7 +50,12 @@
 PyCObject_AsVoidPtr(PyObject *self)
 {
     if (self) {
-        if (self->ob_type == &PyCObject_Type)
+        if (PyCapsule_CheckExact(self)) {
+            const char *name = PyCapsule_GetName(self);
+            const void *value = PyCapsule_GetPointer(self, name);
+            return value;
+        }
+        else if (self->ob_type == &PyCObject_Type)
             return ((PyCObject *)self)->cobject;
         PyErr_SetString(PyExc_TypeError,
                         "PyCObject_AsVoidPtr with non-C-object");


From stefan_ml at behnel.de  Thu Mar 25 20:15:28 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Mar 2010 20:15:28 +0100
Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk:
 Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst
 Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h
 Include/cStringIO.h Include/cobject.h Include/datetime.h
 Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h
 Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c
 Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c
 Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c
 Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c
 Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h
 Modules/unicodedata.c Objects/capsule.c Objects/object.c
 Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def
In-Reply-To: <4BAB2882.6070508@egenix.com>
References: <20100325005454.B1AC5FCCD@mail.python.org>
	<4BAB2882.6070508@egenix.com>
Message-ID: <hogcog$cpg$1@dough.gmane.org>

M.-A. Lemburg, 25.03.2010 10:10:
> larry.hastings wrote:
>> Author: larry.hastings
>> Date: Thu Mar 25 01:54:54 2010
>> New Revision: 79397
>>
>> Log:
>> Backported PyCapsule from 3.1, and converted most uses of
>> CObject to PyCapsule.
>
> Backporting PyCapsule is fine, but the changes you made to all
> those PyCObject uses does not look backwards compatible.

I got this in Cython's continuous integration server after this change:


Traceback (most recent call last):
   File "runtests.py", line 19, in <module>
     import cPickle as pickle
TypeError: PyCObject_AsVoidPtr with non-C-object


Although this looks like an internal problem in CPython, I'm certainly 
against any backwards incompatible changes of this size in the 2.x series.

Stefan


From v+python at g.nevcal.com  Thu Mar 25 20:31:28 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 25 Mar 2010 12:31:28 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<4BAB4764.4000500@gmail.com>
	<4BAB597D.1060508@jcea.es>	<4BAB6E70.3050008@gmail.com>	<5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com>	<4BAB7BC0.8020306@gmail.com>
	<5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com>
Message-ID: <4BABBA10.6000300@g.nevcal.com>

On 3/25/2010 8:13 AM, Mark Dickinson wrote:
> On Thu, Mar 25, 2010 at 3:05 PM, Nick Coghlan<ncoghlan at gmail.com>  wrote:
>    
>> Mark Dickinson wrote:
>>      
>>> On Thu, Mar 25, 2010 at 2:08 PM, Nick Coghlan<ncoghlan at gmail.com>  wrote:
>>>        
>>>> Jesus Cea wrote:
>>>>          
>>>>> But IEEE 754 was created by pretty clever guys and sure they had a
>>>>> reason for define things in the way they are. Probably we are missing
>>>>> something.
>>>>>            
>>>> Yes, this is where their "implementable in a hardware circuit" focus
>>>> comes in. They were primarily thinking of a floating point
>>>> representation where the 32/64 bits are *it* - you can't have "multiple
>>>> NaNs" because you don't have the bits available to describe them.
>>>>          
>>> I'm not so sure about this:  standard 64-bit binary IEEE 754 doubles
>>> allow for 2**53-2 different nans (2**52-2 signaling nans, 2**52 quiet
>>> nans):  anything with bit pattern (msb to lsb)
>>>
>>> x1111111 1111xxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx
>>>
>>> is an infinity or a nan, and there are only 2 infinities.
>>>        
>> I stand corrected :)
>>
>> It still seems to me that the problems mostly arise when we're trying to
>> get floats and Decimals to behave like Python *objects* (i.e. with
>> reflexive equality) rather than like IEEE defined numbers.
>>
>> It's an extra element that isn't part of the problem the numeric
>> standards are trying to solve.
>>      
> Agreed.  We don't have to be "missing something";  rather, the IEEE
> folks (quite understandably) almost certainly didn't anticipate this
> kind of usage.  So I'll concede that it's reasonable to consider
> deviating from the standard in the light of this.
>    

It is my understand that even bit-for-bit identical NaN values will 
compare unequal according to IEEE 754 rules.

I would have no problem with Python interning each encountered NaN 
value, to avoid having bit-for-bit identical NaN values with different 
Python IDs, but having them compare equal seems inappropriate.

Glenn

From gruszczy at gmail.com  Thu Mar 25 22:29:39 2010
From: gruszczy at gmail.com (=?UTF-8?Q?Filip_Gruszczy=C5=84ski?=)
Date: Thu, 25 Mar 2010 22:29:39 +0100
Subject: [Python-Dev] Issue 6081: format string using mapping rather
	than kwargs
In-Reply-To: <4BAB878A.3080106@trueblade.com>
References: <1be78d221003241618m29b4ce0bx21c4321b7a780c70@mail.gmail.com> 
	<4BAB4253.8090605@gmail.com> <4BAB5D9C.1010001@trueblade.com> 
	<4BAB703E.10701@gmail.com> <4BAB78F6.8080905@trueblade.com> 
	<4BAB7C8C.6020004@gmail.com> <4BAB878A.3080106@trueblade.com>
Message-ID: <1be78d221003251429v2652e330tff1e7577ce2a03e@mail.gmail.com>

>> I'm inclined to leave it alone unless/until Raymond or somebody else
>> steps up to really champion it.
>
> I'm okay with that.

And I am looking for another bug, that I could get some python-fu from ;-)

-- 
Filip Gruszczy?ski

From steve at pearwood.info  Thu Mar 25 23:05:10 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 26 Mar 2010 09:05:10 +1100
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <4BAB482F.4030608@gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<201003252118.43146.steve@pearwood.info>
	<4BAB482F.4030608@gmail.com>
Message-ID: <201003260905.11105.steve@pearwood.info>

On Thu, 25 Mar 2010 10:25:35 pm Nick Coghlan wrote:
> Steven D'Aprano wrote:
> > I'd like to turn the question around ... what algorithms are there
> > that rely on NaN == NaN being True?
>
> Absolutely anything that expects "x is y" to imply that "x == y". The
> builtin containers enforce this by checking identity before they
> check equality, but there are plenty of algorithms that won't.

Fair point, but I was actually thinking about mathematical algorithms. 

Builtin containers may also fail with any object that violates the 
expectation that identity implies equality, e.g.:

class AlwaysDifferent:
    def __eq__(self, other):
        return False
    def __ne__(self, other):
        return True


I don't see this as a problem -- if you choose to use such objects, 
you're responsible for understanding what is going on. If you choose to 
use floats, then you need to understand that NANs are weird.

Personally, I'm less concerned about sets of floats ending up with 
strange combinations of NANs than I am about the possibility of 
disastrous maths errors caused by allowing NANs to test as equal. 
Here's a simplistic example:

def func(a, b):
    if a == b:
        return 1.0
    return math.sin(a*b)**(a-b)

(I say "simplistic" because it currently fails for a=b=INF.) Currently 
this function will do the right thing for a, b both NANs:

>>> func(float('nan'), float('nan'))
nan


but making NANs test as equal will cause it to give the wrong answer. I 
fear that kind of error far more than the current funny behaviour of 
builtin containers with NANs.



-- 
Steven D'Aprano

From greg.ewing at canterbury.ac.nz  Thu Mar 25 23:45:51 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 26 Mar 2010 11:45:51 +1300
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <hof1br$8s5$1@dough.gmane.org>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>
	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>
	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>
	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>
	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>
	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>
	<hof1br$8s5$1@dough.gmane.org>
Message-ID: <4BABE79F.7010802@canterbury.ac.nz>

Georg Brandl wrote:
> Thinking of each value created by float('nan') as
> a different nan makes sense to my naive mind, and it also explains
> nicely the behavior present right now.

Not entirely:

   x = float('NaN')
   y = x
   if x == y:
     ...

There it's hard to argue that the NaNs being compared
result from different operations.

It does suggest a potential compromise, though: a single
NaN object compares equal to itself, but different NaN
objects are never equal (more or less what dict membership
testing does now, but extended to all == comparisons).

Whether that's a *sane* compromise I'm not sure.

-- 
Greg

From solipsis at pitrou.net  Fri Mar 26 00:06:06 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 Mar 2010 23:06:06 +0000 (UTC)
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<201003252118.43146.steve@pearwood.info>
	<4BAB482F.4030608@gmail.com>
	<201003260905.11105.steve@pearwood.info>
Message-ID: <loom.20100326T000152-409@post.gmane.org>

Steven D'Aprano <steve <at> pearwood.info> writes:
> 
> Personally, I'm less concerned about sets of floats ending up with 
> strange combinations of NANs than I am about the possibility of 
> disastrous maths errors caused by allowing NANs to test as equal. 
> Here's a simplistic example:

You just said "if you choose to use floats, then you need to understand that
NANs are weird". I wonder why this saying shouldn't apply to your "simplistic
example" of NAN usage.

(is your example even from real life?)



From rhamph at gmail.com  Fri Mar 26 00:10:56 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Thu, 25 Mar 2010 17:10:56 -0600
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <201003252118.43146.steve@pearwood.info>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
	<201003252118.43146.steve@pearwood.info>
Message-ID: <aac2c7cb1003251610r3a3ba9ddo640acc25b3fc6d66@mail.gmail.com>

On Thu, Mar 25, 2010 at 04:18, Steven D'Aprano <steve at pearwood.info> wrote:
> def myfunc(x, y):
> ? ?if x == y:
> ? ? ? ?return 1.0
> ? ?else:
> ? ? ? ?return something_complicated**(x-y)
>
>
> Optimising floating point code is fraught with dangers (the above fails
> for x=y=INF as well as NAN) but anything that make Not A Numbers
> pretend to be numbers is a bad thing.

What about this:

def myfunc(x):
    if x >= THRESHOLD:
        return 1.0
    else:
        return something_complicated(x)

If one behaves right it's more likely a fluke, not a designed in
feature.  It's certainly not obvious without covering every comparison
with comments.

Maybe that's the solution.  Signal by default on comparison, but add a
collection of naneq/naneg/etc functions (math module, methods,
whatever) that use a particular quiet mapping, making the whole thing
explicit?


-- 
Adam Olsen, aka Rhamphoryncus

From guido at python.org  Fri Mar 26 00:14:30 2010
From: guido at python.org (Guido van Rossum)
Date: Thu, 25 Mar 2010 16:14:30 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BABBA10.6000300@g.nevcal.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> 
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com> 
	<5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com> 
	<4BAB7BC0.8020306@gmail.com>
	<5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com> 
	<4BABBA10.6000300@g.nevcal.com>
Message-ID: <ca471dc21003251614j34a2146cje1b725eb5a5c3ff3@mail.gmail.com>

On Thu, Mar 25, 2010 at 12:31 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
> It is my understand that even bit-for-bit identical NaN values will compare
> unequal according to IEEE 754 rules.
>
> I would have no problem with Python interning each encountered NaN value, to
> avoid having bit-for-bit identical NaN values with different Python IDs, but
> having them compare equal seems inappropriate.

Let's please not intern NaNs. Interning is for performance. If you
have enough NaNs to affect your performance I think you have bigger
worries!

-- 
--Guido van Rossum (python.org/~guido)

From g.brandl at gmx.net  Fri Mar 26 00:21:59 2010
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 25 Mar 2010 23:21:59 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BABE79F.7010802@canterbury.ac.nz>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>	<hof1br$8s5$1@dough.gmane.org>
	<4BABE79F.7010802@canterbury.ac.nz>
Message-ID: <hogr7g$2in$1@dough.gmane.org>

Am 25.03.2010 22:45, schrieb Greg Ewing:
> Georg Brandl wrote:
>> Thinking of each value created by float('nan') as
>> a different nan makes sense to my naive mind, and it also explains
>> nicely the behavior present right now.
> 
> Not entirely:
> 
>    x = float('NaN')
>    y = x
>    if x == y:
>      ...
> 
> There it's hard to argue that the NaNs being compared
> result from different operations.
>
> It does suggest a potential compromise, though: a single
> NaN object compares equal to itself, but different NaN
> objects are never equal (more or less what dict membership
> testing does now, but extended to all == comparisons).
> 
> Whether that's a *sane* compromise I'm not sure.

FWIW, I like it.

Georg


From raymond.hettinger at gmail.com  Fri Mar 26 01:00:03 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Thu, 25 Mar 2010 17:00:03 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <hogr7g$2in$1@dough.gmane.org>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<d38f5331003241431y5149fed6s7ba3ba4a181a5ff7@mail.gmail.com>	<AB3214E2-46DC-44FD-8545-6C3FF08FA88D@gmail.com>	<d38f5331003241530r1c4e60ceo6979d31b67634f98@mail.gmail.com>	<5c6f2a5d1003241531m54e73c8dhd844369fe8cf155c@mail.gmail.com>	<d38f5331003241536n60a14757v20983a9ec7936a2d@mail.gmail.com>	<5c6f2a5d1003241547jefbc810off2db2ba07e4d2e0@mail.gmail.com>	<hof1br$8s5$1@dough.gmane.org>
	<4BABE79F.7010802@canterbury.ac.nz> <hogr7g$2in$1@dough.gmane.org>
Message-ID: <71121792-9ED3-40A3-87B1-5ED7EEDE1B65@gmail.com>


On Mar 25, 2010, at 4:21 PM, Georg Brandl wrote:

> Am 25.03.2010 22:45, schrieb Greg Ewing:
>> Georg Brandl wrote:
>>> Thinking of each value created by float('nan') as
>>> a different nan makes sense to my naive mind, and it also explains
>>> nicely the behavior present right now.
>> 
>> Not entirely:
>> 
>>   x = float('NaN')
>>   y = x
>>   if x == y:
>>     ...
>> 
>> There it's hard to argue that the NaNs being compared
>> result from different operations.
>> 
>> It does suggest a potential compromise, though: a single
>> NaN object compares equal to itself, but different NaN
>> objects are never equal (more or less what dict membership
>> testing does now, but extended to all == comparisons).
>> 
>> Whether that's a *sane* compromise I'm not sure.
> 
> FWIW, I like it.
> 
> Georg
> 

+1

Raymond


From steve at pearwood.info  Fri Mar 26 01:57:38 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 26 Mar 2010 11:57:38 +1100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
Message-ID: <201003261157.39250.steve@pearwood.info>

On Thu, 25 Mar 2010 06:26:11 am Mark Dickinson wrote:

> Here's an interesting recent blog post on this subject, from the
> creator of Eiffel:
>
> http://bertrandmeyer.com/2010/02/06/reflexivity-and-other-pillars-of-
>civilization/

Sorry, but he lost me right at the beginning when he quoted someone 
else:

    "there is no reason to believe that the result of one 
    calculation with unclear value should match that of 
    another calculation with unclear value" 

and then argued:

    "The exact same argument can be used to assert that the 
    result should not be False:  

    ? there is no reason to believe that the result of one 
    calculation with unclear value should not match that of 
    another calculation with unclear value.

    Just as convincing! Both arguments complement each other: 
    there is no compelling reason for demanding that the 
    values be equal; and there is no compelling argument either 
    to demand that they be different. If you ignore one of the 
    two sides, you are biased."

This whole argument is invalid on at least three levels. I'll get the 
first two out the way briefly

#1: Bertrand starts by treating NANs as "unclear values", and concludes 
that we shouldn't prefer "two unclear values are different" as more 
compelling than "two unclear values are the same". But this is 
ridiculous -- if you ask me a pair of questions, and I answer "I'm not 
sure" to both of them, why would you assume that the right answer to 
both questions is actually the same?

#2: But in fact NANs aren't "unclear values", they are not values at 
all. The answer to "what is the non-complex logarithm of -1?" is 
not "I'm not sure" but "there is no such value". Bertrand spends an 
awful lot of time trying to demonstrate why the reflexivity of equality 
(every x is equal to itself) should apply to NANs as well as the other 
floats, but RoE is a property of equivalence relations, which does not 
(and should not) hold for "there is no such value".

By analogy: the Lizard King of Russia does not exist; the Vampire Queen 
of New Orleans also does not exist. We don't therefore conclude that 
the Lizard King and the Vampire Queen are therefore the same person.

#3: We could, if we wish, violate the IEEE standard and treat equality 
of NANs as an equivalence relation. It's our language, we're free to 
follow whatever standards we like, and reflexivity of equality is a 
very useful axiom to have. Since it applies to all non-NAN floats (and 
virtually every object in Python, other than those with funny __eq__ 
methods), perhaps we should extend it to NANs as well?

I hope to convince you that the cost of doing so is worse than the 
disease. Since NANs are usually found in mathematical contexts, we 
should follow the IEEE standard even at the cost of rare anomalies in 
non-mathematical code containing NANs.

Simply put: we should treat "two unclear values are different" as more 
compelling than "two unclear values are the same" as it leads to fewer, 
smaller, errors. Consider:

log(-1) = NAN  # maths equality, not assignment
log(-2) = NAN

If we allow NAN = NAN, then we permit the error:

log(-1) = NAN = log(-2)
therefore log(-1) = log(-2)
and 1 = 2

But if make NAN != NAN, then we get:

log(-1) != log(-2)

and all of mathematics does not collapse into a pile of rubble. I think 
that is a fairly compelling reason to prefer inequality over equality.

One objection might be that while log(-1) and log(-2) should be 
considered different NANs, surely NANs should be equal to themselves?

-1 = -1
implies log(-1) = log(-1)

But consider the practicalities: there are far more floats than 
available NAN payloads. We simply can't map every invalid calculation 
to a unique NAN, and therefore there *must* be cases like:

log(-123.456789e-8) = log(-9.876e47)
implies 123.456789e-8 = 9.876e47

So we mustn't consider NANs equal just because their payloads are equal.

What about identity? Even if we don't dare allow this:

x = log(-1)  # assignment
y = log(-1)  # another NAN with the same payload
assert x is not y
assert x == y

surely we can allow this?

assert x == x

But this is dangerous. Don't be fooled by the simplicity of the above 
example. Just because you have two references to the same (as in 
identity) NAN, doesn't mean they represent "the same thing" or came 
from the same place:

data = [1, 2, float('nan'), float('nan'), 3]
x = harmonic_mean(data)
y = 1 - geometric_mean(data)

It is an accident of implementation whether x and y happen to be the 
same object or not. Why should their inequality depend on such a 
fragile thing?

In fact, identity of NANs is itself an implementation quirk of 
programming languages like Python: logically, NANs don't have identity 
at all.

To put it another way: all ONEs are the same ONE, even if they come from 
different sources, are in different memory locations, or have different 
identities; but all NANs are different, even if they come from the same 
source, are in the same memory location, or have the same identity.

The fundamental problem here is that NANs are not values. If you treat 
them as if they were values, then you want reflexivity of equality. But 
they're not -- they're *signals* for "your calculation has gone screwy 
and the result you get is garbage", so to speak. You shouldn't even 
think of a specific NAN as a piece of specific garbage, but merely a 
label on the *kind* of garbage you've got (the payload): INF-INF is, in 
some sense, a different kind of error to log(-1).

In the same way you might say "INF-INF could be any number at all, 
therefore we return NAN", you might say "since INF-INF could be 
anything, there's no reason to think that INF-INF == INF-INF."




-- 
Steven D'Aprano

From steve at pearwood.info  Fri Mar 26 02:10:54 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 26 Mar 2010 12:10:54 +1100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BABE79F.7010802@canterbury.ac.nz>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<hof1br$8s5$1@dough.gmane.org> <4BABE79F.7010802@canterbury.ac.nz>
Message-ID: <201003261210.54863.steve@pearwood.info>

On Fri, 26 Mar 2010 09:45:51 am Greg Ewing wrote:
> Georg Brandl wrote:
> > Thinking of each value created by float('nan') as
> > a different nan makes sense to my naive mind, and it also explains
> > nicely the behavior present right now.
>
> Not entirely:
>
>    x = float('NaN')
>    y = x
>    if x == y:
>      ...
>
> There it's hard to argue that the NaNs being compared
> result from different operations.

But unlike us, the equality operator only has a pinhole view of the 
operands. It can't distinguish between your example and this:

x = float('nan')
y = some_complex_calculation(x)
if x == y:
    ...

where y merely happens to end up with the same object as x by some quirk 
of implementation.


> It does suggest a potential compromise, though: a single
> NaN object compares equal to itself, but different NaN
> objects are never equal (more or less what dict membership
> testing does now, but extended to all == comparisons).
>
> Whether that's a *sane* compromise I'm not sure.


What do we do with Decimal? Aren't we committed to matching the Decimal 
standard, in which case aren't we committed to this?

x = Decimal('nan')
assert x != x

If that's the case, then float NANs and Decimal NANs will behave 
differently. I think that's a mistake.


For what it's worth, I'm -1 on allowing NANs to test equal to any NAN, 
including itself. However, I would be -0 on the following compromise:

Allow NANs to test equal to themselves (identity testing).
math module to grow a function ieee_equals(x, y) that keeps the current 
behaviour.



-- 
Steven D'Aprano

From greg.ewing at canterbury.ac.nz  Fri Mar 26 02:15:54 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 26 Mar 2010 14:15:54 +1300
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <201003252118.43146.steve@pearwood.info>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<0E3C6281-64C3-40FB-8672-9AD6FAE47005@gmail.com>
	<d38f5331003241126l2f880068ka8eb401b23df6905@mail.gmail.com>
	<201003252118.43146.steve@pearwood.info>
Message-ID: <4BAC0ACA.7030401@canterbury.ac.nz>

Steven D'Aprano wrote:

> I'd like to turn the question around ... what algorithms are there that 
> rely on NaN == NaN being True?

That seems to be a straw question, since AFAIK nobody
has suggested that there are any such algorithms.

On the other hand, it has been claimed that some
algorithms exist that benefit from Nan == NaN being
false, so it's fair to ask for examples to back
that up.

-- 
Greg

From pje at telecommunity.com  Fri Mar 26 02:19:06 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 25 Mar 2010 21:19:06 -0400
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003261157.39250.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
Message-ID: <20100326011901.6F4133A4080@sparrow.telecommunity.com>

At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote:
>But they're not -- they're *signals* for "your calculation has gone 
>screwy and the result you get is garbage", so to speak. You 
>shouldn't even think of a specific NAN as a piece of specific 
>garbage, but merely a label on the *kind* of garbage you've got (the 
>payload): INF-INF is, in some sense, a different kind of error to 
>log(-1). In the same way you might say "INF-INF could be any number 
>at all, therefore we return NAN", you might say "since INF-INF could 
>be anything, there's no reason to think that INF-INF == INF-INF."

So, are you suggesting that maybe the Pythonic thing to do in that 
case would be to cause any operation on a NAN (including perhaps 
comparison) to fail, rather than allowing garbage to silently propagate?

In other words, if NAN is only a signal that you have garbage, is 
there really any reason to keep it as an *object*, instead of simply 
raising an exception?  Then, you could at least identify what 
calculation created the garbage, instead of it percolating up through 
other calculations.

In low-level languages like C or Fortran, it obviously makes sense to 
represent NAN as a value, because there's no other way to represent 
it.  But in a language with exceptions, is there a use case for it 
existing as a value?


From fijall at gmail.com  Fri Mar 26 02:31:57 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 25 Mar 2010 19:31:57 -0600
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003261157.39250.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com> 
	<201003261157.39250.steve@pearwood.info>
Message-ID: <693bc9ab1003251831l226687d2m9331b6dc086f0ab3@mail.gmail.com>

> -1 = -1
> implies log(-1) = log(-1)

Mathematically speaking this is incorrect.

x = y
implies
log(x) = log(y)
for x > 0 and y > 0

Cheers,
fijal

From jcea at jcea.es  Fri Mar 26 02:37:48 2010
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 26 Mar 2010 02:37:48 +0100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003261157.39250.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
Message-ID: <4BAC0FEC.8060308@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/26/2010 01:57 AM, Steven D'Aprano wrote:

> In fact, identity of NANs is itself an implementation quirk of 
> programming languages like Python: logically, NANs don't have identity 
> at all.
> 
> To put it another way: all ONEs are the same ONE, even if they come from 
> different sources, are in different memory locations, or have different 
> identities; but all NANs are different, even if they come from the same 
> source, are in the same memory location, or have the same identity.

+inf. Bravo!.

:-)

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS6wP7Jlgi5GaxT1NAQJXFgP+KHshsGGOM1GO/f1lEazul1+vipw6tUuF
Hf8Pm39srX3pKV7OUabWJWBaRURnJWIHEymjwILBwVo9X4604klbTRi3MrqbOeZv
G8NFeMzagXLwEOlAnPbVcfDb3KcW4C/Zm0A5TOPY4X5T/8FRmhfbrC4Ip2klyyMh
24fGUWTr5DI=
=d1RL
-----END PGP SIGNATURE-----

From rhamph at gmail.com  Fri Mar 26 02:53:31 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Thu, 25 Mar 2010 19:53:31 -0600
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003261157.39250.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
Message-ID: <aac2c7cb1003251853o7a550e80p22e64f2cad58f20f@mail.gmail.com>

On Thu, Mar 25, 2010 at 18:57, Steven D'Aprano <steve at pearwood.info> wrote:
> Simply put: we should treat "two unclear values are different" as more
> compelling than "two unclear values are the same" as it leads to fewer,
> smaller, errors. Consider:
>
> log(-1) = NAN ?# maths equality, not assignment
> log(-2) = NAN
>
> If we allow NAN = NAN, then we permit the error:
>
> log(-1) = NAN = log(-2)
> therefore log(-1) = log(-2)
> and 1 = 2
>
> But if make NAN != NAN, then we get:
>
> log(-1) != log(-2)
>
> and all of mathematics does not collapse into a pile of rubble. I think
> that is a fairly compelling reason to prefer inequality over equality.
>
> One objection might be that while log(-1) and log(-2) should be
> considered different NANs, surely NANs should be equal to themselves?
>
> -1 = -1
> implies log(-1) = log(-1)

IMO, this just shows how ludicrous it is to compare NaNs.  No matter
what we do it will imply some insane mathematical consequence implied
and code that will break.

They are, after all, an error passed silently.

Why is it complex can raise an exception when sorted, forcing you to
use a sane (and explicit) method, but for NaN it's okay to silently
fail?


-- 
Adam Olsen, aka Rhamphoryncus

From v+python at g.nevcal.com  Fri Mar 26 02:56:00 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 25 Mar 2010 18:56:00 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <ca471dc21003251614j34a2146cje1b725eb5a5c3ff3@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<5c6f2a5d1003250722g28b130edxda76a1f8b6f9f0dc@mail.gmail.com>
	<4BAB7BC0.8020306@gmail.com>
	<5c6f2a5d1003250813h15e5610bseb4a9b3b541585ff@mail.gmail.com>
	<4BABBA10.6000300@g.nevcal.com>
	<ca471dc21003251614j34a2146cje1b725eb5a5c3ff3@mail.gmail.com>
Message-ID: <4BAC1430.4030301@g.nevcal.com>

On 3/25/2010 4:14 PM, Guido van Rossum wrote:
> On Thu, Mar 25, 2010 at 12:31 PM, Glenn Linderman<v+python at g.nevcal.com>  wrote:
>    
>> It is my understand that even bit-for-bit identical NaN values will compare
>> unequal according to IEEE 754 rules.
>>
>> I would have no problem with Python interning each encountered NaN value, to
>> avoid having bit-for-bit identical NaN values with different Python IDs, but
>> having them compare equal seems inappropriate.
>>      
> Let's please not intern NaNs. Interning is for performance. If you
> have enough NaNs to affect your performance I think you have bigger
> worries!
>    

I'm OK with interning NaNs, and I'm OK with not interning NaNs.  But I 
would much prefer to see Python conform to the IEEE 754 for float, and 
the Decimal standard for Decimal, where at all possible, including 
keeping NaNs not comparing equal to anything, themselves included.

And retaining a mode where Decimal is standoffish, and doesn't accept 
non-Decimal comparisons or arithmetic seems like a big win for program 
correctness to me.

Glenn

From greg.ewing at canterbury.ac.nz  Fri Mar 26 03:25:30 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 26 Mar 2010 15:25:30 +1300
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>
Message-ID: <4BAC1B1A.6030500@canterbury.ac.nz>

Curt Hagenlocher wrote:

> Wait, what? I haven't been paying much attention, but this is backwards.
> There are multiple representations of NaN in the IEEE encoding;

I think Nick's point is that there aren't enough bits
to give the result of every operation its own unique
NaN.

The payload of a NaN in typical hardware implementations
is quite small, because it has to fit into the exponent
field.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Mar 26 05:32:46 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 26 Mar 2010 17:32:46 +1300
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003261157.39250.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
Message-ID: <4BAC38EE.9090600@canterbury.ac.nz>

Steven D'Aprano wrote:

> By analogy: the Lizard King of Russia does not exist; the Vampire Queen 
> of New Orleans also does not exist. We don't therefore conclude that 
> the Lizard King and the Vampire Queen are therefore the same person.

But it's equally invalid to say that they're *not* the
same person, because it's meaningless to say *anything*
about the properties of nonexistent people.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Mar 26 05:35:55 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 26 Mar 2010 17:35:55 +1300
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003261210.54863.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<hof1br$8s5$1@dough.gmane.org> <4BABE79F.7010802@canterbury.ac.nz>
	<201003261210.54863.steve@pearwood.info>
Message-ID: <4BAC39AB.2040400@canterbury.ac.nz>

Steven D'Aprano wrote:

> What do we do with Decimal? Aren't we committed to matching the Decimal 
> standard,

It's been pointed out that the Decimal standard only defines
some abstract operations, and doesn't mandate that they
be mapped onto any particular language syntax. That gives
us enough flexibility to make == do what we want and still
claim compliance with the standard.

BTW, does IEEE754 give us the same flexibility? If so,
we may not have much of a problem in the first place.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Mar 26 05:40:23 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 26 Mar 2010 17:40:23 +1300
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAC1B1A.6030500@canterbury.ac.nz>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<4BAB4764.4000500@gmail.com> <4BAB597D.1060508@jcea.es>
	<4BAB6E70.3050008@gmail.com>
	<d2155e361003250719g532b01d2h4d1a3b697820bf0@mail.gmail.com>
	<4BAC1B1A.6030500@canterbury.ac.nz>
Message-ID: <4BAC3AB7.5040809@canterbury.ac.nz>

I impulsively wrote:

> The payload of a NaN in typical hardware implementations
> is quite small, because it has to fit into the exponent
> field.

...which turns out to be precisely wrong. Some day I'll
learn to wait until somebody else in the thread has checked
the facts for me before posting. :-)

-- 
Greg


From stephen at xemacs.org  Fri Mar 26 05:39:55 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 26 Mar 2010 13:39:55 +0900
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003261210.54863.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<hof1br$8s5$1@dough.gmane.org> <4BABE79F.7010802@canterbury.ac.nz>
	<201003261210.54863.steve@pearwood.info>
Message-ID: <87ocibg0h0.fsf@uwakimon.sk.tsukuba.ac.jp>

Steven D'Aprano writes:

 > But unlike us, the equality operator only has a pinhole view of the 
 > operands. It can't distinguish between your example and this:
 > 
 > x = float('nan')
 > y = some_complex_calculation(x)
 > if x == y:
 >     ...
 > 
 > where y merely happens to end up with the same object as x by some quirk 
 > of implementation.

Note that Mark has already provided a related example of such a quirk
(Decimal(-1).sqrt(), I think it was).

From v+python at g.nevcal.com  Fri Mar 26 05:47:53 2010
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Thu, 25 Mar 2010 21:47:53 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAC39AB.2040400@canterbury.ac.nz>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<hof1br$8s5$1@dough.gmane.org>
	<4BABE79F.7010802@canterbury.ac.nz>	<201003261210.54863.steve@pearwood.info>
	<4BAC39AB.2040400@canterbury.ac.nz>
Message-ID: <4BAC3C79.400@g.nevcal.com>

On 3/25/2010 9:35 PM, Greg Ewing wrote:
> Steven D'Aprano wrote:
>
>> What do we do with Decimal? Aren't we committed to matching the 
>> Decimal standard,
>
> It's been pointed out that the Decimal standard only defines
> some abstract operations, and doesn't mandate that they
> be mapped onto any particular language syntax. That gives
> us enough flexibility to make == do what we want and still
> claim compliance with the standard.
>
> BTW, does IEEE754 give us the same flexibility? If so,
> we may not have much of a problem in the first place.
>

I propose that the abstract Decimal operation for addition be mapped to 
the syntax of operator - and that the abstract Decimal operation for 
subtraction be mapped to the syntax of operator +. Then people will have 
to actually read the manual to learn how to use the Decimal type in 
Python, rather than assuming that things might work the way they expect. 
This will lead to more robust and correct programs, because people will 
have read the manual. Or at least it seems that it should work that way...

?? ? (Hmm, that might not render consistently for everyone, so I'll 
throw in a couple :) :) also.)

Glenn



From mal at egenix.com  Fri Mar 26 10:20:08 2010
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 26 Mar 2010 10:20:08 +0100
Subject: [Python-Dev] [Python-checkins] r79397 -
 in	python/trunk:	Doc/c-api/capsule.rst
 Doc/c-api/cobject.rst	Doc/c-api/concrete.rst	Doc/data/refcounts.dat	Doc/extending/extending.rst
 Include/Python.h	Include/cStringIO.h	Include/cobject.h	Include/datetime.h
 Include/py_curses.h	Include/pycapsule.h	Include/pyexpat.h	Include/ucnhash.h
 Lib/test/test_sys.py	Makefile.pre.in	Misc/NEWS	Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c	Modules/_ctypes/ctypes.h	Modules/_cursesmodule.c
 Modules/_elementtree.c	Modules/_testcapimodule.c	Modules/cStringIO.c	Modules/cjkcodecs/cjkcodecs.h
 Modules/cjkcodecs/multibytecodec.c	Modules/cjkcodecs/multibytecodec.h	Modules/datetimemodule.c
 Modules/pyexpat.c	Modules/socketmodule.c	Modules/socketmodule.h	Modules/unicodedata.c
 Objects/capsule.c	Objects/object.c	Objects/unicodeobject.c	PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj	PC/os2emx/python27.def	PC/os2vacpp/python.def
 Python/compile.c Python/getargs.c
In-Reply-To: <4BAB9FAE.1070206@hastings.org>
References: <20100325005454.B1AC5FCCD@mail.python.org>	<4BAB2882.6070508@egenix.com>
	<4BAB9FAE.1070206@hastings.org>
Message-ID: <4BAC7C48.3080600@egenix.com>

Larry Hastings wrote:
> 
> M.-A. Lemburg wrote:
>> Backporting PyCapsule is fine, but the changes you made to all
>> those PyCObject uses does not look backwards compatible.
>>
>> The C APIs exposed by the modules (e.g. the datetime module)
>> are used in lots of 3rd party extension modules and changing
>> them from PyCObject to PyCapsule is a major change in the
>> module API.
> 
> You're right, my changes aren't backwards compatible.  I thought it was
> reasonable for four reasons:

Just as reminder of the process we have in place for such changes:
Please discuss any major breakage on python-dev before checking in
the patch.

> 1. The CObject API isn't safe.  It's easy to crash Python 2.6 in just a
> few lines by mixing and matching CObjects.  Switching Python to capsules
> prevents a class of exploits.  I've included a script at the bottom of
> this message that demonstrates three such crashes.  The script runs in
> Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

That's good, but then again: deliberate wrong use of APIs will
always cause crashes and at least I don't know of any report about
PyCObjects posing a problem in all their years of existence.

> 2. As I just mentioned, Python 3.1 already uses capsules everywhere
> instead of CObjects.  Since part of the purpose of Python 2.7 is to
> prepare developers for the to upgrade to 3.1, getting them to switch to
> capsules now is just one more way they are prepared.

Sure, but forcing them is not a good idea, even less so if you can easily
expose both a PyCObject and PyCapsule interface to the same C API.

> 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we
> ever made a 2.8 I want to remove it completely.

Please remember that PyCObjects are not only used internally
in CPython, but also in other 3rd party modules to expose C APIs
and those will generally have to support more than just the latest
Python release.

If you deprecate those C APIs, the process will at least have to cover
one more release, i.e. run through the whole deprecation process:

1. pending deprecation (2.7)
2. deprecation (2.8)
3. removal (2.9)

I think it's better to add a -3 warning when using PyCObjects in
2.7.

> 4. When Python publishes an API using a CObject, it describes the thing
> the CObject points to in a header file.  In nearly all cases that header
> file also provides a macro or inline function that does the importing
> work for you.  I changed those to use capsules too.  So if the
> third-party code uses the macro or inline function, all you need do is
> recompile it against 2.7 and it works fine.  Sadly I know of one
> exception: pyexpat.expat_CAPI.  The header file just describes the
> struct pointed to by the CObject, but callers

I know about those macros... I introduced that idea with mxDateTime
and then added the same logic in CPython in a couple of places
a long while ago, e.g. socketmodule.h. IIRC, Jim Fulton added
PyCObjects for exchanging module C APIs a few years before that.

Later on PyCObject_Import() was added to simplify the C API
import a bit.

A recompile is certainly a possibility to have 3rd party
modules switch to the capsule interfaces of the stdlib
modules, but we should still be more careful about this.

Wouldn't it be possible to have PyCObject_AsVoidPtr() et al.
work with PyCapsules as well ?

After all, the caller is only interested in the pointer and
doesn't really care about what object was used to wrap it.

> I can suggest four ways to ameliorate the problem.
> 
> First, we could do as Antoine Pitrou suggests on the bug (issue 7992):
> wherever the CObject used to be published as a module attribute to
> expose an API, we could provide both a CObject and a capsule; internally
> Python would only use the capsules.  This would allow third-party
> libraries to run against 2.7 unchanged.  The major problem with this is
> that third-party libraries would still be vulnerable to the
> mix-and-match CObject crash. 

True, but we've been happy with this vulnerability for years, just
as we've been happy with the fact that it's easy to crash the
VM by passing hand-crafted byte-code to it, or using ctypes to
call an OS function with the wrong parameters, etc.

Like I said: there are many ways to deliberately crash Python.

We don't have a concept of interface signatures in Python,
it's mostly based on trust.

> A secondary, minor concern: obviously we'd
> store the CObject attribute with the existing name, and the capsule
> attribute would have to get some new name.  But in Python 3.1, these
> attributes already expose a capsule.  Therefore, people who convert to
> using the capsules now would have to convert again when moving to 3.1.

This should be manageable with some aliasing.

> Second, we could make CObject internally support unpacking capsules.  If
> you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and
> return the pointer within.  (We could probably also map the capsule
> "context" to the CObject "desc", if any of the Python use cases needed
> it.)  I wouldn't change anything else about CObjects; creating and using
> them would continue to work as normal.  This would also allow
> third-party libraries to run against Python 2.7 unchanged.  The only
> problem is that it's unsafe, as indeed allowing any use of
> PyCObject_AsVoidPtr() is unsafe.

That's a good idea. Yes, playing with fire is unsafe, but fire is
useful in a lot of places as well :-)

> Third, I've been pondering writing a set of preprocessor macros, shipped
> in their own header file distributed independently of Python and
> released to the public domain, that would make it easy to use either
> CObjects or capsules depending on what version of Python you were
> compiling against.  Obviously, using these macros would require a source
> code change in the third-party library.  But these macros would make it
> a five-minute change.  This could compliment the first or second
> approaches.

I'm not sure whether that would worth the effort. 3rd party modules
for Python 2.x are likely not going to use them, since they typically
have to support more than just Python 2.7.

E.g. look at Plone - they are still running on Python 2.4 and use
lots of 3rd party modules. It took Zope years to switch from 2.4 to
2.6. You have the same situation with other large systems, whether
OSS or not.

If a required 3rd party module would suddenly only support Python 2.7,
developers would either have to find a replacement that (still) works
for them (rather unlikely), or try to port their whole app to 2.7,
which often enough is not easily possible due to other restrictions.

This aspect is often not considered when discussing such backwards
incompatible changes - at least not in the early stages. Fortunately,
we have so far - and most of the time - found a way to keep everyone
happy.

> Fourth, we could back out of the changes to published APIs and convert
> them back to CObjects.  -1.

See above. That's still an option as well, but I agree that either
making PyCObjects compatible with PyCapsules (via changes to
the PyCObject functions) or having them exist side-by-side is
a better option.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 26 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From stefan_ml at behnel.de  Fri Mar 26 11:03:32 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 26 Mar 2010 11:03:32 +0100
Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk:
 Doc/c-api/capsule.rst Doc/c-api/cobject.rst Doc/c-api/concrete.rst
 Doc/data/refcounts.dat Doc/extending/extending.rst Include/Python.h
 Include/cStringIO.h Include/cobject.h Include/datetime.h
 Include/py_curses.h Include/pycapsule.h Include/pyexpat.h Include/ucnhash.h
 Lib/test/test_sys.py Makefile.pre.in Misc/NEWS Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_cursesmodule.c
 Modules/_elementtree.c Modules/_testcapimodule.c Modules/cStringIO.c
 Modules/cjkcodecs/cjkcodecs.h Modules/cjkcodecs/multibytecodec.c
 Modules/cjkcodecs/multibytecodec.h Modules/datetimemodule.c
 Modules/pyexpat.c Modules/socketmodule.c Modules/socketmodule.h
 Modules/unicodedata.c Objects/capsule.c Objects/object.c
 Objects/unicodeobject.c PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj PC/os2emx/python27.def
In-Reply-To: <4BAC7C48.3080600@egenix.com>
References: <20100325005454.B1AC5FCCD@mail.python.org>	<4BAB2882.6070508@egenix.com>	<4BAB9FAE.1070206@hastings.org>
	<4BAC7C48.3080600@egenix.com>
Message-ID: <hoi0pk$4lm$1@dough.gmane.org>

M.-A. Lemburg, 26.03.2010 10:20:
> Larry Hastings wrote:
>> 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we
>> ever made a 2.8 I want to remove it completely.
>
> Please remember that PyCObjects are not only used internally
> in CPython, but also in other 3rd party modules to expose C APIs
> and those will generally have to support more than just the latest
> Python release.

The removal of PyCObject from Py3.2 pretty much caught the Cython project 
by surprise, as they had been The One Obvious Way for public C-APIs for 
ages. We quickly switched to PyCapsule back then, but only for newer 3.x 
versions. This is easy to do when your code relies on Cython (or its 
generated header files) for importing and exporting this C-API, as a 
rebuild will just do all the work for you. However, if you have to do this 
manually, you will basically have to write all the code yourself that 
Cython now generates.

If this can be done using preprocessor macros, fine. But the underlying ABI 
should not be changed in Py2.x. The normal 2.x approach would be to add the 
new API *in addition* to the old one, potentially by providing macros that 
map the new functionality to the old one, but not the other way round. That 
would allow code to be written that works in both Py2.7 and Py3.2+, without 
breaking backwards compatibility with existing Py2.x libraries.

And, no, recompilation is not always an option and certainly shouldn't be 
required for the soon to be discontinued 2.x line. The *only* reason for 
the very existence of 2.7 is to not break backwards compatibility.


> If you deprecate those C APIs, the process will at least have to cover
> one more release, i.e. run through the whole deprecation process:
>
> 1. pending deprecation (2.7)
> 2. deprecation (2.8)
> 3. removal (2.9)

If you replace 2.8 by 3.1 and 2.9 by 3.2, that's more or less what has 
happened already. No need to put any more thoughts into Py2.x for that purpose.

Stefan


From cournape at gmail.com  Fri Mar 26 18:04:06 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 27 Mar 2010 02:04:06 +0900
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <20100326011901.6F4133A4080@sparrow.telecommunity.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
Message-ID: <5b8d13221003261004r249dbfacr6a16a8f2212cc080@mail.gmail.com>

On Fri, Mar 26, 2010 at 10:19 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote:
>>
>> But they're not -- they're *signals* for "your calculation has gone screwy
>> and the result you get is garbage", so to speak. You shouldn't even think of
>> a specific NAN as a piece of specific garbage, but merely a label on the
>> *kind* of garbage you've got (the payload): INF-INF is, in some sense, a
>> different kind of error to log(-1). In the same way you might say "INF-INF
>> could be any number at all, therefore we return NAN", you might say "since
>> INF-INF could be anything, there's no reason to think that INF-INF ==
>> INF-INF."
>
> So, are you suggesting that maybe the Pythonic thing to do in that case
> would be to cause any operation on a NAN (including perhaps comparison) to
> fail, rather than allowing garbage to silently propagate?

Nan behavior being tightly linked to FPU exception handling, I think
this is a good idea. One of the goal of Nan is to avoid many testing
in intermediate computation (for efficiency reason), which may not
really apply to python. Generally, you want to detect
errors/exceptional situations as early as possible, and if you use
python, you don't care about potential slowdown caused by those
checks.

David

From casey at pandora.com  Fri Mar 26 18:40:43 2010
From: casey at pandora.com (Casey Duncan)
Date: Fri, 26 Mar 2010 11:40:43 -0600
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <20100326011901.6F4133A4080@sparrow.telecommunity.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
Message-ID: <AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>


On Mar 25, 2010, at 7:19 PM, P.J. Eby wrote:

> At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote:
>> But they're not -- they're *signals* for "your calculation has gone screwy and the result you get is garbage", so to speak. You shouldn't even think of a specific NAN as a piece of specific garbage, but merely a label on the *kind* of garbage you've got (the payload): INF-INF is, in some sense, a different kind of error to log(-1). In the same way you might say "INF-INF could be any number at all, therefore we return NAN", you might say "since INF-INF could be anything, there's no reason to think that INF-INF == INF-INF."
> 
> So, are you suggesting that maybe the Pythonic thing to do in that case would be to cause any operation on a NAN (including perhaps comparison) to fail, rather than allowing garbage to silently propagate?
> 
> In other words, if NAN is only a signal that you have garbage, is there really any reason to keep it as an *object*, instead of simply raising an exception?  Then, you could at least identify what calculation created the garbage, instead of it percolating up through other calculations.
> 
> In low-level languages like C or Fortran, it obviously makes sense to represent NAN as a value, because there's no other way to represent it.  But in a language with exceptions, is there a use case for it existing as a value?

If a NaN object is allowed to exist, that is a float operation that does not return a real number does not itself raise an exception immediately, then it will always be possible to get (seemingly) nonsensical behavior when it is used in containers that do not themselves "operate" on their elements.

So even provided that performing any "operation" on a NaN object raises an exception, it would still be possible to add such an object to a list or tuple and have subsequent containment checks for that object return false. So this "solution" would simply narrow the problem posed, but not eliminate it.

None of the solution posed seem very ideal, in particular when they deviate from the standard in arbitrary ways someone deems "better". It's obvious to me that no ideal solution exists so long as you attempt to represent non-numeric values in a numeric type. So unless you simply eliminate NaNs (thus breaking the standard), you are going to confuse somebody. And I think having float deviate from the IEEE standard is ill advised unless there is no alternative (i.e., the standard cannot be practically implemented), and breaking it will confuse people too (and probably the ones that know this domain).

I propose that the current behavior stands as is and that the documentation make mention of the fact that NaN values are unordered, thus some float values may not behave intuitively wrt hashing, equality, etc.

The fact of the matter is that using floats as dict keys or set values or even just checking equality is much more complex in practice than you would expect. I mean even representing 1.1 is problematic ;^). Unless the float values you are using are constants, how would you practically use them as dict keys, or hsah set members anyway? I'm not saying it can't be done, but is a hash table with float keys ever a data structure that someone on this list would recommend? If so good luck and god speed 8^)

-Casey



From python-dev at masklinn.net  Fri Mar 26 22:16:19 2010
From: python-dev at masklinn.net (Xavier Morel)
Date: Fri, 26 Mar 2010 22:16:19 +0100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
Message-ID: <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>

On 26 Mar 2010, at 18:40 , Casey Duncan wrote:
> 
> 
> On Mar 25, 2010, at 7:19 PM, P.J. Eby wrote:
> 
>> At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote:
>>> But they're not -- they're *signals* for "your calculation has gone screwy and the result you get is garbage", so to speak. You shouldn't even think of a specific NAN as a piece of specific garbage, but merely a label on the *kind* of garbage you've got (the payload): INF-INF is, in some sense, a different kind of error to log(-1). In the same way you might say "INF-INF could be any number at all, therefore we return NAN", you might say "since INF-INF could be anything, there's no reason to think that INF-INF == INF-INF."
>> 
>> So, are you suggesting that maybe the Pythonic thing to do in that case would be to cause any operation on a NAN (including perhaps comparison) to fail, rather than allowing garbage to silently propagate?
>> 
>> In other words, if NAN is only a signal that you have garbage, is there really any reason to keep it as an *object*, instead of simply raising an exception?  Then, you could at least identify what calculation created the garbage, instead of it percolating up through other calculations.
>> 
>> In low-level languages like C or Fortran, it obviously makes sense to represent NAN as a value, because there's no other way to represent it.  But in a language with exceptions, is there a use case for it existing as a value?
> 
> If a NaN object is allowed to exist, that is a float operation that does not return a real number does not itself raise an exception immediately, then it will always be possible to get (seemingly) nonsensical behavior when it is used in containers that do not themselves "operate" on their elements.
How about raising an exception instead of creating nans in the first place, except maybe within specific contexts (so that the IEEE-754 minded can get their nans working as they currently do)?

That way, there cannot be any nan-induced seemingly nonsensical behavior except within known scopes.

From casey at pandora.com  Fri Mar 26 22:39:44 2010
From: casey at pandora.com (Casey Duncan)
Date: Fri, 26 Mar 2010 15:39:44 -0600
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
Message-ID: <16E0E0B5-9E14-4386-815E-742F344ADEA3@pandora.com>


On Mar 26, 2010, at 3:16 PM, Xavier Morel wrote:

> On 26 Mar 2010, at 18:40 , Casey Duncan wrote:
>> 
>> 
>> On Mar 25, 2010, at 7:19 PM, P.J. Eby wrote:
>> 
>>> At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote:
>>>> But they're not -- they're *signals* for "your calculation has gone screwy and the result you get is garbage", so to speak. You shouldn't even think of a specific NAN as a piece of specific garbage, but merely a label on the *kind* of garbage you've got (the payload): INF-INF is, in some sense, a different kind of error to log(-1). In the same way you might say "INF-INF could be any number at all, therefore we return NAN", you might say "since INF-INF could be anything, there's no reason to think that INF-INF == INF-INF."
>>> 
>>> So, are you suggesting that maybe the Pythonic thing to do in that case would be to cause any operation on a NAN (including perhaps comparison) to fail, rather than allowing garbage to silently propagate?
>>> 
>>> In other words, if NAN is only a signal that you have garbage, is there really any reason to keep it as an *object*, instead of simply raising an exception?  Then, you could at least identify what calculation created the garbage, instead of it percolating up through other calculations.
>>> 
>>> In low-level languages like C or Fortran, it obviously makes sense to represent NAN as a value, because there's no other way to represent it.  But in a language with exceptions, is there a use case for it existing as a value?
>> 
>> If a NaN object is allowed to exist, that is a float operation that does not return a real number does not itself raise an exception immediately, then it will always be possible to get (seemingly) nonsensical behavior when it is used in containers that do not themselves "operate" on their elements.
> How about raising an exception instead of creating nans in the first place, except maybe within specific contexts (so that the IEEE-754 minded can get their nans working as they currently do)?
> 
> That way, there cannot be any nan-induced seemingly nonsensical behavior except within known scopes.

Having NaN creation raise an exception would undoubtedly break plenty of existing code that either expects and deals with NaNs itself or works accidentally because the NaNs do not cause harm.

I don't sympathize much with the latter case since they are just hidden bugs probably, but the former makes it hard to justify raising exceptions for NaNs as the default behavior.

But since I assume we're talking Python 3 here, maybe arguments containing the phase "existing code" can be dutifully ignored, I dunno.

-Casey


From raymond.hettinger at gmail.com  Sat Mar 27 00:16:36 2010
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 26 Mar 2010 16:16:36 -0700
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
Message-ID: <A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>


On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote:
>> 
> How about raising an exception instead of creating nans in the first place, except maybe within specific contexts (so that the IEEE-754 minded can get their nans working as they currently do)?

-1  

The numeric community uses NaNs as placeholders in vectorized calculations.
People do use them and there's no point in breaking their code.

Of the ideas I've seen in this thread, only two look reasonable:

* Do nothing.  This is attractive because it doesn't break anything.

* Have float.__eq__(x, y) return True whenever x and y are
   the same NaN object.  This is attractive because it is a
   minimal change that provides a little protection for 
   simple containers.

I support either of those options.


Raymond
   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100326/07695d85/attachment-0001.html>

From steve at pearwood.info  Sat Mar 27 01:05:27 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 27 Mar 2010 11:05:27 +1100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <20100326011901.6F4133A4080@sparrow.telecommunity.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
Message-ID: <201003271105.28852.steve@pearwood.info>

On Fri, 26 Mar 2010 12:19:06 pm P.J. Eby wrote:
> At 11:57 AM 3/26/2010 +1100, Steven D'Aprano wrote:
> >But they're not -- they're *signals* for "your calculation has gone
> >screwy and the result you get is garbage", so to speak. You
> >shouldn't even think of a specific NAN as a piece of specific
> >garbage, but merely a label on the *kind* of garbage you've got (the
> >payload): INF-INF is, in some sense, a different kind of error to
> >log(-1). In the same way you might say "INF-INF could be any number
> >at all, therefore we return NAN", you might say "since INF-INF could
> >be anything, there's no reason to think that INF-INF == INF-INF."
>
> So, are you suggesting that maybe the Pythonic thing to do in that
> case would be to cause any operation on a NAN (including perhaps
> comparison) to fail, rather than allowing garbage to silently
> propagate?

Certainly not. That defeats the whole purpose of NANs. I wish floating 
point calculations in Python would return NANs rather than raise the 
exceptions they do now. I can't speak for others, but in my experience 
NANs are a much nicer way to do maths-related programming. I've 
programmed with a system that supported NANs extensively (Apple's SANE, 
circa 1990), and I miss it so.

Note also that NANs do not necessarily contaminate every expression or 
function call. The standard allows for them to "cancel out", so to 
speak, where it is mathematically justifiable:

>>> nan = float('nan')
>>> 1.0**nan
1.0

so you shouldn't assume that the presence of a NAN in a calculation is 
the kiss of death.


> In other words, if NAN is only a signal that you have garbage, is
> there really any reason to keep it as an *object*, instead of simply
> raising an exception?  Then, you could at least identify what
> calculation created the garbage, instead of it percolating up through
> other calculations.

The standard distinguishes between signalling NANs and quiet NANs (which 
propagate as values). By default, signalling NANs are usually converted 
to quiet NANs, but the caller is supposed to be able to be able to 
change that behaviour to a floating point signal which can be trapped. 
In Python, the equivalent would be an exception.



-- 
Steven D'Aprano

From fuzzyman at voidspace.org.uk  Sat Mar 27 01:59:21 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 27 Mar 2010 00:59:21 +0000
Subject: [Python-Dev] Optional delta argument for assertAlmostEqual
Message-ID: <4BAD5869.6010607@voidspace.org.uk>

Hello all,

A user has suggested an optional argument to 
unittest.TestCase.assertAlmostEqual for specifying a maximum difference 
between the expected and actual values, instead of using rounding.

This sounds great to me as the default implementation of 
assertAlmostEqual has *never* been useful to me (YMMV). In fact one of 
the first things I do on setting up a test suite is provide a TestCase 
that overrides assertAlmostEqual with an implementation that uses a 
delta rather than rounding.

The implementation would be effectively:

assert abs(actual - expected) < delta

This has the advantage that it allows things like:

self.assertAlmostEqual(timeStamp, expected, 
delta=datetime.timedelta(seconds=5))

The issue is this would make the signature of assertAlmostEqual (and its 
negative counterpart):

def assertAlmostEqual(self, first, second, places=7, msg=None, delta=None)

Note that delta comes after msg, which is different to other assert 
methods. To put delta before msg would be backwards incompatible with 
existing uses passing arguments positionally. In Python 3.2 we can make 
delta a keyword argument. Passing both places and delta would be an 
error (TypeError).

Anyway, unless there are strenuous objections I intend to do this.

All the best,

Michael

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.	



From cournape at gmail.com  Sat Mar 27 06:32:07 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 27 Mar 2010 14:32:07 +0900
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
Message-ID: <5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com>

On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote:
>
> How about raising an exception instead of creating nans in the first place,
> except maybe within specific contexts (so that the IEEE-754 minded can get
> their nans working as they currently do)?
>
> -1
> The numeric community uses NaNs as placeholders in vectorized calculations.

But is this relevant to python itself ? In Numpy, we indeed do use and
support NaN, but we have much more control on what happens compared to
python float objects. We can control whether invalid operations raises
an exception or not, we had isnan/isfinite for a long time, and the
fact that nan != nan has never been a real problem AFAIK.

David

From dickinsm at gmail.com  Sat Mar 27 12:12:02 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 27 Mar 2010 11:12:02 +0000
Subject: [Python-Dev] Mixing float and Decimal -- thread reboot
In-Reply-To: <5d44f72f1003241815g109236fco1ab9607544352cf0@mail.gmail.com>
References: <ca471dc21003191450w6429269dr5d36430b6d971126@mail.gmail.com>
	<e04bdf311003191617n3bdbdee1w26988168d7877db0@mail.gmail.com>
	<20100323120959.GA486@mail.bytereef.org>
	<E8F8FF1B-D86D-4EFF-BCB9-D581DEFFB279@gmail.com>
	<4BA9FC18.90102@gmail.com> <20100324193613.GA470@mail.bytereef.org>
	<CC8AF989-04DD-4F1D-BA66-A8900F0B8299@gmail.com>
	<5c6f2a5d1003241409g806d79dw4cd245b97ce47ae9@mail.gmail.com>
	<5d44f72f1003241815g109236fco1ab9607544352cf0@mail.gmail.com>
Message-ID: <5c6f2a5d1003270412r2756f4d0ue0e2a4603166a96b@mail.gmail.com>

On Thu, Mar 25, 2010 at 1:15 AM, Jeffrey Yasskin <jyasskin at gmail.com> wrote:
> On Wed, Mar 24, 2010 at 2:09 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
>> Slight change of topic. ?I've been implementing the extra comparisons
>> required for the Decimal type and found an anomaly while testing.
>> Currently in py3k, order comparisons (but not ==, !=) between a
>> complex number and another complex, float or int raise TypeError:
>>
>>>>> z = complex(0, 0)
>>>>> z < int()
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> TypeError: unorderable types: complex() < int()
>>>>> z < float()
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> TypeError: unorderable types: complex() < float()
>>>>> z < complex()
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> TypeError: unorderable types: complex() < complex()
>>
>> But Fraction is the odd man out: ?a comparison between a Fraction and
>> a complex raises a TypeError for complex numbers with nonzero
>> imaginary component, but returns a boolean value if the complex number
>> has zero imaginary component:
>>
>>>>> z < Fraction()
>> False
>>>>> complex(0, 1) < Fraction()
>> Traceback (most recent call last):
>> ?File "<stdin>", line 1, in <module>
>> TypeError: unorderable types: complex() < Fraction()
>>
>> I'm tempted to call this Fraction behaviour a bug, but maybe it arises
>> from the numeric integration themes of PEP 3141. ?Any ideas?
>
> I'd call it a bug.
>

Thanks, Jeffrey (and everyone else who answered).  Fixed in r79456
(py3k) and r79455 (trunk).

Mark

From dickinsm at gmail.com  Sat Mar 27 13:22:20 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 27 Mar 2010 12:22:20 +0000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
Message-ID: <5c6f2a5d1003270522p2a84cce5seb4c85e824d14358@mail.gmail.com>

On Fri, Mar 26, 2010 at 11:16 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> Of the ideas I've seen in this thread, only two look reasonable:
> * Do nothing. ?This is attractive because it doesn't break anything.
> * Have float.__eq__(x, y) return True whenever x and y are
> ?? the same NaN object. ?This is attractive because it is a
> ?? minimal change that provides a little protection for
> ?? simple containers.
> I support either of those options.

Yes;  those are the only two options I've seen that seem workable.  Of
the two, I prefer the first (do nothing), but would be content with
second.

I'd be interested to know whether there's any real-life code that's
suffering as a result of nan != nan.  While the nan weirdnesses
certainly exist, I'm having difficulty imagining them turning up in
real code.

Casey Duncan's point that there can't be many good uses for floats as
dict keys or set members is compelling, though there may be
type-agnostic applications that care (e.g., memoizing).  Similarly,
putting floats into a list must be very common, but I'd guess that
checking whether a given float is in a list doesn't happen that often.
 I suspect that (nan+container)-related oddities turn up infrequently
enough to make it not worth fixing.

By the way, for those suggesting that any operation producing a nan
raise an exception instead:  Python's math module actually does go out
of its way to protect naive users from nans.  You can't get a nan out
of any of the math module functions without having put a nan in in the
first place.  Invalid operations like math.sqrt(-1), math.log(-1),
consistently raise ValueError rather than return a nan.  Ideally I'd
like to see this behaviour extended to arithmetic as well, so that
e.g., float('inf')/float('inf') raises instead of producing
float('nan') (and similarly 1e300 * 1e300 raises OverflowError instead
of producing an infinity), but there are backwards compatibility
concerns.  But even then, I'd still want it to be possible to produce
nans deliberately when necessary, e.g., by directly calling
float('nan').

Python also needs to be able to handle floating-point data generated
from other sources;  for this alone it should be at least able to read
and write infinities and nans.

Mark

From ncoghlan at gmail.com  Sat Mar 27 14:21:39 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 27 Mar 2010 23:21:39 +1000
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5c6f2a5d1003270522p2a84cce5seb4c85e824d14358@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<201003261157.39250.steve@pearwood.info>	<20100326011901.6F4133A4080@sparrow.telecommunity.com>	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
	<5c6f2a5d1003270522p2a84cce5seb4c85e824d14358@mail.gmail.com>
Message-ID: <4BAE0663.8060303@gmail.com>

Mark Dickinson wrote:
> On Fri, Mar 26, 2010 at 11:16 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>> Of the ideas I've seen in this thread, only two look reasonable:
>> * Do nothing.  This is attractive because it doesn't break anything.
>> * Have float.__eq__(x, y) return True whenever x and y are
>>    the same NaN object.  This is attractive because it is a
>>    minimal change that provides a little protection for
>>    simple containers.
>> I support either of those options.
> 
> Yes;  those are the only two options I've seen that seem workable.  Of
> the two, I prefer the first (do nothing), but would be content with
> second.

I've ended up in the same place as Mark: +1 on retaining the status quo
(possibly with better warnings about the potential oddities of floating
point values being placed in equality-based containers), +0 on changing
NaN equality to check identity first in order to provide reflexivity
under == for these two types.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From dickinsm at gmail.com  Sat Mar 27 14:45:17 2010
From: dickinsm at gmail.com (Mark Dickinson)
Date: Sat, 27 Mar 2010 13:45:17 +0000
Subject: [Python-Dev] Optional delta argument for assertAlmostEqual
In-Reply-To: <4BAD5869.6010607@voidspace.org.uk>
References: <4BAD5869.6010607@voidspace.org.uk>
Message-ID: <5c6f2a5d1003270645v5a673979s63406f7ed9c87ba7@mail.gmail.com>

On Sat, Mar 27, 2010 at 12:59 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Hello all,
>
> A user has suggested an optional argument to
> unittest.TestCase.assertAlmostEqual for specifying a maximum difference
> between the expected and actual values, instead of using rounding.

+1.

Mark

From greg at krypto.org  Sat Mar 27 18:49:22 2010
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 27 Mar 2010 10:49:22 -0700
Subject: [Python-Dev] Optional delta argument for assertAlmostEqual
In-Reply-To: <4BAD5869.6010607@voidspace.org.uk>
References: <4BAD5869.6010607@voidspace.org.uk>
Message-ID: <52dc1c821003271049t2b7056a0yf97ff0bbc0d875df@mail.gmail.com>

On Fri, Mar 26, 2010 at 5:59 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Hello all,
>
> A user has suggested an optional argument to
> unittest.TestCase.assertAlmostEqual for specifying a maximum difference
> between the expected and actual values, instead of using rounding.
>
> This sounds great to me as the default implementation of assertAlmostEqual
> has *never* been useful to me (YMMV). In fact one of the first things I do
> on setting up a test suite is provide a TestCase that overrides
> assertAlmostEqual with an implementation that uses a delta rather than
> rounding.
>
> The implementation would be effectively:
>
> assert abs(actual - expected) < delta
>
> This has the advantage that it allows things like:
>
> self.assertAlmostEqual(timeStamp, expected,
> delta=datetime.timedelta(seconds=5))
>
> The issue is this would make the signature of assertAlmostEqual (and its
> negative counterpart):
>
> def assertAlmostEqual(self, first, second, places=7, msg=None, delta=None)
>
> Note that delta comes after msg, which is different to other assert methods.
> To put delta before msg would be backwards incompatible with existing uses
> passing arguments positionally. In Python 3.2 we can make delta a keyword
> argument. Passing both places and delta would be an error (TypeError).
>
> Anyway, unless there are strenuous objections I intend to do this.

+1 sounds good to me.

>
> All the best,
>
> Michael
>
> --
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
> READ CAREFULLY. By accepting and reading this email you agree, on behalf of
> your employer, to release me from all obligations and waivers arising from
> any and all NON-NEGOTIATED agreements, licenses, terms-of-service,
> shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure,
> non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have
> entered into with your employer, its partners, licensors, agents and
> assigns, in perpetuity, without prejudice to my ongoing rights and
> privileges. You further represent that you have the authority to release me
> from any BOGUS AGREEMENTS on behalf of your employer.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>

From rhamph at gmail.com  Sat Mar 27 19:36:27 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Sat, 27 Mar 2010 12:36:27 -0600
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
Message-ID: <aac2c7cb1003271136g5189bef3o687ed7c52333031e@mail.gmail.com>

On Fri, Mar 26, 2010 at 17:16, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> Of the ideas I've seen in this thread, only two look reasonable:
> * Do nothing. ?This is attractive because it doesn't break anything.
> * Have float.__eq__(x, y) return True whenever x and y are
> ?? the same NaN object. ?This is attractive because it is a
> ?? minimal change that provides a little protection for
> ?? simple containers.
> I support either of those options.

What's the flaw in using isnan()?


-- 
Adam Olsen, aka Rhamphoryncus

From robert.kern at gmail.com  Sun Mar 28 01:28:03 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 27 Mar 2010 19:28:03 -0500
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<201003261157.39250.steve@pearwood.info>	<20100326011901.6F4133A4080@sparrow.telecommunity.com>	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
	<5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com>
Message-ID: <hom7rh$a75$4@dough.gmane.org>

On 2010-03-27 00:32 , David Cournapeau wrote:
> On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger
> <raymond.hettinger at gmail.com>  wrote:
>>
>> On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote:
>>
>> How about raising an exception instead of creating nans in the first place,
>> except maybe within specific contexts (so that the IEEE-754 minded can get
>> their nans working as they currently do)?
>>
>> -1
>> The numeric community uses NaNs as placeholders in vectorized calculations.
>
> But is this relevant to python itself ? In Numpy, we indeed do use and
> support NaN, but we have much more control on what happens compared to
> python float objects. We can control whether invalid operations raises
> an exception or not, we had isnan/isfinite for a long time, and the
> fact that nan != nan has never been a real problem AFAIK.

Nonetheless, the closer our float arrays are to Python's float type, the happier 
I will be.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robert.kern at gmail.com  Sun Mar 28 01:27:57 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Sat, 27 Mar 2010 19:27:57 -0500
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <aac2c7cb1003271136g5189bef3o687ed7c52333031e@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<201003261157.39250.steve@pearwood.info>	<20100326011901.6F4133A4080@sparrow.telecommunity.com>	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
	<aac2c7cb1003271136g5189bef3o687ed7c52333031e@mail.gmail.com>
Message-ID: <hom7ra$a75$3@dough.gmane.org>

On 2010-03-27 13:36 , Adam Olsen wrote:
> On Fri, Mar 26, 2010 at 17:16, Raymond Hettinger
> <raymond.hettinger at gmail.com>  wrote:
>> Of the ideas I've seen in this thread, only two look reasonable:
>> * Do nothing.  This is attractive because it doesn't break anything.
>> * Have float.__eq__(x, y) return True whenever x and y are
>>     the same NaN object.  This is attractive because it is a
>>     minimal change that provides a little protection for
>>     simple containers.
>> I support either of those options.
>
> What's the flaw in using isnan()?

There are implicit comparisons being done inside list.__contains__() and other 
such methods. They do not, and should not, know about isnan().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From charles.r.mccreary at gmail.com  Sun Mar 28 05:54:19 2010
From: charles.r.mccreary at gmail.com (Charles McCreary)
Date: Sat, 27 Mar 2010 22:54:19 -0500
Subject: [Python-Dev] Optional delta argument for assertAlmostEqual
In-Reply-To: <4BAD5869.6010607@voidspace.org.uk>
References: <4BAD5869.6010607@voidspace.org.uk>
Message-ID: <8294397f1003272054l21c659e1jdbb02191e86d3f09@mail.gmail.com>

Perhaps not just absolute but relative tolerance, e.g.:

def isclose(x, y, rtol=1.e-5, atol=1.e-8):
    return abs(x-y) <= atol + rtol * abs(y)

On Fri, Mar 26, 2010 at 7:59 PM, Michael Foord <fuzzyman at voidspace.org.uk>wrote:

> Hello all,
>
> A user has suggested an optional argument to
> unittest.TestCase.assertAlmostEqual for specifying a maximum difference
> between the expected and actual values, instead of using rounding.
>
> This sounds great to me as the default implementation of assertAlmostEqual
> has *never* been useful to me (YMMV). In fact one of the first things I do
> on setting up a test suite is provide a TestCase that overrides
> assertAlmostEqual with an implementation that uses a delta rather than
> rounding.
>
> The implementation would be effectively:
>
> assert abs(actual - expected) < delta
>
> This has the advantage that it allows things like:
>
> self.assertAlmostEqual(timeStamp, expected,
> delta=datetime.timedelta(seconds=5))
>
> The issue is this would make the signature of assertAlmostEqual (and its
> negative counterpart):
>
> def assertAlmostEqual(self, first, second, places=7, msg=None, delta=None)
>
> Note that delta comes after msg, which is different to other assert
> methods. To put delta before msg would be backwards incompatible with
> existing uses passing arguments positionally. In Python 3.2 we can make
> delta a keyword argument. Passing both places and delta would be an error
> (TypeError).
>
> Anyway, unless there are strenuous objections I intend to do this.
>
> All the best,
>
> Michael
>
> --
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
> READ CAREFULLY. By accepting and reading this email you agree, on behalf of
> your employer, to release me from all obligations and waivers arising from
> any and all NON-NEGOTIATED agreements, licenses, terms-of-service,
> shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure,
> non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have
> entered into with your employer, its partners, licensors, agents and
> assigns, in perpetuity, without prejudice to my ongoing rights and
> privileges. You further represent that you have the authority to release me
> from any BOGUS AGREEMENTS on behalf of your employer.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/charles.r.mccreary%40gmail.com
>



-- 
Charles McCreary P.E.
CRM Engineering
903.643.3490 - office
903.224.5701 - mobile/GV
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100327/ad5846a3/attachment.html>

From rhamph at gmail.com  Sun Mar 28 08:32:46 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Sun, 28 Mar 2010 00:32:46 -0600
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <hom7ra$a75$3@dough.gmane.org>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
	<aac2c7cb1003271136g5189bef3o687ed7c52333031e@mail.gmail.com>
	<hom7ra$a75$3@dough.gmane.org>
Message-ID: <aac2c7cb1003272332q4b8a0a6ag41d48e8a3bb444d2@mail.gmail.com>

On Sat, Mar 27, 2010 at 18:27, Robert Kern <robert.kern at gmail.com> wrote:
> On 2010-03-27 13:36 , Adam Olsen wrote:
>> What's the flaw in using isnan()?
>
> There are implicit comparisons being done inside list.__contains__() and
> other such methods. They do not, and should not, know about isnan().

Those methods should raise an exception.  Conceptually, NaN should
contaminate the result and make list.__contains__() return some
"unsortable", but we don't want to bend the whole language backwards
just for one obscure feature, especially when we have a much better
approach most of the time (exceptions).

The reason why NaN's current behaviour is so disturbing is that it
increases the mental load of everybody dealing with floats.  When you
write new code or debug a program you have to ask yourself what might
happen if a NaN is produced.  When maintaining existing code you have
to figure out if it's written a specific way to get NaN to work right,
or if it's even a fluke that NaN's work right, even if it was never
intended for NaNs or never sees them on developer machines.  This is
all the subtlety we work so hard to avoid normally, so why make an
exception here?  NaNs themselves have use cases, but their subtlety
doesn't.


-- 
Adam Olsen, aka Rhamphoryncus

From steve at pearwood.info  Sun Mar 28 09:06:06 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 28 Mar 2010 18:06:06 +1100
Subject: [Python-Dev] Optional delta argument for assertAlmostEqual
In-Reply-To: <8294397f1003272054l21c659e1jdbb02191e86d3f09@mail.gmail.com>
References: <4BAD5869.6010607@voidspace.org.uk>
	<8294397f1003272054l21c659e1jdbb02191e86d3f09@mail.gmail.com>
Message-ID: <201003281806.07478.steve@pearwood.info>

On Sun, 28 Mar 2010 02:54:19 pm Charles McCreary wrote:
> Perhaps not just absolute but relative tolerance, e.g.:
>
> def isclose(x, y, rtol=1.e-5, atol=1.e-8):
>     return abs(x-y) <= atol + rtol * abs(y)

I'm not sure why you add the tolerances like that. Surely a more 
appropriate approach is to compare the values against the tolerances 
individually, and return True if they meet either condition.

For what it's worth, here's a recipe I wrote for approximately equal, 
following someone's complaint on comp.lang.python that Python has no 
approximately-equal operator.

http://code.activestate.com/recipes/577124-approximately-equal/

Comments and criticism welcome.

I've never found unittest.TestCase.assertAlmostEqual to be useful. I 
generally write my own comparison function, then call:

assert_(compare(x, y))

as needed. I'm +0.5 on Michael's suggested change -- it will make 
assertAlmostEqual marginally more useful, which is a plus, but I'm not 
sure that absolute tolerances are better than relative. But how general 
do we want to be? "Almost equal" means something different to everyone.



> On Fri, Mar 26, 2010 at 7:59 PM, Michael Foord 
<fuzzyman at voidspace.org.uk>wrote:
> > Hello all,
> >
> > A user has suggested an optional argument to
> > unittest.TestCase.assertAlmostEqual for specifying a maximum
> > difference between the expected and actual values, instead of using
> > rounding.
[snip]



-- 
Steven D'Aprano

From charles.r.mccreary at gmail.com  Sun Mar 28 16:35:58 2010
From: charles.r.mccreary at gmail.com (Charles McCreary)
Date: Sun, 28 Mar 2010 09:35:58 -0500
Subject: [Python-Dev] Optional delta argument for assertAlmostEqual
In-Reply-To: <201003281806.07478.steve@pearwood.info>
References: <4BAD5869.6010607@voidspace.org.uk>
	<8294397f1003272054l21c659e1jdbb02191e86d3f09@mail.gmail.com>
	<201003281806.07478.steve@pearwood.info>
Message-ID: <8294397f1003280735x1f6acd04waf63074f6666c5fd@mail.gmail.com>

A good summary on floating point comparison algorithms can be found at

http://www.boost.org/doc/libs/1_34_0/libs/test/doc/components/test_tools/floating_point_comparison.html

I, too, redefine almostEquals so that the comparison is more robust

+1

On Sun, Mar 28, 2010 at 2:06 AM, Steven D'Aprano <steve at pearwood.info>wrote:

> On Sun, 28 Mar 2010 02:54:19 pm Charles McCreary wrote:
> > Perhaps not just absolute but relative tolerance, e.g.:
> >
> > def isclose(x, y, rtol=1.e-5, atol=1.e-8):
> >     return abs(x-y) <= atol + rtol * abs(y)
>
> I'm not sure why you add the tolerances like that. Surely a more
> appropriate approach is to compare the values against the tolerances
> individually, and return True if they meet either condition.
>
> For what it's worth, here's a recipe I wrote for approximately equal,
> following someone's complaint on comp.lang.python that Python has no
> approximately-equal operator.
>
> http://code.activestate.com/recipes/577124-approximately-equal/
>
> Comments and criticism welcome.
>
> I've never found unittest.TestCase.assertAlmostEqual to be useful. I
> generally write my own comparison function, then call:
>
> assert_(compare(x, y))
>
> as needed. I'm +0.5 on Michael's suggested change -- it will make
> assertAlmostEqual marginally more useful, which is a plus, but I'm not
> sure that absolute tolerances are better than relative. But how general
> do we want to be? "Almost equal" means something different to everyone.
>
>
>
> > On Fri, Mar 26, 2010 at 7:59 PM, Michael Foord
> <fuzzyman at voidspace.org.uk>wrote:
> > > Hello all,
> > >
> > > A user has suggested an optional argument to
> > > unittest.TestCase.assertAlmostEqual for specifying a maximum
> > > difference between the expected and actual values, instead of using
> > > rounding.
> [snip]
>
>
>
> --
> Steven D'Aprano
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/charles.r.mccreary%40gmail.com
>



-- 
Charles McCreary P.E.
CRM Engineering
903.643.3490 - office
903.224.5701 - mobile/GV
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100328/0581bad0/attachment.html>

From debatem1 at gmail.com  Sun Mar 28 20:09:39 2010
From: debatem1 at gmail.com (geremy condra)
Date: Sun, 28 Mar 2010 14:09:39 -0400
Subject: [Python-Dev] Optional delta argument for assertAlmostEqual
In-Reply-To: <4BAD5869.6010607@voidspace.org.uk>
References: <4BAD5869.6010607@voidspace.org.uk>
Message-ID: <q2sf3cc57c61003281109zb9d852d4o426b14a20fbb2905@mail.gmail.com>

On Fri, Mar 26, 2010 at 8:59 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Hello all,
>
> A user has suggested an optional argument to
> unittest.TestCase.assertAlmostEqual for specifying a maximum difference
> between the expected and actual values, instead of using rounding.
>
> This sounds great to me as the default implementation of assertAlmostEqual
> has *never* been useful to me (YMMV). In fact one of the first things I do
> on setting up a test suite is provide a TestCase that overrides
> assertAlmostEqual with an implementation that uses a delta rather than
> rounding.
>
> The implementation would be effectively:
>
> assert abs(actual - expected) < delta
>
> This has the advantage that it allows things like:
>
> self.assertAlmostEqual(timeStamp, expected,
> delta=datetime.timedelta(seconds=5))
>
> The issue is this would make the signature of assertAlmostEqual (and its
> negative counterpart):
>
> def assertAlmostEqual(self, first, second, places=7, msg=None, delta=None)
>
> Note that delta comes after msg, which is different to other assert methods.
> To put delta before msg would be backwards incompatible with existing uses
> passing arguments positionally. In Python 3.2 we can make delta a keyword
> argument. Passing both places and delta would be an error (TypeError).
>
> Anyway, unless there are strenuous objections I intend to do this.
>
> All the best,
>
> Michael

+1, this would be very helpful to me.

Geremy Condra

From larry at hastings.org  Sun Mar 28 21:25:37 2010
From: larry at hastings.org (Larry Hastings)
Date: Sun, 28 Mar 2010 12:25:37 -0700
Subject: [Python-Dev] [Python-checkins] r79397 -
 in	python/trunk:	Doc/c-api/capsule.rst
 Doc/c-api/cobject.rst	Doc/c-api/concrete.rst	Doc/data/refcounts.dat	Doc/extending/extending.rst
 Include/Python.h	Include/cStringIO.h	Include/cobject.h	Include/datetime.h
 Include/py_curses.h	Include/pycapsule.h	Include/pyexpat.h	Include/ucnhash.h
 Lib/test/test_sys.py	Makefile.pre.in	Misc/NEWS	Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c	Modules/_ctypes/ctypes.h	Modules/_cursesmodule.c
 Modules/_elementtree.c	Modules/_testcapimodule.c	Modules/cStringIO.c	Modules/cjkcodecs/cjkcodecs.h
 Modules/cjkcodecs/multibytecodec.c	Modules/cjkcodecs/multibytecodec.h	Modules/datetimemodule.c
 Modules/pyexpat.c	Modules/socketmodule.c	Modules/socketmodule.h	Modules/unicodedata.c
 Objects/capsule.c	Objects/object.c	Objects/unicodeobject.c	PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj	PC/os2emx/python27.def	PC/os2vacpp/python.def
 Python/compile.c Python/getargs.c
In-Reply-To: <4BAC7C48.3080600@egenix.com>
References: <20100325005454.B1AC5FCCD@mail.python.org>	<4BAB2882.6070508@egenix.com>
	<4BAB9FAE.1070206@hastings.org> <4BAC7C48.3080600@egenix.com>
Message-ID: <4BAFAD31.2050300@hastings.org>



Well, it's been a couple of days, and nobody has stepped forward to 
speak in favor of my aggressive PyCapsule schedule.  So I will bow to 
the wishes of those who have spoken up, and scale it back.  
Specifically, I propose writing a patch that will:

    * allow the CObject API to (unsafely) dereference a capsule
    * add support for the new files added by capsule to the Visual
      Studio builds (fixes an unrelated second complaint about my checkin)
    * add a -3 warning for calls to CObject
    * add a PendingDeprecation warning to CObject


I've gotten in the habit of doing code reviews at work, and I now 
greatly prefer working that way.  Can I get a volunteer to review the 
patch when it's ready?   Please email me directly.  Thanks!


M.-A. Lemburg wrote:
> Just as reminder of the process we have in place for such changes:
> Please discuss any major breakage on python-dev before checking in
> the patch.
>   

I'm aware this is a good idea.  I simply didn't consider this a major 
breakage.  Recompiling against the 2.7 header files fixes it for 
everybody.  (Except external users of pyexpat, if any exist.  Google 
doesn't show any, though this is not proof that they don't exist.)

If you suggest that any breakage with previous versions is worth 
mentioning, no matter how small, then I'll remember that in the future.  
Certainly the Python community has had many thrilling and dynamic 
conversations over minutae, so I guess it wouldn't be that surprising if 
this were true.


>> 1. The CObject API isn't safe.  It's easy to crash Python 2.6 in just a
>> few lines by mixing and matching CObjects.  Switching Python to capsules
>> prevents a class of exploits.  I've included a script at the bottom of
>> this message that demonstrates three such crashes.  The script runs in
>> Python 2 and 3, but 3.1 doesn't crash because it's using capsules.
>>     
>
> That's good, but then again: deliberate wrong use of APIs will
> always cause crashes and at least I don't know of any report about
> PyCObjects posing a problem in all their years of existence.
>   
> [...] we've been happy with this vulnerability for years, just
> as we've been happy with the fact that it's easy to crash the
> VM by passing hand-crafted byte-code to it, or using ctypes to
> call an OS function with the wrong parameters, etc.
>   

I'm surprised you find that acceptable.  I thought the community was in 
agreement: our goal is to make the interpreter uncrashable from pure Python.

I would describe ctypes as "impure" Python.  If you wanted to allow 
untrusted developers to run code through a Python interpreter, you'd 
harden it: take away ctypes, and the ability to feed in arbitrary 
bytecode, and reduce the stack limit.  You probably wouldn't take away 
"datetime", "cStringIO", and "cPickle"--yet I can crash 2.6 using those.


>> Third, I've been pondering writing a set of preprocessor macros, shipped
>> in their own header file distributed independently of Python and
>> released to the public domain, that would make it easy to use either
>> CObjects or capsules depending on what version of Python you were
>> compiling against.  Obviously, using these macros would require a source
>> code change in the third-party library.  But these macros would make it
>> a five-minute change.  This could compliment the first or second
>> approaches.
>>     
> I'm not sure whether that would worth the effort. 3rd party modules
> for Python 2.x are likely not going to use them, since they typically
> have to support more than just Python 2.7.
>   

The point of these macros would be to support older versions of Python.  
The macros would boil down to either PyCapsule or CObject calls, 
depending on what version of Python you were compiling against.  For 
Python 2.6 and before it would alway use CObjects.  One set of macros 
would switch to PyCapsule when building against 2.7; another would use 
CObject permanently in 2.x.  (Or maybe it should switch in 2.8?  2.9?  
But now we're in the realm of speculative fiction.)

I'd be happy to write these if there was demand, but my sneaking 
suspicion is that nobody would use them.  If you're genuinely interested 
in these macros please email me privately.


/larry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100328/d888c7a6/attachment-0001.html>

From steve at pearwood.info  Sun Mar 28 23:17:25 2010
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 29 Mar 2010 08:17:25 +1100
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <aac2c7cb1003272332q4b8a0a6ag41d48e8a3bb444d2@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<hom7ra$a75$3@dough.gmane.org>
	<aac2c7cb1003272332q4b8a0a6ag41d48e8a3bb444d2@mail.gmail.com>
Message-ID: <201003290817.25870.steve@pearwood.info>

On Sun, 28 Mar 2010 05:32:46 pm Adam Olsen wrote:
> On Sat, Mar 27, 2010 at 18:27, Robert Kern <robert.kern at gmail.com> 
wrote:
> > On 2010-03-27 13:36 , Adam Olsen wrote:
> >> What's the flaw in using isnan()?
> >
> > There are implicit comparisons being done inside
> > list.__contains__() and other such methods. They do not, and should
> > not, know about isnan().
>
> Those methods should raise an exception.  Conceptually, NaN should
> contaminate the result and make list.__contains__() return some
> "unsortable", but we don't want to bend the whole language backwards
> just for one obscure feature, especially when we have a much better
> approach most of the time (exceptions).

I disagree -- if I ask:

3.0 in [1.0, 2.0, float('nan'), 3.0]

I should get True, not an exception. Comparing NANs for equality isn't 
an error.

+1 on leaving the behaviour alone -- the surprising behaviour people 
have pointed out with NANs in lists, dicts and sets occurs more often 
in theory than in practice. 



-- 
Steven D'Aprano

From greg.ewing at canterbury.ac.nz  Mon Mar 29 01:55:55 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 29 Mar 2010 11:55:55 +1200
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <201003290817.25870.steve@pearwood.info>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<hom7ra$a75$3@dough.gmane.org>
	<aac2c7cb1003272332q4b8a0a6ag41d48e8a3bb444d2@mail.gmail.com>
	<201003290817.25870.steve@pearwood.info>
Message-ID: <4BAFEC8B.8020009@canterbury.ac.nz>

Steven D'Aprano wrote:

> I disagree -- if I ask:
> 
> 3.0 in [1.0, 2.0, float('nan'), 3.0]
> 
> I should get True, not an exception.

Yes, I don't think anyone would disagree that NaN should compare
unequal to anything that isn't a NaN. Problems only arise when
comparing two NaNs.

-- 
Greg


From rnk at mit.edu  Mon Mar 29 04:13:23 2010
From: rnk at mit.edu (Reid Kleckner)
Date: Sun, 28 Mar 2010 22:13:23 -0400
Subject: [Python-Dev] [Python-checkins] r79397 - in python/trunk:
	Doc/c-api/capsule.rst Doc/c-api/cobject.rst
	Doc/c-api/concrete.rst Doc/data/refcounts.dat
	Doc/extending/extending.rst Include/Python.h
	Include/cStringIO.h Include/cobject.h Include/datetime
Message-ID: <9a9942201003281913u77f7fa31v6ec3b04cfeea5112@mail.gmail.com>

On Sun, Mar 28, 2010 at 3:25 PM, Larry Hastings <larry at hastings.org> wrote:
> M.-A. Lemburg wrote:
>
> Just as reminder of the process we have in place for such changes:
> Please discuss any major breakage on python-dev before checking in
> the patch.
>
>
> I'm aware this is a good idea.? I simply didn't consider this a major breakage.? Recompiling against the 2.7 header files fixes it for everybody.? (Except external users of pyexpat, if any exist.? Google doesn't show any, though this is not proof that they don't exist.)
>
> If you suggest that any breakage with previous versions is worth mentioning, no matter how small, then I'll remember that in the future.? Certainly the Python community has had many thrilling and dynamic conversations over minutae, so I guess it wouldn't be that surprising if this were true.

I'm curious what is considered reasonable/unreasonable breakage.  If
this breakage just requires a recompile, then doesn't it just
introduce an ABI incompatibility?  Aren't those allowed every minor
(point) release?  Or do people believe that this is more than an ABI
change?

Reid

From rhamph at gmail.com  Mon Mar 29 07:04:33 2010
From: rhamph at gmail.com (Adam Olsen)
Date: Sun, 28 Mar 2010 23:04:33 -0600
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <4BAFEC8B.8020009@canterbury.ac.nz>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<hom7ra$a75$3@dough.gmane.org>
	<aac2c7cb1003272332q4b8a0a6ag41d48e8a3bb444d2@mail.gmail.com>
	<201003290817.25870.steve@pearwood.info>
	<4BAFEC8B.8020009@canterbury.ac.nz>
Message-ID: <aac2c7cb1003282204s18cfa72dpe3fa82465e41d88c@mail.gmail.com>

On Sun, Mar 28, 2010 at 17:55, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Steven D'Aprano wrote:
>
>> I disagree -- if I ask:
>>
>> 3.0 in [1.0, 2.0, float('nan'), 3.0]
>>
>> I should get True, not an exception.
>
> Yes, I don't think anyone would disagree that NaN should compare
> unequal to anything that isn't a NaN. Problems only arise when
> comparing two NaNs.

NaN includes real numbers.  Although a NaN is originally produced for
results that are not real numbers, further operations could produce a
real number; we'd never know as NaN has no precision.  Extending with
complex numbers instead gives enough precision to show how this can
happen.


-- 
Adam Olsen, aka Rhamphoryncus

From techtonik at gmail.com  Mon Mar 29 07:55:32 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 08:55:32 +0300
Subject: [Python-Dev] Request for commit access
In-Reply-To: <4BA94202.6060502@v.loewis.de>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>
	<20100323152748.50ECE1FD33F@kimball.webabinitio.net>
	<4BA94202.6060502@v.loewis.de>
Message-ID: <d34314101003282255s69d7e9fbga61dec255462aa9f@mail.gmail.com>

On Wed, Mar 24, 2010 at 12:34 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>
> Procedurally, I wonder where people got the notion from that you can or
> need to apply for commit access. IIUC, it used to be the case that you
> would be recommended for commit access, by some (more or less senior)
> fellow committer. That person then would work on actually getting commit
> access to the new committer - perhaps by first asking other people in
> private, to avoid any public embarrassment if access is ultimately
> denied. IMO, that committer should then also mentor the new guy, both by
> helping out in difficult cases, and by closely following commits to see
> whether (possibly unstated) conventions are being followed.

That's an ideal case, but it doesn't work, because more or less senior
committers are already too busy. If they do not even have time to
review issues, followup on patches  - how can they monitor who reached
the appropriate karma level?
-- 
anatoly t.

From cournape at gmail.com  Mon Mar 29 08:17:34 2010
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 29 Mar 2010 15:17:34 +0900
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <hom7rh$a75$4@dough.gmane.org>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>
	<201003261157.39250.steve@pearwood.info>
	<20100326011901.6F4133A4080@sparrow.telecommunity.com>
	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>
	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>
	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>
	<5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com>
	<hom7rh$a75$4@dough.gmane.org>
Message-ID: <5b8d13221003282317h32b2807bm467df310fe56b285@mail.gmail.com>

On Sun, Mar 28, 2010 at 9:28 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On 2010-03-27 00:32 , David Cournapeau wrote:
>>
>> On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger
>> <raymond.hettinger at gmail.com> ?wrote:
>>>
>>> On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote:
>>>
>>> How about raising an exception instead of creating nans in the first
>>> place,
>>> except maybe within specific contexts (so that the IEEE-754 minded can
>>> get
>>> their nans working as they currently do)?
>>>
>>> -1
>>> The numeric community uses NaNs as placeholders in vectorized
>>> calculations.
>>
>> But is this relevant to python itself ? In Numpy, we indeed do use and
>> support NaN, but we have much more control on what happens compared to
>> python float objects. We can control whether invalid operations raises
>> an exception or not, we had isnan/isfinite for a long time, and the
>> fact that nan != nan has never been a real problem AFAIK.
>
> Nonetheless, the closer our float arrays are to Python's float type, the
> happier I will be.

Me too, but I don't see how to reconcile this with the intent of
simplifying nan handling because they are not intuitive, which seems
to be the goal of this discussion.

David

From martin at v.loewis.de  Mon Mar 29 08:19:01 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 29 Mar 2010 08:19:01 +0200
Subject: [Python-Dev] Request for commit access
In-Reply-To: <d34314101003282255s69d7e9fbga61dec255462aa9f@mail.gmail.com>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>	
	<20100323152748.50ECE1FD33F@kimball.webabinitio.net>	
	<4BA94202.6060502@v.loewis.de>
	<d34314101003282255s69d7e9fbga61dec255462aa9f@mail.gmail.com>
Message-ID: <4BB04655.6060008@v.loewis.de>

> That's an ideal case, but it doesn't work, because more or less senior
> committers are already too busy. If they do not even have time to
> review issues, followup on patches  - how can they monitor who reached
> the appropriate karma level?

The practice proves to be different. In the recent additions, the new
developer had been approached by some committer, suggesting that they
apply. So the initiative actually started from a committer who thought
that the karma level was appropriate. All I'm asking for is that then
this committer takes the issues in his own hands, rather than letting
the new developer ask for himself.

Regards,
Martin

From techtonik at gmail.com  Mon Mar 29 09:30:03 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 10:30:03 +0300
Subject: [Python-Dev] Bootstrap script for package management tool in Python
	2.7 (Was: Re: [Distutils] At least one package management
	tool for 2.7)
Message-ID: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>

So, there won't be any package management tool shipped with Python 2.7
and users will have to download and install `setuptools` manually as
before:

  "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python
setup.py install"


Therefore I still propose shipping bootstrap package that instruct
user how to download and install an actual package  management tool
when users tries to use it. So far I know only one stable tool -
`easy_install` - a part of `setuptools` package.

The required behavior for very basic user friendliness:
1. user installs Python 2.7
2. user issues `python -m easy_install something`
3. user gets message
'easy_install' tool is not installed on this system. To make it
available, download and install `setuptools` package from
http://pypi.python.org/pypi/setuptools/

4. the screen is paused before exit (for windows systems)

Other design notes:
1. if package tries to import `easy_install` module used for
bootstrap, it gets the same ImportException as if there were no
`easy_install` at all
2. bootstrap module is overwritten by actual package when users installs it


So, do we need a PEP for that? How else can I know if consensus is
reached? Anybody is willing to elaborate on implementation?


P.S. Please be careful to reply to relevant lists
-- 
anatoly t.



On Mon, Mar 29, 2010 at 9:37 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> 2010/3/29 anatoly techtonik <techtonik at gmail.com>:
> [..]
>> It is really hard to follow. You should at least change subjects when
>> switching topic.
>
> I was talking about the work going on and the decisions taken lately.
>
> I never change topics of threads mails when there's less than 100 mails,
> because I find it way more confusing :)
>
>>
>> So, what is the verdict? Will there be a package management tool or
>> bootstrap package for it shipped with Python 2.7 or not?
>
> As I said in the mail you've quoted, all improvements are made in
> Distutils2. So the answer is no.
> Python 2.7b1 is due in less than a week anyways, so any new
> development on this topic will happen after.
>
> Basically Python 2.7 == Python 2.6 in term of packaging.
>
> Regards
> Tarek
>
> --
> Tarek Ziad? | http://ziade.org
>

From techtonik at gmail.com  Mon Mar 29 09:40:06 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 10:40:06 +0300
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <20100319023630.GC1617@idyll.org>
References: <20100319023630.GC1617@idyll.org>
Message-ID: <d34314101003290040m101343a0t14d3584b67ab0693@mail.gmail.com>

I would vote for allowing student work on community infrastructure
tasks. Tracker, Wiki, Web site management tools are all outdated and
everybody who cares agrees that they've seen a better tools.

-- 
anatoly t.



On Fri, Mar 19, 2010 at 4:36 AM, C. Titus Brown <ctb at msu.edu> wrote:
> Hi all,
>
> once again, the PSF has been accepted as a mentoring foundation for the Google
> Summer of Code! ?This year, we're going to emphasize python 3 porting, so
> please think of projects you'd like to see tackled.
>
> Please submit ideas for projects as soon as possible, as students will be able
> to start applying soon. ?You can add them to this page:
>
> ? http://wiki.python.org/moin/SummerOfCode/2010
>
> You can subscribe to the mentors list here,
>
> ? http://mail.python.org/mailman/listinfo/soc2010-mentors
>
> and the general discussion list (students included) here:
>
> ? http://mail.python.org/mailman/listinfo/soc2010-general
>
> thanks,
> --titus
> --
> C. Titus Brown, ctb at msu.edu
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/techtonik%40gmail.com
>

From ziade.tarek at gmail.com  Mon Mar 29 09:55:08 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 29 Mar 2010 09:55:08 +0200
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
Message-ID: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com>

On Mon, Mar 29, 2010 at 9:30 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> So, there won't be any package management tool shipped with Python 2.7
> and users will have to download and install `setuptools` manually as
> before:
>
> ?"search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python
> setup.py install"
>
>
> Therefore I still propose shipping bootstrap package that instruct
> user how to download and install an actual package ?management tool
> when users tries to use it. So far I know only one stable tool -
> `easy_install` - a part of `setuptools` package.

There are other tools to install something in a vanilla Python :

- pip  (which also have an uninstall feature)
- distutils
- distribute

>
> The required behavior for very basic user friendliness:
> 1. user installs Python 2.7
> 2. user issues `python -m easy_install something`
> 3. user gets message
> 'easy_install' tool is not installed on this system. To make it
> available, download and install `setuptools` package from
> http://pypi.python.org/pypi/setuptools/

Are you thinking about something generic ?

Like, being able to call :

$ python -m ANYTHING

And get an error message saying that the ANYTHING script is not installed ?
Then have a registry somewhere to get the name of the project that
owns the ANYTHING script ?


[..]
> So, do we need a PEP for that? How else can I know if consensus is
> reached? Anybody is willing to elaborate on implementation?

I see two paths here:
1 - do you want to define a general mechanism for Python to install scripts ?
2 - are you just suggesting to have this mechanism only for Package Managers ?

In case of 2), what you would need to do is propose an extension to
PEP 376, we are currently working
on, then we can add it there once people have discussed it on
distutils-SIG, or maybe create a new PEP if the topic is too big to
fit in 376.

>
>
> P.S. Please be careful to reply to relevant lists

Yes, so if it's 2) let's keep this thread in Distutils-SIG.
Cross-posting like this makes it hard to follow.

Tarek

-- 
Tarek Ziad? | http://ziade.org

From regebro at gmail.com  Mon Mar 29 09:58:59 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 29 Mar 2010 09:58:59 +0200
Subject: [Python-Dev] [Distutils] Bootstrap script for package
	management tool in Python 2.7 (Was: Re: At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
Message-ID: <319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com>

On Mon, Mar 29, 2010 at 09:30, anatoly techtonik <techtonik at gmail.com> wrote:
> Therefore I still propose shipping bootstrap package that instruct
> user how to download and install an actual package ?management tool
> when users tries to use it. So far I know only one stable tool -
> `easy_install` - a part of `setuptools` package.

We can't ship *A* bootstrap script until there is *A* package
management tool in the Python world. Currently there are three. :)

In short: The tools that exist are not ready yet.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From techtonik at gmail.com  Mon Mar 29 11:02:10 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 12:02:10 +0300
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com>
Message-ID: <d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>

On Mon, Mar 29, 2010 at 10:55 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>>
>> Therefore I still propose shipping bootstrap package that instruct
>> user how to download and install an actual package ?management tool
>> when users tries to use it. So far I know only one stable tool -
>> `easy_install` - a part of `setuptools` package.
>
> There are other tools to install something in a vanilla Python :
>
> - pip ?(which also have an uninstall feature)
> - distutils
> - distribute

distutils is not a `package management` tool, because it doesn't know
anything even about installed packages, not saying anything about
dependencies.

`pip` and `distribute` are unknown for a vast majority of Python
users, so if you have a perspective replacement for `easy_install` -
it can be said in bootstrap package message. There is no problem with
packages that require `setuptools` either - they will require
`setuptools` as dependency anyway.

For now there are two questions:
1. Are they stable enough for the replacement of user command line
`easy_install` tool?
2. Which one is the recommended?

P.S. Should there be an accessible FAQ in addition to ML?

>>
>> The required behavior for very basic user friendliness:
>> 1. user installs Python 2.7
>> 2. user issues `python -m easy_install something`
>> 3. user gets message
>> 'easy_install' tool is not installed on this system. To make it
>> available, download and install `setuptools` package from
>> http://pypi.python.org/pypi/setuptools/
>
> Are you thinking about something generic ?
>
> Like, being able to call :
>
> $ python -m ANYTHING
>
> And get an error message saying that the ANYTHING script is not installed ?
> Then have a registry somewhere to get the name of the project that
> owns the ANYTHING script ?
>

No. To get something in 2.7 I would refrain from developing into this
direction for now.

> [..]
>> So, do we need a PEP for that? How else can I know if consensus is
>> reached? Anybody is willing to elaborate on implementation?
>
> I see two paths here:
> 1 - do you want to define a general mechanism for Python to install scripts ?
> 2 - are you just suggesting to have this mechanism only for Package Managers ?

3 - I want current "best practice" for installing Python modules with
dependencies from PyPI to be shipped with Python 2.7 by default

Answering your questions:
1. I want to have some user-friendly way for installing Python
scripts, but I am more concerned that I will miss `easy_install` in
Python 2.7
2. Bootstrap script is aimed at users. Package managers are assumed to
already have their package
at PyPI and provide install instructions that involve `easy_install`,
so nothing is required to be done on their part.

>
> In case of 2), what you would need to do is propose an extension to
> PEP 376, we are currently working
> on, then we can add it there once people have discussed it on
> distutils-SIG, or maybe create a new PEP if the topic is too big to
> fit in 376.

PEP 376 is completely irrelevant to user side boot package proposal.
This proposal is to recommend whatever package managing tool you think
user need and instruct how to get the tool. The boot package doesn't
know anything at all about how packages are managed.


>> P.S. Please be careful to reply to relevant lists
>
> Yes, so if it's 2) let's keep this thread in Distutils-SIG.
> Cross-posting like this makes it hard to follow.

My vision is that decision about having bootstrap package or not in
2.7 should be in python-dev and specific packaging, implementation and
pip/distutils/distribute questions in distutils-sig.

-- 
anatoly t.

From ziade.tarek at gmail.com  Mon Mar 29 11:15:06 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 29 Mar 2010 11:15:06 +0200
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com>
	<d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>
Message-ID: <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com>

On Mon, Mar 29, 2010 at 11:02 AM, anatoly techtonik <techtonik at gmail.com> wrote:
[..]
> distutils is not a `package management` tool, because it doesn't know
> anything even about installed packages, not saying anything about
> dependencies.

At this point, no one knows anything about installed packages at the
Python level.
Keeping track of installed projects is a feature done within each
package managment system.

And the whole purpose of PEP 376 is to define a database of what's
installed, for the sake of interoperability.

>
> `pip` and `distribute` are unknown for a vast majority of Python
> users, so if you have a perspective replacement for `easy_install` -

Depending on how you call a Python user, I disagree here. Many people
use pip and distribute.

The first one because it has an uninstall feature among other things.
The second one because it fixes some bugs of setuptools and provides
Python 3 support


>
> For now there are two questions:
> 1. Are they stable enough for the replacement of user command line
> `easy_install` tool?
> 2. Which one is the recommended?
>
> P.S. Should there be an accessible FAQ in addition to ML?

This FAQ work has been started in th "HitchHicker's guide to
Packaging" you can find here:

http://guide.python-distribute.org

[..]
> No. To get something in 2.7 I would refrain from developing into this
> direction for now.
>
>> [..]
>>> So, do we need a PEP for that? How else can I know if consensus is
>>> reached? Anybody is willing to elaborate on implementation?
>>
>> I see two paths here:
>> 1 - do you want to define a general mechanism for Python to install scripts ?
>> 2 - are you just suggesting to have this mechanism only for Package Managers ?
>
> 3 - I want current "best practice" for installing Python modules with
> dependencies from PyPI to be shipped with Python 2.7 by default

Again, any new code work will not happen because 2.7 is due in less
than a week. Things are happening in Distutils2.

Now, for the "best practice" documentation, I think the guide is the
best plce to look at.


[..]
> PEP 376 is completely irrelevant to user side boot package proposal.

It is, since it proposes an uninstall script called via -m. So having
a install script called by -m is definitely its business.


> My vision is that decision about having bootstrap package or not in
> 2.7 should be in python-dev and specific packaging, implementation and
> pip/distutils/distribute questions in distutils-sig.

If the mentioned bootstrap script is about a package managment system,
you should keep the discussion in distutils-SIG I think.

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From ziade.tarek at gmail.com  Mon Mar 29 11:19:39 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 29 Mar 2010 11:19:39 +0200
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com>
	<d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>
	<94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com>
Message-ID: <94bdd2611003290219h11f01c27sf09cd085ea20f7a8@mail.gmail.com>

On Mon, Mar 29, 2010 at 11:15 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
[..]
> Depending on how you call a Python user, I disagree here. Many people
> use pip and distribute.

s/how/who     :)

From techtonik at gmail.com  Mon Mar 29 11:24:56 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 12:24:56 +0300
Subject: [Python-Dev] [Distutils] Bootstrap script for package
	management tool in Python 2.7 (Was: Re: At least one package
	management tool for 2.7)
In-Reply-To: <319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com>
Message-ID: <d34314101003290224ne71ebd5q40ba60b1b86a17d2@mail.gmail.com>

2010/3/29 Lennart Regebro <regebro at gmail.com>:
> On Mon, Mar 29, 2010 at 09:30, anatoly techtonik <techtonik at gmail.com> wrote:
>> Therefore I still propose shipping bootstrap package that instruct
>> user how to download and install an actual package ?management tool
>> when users tries to use it. So far I know only one stable tool -
>> `easy_install` - a part of `setuptools` package.
>
> We can't ship *A* bootstrap script until there is *A* package
> management tool in the Python world. Currently there are three. :)
>
> In short: The tools that exist are not ready yet.

Ok. How about shipping bootstrap script only for `easy_install` tool for now?

-- 
anatoly t.

From ziade.tarek at gmail.com  Mon Mar 29 11:38:46 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 29 Mar 2010 11:38:46 +0200
Subject: [Python-Dev] [Distutils] Bootstrap script for package
	management tool in Python 2.7 (Was: Re: At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290224ne71ebd5q40ba60b1b86a17d2@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com>
	<d34314101003290224ne71ebd5q40ba60b1b86a17d2@mail.gmail.com>
Message-ID: <94bdd2611003290238x6455c0d0xefa9afb2d2620064@mail.gmail.com>

2010/3/29 anatoly techtonik <techtonik at gmail.com>:
> 2010/3/29 Lennart Regebro <regebro at gmail.com>:
>> On Mon, Mar 29, 2010 at 09:30, anatoly techtonik <techtonik at gmail.com> wrote:
>>> Therefore I still propose shipping bootstrap package that instruct
>>> user how to download and install an actual package ?management tool
>>> when users tries to use it. So far I know only one stable tool -
>>> `easy_install` - a part of `setuptools` package.
>>
>> We can't ship *A* bootstrap script until there is *A* package
>> management tool in the Python world. Currently there are three. :)
>>
>> In short: The tools that exist are not ready yet.
>
> Ok. How about shipping bootstrap script only for `easy_install` tool for now?

-1

I don't see how this will improve the current state. It will make it
worse since the future of packaging for the stldib is being built in
distutils2, and also in Pip.

People that want to use easy_install can install it. It's very simple
and doesn't require more work than what you have described through a
-m call.

The instructions you want to add in that bootstrap script to inform
people belong to the documentation imho. And that's one of the goal of
the packaging guide we are building.

Another suggestion could be to add in the Python installer a notice
about the guide, saying :

"Hey, we are working on packaging right now. Python currently doesn't
ship with a full-featured package manager, you can have a look in this
guide though, for guidance"

I still think you should remove python-dev from the cc list  ;)

Tarek

-- 
Tarek Ziad? | http://ziade.org

From ncoghlan at gmail.com  Mon Mar 29 13:56:27 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Mar 2010 21:56:27 +1000
Subject: [Python-Dev] Request for commit access
In-Reply-To: <4BB04655.6060008@v.loewis.de>
References: <cf9f31f21003230750h48cde42nc9d6a69318c30955@mail.gmail.com>		<20100323152748.50ECE1FD33F@kimball.webabinitio.net>		<4BA94202.6060502@v.loewis.de>	<d34314101003282255s69d7e9fbga61dec255462aa9f@mail.gmail.com>
	<4BB04655.6060008@v.loewis.de>
Message-ID: <4BB0956B.5030904@gmail.com>

Martin v. L?wis wrote:
>> That's an ideal case, but it doesn't work, because more or less senior
>> committers are already too busy. If they do not even have time to
>> review issues, followup on patches  - how can they monitor who reached
>> the appropriate karma level?
> 
> The practice proves to be different. In the recent additions, the new
> developer had been approached by some committer, suggesting that they
> apply. So the initiative actually started from a committer who thought
> that the karma level was appropriate. All I'm asking for is that then
> this committer takes the issues in his own hands, rather than letting
> the new developer ask for himself.

Agreed, since there is a bit of a commitment on the sponsor's part to
help coach the new committer in the early days (most new committers
don't need much coaching, but it's still good practice to have a
specific existing committer looking at their work for a while, in
addition to those of us that just try to review everything that goes by
on the checkin list).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon Mar 29 13:58:41 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Mar 2010 21:58:41 +1000
Subject: [Python-Dev] [Python-checkins] r79397 -
 in	python/trunk:	Doc/c-api/capsule.rst
 Doc/c-api/cobject.rst	Doc/c-api/concrete.rst	Doc/data/refcounts.dat	Doc/extending/extending.rst
 Include/Python.h	Include/cStringIO.h	Include/cobject.h	Include/datetime.h
 Include/py_curses.h	Include/pycapsule.h	Include/pyexpat.h	Include/ucnhash.h
 Lib/test/test_sys.py	Makefile.pre.in	Misc/NEWS	Modules/_ctypes/callproc.c
 Modules/_ctypes/cfield.c	Modules/_ctypes/ctypes.h	Modules/_cursesmodule.c
 Modules/_elementtree.c	Modules/_testcapimodule.c	Modules/cStringIO.c	Modules/cjkcodecs/cjkcodecs.h
 Modules/cjkcodecs/multibytecodec.c	Modules/cjkcodecs/multibytecodec.h	Modules/datetimemodule.c
 Modules/pyexpat.c	Modules/socketmodule.c	Modules/socketmodule.h	Modules/unicodedata.c
 Objects/capsule.c	Objects/object.c	Objects/unicodeobject.c	PC/VS7.1/pythoncore.vcproj
 PC/VS8.0/pythoncore.vcproj	PC/os2emx/python27.def	PC/os2vacpp/python.def
 Python/compile.c Python/getargs.c
In-Reply-To: <4BAFAD31.2050300@hastings.org>
References: <20100325005454.B1AC5FCCD@mail.python.org>	<4BAB2882.6070508@egenix.com>	<4BAB9FAE.1070206@hastings.org>
	<4BAC7C48.3080600@egenix.com> <4BAFAD31.2050300@hastings.org>
Message-ID: <4BB095F1.5090005@gmail.com>

Larry Hastings wrote:
>     * allow the CObject API to (unsafely) dereference a capsule
>     * add support for the new files added by capsule to the Visual
>       Studio builds (fixes an unrelated second complaint about my checkin)
>     * add a -3 warning for calls to CObject
>     * add a PendingDeprecation warning to CObject

The last item probably isn't necessary given the -3 warning.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Mon Mar 29 14:10:52 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 29 Mar 2010 22:10:52 +1000
Subject: [Python-Dev] Bootstrap script for package management tool in
 Python	2.7 (Was: Re: [Distutils] At least one package management	tool for
 2.7)
In-Reply-To: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
Message-ID: <4BB098CC.1000708@gmail.com>

anatoly techtonik wrote:
> So, there won't be any package management tool shipped with Python 2.7
> and users will have to download and install `setuptools` manually as
> before:

Until the discussed package management tools support a robust inventory
and uninstallation system that plays well with directly installed Python
packages, you won't find widespread support on python-dev for endorsing
any of them.

Yes, the people who use them love them for good and valid reasons, but
those who absolutely detest them also do so for good and valid reasons.
While this is still the case, it would be highly inappropriate for
python-dev to include bootstrap scripts that direct users to these
systems, as they're all flawed in their current incarnations (this isn't
the packaging systems' fault - the flaws are largely due to a lack of
supporting infrastructure in the standard library).

The distutils2 work and the various metadata PEPs that have been
approved recently are all about addressing those limitations in the
infrastructure. With those in place and flowing through the Python
package management ecosystem, bootstrapping interoperable package
management tools is likely to become a reasonable option in the future.
But we aren't there yet, and won't be for 2.7 or 3.2. From an outsider's
perspective, the 3.3 time frame appears to be very possible though.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ctb at msu.edu  Mon Mar 29 14:30:40 2010
From: ctb at msu.edu (C. Titus Brown)
Date: Mon, 29 Mar 2010 05:30:40 -0700
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <d34314101003290040m101343a0t14d3584b67ab0693@mail.gmail.com>
References: <20100319023630.GC1617@idyll.org>
	<d34314101003290040m101343a0t14d3584b67ab0693@mail.gmail.com>
Message-ID: <20100329123039.GA23643@idyll.org>

On Mon, Mar 29, 2010 at 10:40:06AM +0300, anatoly techtonik wrote:
> I would vote for allowing student work on community infrastructure
> tasks. Tracker, Wiki, Web site management tools are all outdated and
> everybody who cares agrees that they've seen a better tools.

As long as it's programming, it's allowed by Google.  So let's find a good
student or two, and outline a few good projects!

I do worry that that kind of work is difficult to evaluate, and requires really
great communication on both sides...

--titus

From techtonik at gmail.com  Mon Mar 29 15:45:40 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 16:45:40 +0300
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com>
	<d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>
	<94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com>
Message-ID: <d34314101003290645if3235c4g9bc068165d0eac05@mail.gmail.com>

On Mon, Mar 29, 2010 at 12:15 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> [..]
>> distutils is not a `package management` tool, because it doesn't know
>> anything even about installed packages, not saying anything about
>> dependencies.
>
> At this point, no one knows anything about installed packages at the
> Python level.

Users do not care about this, and `distutils` doesn't know this even
at package level.

> Keeping track of installed projects is a feature done within each
> package managment system.
>
> And the whole purpose of PEP 376 is to define a database of what's
> installed, for the sake of interoperability.

That's great. When it will be ready everybody would be happy to make
their package management tool compliant.

>>
>> `pip` and `distribute` are unknown for a vast majority of Python
>> users, so if you have a perspective replacement for `easy_install` -
>
> Depending on how you call a Python user, I disagree here. Many people
> use pip and distribute.
>
> The first one because it has an uninstall feature among other things.
> The second one because it fixes some bugs of setuptools and provides
> Python 3 support

I do not mind if we can distribute three stubs, they will also serve
as pointers for a better way of packaging when an ultimate tool is
finally born. Me personally is willing to elaborate for `easy_install`
stub in 2.7.

>>
>> For now there are two questions:
>> 1. Are they stable enough for the replacement of user command line
>> `easy_install` tool?
>> 2. Which one is the recommended?
>>
>> P.S. Should there be an accessible FAQ in addition to ML?
>
> This FAQ work has been started in th "HitchHicker's guide to
> Packaging" you can find here:
>
> http://guide.python-distribute.org

I can see any FAQ. To me the FAQ is something that could be posted to
distutils ML once a month to reflect current state of packaging. It
should also carry version number. So anybody can comment on the FAQ,
ask another question or ask to make a change.

> Again, any new code work will not happen because 2.7 is due in less
> than a week. Things are happening in Distutils2.

That doesn't solve the problem. Bootstrap script can be written in one
day. What we need is a consensus whatever this script is welcomed in
2.7 or not? Who is the person to make the decision?

> Now, for the "best practice" documentation, I think the guide is the
> best plce to look at.

Let's refer to original user story:
"I installed Python and need a quick way to install my packages on top of it."
"I execute `easy_install something` as said in installation manual,
but nothing/error happens."

> [..]
>> PEP 376 is completely irrelevant to user side boot package proposal.
>
> It is, since it proposes an uninstall script called via -m. So having
> a install script called by -m is definitely its business.

The scope of my proposal is a bootstrap script that instructs user how
to install the package management tool of user's choice if this tool
is not yet installed on user system (this is the case with new Python
installation).  If you'll come up with a better way of package
management - you will update this bootstrap script with relevant
information in future Python releases.

Right now it is essential to get this _feature_ in Python 2.7 until
2.7 is frozen for new features. The script that shows message upon
invocation and is overwritten by the version of the package it
proposes to install. Is it hard to do?

>> My vision is that decision about having bootstrap package or not in
>> 2.7 should be in python-dev and specific packaging, implementation and
>> pip/distutils/distribute questions in distutils-sig.
>
> If the mentioned bootstrap script is about a package managment system,
> you should keep the discussion in distutils-SIG I think.

If there will be no bootstrap script in 2.7, I won't have any
motivation to continue discussion until 2.8 deadline is near. Of
course I will be disappointed, because I will have to explain
everything to 2.7 users again and again instead of letting them
execute one command and follow its instructions.
-- 
anatoly t.

From ziade.tarek at gmail.com  Mon Mar 29 15:50:18 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 29 Mar 2010 15:50:18 +0200
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290645if3235c4g9bc068165d0eac05@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com>
	<d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>
	<94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com>
	<d34314101003290645if3235c4g9bc068165d0eac05@mail.gmail.com>
Message-ID: <94bdd2611003290650x27a074b5j586fdd1f05beebe2@mail.gmail.com>

Anatoly, I am now answering only in Distutils-SIG.

On Mon, Mar 29, 2010 at 3:45 PM, anatoly techtonik <techtonik at gmail.com> wrote:
[..]

From rrr at ronadam.com  Mon Mar 29 15:52:39 2010
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 29 Mar 2010 08:52:39 -0500
Subject: [Python-Dev] Bootstrap script for package management tool in
 Python 2.7 (Was: Re: At least one package management tool for 2.7)
In-Reply-To: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
Message-ID: <4BB0B0A7.4060405@ronadam.com>



anatoly techtonik wrote:
> So, there won't be any package management tool shipped with Python 2.7
> and users will have to download and install `setuptools` manually as
> before:
> 
>   "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python
> setup.py install"
> 
> 
> Therefore I still propose shipping bootstrap package that instruct
> user how to download and install an actual package  management tool
> when users tries to use it. So far I know only one stable tool -
> `easy_install` - a part of `setuptools` package.
> 
> The required behavior for very basic user friendliness:
> 1. user installs Python 2.7
> 2. user issues `python -m easy_install something`
> 3. user gets message
> 'easy_install' tool is not installed on this system. To make it
> available, download and install `setuptools` package from
> http://pypi.python.org/pypi/setuptools/
> 
> 4. the screen is paused before exit (for windows systems)
> 
> Other design notes:
> 1. if package tries to import `easy_install` module used for
> bootstrap, it gets the same ImportException as if there were no
> `easy_install` at all
> 2. bootstrap module is overwritten by actual package when users installs it
> 
> 
> So, do we need a PEP for that? How else can I know if consensus is
> reached? Anybody is willing to elaborate on implementation?
> 
> 
> P.S. Please be careful to reply to relevant lists

An even lighter option would be to add an item to pythons 'help' feature.

Currently help("PACKAGES") == help("import")

It may be enough at this time to add a "PACKAGES" help entry that gives an 
overview of packages and hints on installing them.  Then "import" can be a 
related help topic for "PACKAGES".

Ron








From regebro at gmail.com  Mon Mar 29 15:57:04 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 29 Mar 2010 15:57:04 +0200
Subject: [Python-Dev] [Distutils] Bootstrap script for package
	management tool in Python 2.7 (Was: Re: At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com> 
	<94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com> 
	<d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>
Message-ID: <319e029f1003290657r2a15dbc7s131b83f51f731218@mail.gmail.com>

On Mon, Mar 29, 2010 at 11:02, anatoly techtonik <techtonik at gmail.com> wrote:
> distutils is not a `package management` tool, because it doesn't know
> anything even about installed packages

With that definition, there are no packaga management tools for
Python. So it's going to be pretty hard to ship one with 2.7.

> `pip` and `distribute` are unknown for a vast majority of Python
> users, so if you have a perspective replacement for `easy_install` -
> it can be said in bootstrap package message. There is no problem with
> packages that require `setuptools` either - they will require
> `setuptools` as dependency anyway.
>
> For now there are two questions:
> 1. Are they stable enough for the replacement of user command line
> `easy_install` tool?

Yes. But that's the wrong question. The correct question is: Are they
stable enough to be included in standard library. And the answer is
"no".

> 2. Which one is the recommended?

Both.

-- 
Lennart Regebro: http://regebro.wordpress.com/
Python 3 Porting: http://python-incompatibility.googlecode.com/
+33 661 58 14 64

From cournape at gmail.com  Mon Mar 29 15:58:32 2010
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 29 Mar 2010 22:58:32 +0900
Subject: [Python-Dev] [Distutils] Bootstrap script for package
	management tool in Python 2.7 (Was: Re: At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290645if3235c4g9bc068165d0eac05@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<94bdd2611003290055k7daa167ewdf7e7226fcb2353f@mail.gmail.com>
	<d34314101003290202t702fd978q9d3fe2caf5217c6d@mail.gmail.com>
	<94bdd2611003290215s72ea681dp246505b8894695@mail.gmail.com>
	<d34314101003290645if3235c4g9bc068165d0eac05@mail.gmail.com>
Message-ID: <5b8d13221003290658j4734d46atd373b0d389367d4@mail.gmail.com>

On Mon, Mar 29, 2010 at 10:45 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> On Mon, Mar 29, 2010 at 12:15 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> [..]
>>> distutils is not a `package management` tool, because it doesn't know
>>> anything even about installed packages, not saying anything about
>>> dependencies.
>>
>> At this point, no one knows anything about installed packages at the
>> Python level.
>
> Users do not care about this, and `distutils` doesn't know this even
> at package level.
>
>> Keeping track of installed projects is a feature done within each
>> package managment system.
>>
>> And the whole purpose of PEP 376 is to define a database of what's
>> installed, for the sake of interoperability.
>
> That's great. When it will be ready everybody would be happy to make
> their package management tool compliant.
>
>>>
>>> `pip` and `distribute` are unknown for a vast majority of Python
>>> users, so if you have a perspective replacement for `easy_install` -
>>
>> Depending on how you call a Python user, I disagree here. Many people
>> use pip and distribute.
>>
>> The first one because it has an uninstall feature among other things.
>> The second one because it fixes some bugs of setuptools and provides
>> Python 3 support
>
> I do not mind if we can distribute three stubs, they will also serve
> as pointers for a better way of packaging when an ultimate tool is
> finally born. Me personally is willing to elaborate for `easy_install`
> stub in 2.7.
>
>>>
>>> For now there are two questions:
>>> 1. Are they stable enough for the replacement of user command line
>>> `easy_install` tool?
>>> 2. Which one is the recommended?
>>>
>>> P.S. Should there be an accessible FAQ in addition to ML?
>>
>> This FAQ work has been started in th "HitchHicker's guide to
>> Packaging" you can find here:
>>
>> http://guide.python-distribute.org
>
> I can see any FAQ. To me the FAQ is something that could be posted to
> distutils ML once a month to reflect current state of packaging. It
> should also carry version number. So anybody can comment on the FAQ,
> ask another question or ask to make a change.
>
>> Again, any new code work will not happen because 2.7 is due in less
>> than a week. Things are happening in Distutils2.
>
> That doesn't solve the problem. Bootstrap script can be written in one
> day. What we need is a consensus whatever this script is welcomed in
> 2.7 or not? Who is the person to make the decision?
>
>> Now, for the "best practice" documentation, I think the guide is the
>> best plce to look at.
>
> Let's refer to original user story:
> "I installed Python and need a quick way to install my packages on top of it."

python setup.py install works well, and has for almost a decade.

If you need setuptools, you can include ez_setup.py, which does
exactly what you want, without adding a hugely controversial feature
to python proper. You do something like:

try:
     import setuptools
except ImportError:
     print "Run ez_setup.py first"
     ....

And you're done,

cheers,

David

From regebro at gmail.com  Mon Mar 29 15:59:47 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 29 Mar 2010 15:59:47 +0200
Subject: [Python-Dev] [Distutils] Bootstrap script for package
	management tool in Python 2.7 (Was: Re: At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290224ne71ebd5q40ba60b1b86a17d2@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com> 
	<319e029f1003290058tf8f840bm23e454386570ea9a@mail.gmail.com> 
	<d34314101003290224ne71ebd5q40ba60b1b86a17d2@mail.gmail.com>
Message-ID: <319e029f1003290659j2243ef5dv473805c9df67b250@mail.gmail.com>

2010/3/29 anatoly techtonik <techtonik at gmail.com>:
> Ok. How about shipping bootstrap script only for `easy_install` tool for now?

Since there are many who are of the opinion that easy_install isn't
very good, and pip should be used instead, that would be a bad idea.


Just drop it. 2.7 will not have a package management tool or bootstrap
script to one, because there is currently not one that is good enough
to be the standard. People are working on making the situation good
enough, this debate is wasting their time.

The end.

-- 
Lennart Regebro: http://regebro.wordpress.com/
Python 3 Porting: http://python-incompatibility.googlecode.com/
+33 661 58 14 64

From techtonik at gmail.com  Mon Mar 29 16:05:33 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 17:05:33 +0300
Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for
	package management tool in Python 2.7)
Message-ID: <d34314101003290705k2b6d8256n8189728ee7bded58@mail.gmail.com>

On Mon, Mar 29, 2010 at 4:50 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> Anatoly, I am now answering only in Distutils-SIG.
>
> On Mon, Mar 29, 2010 at 3:45 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> [..]

Seems like I start to hate mailing lists even more with all this
message duplication and thread following nightmare. Why can't people
here create a Google Groups mirror?

-- 
anatoly t.

From solipsis at pitrou.net  Mon Mar 29 16:15:35 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 29 Mar 2010 14:15:35 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?Google_Groups_mirror_=28Was=3A_Bootstrap_s?=
	=?utf-8?q?cript_for=09package_management_tool_in_Python_2=2E7=29?=
References: <d34314101003290705k2b6d8256n8189728ee7bded58@mail.gmail.com>
Message-ID: <loom.20100329T161517-959@post.gmane.org>

anatoly techtonik <techtonik <at> gmail.com> writes:
> 
> Seems like I start to hate mailing lists even more with all this
> message duplication and thread following nightmare. Why can't people
> here create a Google Groups mirror?

There are already gmane mirrors, such as:
http://news.gmane.org/gmane.comp.python.devel


From techtonik at gmail.com  Mon Mar 29 16:21:39 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 17:21:39 +0300
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <4BB098CC.1000708@gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
	<4BB098CC.1000708@gmail.com>
Message-ID: <d34314101003290721j4e64b8dbsa9616d196a5c975@mail.gmail.com>

2010/3/29 Nick Coghlan <ncoghlan at gmail.com>:
> anatoly techtonik wrote:
>> So, there won't be any package management tool shipped with Python 2.7
>> and users will have to download and install `setuptools` manually as
>> before:
>
> Until the discussed package management tools support a robust inventory
> and uninstallation system that plays well with directly installed Python
> packages, you won't find widespread support on python-dev for endorsing
> any of them.

This wasn't the question, but the summary. I would greatly appreciate
if you could provide your feedback on the second part of my letter
that started with "Therefore..."


> Yes, the people who use them love them for good and valid reasons, but
> those who absolutely detest them also do so for good and valid reasons.
> While this is still the case, it would be highly inappropriate for
> python-dev to include bootstrap scripts that direct users to these

Scripts do not _direct_ users - they _help_ users, already directed by
some installation instruction or tutorial, to find and install package
management system they are _trying_ to use.

> The distutils2 work and the various metadata PEPs that have been
> approved recently are all about addressing those limitations in the
> infrastructure. With those in place and flowing through the Python
> package management ecosystem, bootstrapping interoperable package
> management tools is likely to become a reasonable option in the future.

Bootstrap tools evolve together with packaging situation. You may
deprecate package management tool in your future bootstrap scripts if
it is "incompatible" with this Python release. It is just user message
that is a flexible as the mind of its author.

> But we aren't there yet, and won't be for 2.7 or 3.2. From an outsider's
> perspective, the 3.3 time frame appears to be very possible though.

Just a thought about user story my customers would likely write if I
shipped Python as a product:
"As a user, I think Python is suxx, because it makes its users suffer
for a long time from packaging disorder".

-- 
anatoly t.

From regebro at gmail.com  Mon Mar 29 16:30:36 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 29 Mar 2010 16:30:36 +0200
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <d34314101003290721j4e64b8dbsa9616d196a5c975@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com> 
	<4BB098CC.1000708@gmail.com>
	<d34314101003290721j4e64b8dbsa9616d196a5c975@mail.gmail.com>
Message-ID: <319e029f1003290730y1a3d3e78s2bf07c284ff2885b@mail.gmail.com>

On Mon, Mar 29, 2010 at 16:21, anatoly techtonik <techtonik at gmail.com> wrote:
> Just a thought about user story my customers would likely write if I
> shipped Python as a product:
> "As a user, I think Python is suxx, because it makes its users suffer
> for a long time from packaging disorder".

So fix it. Python is open source. It's created by it's users. If there
is something you think it lacking, you step up and help make it
better.

But the packaging solution will not be ready for 2.7. It's a really
hard problem. Your help is appreciated. Complaints are not.

-- 
Lennart Regebro: http://regebro.wordpress.com/
Python 3 Porting: http://python-incompatibility.googlecode.com/
+33 661 58 14 64

From techtonik at gmail.com  Mon Mar 29 16:32:31 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Mon, 29 Mar 2010 17:32:31 +0300
Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for
	package management tool in Python 2.7)
In-Reply-To: <loom.20100329T161517-959@post.gmane.org>
References: <d34314101003290705k2b6d8256n8189728ee7bded58@mail.gmail.com>
	<loom.20100329T161517-959@post.gmane.org>
Message-ID: <d34314101003290732m6b505e26m82b56c75510faf00@mail.gmail.com>

On Mon, Mar 29, 2010 at 5:15 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> anatoly techtonik <techtonik <at> gmail.com> writes:
>>
>> Seems like I start to hate mailing lists even more with all this
>> message duplication and thread following nightmare. Why can't people
>> here create a Google Groups mirror?
>
> There are already gmane mirrors, such as:
> http://news.gmane.org/gmane.comp.python.devel

Do you use it yourself for thread subscription?

My point is that gmane mouse clicking interface is insane. They can
not fold quotations. And if you'll look attentively you won't find
most of messages from ongoing discussion about bootstrap script for
package management.
-- 
anatoly t.

From solipsis at pitrou.net  Mon Mar 29 16:43:35 2010
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 29 Mar 2010 16:43:35 +0200
Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for
 package management tool in Python 2.7)
In-Reply-To: <d34314101003290732m6b505e26m82b56c75510faf00@mail.gmail.com>
References: <d34314101003290705k2b6d8256n8189728ee7bded58@mail.gmail.com>
	<loom.20100329T161517-959@post.gmane.org>
	<d34314101003290732m6b505e26m82b56c75510faf00@mail.gmail.com>
Message-ID: <1269873815.3451.4.camel@localhost>


> > There are already gmane mirrors, such as:
> > http://news.gmane.org/gmane.comp.python.devel
> 
> Do you use it yourself for thread subscription?

I don't think there's any "thread subscription", if you mean subscribing
to individual threads.

> My point is that gmane mouse clicking interface is insane. They can
> not fold quotations.

Neither can some e-mail clients. It's up to posters to quote (that is,
strip quotes) properly.
Gmane also allows you to use NNTP, though, perhaps you can find an NNTP
client which has the desired quoting UI :)

> And if you'll look attentively you won't find
> most of messages from ongoing discussion about bootstrap script for
> package management.

Yes, gmane seems to have some strange "duplicate suppression" algorithm
which doesn't go well with cross-posting.




From robert.kern at gmail.com  Mon Mar 29 16:41:58 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 29 Mar 2010 09:41:58 -0500
Subject: [Python-Dev] Why is nan != nan?
In-Reply-To: <5b8d13221003282317h32b2807bm467df310fe56b285@mail.gmail.com>
References: <5c6f2a5d1003241226y198bc036le862cbd0a4e886e4@mail.gmail.com>	<201003261157.39250.steve@pearwood.info>	<20100326011901.6F4133A4080@sparrow.telecommunity.com>	<AEC73B60-9594-492C-9C6F-0159FA3380E8@pandora.com>	<15ABBE39-E603-4F0C-A33D-9D61466A344D@masklinn.net>	<A029297B-C678-45C3-9A74-2337642B39A5@gmail.com>	<5b8d13221003262232m11c30a9ata3b17410f3d7c53@mail.gmail.com>	<hom7rh$a75$4@dough.gmane.org>
	<5b8d13221003282317h32b2807bm467df310fe56b285@mail.gmail.com>
Message-ID: <hoqe7m$4pg$1@dough.gmane.org>

On 2010-03-29 01:17 AM, David Cournapeau wrote:
> On Sun, Mar 28, 2010 at 9:28 AM, Robert Kern<robert.kern at gmail.com>  wrote:
>> On 2010-03-27 00:32 , David Cournapeau wrote:
>>>
>>> On Sat, Mar 27, 2010 at 8:16 AM, Raymond Hettinger
>>> <raymond.hettinger at gmail.com>    wrote:
>>>>
>>>> On Mar 26, 2010, at 2:16 PM, Xavier Morel wrote:
>>>>
>>>> How about raising an exception instead of creating nans in the first
>>>> place,
>>>> except maybe within specific contexts (so that the IEEE-754 minded can
>>>> get
>>>> their nans working as they currently do)?
>>>>
>>>> -1
>>>> The numeric community uses NaNs as placeholders in vectorized
>>>> calculations.
>>>
>>> But is this relevant to python itself ? In Numpy, we indeed do use and
>>> support NaN, but we have much more control on what happens compared to
>>> python float objects. We can control whether invalid operations raises
>>> an exception or not, we had isnan/isfinite for a long time, and the
>>> fact that nan != nan has never been a real problem AFAIK.
>>
>> Nonetheless, the closer our float arrays are to Python's float type, the
>> happier I will be.
>
> Me too, but I don't see how to reconcile this with the intent of
> simplifying nan handling because they are not intuitive, which seems
> to be the goal of this discussion.

"Do nothing" is still on the table, I think.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From Jeffrey.Pry at sig.com  Mon Mar 29 17:02:13 2010
From: Jeffrey.Pry at sig.com (Pry, Jeffrey)
Date: Mon, 29 Mar 2010 11:02:13 -0400
Subject: [Python-Dev] OOP On Web
Message-ID: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com>

Hello,

Nice to meet everyone!

I have two files (class.py and object.py). Now, within object.py I create two objects from the class.py file. The class.py file contains the necessary code to make a connection to a FTP server. The two objects pass the credentials, etc. My question is, if I wanted to display a dropdown box on a webpage listing all instances of the object (most likely five servers), how would I do that. Would I do it at the class level, the object level, or at some other level. Forgive me if this is a somewhat basic question.

Thank you for your help!

Jeffrey Kevin Pry

________________________________
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100329/e7206d65/attachment.html>

From fuzzyman at voidspace.org.uk  Mon Mar 29 18:25:08 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 29 Mar 2010 17:25:08 +0100
Subject: [Python-Dev] OOP On Web
In-Reply-To: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com>
References: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com>
Message-ID: <4BB0D464.8090305@voidspace.org.uk>

On 29/03/2010 16:02, Pry, Jeffrey wrote:
>
> Hello,
>
> Nice to meet everyone!
>

Hello Jeffrey,

This mailing list is for the development *of* Python, not for developing 
*with* Python. More appropriate places to ask are the Python-tutor 
mailing list or the comp.lang.python newsgroup. Both of these have 
google and gmane web interfaces as well I believe.

You can find more information at:

     http://www.python.org/community/lists/

All the best,

Michael Foord

> I have two files (class.py and object.py). Now, within object.py I 
> create two objects from the class.py file. The class.py file contains 
> the necessary code to make a connection to a FTP server. The two 
> objects pass the credentials, etc. My question is, if I wanted to 
> display a dropdown box on a webpage listing all instances of the 
> object (most likely five servers), how would I do that. Would I do it 
> at the class level, the object level, or at some other level. Forgive 
> me if this is a somewhat basic question.
>
> Thank you for your help!
>
> Jeffrey Kevin Pry
>
>
> ------------------------------------------------------------------------
> IMPORTANT: The information contained in this email and/or its 
> attachments is confidential. If you are not the intended recipient, 
> please notify the sender immediately by reply and immediately delete 
> this message and all its attachments. Any review, use, reproduction, 
> disclosure or dissemination of this message or any attachment by an 
> unintended recipient is strictly prohibited. Neither this message nor 
> any attachment is intended as or should be construed as an offer, 
> solicitation or recommendation to buy or sell any security or other 
> financial instrument. Neither the sender, his or her employer nor any 
> of their respective affiliates makes any warranties as to the 
> completeness or accuracy of any of the information contained herein or 
> that this message or any of its attachments is free of viruses.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100329/cf2cb2b2/attachment.html>

From phd at phd.pp.ru  Mon Mar 29 18:28:37 2010
From: phd at phd.pp.ru (Oleg Broytman)
Date: Mon, 29 Mar 2010 20:28:37 +0400
Subject: [Python-Dev] OOP On Web
In-Reply-To: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com>
References: <26AD5A26C15C3948AEF795C346F7E9ED03EC949DCD@msgbal518.ds.susq.com>
Message-ID: <20100329162837.GA931@phd.pp.ru>

Hello.

   We'are sorry but we cannot help you. This mailing list is to work on
developing Python (fixing bugs and adding new features to Python itself); if
you're having problems using Python, please find another forum. Probably
python-list (comp.lang.python) news group/mailing list is the best place.
See http://www.python.org/community/lists/ for other lists/news groups/fora.

On Mon, Mar 29, 2010 at 11:02:13AM -0400, Pry, Jeffrey wrote:
Thank you for understanding.
> Hello,
> 
> Nice to meet everyone!
> 
> I have two files (class.py and object.py). Now, within object.py I create two objects from the class.py file. The class.py file contains the necessary code to make a connection to a FTP server. The two objects pass the credentials, etc. My question is, if I wanted to display a dropdown box on a webpage listing all instances of the object (most likely five servers), how would I do that. Would I do it at the class level, the object level, or at some other level. Forgive me if this is a somewhat basic question.
> 
> Thank you for your help!
> 
> Jeffrey Kevin Pry
> 
> ________________________________
> IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/phd%40phd.pp.ru


Oleg.
-- 
     Oleg Broytman            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From michael at voidspace.org.uk  Mon Mar 29 19:16:50 2010
From: michael at voidspace.org.uk (Michael Foord)
Date: Mon, 29 Mar 2010 18:16:50 +0100
Subject: [Python-Dev] WeakSet in Python 2.7
Message-ID: <4BB0E082.8040602@voidspace.org.uk>

Hello all,

Python 3 includes a WeakSet implementation. Any objections to me 
backporting this to 2.7?

http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet

I needed this in unittest and used a WeakKeyDictionary instead.

Michael

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From larry at hastings.org  Mon Mar 29 19:26:05 2010
From: larry at hastings.org (Larry Hastings)
Date: Mon, 29 Mar 2010 10:26:05 -0700
Subject: [Python-Dev] Bootstrap script for package management tool in
 Python 2.7 (Was: Re: [Distutils] At least one package management tool for
 2.7)
In-Reply-To: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
Message-ID: <4BB0E2AD.6020407@hastings.org>

anatoly techtonik wrote:
> So, there won't be any package management tool shipped with Python 2.7
> and users will have to download and install `setuptools` manually as
> before:
>
>   "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python
> setup.py install"
>
>
> Therefore I still propose shipping bootstrap package that instruct
> user how to download and install an actual package  management tool
> when users tries to use it.

For what it's worth, Guido prototyped something similar in March of 
2008, but his was an actual bootstrapping tool for package management:

    http://mail.python.org/pipermail/python-dev/2008-March/077837.html

His tool knew how to download a tar file, untar it, and run "python 
setup.py install" on it.  No version numbers, no dependency management, 
simple enough that it should be easy to get right.  Only appropriate for 
bootstrapping into a real package management tool.

The thread ends with him saying "I don't have time to deal with this 
further this week", and I dunno, maybe it just fell off the radar?  I'd 
been thinking about resurrecting the discussion but I didn't have time 
either.

I fear it is too late for 2.7,


//larry//

From skip at pobox.com  Mon Mar 29 19:31:17 2010
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 29 Mar 2010 12:31:17 -0500
Subject: [Python-Dev] Google Groups mirror (Was: Bootstrap script for
 package management tool in Python 2.7)
In-Reply-To: <d34314101003290705k2b6d8256n8189728ee7bded58@mail.gmail.com>
References: <d34314101003290705k2b6d8256n8189728ee7bded58@mail.gmail.com>
Message-ID: <19376.58341.90376.949602@montanaro.dyndns.org>


    anatoly> Why can't people here create a Google Groups mirror?

Whoever "people here" are could, I suppose.  I wouldn't mind a read-only
mirror but would not be a fan of a two-way mirror because Google seems to do
an awful job of weeding out spam.

In any case, the correct place to submit such a request would be
postmaster at python.org.  

Skip


From daniel at stutzbachenterprises.com  Mon Mar 29 20:14:13 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Mon, 29 Mar 2010 13:14:13 -0500
Subject: [Python-Dev] WeakSet in Python 2.7
In-Reply-To: <4BB0E082.8040602@voidspace.org.uk>
References: <4BB0E082.8040602@voidspace.org.uk>
Message-ID: <eae285401003291114h573dd92bjbb927d47d7c83482@mail.gmail.com>

On Mon, Mar 29, 2010 at 12:16 PM, Michael Foord <michael at voidspace.org.uk>wrote:

> Python 3 includes a WeakSet implementation. Any objections to me
> backporting this to 2.7?
>
> http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet
>

Backporting WeakSet would also make it possible to backport the fix for this
reference leak:

http://bugs.python.org/issue2521
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100329/b540dff1/attachment.html>

From fuzzyman at voidspace.org.uk  Mon Mar 29 21:20:11 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 29 Mar 2010 20:20:11 +0100
Subject: [Python-Dev] WeakSet in Python 2.7
In-Reply-To: <4BB0E082.8040602@voidspace.org.uk>
References: <4BB0E082.8040602@voidspace.org.uk>
Message-ID: <4BB0FD6B.6080104@voidspace.org.uk>

On 29/03/2010 18:16, Michael Foord wrote:
> Hello all,
>
> Python 3 includes a WeakSet implementation. Any objections to me 
> backporting this to 2.7?
>
> http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet
>
> I needed this in unittest and used a WeakKeyDictionary instead.

Hmmm... I was under the mistaken impression that this was implemented in 
pure-Python. It's in C so beyond my threshold for simple porting I'm 
afraid. Still if someone else is up for it...

All the best,

Michael

>
> Michael
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From michael at voidspace.org.uk  Mon Mar 29 21:21:04 2010
From: michael at voidspace.org.uk (Michael Foord)
Date: Mon, 29 Mar 2010 20:21:04 +0100
Subject: [Python-Dev] WeakSet in Python 2.7
In-Reply-To: <eae285401003291114h573dd92bjbb927d47d7c83482@mail.gmail.com>
References: <4BB0E082.8040602@voidspace.org.uk>
	<eae285401003291114h573dd92bjbb927d47d7c83482@mail.gmail.com>
Message-ID: <4BB0FDA0.1050207@voidspace.org.uk>

On 29/03/2010 19:14, Daniel Stutzbach wrote:
> On Mon, Mar 29, 2010 at 12:16 PM, Michael Foord 
> <michael at voidspace.org.uk <mailto:michael at voidspace.org.uk>> wrote:
>
>     Python 3 includes a WeakSet implementation. Any objections to me
>     backporting this to 2.7?
>
>     http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet
>
>
> Backporting WeakSet would also make it possible to backport the fix 
> for this reference leak:
>
> http://bugs.python.org/issue2521

It should be possible to fix it with a WeakKeyDictionary instead of WeakSet.

All the best,

Michael Foord

> --
> Daniel Stutzbach, Ph.D.
> President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100329/ade129c9/attachment.html>

From daniel at stutzbachenterprises.com  Mon Mar 29 21:46:10 2010
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Mon, 29 Mar 2010 14:46:10 -0500
Subject: [Python-Dev] WeakSet in Python 2.7
In-Reply-To: <4BB0FDA0.1050207@voidspace.org.uk>
References: <4BB0E082.8040602@voidspace.org.uk>
	<eae285401003291114h573dd92bjbb927d47d7c83482@mail.gmail.com>
	<4BB0FDA0.1050207@voidspace.org.uk>
Message-ID: <eae285401003291246o48417ee8q9c2744f1312e34b4@mail.gmail.com>

On Mon, Mar 29, 2010 at 2:21 PM, Michael Foord <michael at voidspace.org.uk>wrote:

> It should be possible to fix it with a WeakKeyDictionary instead of
> WeakSet.
>

True.  I should have said "Backporting WeakSet would make it *easier* to
backport the fix ..." :-)
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100329/517ccbb1/attachment-0001.html>

From michael at voidspace.org.uk  Mon Mar 29 21:46:52 2010
From: michael at voidspace.org.uk (Michael Foord)
Date: Mon, 29 Mar 2010 20:46:52 +0100
Subject: [Python-Dev] WeakSet in Python 2.7
In-Reply-To: <4BB0FD6B.6080104@voidspace.org.uk>
References: <4BB0E082.8040602@voidspace.org.uk>
	<4BB0FD6B.6080104@voidspace.org.uk>
Message-ID: <4BB103AC.7090508@voidspace.org.uk>

On 29/03/2010 20:20, Michael Foord wrote:
> On 29/03/2010 18:16, Michael Foord wrote:
>> Hello all,
>>
>> Python 3 includes a WeakSet implementation. Any objections to me 
>> backporting this to 2.7?
>>
>> http://docs.python.org/py3k/library/weakref.html#weakref.WeakSet
>>
>> I needed this in unittest and used a WeakKeyDictionary instead.
>
> Hmmm... I was under the mistaken impression that this was implemented 
> in pure-Python. It's in C so beyond my threshold for simple porting 
> I'm afraid. Still if someone else is up for it...

No, my mistake - it *is* implemented in Python. I'm porting it now.

Michael

>
> All the best,
>
> Michael
>
>>
>> Michael
>>
>
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From martin at v.loewis.de  Mon Mar 29 22:06:20 2010
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 29 Mar 2010 22:06:20 +0200
Subject: [Python-Dev] [Distutils] Bootstrap script for package
 management tool in Python 2.7 (Was: Re: At least one package
 management	tool for 2.7)
In-Reply-To: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com>
Message-ID: <4BB1083C.30509@v.loewis.de>

anatoly techtonik wrote:
> So, there won't be any package management tool shipped with Python 2.7
> and users will have to download and install `setuptools` manually as
> before:
[...]
> So, do we need a PEP for that? How else can I know if consensus is
> reached? Anybody is willing to elaborate on implementation?

Notice that the first beta release of Python 2.7 will be made on April
3. Any new feature that you propose must be committed by then.

I suggest focusing on Python 3.2 instead.

Regards,
Martin


From fuzzyman at voidspace.org.uk  Mon Mar 29 22:43:01 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 29 Mar 2010 21:43:01 +0100
Subject: [Python-Dev] WeakSet in Python 2.7
In-Reply-To: <eae285401003291246o48417ee8q9c2744f1312e34b4@mail.gmail.com>
References: <4BB0E082.8040602@voidspace.org.uk>	<eae285401003291114h573dd92bjbb927d47d7c83482@mail.gmail.com>	<4BB0FDA0.1050207@voidspace.org.uk>
	<eae285401003291246o48417ee8q9c2744f1312e34b4@mail.gmail.com>
Message-ID: <4BB110D5.7090802@voidspace.org.uk>

On 29/03/2010 20:46, Daniel Stutzbach wrote:
> On Mon, Mar 29, 2010 at 2:21 PM, Michael Foord 
> <michael at voidspace.org.uk <mailto:michael at voidspace.org.uk>> wrote:
>
>     It should be possible to fix it with a WeakKeyDictionary instead
>     of WeakSet.
>
>
> True.  I should have said "Backporting WeakSet would make it *easier* 
> to backport the fix ..." :-)

WeakSet is now in.

Michael

> --
> Daniel Stutzbach, Ph.D.
> President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100329/f5e95629/attachment.html>

From ianb at colorstudy.com  Mon Mar 29 23:45:00 2010
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 29 Mar 2010 15:45:00 -0600
Subject: [Python-Dev] Bootstrap script for package management tool in
	Python 2.7 (Was: Re: [Distutils] At least one package
	management tool for 2.7)
In-Reply-To: <4BB0E2AD.6020407@hastings.org>
References: <d34314101003290030g6623531er4a70f911b1423767@mail.gmail.com> 
	<4BB0E2AD.6020407@hastings.org>
Message-ID: <b654cd2e1003291445w1dfcc63sa35043499613185e@mail.gmail.com>

On Mon, Mar 29, 2010 at 11:26 AM, Larry Hastings <larry at hastings.org> wrote:

> anatoly techtonik wrote:
>
>> So, there won't be any package management tool shipped with Python 2.7
>> and users will have to download and install `setuptools` manually as
>> before:
>>
>>  "search" -> "download" -> "unzip" -> "cmd" -> "cd" -> "python
>> setup.py install"
>>
>>
>> Therefore I still propose shipping bootstrap package that instruct
>> user how to download and install an actual package  management tool
>> when users tries to use it.
>>
>
> For what it's worth, Guido prototyped something similar in March of 2008,
> but his was an actual bootstrapping tool for package management:
>
>   http://mail.python.org/pipermail/python-dev/2008-March/077837.html
>
> His tool knew how to download a tar file, untar it, and run "python
> setup.py install" on it.  No version numbers, no dependency management,
> simple enough that it should be easy to get right.  Only appropriate for
> bootstrapping into a real package management tool.
>
> The thread ends with him saying "I don't have time to deal with this
> further this week", and I dunno, maybe it just fell off the radar?  I'd been
> thinking about resurrecting the discussion but I didn't have time either.
>

I would consider this bootstrap to be quite workable, though I would add
that any extra option to the bootstrap script should be passed to setup.py
install, and the download should be cached (so you can do -h and not have to
re-download the package once you figure out the extra options -- at least a
--user option is reasonable here for people without root).  Specifically
targeting this bootstrap for tools like pip and virtualenv is no problem.

I think looking around PyPI etc is kind of more than I'd bother with.  Those
things change, this bootstrap code won't change, it could cause unnecessary
future pain.  Maybe (*maybe*) it could look in
http://pypi.python.org/well-known-packages/PACKAGE_NAME and so we can have
it install a certain small number of things quickly that way -- if the URL
it looks to is targeted only for the bootstrap script itself then we don't
have to worry about compatibility problems as much.

Oh... then i can think of a half dozen other options it could take, and then
it becomes an installer.  Blech.  OK, I'd be willing to cut off the options
at --user (which I think is a minimum... maybe --prefix too), and maybe some
simple package detection so people could write "python -m boostrap
Setuptools --user" -- entirely based on some well-known URL baked into
bootstrap.py, where the URL is independent of any other service (and so is
least likely to cause future problems or ambiguities).

An advantage to this kind of bootstrapper is that as future packaging
systems are developed there's a clear way to get started with them, without
prematurely baking anything in to Python.

-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://twitter.com/ianbicking
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100329/af8a9e14/attachment.html>

From turturici.valerio at gmail.com  Tue Mar 30 01:06:13 2010
From: turturici.valerio at gmail.com (Valerio Turturici)
Date: Tue, 30 Mar 2010 01:06:13 +0200
Subject: [Python-Dev] Contribute
Message-ID: <4BB13265.9020508@gmail.com>

Hello everyone,
my name's Valerio, i'm italian and i study computer science at 
University of Pisa. I'm honored to speak and learn from people like you.
My knowledge of C language is not very advanced, but i know Python well 
enough. I really like to contribute, i can learn and improve a lot. This 
is the first time that i participate in a serious, large and complex 
project.

Have you any advice for me? :)



From benjamin at python.org  Tue Mar 30 00:11:06 2010
From: benjamin at python.org (Benjamin Peterson)
Date: Mon, 29 Mar 2010 16:11:06 -0600
Subject: [Python-Dev] Contribute
In-Reply-To: <4BB13265.9020508@gmail.com>
References: <4BB13265.9020508@gmail.com>
Message-ID: <1afaf6161003291511u500063f9wc87263610ea90e14@mail.gmail.com>

2010/3/29 Valerio Turturici <turturici.valerio at gmail.com>:
> Hello everyone,
> my name's Valerio, i'm italian and i study computer science at University of
> Pisa. I'm honored to speak and learn from people like you.
> My knowledge of C language is not very advanced, but i know Python well
> enough. I really like to contribute, i can learn and improve a lot. This is
> the first time that i participate in a serious, large and complex project.
>
> Have you any advice for me? :)

Pick a bug and provide a patch.



-- 
Regards,
Benjamin

From ncoghlan at gmail.com  Tue Mar 30 00:24:41 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Mar 2010 08:24:41 +1000
Subject: [Python-Dev] Contribute
In-Reply-To: <4BB13265.9020508@gmail.com>
References: <4BB13265.9020508@gmail.com>
Message-ID: <4BB128A9.6010704@gmail.com>

Valerio Turturici wrote:
> Hello everyone,
> my name's Valerio, i'm italian and i study computer science at
> University of Pisa. I'm honored to speak and learn from people like you.
> My knowledge of C language is not very advanced, but i know Python well
> enough. I really like to contribute, i can learn and improve a lot. This
> is the first time that i participate in a serious, large and complex
> project.
> 
> Have you any advice for me? :)

Head over to http://bugs.python.org and look for items with the "easy"
tag. Alternatively, look for items with attached patches and download,
apply and test them out to see if they work as advertised.

I think there's a page on the wiki with suggestions on how to get
started, but I don't have a link handy - hopefully someone else will
chime in with it.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From fuzzyman at voidspace.org.uk  Tue Mar 30 00:27:38 2010
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 29 Mar 2010 23:27:38 +0100
Subject: [Python-Dev] Contribute
In-Reply-To: <4BB13265.9020508@gmail.com>
References: <4BB13265.9020508@gmail.com>
Message-ID: <4BB1295A.7030108@voidspace.org.uk>

On 30/03/2010 00:06, Valerio Turturici wrote:
> Hello everyone,
> my name's Valerio, i'm italian and i study computer science at 
> University of Pisa. I'm honored to speak and learn from people like you.
> My knowledge of C language is not very advanced, but i know Python 
> well enough. I really like to contribute, i can learn and improve a 
> lot. This is the first time that i participate in a serious, large and 
> complex project.
>
> Have you any advice for me? :)

The #python-dev IRC channel on freenode is a useful place to hangout if 
you have realtime questions.

Michael

>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk 
>


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.



From skip at pobox.com  Tue Mar 30 00:30:24 2010
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 29 Mar 2010 17:30:24 -0500
Subject: [Python-Dev] Contribute
In-Reply-To: <4BB128A9.6010704@gmail.com>
References: <4BB13265.9020508@gmail.com> <4BB128A9.6010704@gmail.com>
Message-ID: <19377.10752.20195.712556@montanaro.dyndns.org>


    Nick> I think there's a page on the wiki with suggestions on how to get
    Nick> started, but I don't have a link handy - hopefully someone else
    Nick> will chime in with it.

:ring: :ring: :ring:

    http://www.python.org/dev/contributing/

Skip

From matthew at matthewwilkes.co.uk  Tue Mar 30 00:36:23 2010
From: matthew at matthewwilkes.co.uk (Matthew Wilkes)
Date: Mon, 29 Mar 2010 23:36:23 +0100
Subject: [Python-Dev] Contribute
In-Reply-To: <4BB13265.9020508@gmail.com>
References: <4BB13265.9020508@gmail.com>
Message-ID: <791240A6-C179-4685-AFE3-6BCFC5276219@matthewwilkes.co.uk>


On 2010-03-30, at 0006, Valerio Turturici wrote:

> Have you any advice for me? :)

Also, have a look at Google's Summer of Code programme.  It is a good  
way to get involved with a large project.

Matthw

From anand.shashwat at gmail.com  Tue Mar 30 00:36:35 2010
From: anand.shashwat at gmail.com (Shashwat Anand)
Date: Tue, 30 Mar 2010 04:06:35 +0530
Subject: [Python-Dev] Contribute
In-Reply-To: <19377.10752.20195.712556@montanaro.dyndns.org>
References: <4BB13265.9020508@gmail.com> <4BB128A9.6010704@gmail.com> 
	<19377.10752.20195.712556@montanaro.dyndns.org>
Message-ID: <d4ab53de1003291536t1382f231w301307282039a27f@mail.gmail.com>

Python should have something like gnome-love (
http://live.gnome.org/GnomeLove ). Some bugs on bugzilla is tagged with
gnome-love which are fairly easy and especially created to help dive in new
developers.


On Tue, Mar 30, 2010 at 4:00 AM, <skip at pobox.com> wrote:

>
>    Nick> I think there's a page on the wiki with suggestions on how to get
>    Nick> started, but I don't have a link handy - hopefully someone else
>    Nick> will chime in with it.
>
> :ring: :ring: :ring:
>
>    http://www.python.org/dev/contributing/
>
> Skip
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/anand.shashwat%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100330/ef4bb782/attachment.html>

From skip at pobox.com  Tue Mar 30 01:20:01 2010
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 29 Mar 2010 18:20:01 -0500
Subject: [Python-Dev] Contribute
In-Reply-To: <d4ab53de1003291536t1382f231w301307282039a27f@mail.gmail.com>
References: <4BB13265.9020508@gmail.com> <4BB128A9.6010704@gmail.com>
	<19377.10752.20195.712556@montanaro.dyndns.org>
	<d4ab53de1003291536t1382f231w301307282039a27f@mail.gmail.com>
Message-ID: <19377.13729.368146.37884@montanaro.dyndns.org>


    Shashwat> Python should have something like gnome-love (
    Shashwat> http://live.gnome.org/GnomeLove ). Some bugs on bugzilla is
    Shashwat> tagged with gnome-love which are fairly easy and especially
    Shashwat> created to help dive in new developers.

I don't know what gnome love is (and would be hesitant to click on any such
links, especially at work) but on bugs.python.org such items are tagged
"easy".  Right on the front page is a link with the label "Show Easy".

Skip

From techtonik at gmail.com  Wed Mar 31 09:14:56 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 31 Mar 2010 10:14:56 +0300
Subject: [Python-Dev] Contribute
In-Reply-To: <19377.13729.368146.37884@montanaro.dyndns.org>
References: <4BB13265.9020508@gmail.com> <4BB128A9.6010704@gmail.com>
	<19377.10752.20195.712556@montanaro.dyndns.org>
	<d4ab53de1003291536t1382f231w301307282039a27f@mail.gmail.com>
	<19377.13729.368146.37884@montanaro.dyndns.org>
Message-ID: <z2ud34314101003310014l9b6e7820gd360fef5b38020e3@mail.gmail.com>

On Tue, Mar 30, 2010 at 2:20 AM,  <skip at pobox.com> wrote:
>
> ? ?Shashwat> Python should have something like gnome-love (
> ? ?Shashwat> http://live.gnome.org/GnomeLove ). Some bugs on bugzilla is
> ? ?Shashwat> tagged with gnome-love which are fairly easy and especially
> ? ?Shashwat> created to help dive in new developers.
>
> I don't know what gnome love is (and would be hesitant to click on any such
> links, especially at work) but on bugs.python.org such items are tagged
> "easy". ?Right on the front page is a link with the label "Show Easy".

The difference between "gnome-love" and "easy" tags is that the latter
can only be flagged by code devs (who usually do not have much time to
review bugs).
-- 
anatoly t.

From techtonik at gmail.com  Wed Mar 31 10:06:34 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 31 Mar 2010 11:06:34 +0300
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <20100329123039.GA23643@idyll.org>
References: <20100319023630.GC1617@idyll.org>
	<d34314101003290040m101343a0t14d3584b67ab0693@mail.gmail.com>
	<20100329123039.GA23643@idyll.org>
Message-ID: <v2sd34314101003310106n660a8a71j86ec80c7f72d82a3@mail.gmail.com>

On Mon, Mar 29, 2010 at 3:30 PM, C. Titus Brown <ctb at msu.edu> wrote:
>
>> I would vote for allowing student work on community infrastructure
>> tasks. Tracker, Wiki, Web site management tools are all outdated and
>> everybody who cares agrees that they've seen a better tools.
>
> As long as it's programming, it's allowed by Google. ?So let's find a good
> student or two, and outline a few good projects!
>
> I do worry that that kind of work is difficult to evaluate, and requires really
> great communication on both sides...

First we need to compile a list of things to do into one big list. I
see the major problem that there is no "dashboard" that gives an
overview of available/supported community services and their status.
Services that are parts of python.org and those that linked and often
used, but not parts.

Status of service is the amount of opened/languishing
bugs/enhancements. Some services don't have trackers at all. For
example, infrastructure proposals, web site patches is nowhere to
track.

Service on a dashboard should be accompanied by contact points info.

Service should list location of primary repository and mirrors.

Feature creep:
- Service can be monitored for online/offline status.
- Service may have maintenance scripts, which community members can
see, inspect scheduled times for the next run and results of
execution.
- Service may publish stats for further analysis.


After dashboard is ready, it would be nice to unify all Services:
1. Add OpenID/Google support
2. Add searchable Google Groups mirrors
3. Add Google search form to Mailman web interface for hosted groups
4. Register services search with Google (see attach)


Then we need to find a contact point - a person who could be most
helpful for each service and help other students with other specific
tasks. I am not such person for neither service, but I am interested
to help push progress forward.
-- 
anatoly t.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: search_search_form.png
Type: image/png
Size: 16969 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100331/23fa2dbf/attachment-0001.png>

From stephen at xemacs.org  Wed Mar 31 10:41:47 2010
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 31 Mar 2010 17:41:47 +0900
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <d34314101003290040m101343a0t14d3584b67ab0693@mail.gmail.com>
References: <20100319023630.GC1617@idyll.org>
	<d34314101003290040m101343a0t14d3584b67ab0693@mail.gmail.com>
Message-ID: <87wrwssx10.fsf@uwakimon.sk.tsukuba.ac.jp>

anatoly techtonik writes:

 > I would vote for allowing student work on community infrastructure
 > tasks. Tracker, Wiki, Web site management tools are all outdated and
 > everybody who cares agrees that they've seen a better tools.

I've also seen *much* worse tools in actual use.  You don't have to
look any farther than macports.org and GNU Savannah for examples of
implementations of individual tools that are much worse, and overall
usability that is clearly worse, than Python's environment.

Over and over again I have seen short-term volunteers pick the best-
reputed software of the day, run out of steam just getting the old
data ported over to the new host software, leaving behind a disgrace.
(This is what happened to MacPorts Trac, it would seem -- it took 4
years for them to get an explanation of how to search for bugs on a
given port on the top page of the issue tracker, and you still have to
type queries like "?PORT=python26" by hand in the browser address
field![1])  In other cases, the old data is never successfully ported
at all (a common way to migrate from one VCS to another).  So while
improving the infrastructure is clearly a good thing, it is not a good
idea to have people without *long-term* commitment to maintenance
*changing the tools themselves*.

I would recommend changing the tools only if *current* maintainers are
either planning to step down (and so we face the problem of how to
support maintenance in the future in any case), or are willing to
supervise (ie, the people who will come back and fix problems in the
future find the proposed "improvements" to be real improvements).  Eg,
MvL should be intimately involved in any move to use different
software for the tracker.  If the GSoC student(s) is (are) willing to
work within the constraints of the existing software (ie, incremental
improvements), then the constraints on mentors could be substantially
relaxed to any of the senior folks who have recently contributed in
those areas.


Footnotes: 
[1]  Actually, it may be even worse than it looks -- IIUC, macosforge
has paid staff!


From ctb at msu.edu  Wed Mar 31 15:29:11 2010
From: ctb at msu.edu (C. Titus Brown)
Date: Wed, 31 Mar 2010 06:29:11 -0700
Subject: [Python-Dev] GSoC 2010 is on -- projects?
In-Reply-To: <87wrwssx10.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20100319023630.GC1617@idyll.org>
	<d34314101003290040m101343a0t14d3584b67ab0693@mail.gmail.com>
	<87wrwssx10.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20100331132911.GA8170@idyll.org>

On Wed, Mar 31, 2010 at 05:41:47PM +0900, Stephen J. Turnbull wrote:
> anatoly techtonik writes:
> 
>  > I would vote for allowing student work on community infrastructure
>  > tasks. Tracker, Wiki, Web site management tools are all outdated and
>  > everybody who cares agrees that they've seen a better tools.

[ munch ]

> I would recommend changing the tools only if *current* maintainers are
> either planning to step down (and so we face the problem of how to
> support maintenance in the future in any case), or are willing to
> supervise (ie, the people who will come back and fix problems in the
> future find the proposed "improvements" to be real improvements).  Eg,
> MvL should be intimately involved in any move to use different
> software for the tracker.  If the GSoC student(s) is (are) willing to
> work within the constraints of the existing software (ie, incremental
> improvements), then the constraints on mentors could be substantially
> relaxed to any of the senior folks who have recently contributed in
> those areas.

Agreed.

It'd be great to have students work on prototyping improvements, with an
eye to making changes that can then be evaluated in terms of maintainability,
extensibility, etc.  Having them actually change the infrastructure itself
seems to me like a bad idea :)

--titus
-- 
C. Titus Brown, ctb at msu.edu

From ezio.melotti at gmail.com  Wed Mar 31 16:58:59 2010
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Wed, 31 Mar 2010 17:58:59 +0300
Subject: [Python-Dev] Add -3 and -Wd to the buildbots
Message-ID: <4BB36333.6020806@gmail.com>

Hi,
now that the py3k warnings are fixed (see 
http://bugs.python.org/issue7092) we should run the tests on the trunk 
buildbots with the -3 flags, in order to check if new warnings are 
introduced and possibly to uncover other ones.
It might be a good idea to add the -Wd flag too, both on trunk and py3k.

-Ezio

From screwdriver at lxnt.info  Wed Mar 31 17:23:27 2010
From: screwdriver at lxnt.info (Alexander Sabourenkov)
Date: Wed, 31 Mar 2010 19:23:27 +0400
Subject: [Python-Dev] Replacing threads with swapcontext()
Message-ID: <4BB368EF.8010206@lxnt.info>

Hello.

I have went and replaced Python threads with explicitly-scheduled 
coroutines using swapcontext() family of functions (basically out of 
frustration with the greenlet).

I have two questions.

1. Is it as dangerous to switch C stacks and not switch PyThreadState as 
I suspect? In particular, how tightly is PyThreadState::frame coupled to 
current C stack?

2. How do you all look at merging my work as another threading-related 
configure switch? It's only another thread_something.h file and a small 
patch to threadmodule.c.


Project is at http://code.google.com/p/coev/

-- 

./lxnt


From brett at python.org  Wed Mar 31 20:08:00 2010
From: brett at python.org (Brett Cannon)
Date: Wed, 31 Mar 2010 11:08:00 -0700
Subject: [Python-Dev] Add -3 and -Wd to the buildbots
In-Reply-To: <4BB36333.6020806@gmail.com>
References: <4BB36333.6020806@gmail.com>
Message-ID: <m2lbbaeab101003311108k32c19005k1b9192aa905880fe@mail.gmail.com>

On Wed, Mar 31, 2010 at 07:58, Ezio Melotti <ezio.melotti at gmail.com> wrote:

> Hi,
> now that the py3k warnings are fixed (see http://bugs.python.org/issue7092)
> we should run the tests on the trunk buildbots with the -3 flags, in order
> to check if new warnings are introduced and possibly to uncover other ones.
> It might be a good idea to add the -Wd flag too, both on trunk and py3k.
>

+1 from me.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100331/83cf2c5f/attachment.html>

From florent.xicluna at gmail.com  Wed Mar 31 22:03:52 2010
From: florent.xicluna at gmail.com (Florent Xicluna)
Date: Wed, 31 Mar 2010 22:03:52 +0200
Subject: [Python-Dev] Add -3 and -Wd to the buildbots
In-Reply-To: <m2lbbaeab101003311108k32c19005k1b9192aa905880fe@mail.gmail.com>
References: <4BB36333.6020806@gmail.com>
	<m2lbbaeab101003311108k32c19005k1b9192aa905880fe@mail.gmail.com>
Message-ID: <y2ye43810331003311303v5fe8a7c2t223ef729d5cd30ef@mail.gmail.com>

2010/3/31 Brett Cannon:
> On Wed, Mar 31, 2010 at 07:58, Ezio Melotti wrote:
>> Hi,
>> now that the py3k warnings are fixed (see
>> http://bugs.python.org/issue7092) we should run the tests on the trunk
>> buildbots with the -3 flags, in order to check if new warnings are
>> introduced and possibly to uncover other ones.
>> It might be a good idea to add the -Wd flag too, both on trunk and py3k.
>
> +1 from me.


I agree, it could help to catch unexpected warnings or bugs earlier in
the process.


-- 
Florent

From martin at v.loewis.de  Wed Mar 31 23:05:56 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 31 Mar 2010 23:05:56 +0200
Subject: [Python-Dev] Add -3 and -Wd to the buildbots
In-Reply-To: <4BB36333.6020806@gmail.com>
References: <4BB36333.6020806@gmail.com>
Message-ID: <4BB3B934.4030003@v.loewis.de>

> now that the py3k warnings are fixed (see
> http://bugs.python.org/issue7092) we should run the tests on the trunk
> buildbots with the -3 flags, in order to check if new warnings are
> introduced and possibly to uncover other ones.
> It might be a good idea to add the -Wd flag too, both on trunk and py3k.

Not sure what you are asking for. If you want confirmation that this
change is desirable, it seems you got that already. If you are asking
for someone to make that change - go ahead and make it yourself. You
just need to change the buildbottest target.

Regards,
Martin

From ncoghlan at gmail.com  Wed Mar 31 23:53:50 2010
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 01 Apr 2010 07:53:50 +1000
Subject: [Python-Dev] Replacing threads with swapcontext()
In-Reply-To: <4BB368EF.8010206@lxnt.info>
References: <4BB368EF.8010206@lxnt.info>
Message-ID: <4BB3C46E.8000507@gmail.com>

Alexander Sabourenkov wrote:
> Hello.
> 
> I have went and replaced Python threads with explicitly-scheduled
> coroutines using swapcontext() family of functions (basically out of
> frustration with the greenlet).
> 
> I have two questions.
> 
> 1. Is it as dangerous to switch C stacks and not switch PyThreadState as
> I suspect? In particular, how tightly is PyThreadState::frame coupled to
> current C stack?

Off the top of my head, recursion limit checks will definitely go
haywire, and I would expect threading.local() and exception handling to
explode at some point (although the latter depends on what you mean by
"switching C stacks" - if you really are just fiddling the stack pointer
without affecting the thread local heap storage mechanisms, the latter
two should be fine. Still doesn't sound like a good idea though).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------